Topological Constraints on the Universal Approximation of Neural Networks

The title may seem like a contradiction given that there is such a thing as the Universal Approximation Theorem which simply states that a neural network with a single hidden layer of finite width(i.e finite number of neurons) can approximate any function on a compact set of \mathbb{R}^{n} given that the activation function is non-constant,bounded and continuous.

Needless to say, I haven’t found any kind of flaws in the existing proofs(see Kolmogrov or Cybenko). However, I thought of something a little simple and scoured the internet for an answer.

What if we allow an arbitrary number of hidden layers and bound the dimension of the hidden layers making them ‘Deep, Skinny Neural Networks’? Would that be a universal approximator?

Continue reading “Topological Constraints on the Universal Approximation of Neural Networks”

Advertisements

Hopf Fibrations-Construction and Quaternion Interpretations (I)

Before I begin discussing the Hopf fibration of the 3-spbhere, one of the simplest yet deeply profound example of a non-trivial fiber bundle, I’d like to recall the definition of a fiber bundle.

Let E,B,F represent the entire space, base space and the fiber respectively where E,B are connected. If f:E \mapsto B is a continuous surjection onto the base space, then the structure (E,B,F,f) is said to be a fiber bundle if for every x\in E, there exists a neighborhood U \subset B of f(x) such that there exists a homeomorphism \psi:f^{-1}(U) \mapsto U \times F such that \psi \circ \pi_{U}=f.

What this basically means is that locally, the fiber bundle looks like the product B \times F but globally, it may have different topological properties.

A trivial fiber bundle is a fiber bundle which in which the total space is E=B \times F. In fact, any fiber bundle over a contractible CW Complex is trivial.

Continue reading “Hopf Fibrations-Construction and Quaternion Interpretations (I)”

CW Complex Structure of Real Projective Space and Kusner’s Parametrization of Boy’s surface

The real projective space represented by \mathbb{RP}^{n} is the space obtained from \mathbb{R}^{n+1} under the equivalence relation x \sim kx \forall x \in \mathbb{R}^{n+1}. Basically, \mathbb{RP}^{n} is the set of lines which passed through the origin in \mathbb{R}^{n+1}. It can also be understood by identifying antipodal points(points which lie on opposite ends of the diameter) of a unit n-sphere,S^{n}.

 

One very basic yet deeply interesting example of these spaces is \mathbb{RP}^{2}, known as the real projective plane. While \mathbb{RP}^{0} is a points and \mathbb{RP}^{1} is homeomorphic to a circle with infinity, the real projective plane turns out to be far more interesting indeed. It can’t be embedded in \mathbb{R}^{3} and its immersion/s such as Boy’s surface, Roman surface and Cross Cap have far stranger structures than a mere circle as in the case of \mathbb{RP}^{2}. In fact, I will delineate some of the calculations involved in the Kusner-Bryant 3-dimensional parametrization of Boy’s surface. It’s a little fascinating how much complexity can be added to mathematical structures upon generalization especially in the case of the projective space which I believe have a remarkably simple and ‘non-threatening’ definition.

Continue reading “CW Complex Structure of Real Projective Space and Kusner’s Parametrization of Boy’s surface”

Moore-Smith Sequences/Nets in General Topology

Nets/Moore-Smith Sequences

Sequences are common objects in the field of topology. Often, sequences can help identify continuous functions, limit points and compact spaces in metric spaces.

Moore-Smith sequences(or nets) are essentially a generalization of the sequence for an arbitrary topological space and we can see that many foundational theorems of general topology can be stated in terms of nets.

So first, let’s recall what a partial order and direct set is:

A partial order is an order realtion \leq satisfying:-

  • \alpha \leq \alpha
  • If \alpha \leq \beta and \beta \leq \alpha, then \alpha=\beta.
  • If \alpha \leq \beta,\beta \leq \gamma, then \alpha \leq \gamma.

 

A directed set I is a set with a partial order relation \leq such that for any \alpha,\beta \in I,there exists \gamma such that \alpha \leq \gamma,\beta \leq \gamma.

It’s best to think of a directed set as a sort of analogue to an indexing set.

Continue reading “Moore-Smith Sequences/Nets in General Topology”