Introduction to Neural Networks

Neural networks have been here since the early 1980s. 

The objectives were to relate neural networks to linear classifiers, define the structure of a multilayer network, and train the weights of a network.

The neuron generalizes a linear classifier. There are features Xt = [x1 ... xm] and weights Wt = [w1 ... wm].

there is a linear classifier d = sign(XtW). >0 is 1 and -1 if XtW < 0. It can be pictured like the following, denoted by a few weights: 

We can use a similar representation for a neuron, the basic unit that is involved in building neural networks. 

We use a nonlinear fuction sigma. There is ReLu, Logistic, and the sin functions that can constitute this neural network. 

Xt(w) goes into the input of the ReLU network.

A neural network involves a network of neurons with an input layer, a hidden layer, and an output layer. The hidden nodes are the nonlinearity applied as the weighted sum of all of the values. "Deep learning neural networks" has many, many hidden layers. 

2 issues must be addressed. 

1. I have to make a decision on the network structure (layers, hidden layers, etc.) In general, this is an open question and people are trying to understand how neural networks actually work.

Universal approximation theorem: A 3-layer network can approximate any function arbitrarily well givenn enough hidden nodes and the right weights. 

2. How do I choose these weights? This question has much clearer andwer. I can use stochastic gradient descent, and then backpropagation, going backwards from the output layer to an input layer to adjust the weights.  

Backpropagation updates the weights in sequence. We can write this as an optimization problem that minimize the squared errror between the true layer and training stack. 

The process is to Guess weights, and randomly choose one of the N training samples.

We can propogate that feature forward to the network to calculate the hidden and output nodes. Then I can converge using gradient descent. 

The chain rule is essential for deriving gradients. Equation is as follows: