#8: Neural Networks content
XOR, Backpropagation Algorithm
Lecture Outline
- Motivation behind neural networks
- The XOR problem
- Multi-Layer Perceptron model
- Backpropagation algorithm
- Regularization and other stuff in NNs
Artificial Neuron
OR vs XOR
TODO
Activation Functions
Introduce non-linearity to the model
Choosing activation function: an art not a science
- Stick to one activation function for all layers
Two-Layer Neural Network
Single-hidden-layer network
Potentially overfits the dataset
Can simulate any line (linear or non-linear)
- M= number of hidden layers (depth of network ) ⇒ number of hyperplanes
- Deep learning: many hidden layers
- h= any activation function (sigmoid, ReLU, etc.)
For regression tasks:
K=1,
σ= identity function
For binary classification task:
σ(a)=sigmoid(a)=1+e−a1Model
yk(x,w)=σ(j=0∑Mwkj(2)⋅h(i=0∑Dwji(1)⋅xi))Weight Matrix
- i rows ⇒ # neurons in the next layer
- j columns ⇒ # neurons in the current layer
Each row in
Wi corresponds to the output of the next layer
- σ(wi⋅ai(L−1)+bi)=ai(L)
- σ(wTa(L−1)+b)=a(L)
General Neural Network
A directed acyclic graph from left to right
zk=h(j∑wkjzj)k = unit for which compute activation for
j = input for unit k
Universal Approximator
💡
A two layer neural network can approximate any function
Training Neural Networks
Regression Task
Likelihood function
p(t∣xn,w)p(t∣X,w,β)=N(t∣y(xn,w),β−1)=i=1∏Np(tn∣xn,w,β)Loss Function
E(w)=−logn=1∏Np(tn∣y(xn,w,β)=2βn=1∑N[y(xn,w)−tn]2−2Nlogβ+2Nlog(2π)≈21n=1∑N[y(xn,w)−tn]2Classification Task
Cross Entropy Loss
E(w)=−n=1∑Nj=1∑Ktknlogyk(xn,w)Training Using Gradient Descent
wt+1←wt−η∇E(w) Gradient
∇E(w) is hard to find due to model complexity
Training Using Backpropagation Algorithm
💡
Message passing algo with 2 steps:
➡️ Forward Pass
starting from the inputs successively compute the activations of each unit
- Apply input xn to the network
- Compute the activations of all units (hidden & output)
- Evaluate δkfor all output units
δk=yk−tk - yk= prediction of the model
Example: Single Neuron Per Layer
Formulas
C0a(L)z(L)∂a(L)∂C0∂z(L)∂a(L)∂w(L)∂z(L)=(a(L)−y)2=E0=σ(z(L))=w(L)a(L−1)+b(L)=2(a(L)−y)=σ′(z(L))=a(L−1)C0=(σ(w(L)a(L−1)+b(L))−y)2 ∂w(L)∂C0=∂w(L)∂z(L)∂z(L)∂a(L)∂a(L)∂C0=a(L−1)σ′(z(L))2(a(L)−y)∂b(L)∂C0=∂b(L)∂z(L)∂z(L)∂a(L)∂a(L)∂C0=1σ′(z(L))2(a(L)−y)∂a(L−1)∂C0=∂a(L−1)∂z(L)∂z(L)∂a(L)∂a(L)∂C0=w(L)σ′(z(L))2(a(L)−y)
⬅️ Backward Pass
successively compute the gradients of the error function with respect to the activations and the weights
- First compute the gradient of the output
- Backpropagate δ to compute δ for all hidden unitsδj=∂aj∂En=∂zj∂En⋅∂aj∂zj=[k∑∂ak∂En∂zj∂ak]⋅h′(aj)
Given
∂zjδak=wkj and
δakδEn=δk δj=h′(aj)k∑wkjδk
- Evaluate the derivatives w.r.t. weights wji:∂wji∂En=∂aj∂En∂wji∂aj=δjzi
Regularization
Weight Decay
Early stopping
Stop training as soon as the validation error starts increasing
⇒ Equivalent to some kind of weight decay
Dropout
During each training iteration, remove from the model (turn off) each unit with a probability of
pj turned off ⇒
δj=0 - Validate & test using the full model, scale each unit by the frequency at which it is on
Convolutional Neural Networks (CNNs)
Motivation
Neural Network Convention
Problem: Output should be invariant to translation, rotation, scaling, distortion, elastic deformation, other arbitrary artifacts
Naïve Solution: Fully Connected Networks w/ a large & diverse data to obtain invariance
- Each unit of each layer sees the whole image
- Ignores the key property of images: locality - near by pixels tend to have similar values
- Neural Networks are brittle ⚠️: small variances → big effect on output
- Information can be merged at later stages to get higher order features about the whole image