You will see how stacked neurons learn, and train a tiny network.
Outcomes
What You Will Be Able To Do
Describe what one neuron computes
Explain why activations add curves
Trace a forward pass through layers
Explain backpropagation as passing blame
Run and modify a tiny network
Idea
From A Line To A Neuron
Week 2: price = w times area plus b
A neuron does the same with many inputs
The name comes loosely from biology
Treat it as a label, not an explanation
Mechanism
What One Neuron Computes
Multiply each input by a weight
Add the results and a bias
Pass the total through an activation
total = w1*x1 + w2*x2 + b
output = activation(total)
Example
A Neuron Deciding On A Walk
Negative weight pushes the answer down
Training would find these numbers
x1 = 0.8 # warmth, scaled 0 to 1
x2 = 0 # raining? 1 = yes
total = 2*x1 + (-3)*x2 + (-0.5)
# total = 1.1 (leans yes)
Activation
Why A Bend Is Needed
Only multiply and add gives a straight line
Stacked straight lines are still one line
Real patterns are curved
An activation adds the bend
Activation
Three Common Activations
Sigmoid for single probabilities
ReLU is simple and works well deep
Softmax ends classifiers and language models
Name
Rule
Output range
ReLU
max(0, x)
0 and up
Sigmoid
1 / (1 + e^-x)
0 to 1
Tanh
(e^x - e^-x) / (e^x + e^-x)
-1 to 1
sigmoid(z) = 1 / (1 + e^-z) # 0 to 1
relu(z) = max(0, z) # 0 or z
softmax -> probabilities summing to 1
Structure
Layers
Input layer holds the raw numbers
Hidden layers build intermediate features
Output layer gives the answer
Fully connected: each neuron sees the whole previous layer
inputs -> [hidden layer] -> [hidden layer] -> outputs
x1 \ o o o o o p1
x2 --> o o o ... o o --> p2
Computation
The Forward Pass
Compute layer by layer from input to output
Each layer applies weights, bias, activation
2 inputs, 3 hidden, 1 output: 13 parameters
Count them: 2x3+3 and 3x1+1
Depth
Deep Means Many Layers
Early layers find simple patterns like edges
Later layers combine them into shapes
Deepest layers represent objects and concepts
Networks discover features by themselves
Training
The Training Loop Again
Same loop as week 2
Backpropagation supplies the gradients
for each batch:
out = network(x) # forward pass
loss = compare(out, y) # how wrong?
loss.backward() # gradients
update the weights
Intuition
Backpropagation: Passing Blame
Ask how loss changes if one weight moves
That answer is the weight's gradient
Start at the output and work backward
Each layer takes its share of the blame
Practice
Batches, Epochs, And Starts
Batches: small groups for each update
Epoch: one pass through all batches
Start weights as small random numbers
Identical starts make neurons act identically
Code
Training A Tiny Network
import torch, torch.nn as nn
X = torch.tensor([[0.,0.],[0.,1.],[1.,0.],[1.,1.]])
y = torch.tensor([[0.],[1.],[1.],[0.]])
net = nn.Sequential(nn.Linear(2,4), nn.ReLU(), nn.Linear(4,1), nn.Sigmoid())
opt = torch.optim.SGD(net.parameters(), lr=0.1)
for epoch in range(5000):
loss = nn.functional.binary_cross_entropy(net(X), y)
opt.zero_grad(); loss.backward(); opt.step()
print(net(X).round().flatten())