Neural Networks

From A Line To A Neuron

In week 2 we saw a model that computed price as a number times area plus another number. A neural network is built from many small units that each do something very similar. These units are called neurons, a name borrowed loosely from biology. The resemblance to brain cells is slight, so treat the name as a label and not as an explanation.

An artificial neuron receives several numbers as input. It multiplies each input by a weight, adds the results, adds a bias, and passes the total through an activation function. The weights and the bias are the parameters that training adjusts. The output is a single number that can be sent on to other neurons.

What One Neuron Computes

Suppose a neuron decides whether to go for a walk. It has two inputs: the temperature scaled to a number between 0 and 1, and whether it is raining, which is 1 for rain and 0 for none. It might give the temperature a weight of 2 and the rain a weight of negative 3, with a bias of negative 0.5. For a warm dry day with temperature 0.8 and no rain, the total is 2 times 0.8 plus negative 3 times 0, minus 0.5, which equals 1.1. A positive total leans toward yes.

The weights express how much each input matters and in which direction. A negative weight means that input pushes the answer down. The bias shifts the threshold for when the neuron becomes active. Training changes these values until the outputs match the labels.

Activation Functions

If a neuron only multiplied and added, stacking many of them would still give a straight-line relationship overall. Real patterns are curved. The activation function adds a bend. It takes the total and transforms it in a simple, fixed way.

Three activations are common. The sigmoid squeezes any number into the range from 0 to 1, which is useful for probabilities. The ReLU, short for rectified linear unit, returns zero for negative totals and the total itself for positive ones. It is simple and works well in deep networks. The softmax turns a list of numbers into probabilities that add up to 1, and it is used at the end of classifiers and language models.

With a bend in each neuron, a network of enough neurons can approximate very complicated relationships. This is why neural networks can handle images, speech, and text.

Layers And The Forward Pass

Neurons are arranged in layers. The input layer holds the raw numbers, such as the pixel values of an image. One or more hidden layers sit in the middle. The output layer gives the answer, such as the probability of each category. In a fully connected layer, every neuron receives the outputs of every neuron in the previous layer.

The forward pass is the process of computing the output from the input. Each layer takes the numbers from the layer before, applies weights, biases, and activations, and hands the result on. A network with two inputs, a hidden layer of three neurons, and one output has a small number of parameters you could count by hand: 2 times 3 weights plus 3 biases in the hidden layer, and 3 weights plus 1 bias in the output layer, for 13 in total.

Networks with many hidden layers are called deep, which gives deep learning its name. Early layers tend to capture simple patterns, such as edges in an image, and later layers combine them into more abstract ones, such as shapes and objects.

Learning By Backpropagation

Training a network follows the loop from week 2. Run a forward pass, compute the loss against the label, and adjust the parameters to lower the loss. The question is how to adjust millions of weights sensibly. Backpropagation answers it.

Backpropagation works backward from the loss. It asks, for each weight, how much would the loss change if this weight changed slightly? That quantity is the gradient. Using the chain rule from calculus, it computes the gradients layer by layer, starting at the output and moving toward the input. You can picture it as passing blame backward: the output was wrong by some amount, so the last layer takes its share of the blame, passes the rest to the layer before it, and so on.

Once each weight has its gradient, gradient descent moves it a small step in the direction that lowers the loss. Repeat over many batches of examples, and the network improves. You do not need to do the calculus yourself, because software libraries compute gradients automatically. The idea to hold is that every parameter learns from its own share of the error.

Training In Practice

Data is fed to the network in small groups called batches, because processing everything at once would need too much memory and gives fewer updates. A pass through all batches is an epoch, as in week 2. Training usually runs for many epochs while you watch the training loss and the validation loss.

Several choices matter. The learning rate sets the step size. The number and width of layers sets the model's capacity. Weights start as small random numbers, because identical starting values would make neurons behave identically. Deep networks overfit easily, so the habits from week 2 still apply: hold out a validation set, watch the gap, and stop when validation loss stops improving.

Training large networks uses graphics processing units, or GPUs. A GPU performs very many simple multiplications at the same time, which suits the arithmetic of neural networks.

A small mental model helps here. Think of each hidden neuron as a question the network invents for itself, such as is there a vertical edge here or is this word negative in tone. The weights decide which inputs matter for that question, and later layers combine many such answers into a final decision. No one writes these questions down. They emerge because answering them lowers the loss, which is the whole point of training.

Strengths And Limits

Neural networks learn useful features by themselves. Earlier methods needed a person to design features, while a network can discover them from raw pixels or raw text if given enough data. This is the main reason deep learning displaced older approaches in vision, speech, and language.

They also have costs. They need a lot of data and computing power. Their internal reasoning is hard to inspect, since the knowledge sits in huge tables of numbers. They can be confidently wrong on inputs unlike their training data. Keep these limits in mind in the coming weeks, because large language models inherit all of them.

Next week we look at the designs that made networks work well on text and images, ending with the transformer that powers language models.

Key Terms

Neuron

A unit that multiplies inputs by weights, adds a bias, and applies an activation function.

Weight

A parameter that scales one input and sets how much it matters.

Bias

A parameter added to the weighted sum that shifts when a neuron activates.

Activation function

A fixed bend applied to a neuron's total so networks can learn curved patterns.

Hidden layer

A layer between input and output whose neurons build intermediate features.

Forward pass

The computation that turns an input into an output through all the layers.

Backpropagation

A method that computes each weight's gradient by passing the error backward through the layers.

Batch

A small group of training examples processed together for one update.

Deep learning

Machine learning with neural networks that have many layers.

Exercise: Train A Tiny Network With PyTorch

1. Open a free Google Colab notebook. PyTorch is already installed there.
2. Paste the code from the slide titled Training a tiny network and run it. It learns the XOR function, which a single straight line cannot solve.
3. Watch the printed loss. Write down the loss at epoch 0 and at the last epoch.
4. Change the hidden layer size from 4 to 1 and run again. Record whether the network can still learn XOR.
5. Change the learning rate from 0.1 to 0.001 and note how training speed changes.
6. Write a short paragraph explaining in your own words what the hidden layer adds.

Further Reading

• Neural Networks and Deep Learning, Michael Nielsen, free online book (chapters 1 and 2)
• But what is a neural network?, 3Blue1Brown, video series on neural networks
• Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, MIT Press (chapter 6)
• Learning representations by back-propagating errors, Rumelhart, Hinton, and Williams, Nature, 1986