Week 4 of 12 · Beginner+

Deep Learning To Transformers

You learn how networks represent words as numbers called embeddings and how attention lets each word look at the others. You finish with a clear picture of the transformer, the design behind modern language models.

3 hours18 slides6 min read

Open slides Read study notes

Networks For Different Kinds Of Data Turning Words Into Numbers Embeddings Context Changes Meaning Attention, The Intuition Multiple Heads And The Transformer Block From Transformer To Language Model

You Will Be Able To

Explain why text must be turned into numbers before a network can use it.
Describe an embedding and measure similarity between two embeddings.
Explain attention as a weighted look at other words in the same text.
Name the main parts of a transformer block and what each one does.
Compute cosine similarity between small vectors in Python.

Networks For Different Kinds Of Data

In week 3 we built a fully connected network. It works, but it ignores the structure of the data. Researchers designed special layouts for data with a shape. Convolutional neural networks, or CNNs, slide small filters across an image so that the same pattern, such as an edge, is detected anywhere in the picture. They became the standard for vision after AlexNet in 2012.

Text arrives as a sequence, where order matters. Recurrent neural networks, or RNNs, read one word at a time and carry a memory forward. Variants such as the LSTM improved this. They had two weaknesses: they process words one after another, which is slow to train, and they struggle to remember something from far back in a long passage.

In 2017 a team at Google published the paper Attention Is All You Need. It proposed the transformer, which drops the step-by-step reading and instead lets every word look at every other word at once. Transformers train quickly on modern hardware and handle long-range links well. Almost every large language model since then uses this design.

Architectures By Data Type

Data	Typical network
Images	Convolutional network
Sequences over time	Recurrent network
Text at scale	Transformer

Turning Words Into Numbers

A network works with numbers, so text needs a conversion. The first step splits text into small pieces called tokens, which we study in week 5. For now think of each token as roughly a word or part of a word. Each token in the vocabulary receives an integer id, such as 5012 for the token cat.

A bare id is a poor input, because the number 5012 says nothing about meaning. A cat and a dog are similar, but their ids are unrelated. Another idea, one-hot encoding, gives each token a long list of zeros with a single one. It treats every pair of words as equally different, which also hides meaning.

Embeddings

An embedding is a list of numbers, called a vector, that represents a token. A typical embedding has hundreds or thousands of numbers. The key point is that the numbers are learned during training, so that tokens used in similar ways end up with similar vectors.

You can picture each vector as a point in a space with many dimensions. Words with related meaning sit close together. Cat and dog are near each other and far from carburetor. Directions in the space can also carry meaning. A well-known example from early word embedding research is that the vector for king, minus man, plus woman, lands near the vector for queen. Treat this as an illustration, since real results are messier.

To measure closeness we use cosine similarity. It compares the angle between two vectors and returns a value near 1 for vectors pointing the same way, near 0 for unrelated ones, and below 0 for opposite ones. It ignores the length of the vectors and looks only at direction. The same measure powers the search tools we use in week 7.

Embedding table: one learned vector for each token in the vocabulary.
Similar usage leads to similar vectors.
Cosine similarity compares direction, from -1 to 1.

Context Changes Meaning

A single fixed vector per word has a problem. The word bank means something different in river bank and bank account. The vector should depend on the surrounding words. Transformers solve this by refining each token's vector layer by layer, using the other tokens in the text. After several layers, the vector for bank in a sentence about rivers differs from the vector for bank in a sentence about money.

The mechanism that does the refining is attention.

Attention, The Intuition

Attention lets a token gather information from other tokens. Consider the sentence: The trophy did not fit in the suitcase because it was too big. To understand what it refers to, the model should look at trophy and suitcase and weigh them. Attention does exactly this: each token decides how much to look at every other token, then blends their information using those weights.

A useful way to describe the machinery uses three roles. Each token produces a query, a key, and a value, all vectors computed by learned weights. The query says what this token is looking for. The keys say what each token offers. Comparing a query with every key gives a score for each pair. The scores pass through softmax, from week 3, to become weights that add up to 1. The token's new vector is the weighted sum of the values.

Think of a library. Your query is the topic you want. Each book has a key on its spine. You rate how well each spine matches, then read mostly from the best matches, taking a little from the others. The result is a summary built from the most relevant pieces.

Inside Attention

Tokens become embeddings
Compare query with keys
Scores
Softmax makes weights
Blend the values

How one token gathers context.

Multiple Heads And The Transformer Block

One attention pattern is not enough, because words relate to each other in several ways at once: grammar, reference, topic, and more. Multi-head attention runs several attention operations in parallel, each with its own learned weights, and joins the results. Different heads often specialize in different relationships.

A transformer block combines attention with a small fully connected network, like the one from week 3, that processes each token on its own. Around both sit two helpful tricks. Residual connections add the block's input back to its output, which keeps information flowing through deep stacks. Layer normalization keeps the numbers in a stable range. A full model stacks dozens of identical blocks.

One more detail is position. Attention by itself ignores word order, so the model adds position information to each token's vector. Without it, dog bites man and man bites dog would look the same.

From Transformer To Language Model

A language model predicts the next token. To do this, the transformer uses a causal mask in each attention step, so a token may look only at earlier tokens and never at later ones. This matches the real task, where the future is unknown. The last layer produces a score for every token in the vocabulary, and softmax turns the scores into probabilities for the next token.

Two features explain the success of the design. Attention steps for all positions can run at the same time on a GPU, so very large models can be trained. And performance tends to improve steadily as you add more data, parameters, and computing power. Week 5 shows how a model of this kind is trained on text and then shaped into a chat assistant.

One cost deserves mention. Attention compares every token with every other token, so the work grows quickly as the text gets longer. This is one reason there is a limit on how much text a model can read at once, a limit called the context window.

As a quick check on your understanding, take the sentence The animal did not cross the street because it was tired. When the model processes the token it, attention can give a high weight to animal and a low weight to street, so the new vector for it carries information about the animal. Change tired to wide, and a well-trained model shifts its weight toward street. The sentence structure is nearly the same, but the meaning, and therefore the attention pattern, is different.

Recurrent Networks Versus Transformers

	Recurrent network	Transformer
Reads text	One token at a time	All tokens at once
Long range links	Fade over distance	Direct through attention
Training on GPUs	Hard to parallelise	Easy to parallelise

Key Terms

Token: A small piece of text, roughly a word or part of a word, that a model reads as one unit.
Embedding: A learned list of numbers that represents a token so similar tokens have similar vectors.
Vector: An ordered list of numbers, which can be seen as a point in a many-dimensional space.
Cosine similarity: A measure of how closely two vectors point in the same direction, from -1 to 1.
Attention: A mechanism where each token blends information from other tokens using learned weights.
Query, key, value: The three vectors per token used in attention: what it seeks, what it offers, and what it passes on.
Multi-head attention: Several attention operations run in parallel so the model can track different relationships.
Transformer: A network design built from stacked blocks of attention and small networks, introduced in 2017.
Causal mask: A rule that lets each token see only earlier tokens when predicting the next one.

Exercise: Measure Meaning With Embeddings

Open a free Google Colab notebook and run: pip install sentence-transformers.
Load the model with: from sentence_transformers import SentenceTransformer; m = SentenceTransformer('all-MiniLM-L6-v2').
Write 8 short sentences on three topics, for example cooking, football, and banking. Include one sentence using the word bank in a river sense.
Embed all sentences with m.encode(list_of_sentences) and compute the cosine similarity matrix with sklearn.metrics.pairwise.cosine_similarity.
For each sentence, print its most similar neighbor. Check whether the neighbors share a topic.
Write a short note on any surprising pairs, especially for the river bank sentence, and explain what the result says about context.