What You Will Be Able To Do

• Explain why text becomes numbers
• Describe and compare embeddings
• Explain attention as weighted looking
• Name the parts of a transformer block
• Compute cosine similarity in Python

Attention Is All You Need

• Google researchers publish the transformer
• Every word looks at every other word at once
• Trains fast on GPUs
• Used by nearly every large language model

What An Embedding Is

• A learned vector per token
• Hundreds or thousands of numbers
• Similar usage gives similar vectors

cat -> [ 0.21, -0.40,  0.77, ... ]
dog -> [ 0.19, -0.35,  0.70, ... ]
car -> [-0.62,  0.11, -0.05, ... ]

Meaning As A Map

• Each vector is a point in many dimensions
• Related words sit close together
• Directions can carry meaning
• king minus man plus woman lands near queen

Cosine Similarity

• Compares direction, ignores length
• Used in search and retrieval in week 7

cos(a, b) = (a . b) / (|a| * |b|)

 1  same direction
 0  unrelated
-1  opposite

Same Word, Different Meaning

• river bank versus bank account
• One fixed vector per word is not enough
• Layers refine vectors using nearby words
• Attention does the refining

Attention In One Sentence

• Each token decides how much to look at the others
• It then blends their information by those weights
• The trophy did not fit the suitcase because it was big
• What does it refer to?

Query, Key, Value

• Query: what this token is looking for
• Key: what each token offers
• Value: the information it passes on
• Learned weights produce all three

Scores, Softmax, Blend

• Compare query with every key
• Softmax turns scores into weights
• Weighted sum of values is the result

scores  = query . every key
weights = softmax(scores)    # sums to 1
new_vec = sum(weights * values)

Many Heads

• Words relate in several ways at once
• Run several attentions in parallel
• Each head has its own learned weights
• Heads often specialize

The Transformer Block

• Residual paths keep information flowing
• Normalization keeps numbers stable

tokens
  -> multi-head attention
  -> add input back (residual)
  -> small network per token
  -> add input back, normalize
  -> next block (repeat many times)

From Transformer To Language Model

• Causal mask: look only at earlier tokens
• Last layer scores every vocabulary token
• Softmax gives next-token probabilities
• Repeat to generate text

Why This Design Won

• All positions compute at the same time on GPUs
• Scales steadily with data and parameters
• Handles long-range links well
• Cost: attention work grows with text length

Compute Similarity In Python

import numpy as np
def cos(a, b):
    a, b = np.array(a), np.array(b)
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
print(cos([1, 2, 3], [2, 4, 6]))   # 1.0
print(cos([1, 0], [0, 1]))         # 0.0

Recap

• Text becomes tokens, then embedding vectors
• Similar usage means close vectors
• Attention blends information from other tokens
• Transformer blocks stack attention and small networks

Next Week: How LLMs Work

• Tokens and tokenizers in detail
• Pretraining on large text collections
• Fine-tuning and preference tuning
• The context window

Questions And Exercise

• Exercise: measure meaning with embeddings
• Embed 8 sentences, find nearest neighbors
• Read The Illustrated Transformer
• Bring one question to week 5