How LLMs Work

A Large Language Model In One Paragraph

A large language model, or LLM, is a transformer network, as described in week 4, trained on a very large amount of text to predict the next token. It has billions of parameters. Given some text, it produces a probability for every possible next token, picks one, adds it to the text, and repeats. Fluent answers, translations, summaries, and code all come from this single repeated step.

This sounds too simple to explain what these systems do, and yet predicting the next token well requires learning a great deal about grammar, facts, style, and reasoning patterns found in the training text. The rest of this document explains how the model is built in stages.

Tokens

A model does not read letters or whole words. A tokenizer splits text into tokens, which are chunks drawn from a fixed vocabulary, often 30,000 to 200,000 entries long. Common words are one token. Rare words are split into pieces. A popular method, byte pair encoding, builds the vocabulary by repeatedly merging the pairs of characters that occur most often in a large sample of text.

As a rough rule for English, one token is about four characters, or three quarters of a word. Other languages and code often need more tokens for the same meaning, which makes them costlier and shortens what fits in the context window. Numbers and unusual spellings may be split into odd pieces, which is one reason models can make arithmetic and spelling errors.

Tokens matter in practice because providers price usage per token, and every model has a maximum number of tokens it can handle at once. Different models use different tokenizers, so the same text gives different counts. Always count with the tokenizer of the model you use.

Pretraining

Pretraining is the first and by far the most expensive stage. The model reads huge collections of text, such as web pages, books, articles, and code. For each position in the text, it predicts the next token. Its guess is compared with the real next token, and the loss from week 2 is used to adjust the parameters through backpropagation from week 3. No human labels are needed, because the text itself supplies the answers. This is called self-supervised learning.

After pretraining, we have a base model. It is very good at continuing text. Give it the beginning of a news article and it writes a plausible rest. Give it a question and it may continue with more questions, because that is what lists of questions look like on the web. A base model absorbs facts, but it also absorbs mistakes and biases from its data, and it does not reliably follow instructions.

Scale matters. Studies of scaling laws found that quality improves smoothly as you increase model size, data, and computing power together. A 2022 study from DeepMind, often called the Chinchilla paper, showed that many models were too large for the amount of data they had seen, and that data and model size should grow together.

Parameters In Three GPT Models

billion parameters

Published sizes. GPT-3 is about 1,500 times larger than GPT-1. Source: OpenAI papers and announcements for GPT-1, GPT-2 and GPT-3

Fine-Tuning For Instructions

To make a base model useful as an assistant, developers continue training on a much smaller, carefully chosen set of examples. Each example is an instruction paired with a good response written by people. This stage is called supervised fine-tuning, or instruction tuning. The model learns the format of a helpful exchange: a user asks, the assistant answers directly.

Fine-tuning changes behavior far more than knowledge. The facts come mostly from pretraining, and the manners come from fine-tuning. The same idea is available to builders. You can fine-tune a model on your own examples to teach it a specific style or format, though in many cases a clear prompt, covered in week 6, or retrieval, covered in week 7, is cheaper and enough.

Preference Tuning And RLHF

Writing perfect answers by hand is slow, but comparing two answers is easy. Preference tuning uses that fact. People are shown two or more responses to the same prompt and pick the better one. These choices train a reward model, a network that scores how much a person would like a response. The language model is then trained to produce responses that earn higher scores. This approach is called reinforcement learning from human feedback, or RLHF.

Newer methods skip the separate reward model. Direct preference optimization, or DPO, trains the language model straight from pairs of preferred and rejected answers. Some labs also use written principles and AI-generated feedback to supplement human labels.

Preference tuning makes models more helpful, polite, and safe, but it has side effects. A model trained to please may agree too readily with the user, a behavior called sycophancy. It may also sound confident while being wrong. Remember that the training goal is to match what raters preferred, and that is not always the same as what is true.

Three Training Stages

1. Pretraining on large text
2. Fine-tuning on instructions
3. Preference tuning (RLHF)
4. Chat assistant

How a base model becomes an assistant.

Generating Text: Sampling

At each step the model outputs probabilities for the next token. How we pick from them is called decoding. Always taking the most likely token is called greedy decoding. It is stable but can be dull and repetitive. Sampling picks randomly in proportion to the probabilities, which gives variety.

Temperature controls the spread. A low temperature, near 0, sharpens the probabilities so the top token almost always wins, which suits tasks needing consistent answers. A higher temperature flattens them, giving more varied and sometimes more creative text, along with more errors. Top-p sampling, also called nucleus sampling, restricts the choice to the smallest group of tokens whose probabilities add up to a chosen amount, such as 0.9.

Because of sampling, the same prompt can produce different answers on different runs. This is normal, and it matters for testing, which we cover in week 8.

• Low temperature: consistent, predictable output.
• High temperature: varied output, more risk of errors.
• Top-p: sample only from the most likely group of tokens.

Temperature Settings

Setting	Behaviour	Good for
Low (near 0)	Picks the likeliest token, repeatable	Extraction, code, facts
Medium	Some variety	Everyday writing
High	Varied, riskier	Brainstorming, names

The Context Window

The context window is the maximum number of tokens a model can consider at once. It includes everything: your instructions, the conversation so far, any documents you paste in, and the answer being written. Limits vary by model and have grown over time, from a few thousand tokens to hundreds of thousands and beyond for some models.

The model has no memory outside this window. When a chat appears to remember earlier messages, the application is sending those messages again on each turn. Text that does not fit is cut off or dropped, and the model cannot use it. Even within the window, models can use information near the beginning and end of a long input better than information buried in the middle, so placement matters.

Two practical rules follow. Put the most important instructions where they are easy to find, and include only the text the task needs. When the material is larger than the window, we do not paste everything. We retrieve the relevant pieces first, which is the subject of week 7.

Finally, remember what an LLM is not. It is not a database that looks up facts, and it does not check its output against the world. It produces likely text. That explains both its fluency and its habit of inventing details, often called hallucination. Week 6 shows how to prompt around this, and week 8 how to test for it.

Key Terms

Large language model

A transformer with billions of parameters trained to predict the next token in text.

Tokenizer

A tool that splits text into tokens drawn from a fixed vocabulary.

Pretraining

The first training stage, where the model learns to predict the next token on huge text collections.

Base model

A model after pretraining only, good at continuing text but not at following instructions.

Fine-tuning

Further training on a smaller, curated set of examples to shape behavior.

RLHF

Reinforcement learning from human feedback, where human choices between answers guide training.

Temperature

A setting that controls how random the choice of the next token is.

Context window

The maximum number of tokens a model can consider at once, including input and output.

Hallucination

A fluent output that contains invented or false details.

Exercise: Count Tokens And Watch Sampling

1. Install the tiktoken library in a Colab notebook with pip install tiktoken, or use any free online tokenizer page.
2. Count the tokens in three texts of about 100 words each: plain English, a paragraph in another language you know, and a short piece of Python code. Record the counts.
3. Compare tokens per word for each text and write one sentence explaining the differences.
4. In a free chat assistant, ask for a one-sentence slogan for a coffee shop five separate times. Note how much the answers vary.
5. If you have API access or a playground with a temperature setting, repeat with temperature 0 and temperature 1. Otherwise ask the assistant to answer in the most predictable way and then in the most unusual way.
6. Write three lines on what you saw and when you would prefer low temperature.

Further Reading

• Language Models are Few-Shot Learners, Brown et al., OpenAI, 2020 (GPT-3 paper)
• Training language models to follow instructions with human feedback, Ouyang et al., OpenAI, 2022
• Training Compute-Optimal Large Language Models, Hoffmann et al., DeepMind, 2022
• Intro to Large Language Models, Andrej Karpathy, video lecture
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model, Rafailov et al., Stanford, 2023