Week 7 of 12 · Beginner+

Giving Models Knowledge: Context, Retrieval, Embeddings, Tools

You learn four ways to get facts and abilities into a model that was trained months ago: the context window, retrieval (RAG), embeddings with vector search, and tool use. You finish able to choose between them and to build a small retrieval pipeline.

3 hours18 slides7 min read

Open slides Read study notes

Recap Of Week 6 And The Knowledge Problem The Context Window: The Model's Working Memory Embeddings: Turning Meaning Into Numbers Vector Search Retrieval-Augmented Generation (RAG)Chunking And Why RAG Fails Tool Use: Letting The Model Act Choosing An Approach

You Will Be Able To

Explain why a trained model does not know your private or recent information.
Estimate whether a document fits in a context window and what that costs.
Describe how embeddings turn text into vectors and how cosine similarity finds related text.
Build a tiny retrieval-augmented generation (RAG) pipeline with chunking, search, and a grounded prompt.
Choose between context stuffing, RAG, and tool use for a given problem.

Recap Of Week 6 And The Knowledge Problem

In week 6 you learned to write prompts. A good prompt names the task, gives the model a role, shows an example or two, and says what format the answer should take. You also saw that a prompt is only text, and that the model reads it from top to bottom like any other input.

That raises a practical question. A model learns from a fixed collection of text during training, and training stops on a certain date, called the training cutoff. After that day the model learns nothing new by itself. It has never seen your company handbook, your lecture notes, or this morning's news.

So how do you get new facts in front of the model? This week covers four answers. Each answer trades effort, cost, and reliability differently, and real systems often combine them.

Put the facts directly in the prompt (the context window).
Search a collection of documents and paste only the relevant pieces (retrieval, or RAG).
Use embeddings to make that search match meaning instead of exact words.
Let the model call tools, such as a calculator or a database query, to fetch facts itself.

Four Ways To Add Knowledge

Method	Best when
Paste into the prompt	The text is short
Retrieval (RAG)	Large, changing documents
Tools	Live data or actions are needed
Fine-tuning	You need a style or skill, not facts

The Context Window: The Model'S Working Memory

The context window is the maximum amount of text a model can consider at once, measured in tokens. A token is a small piece of text, roughly three quarters of an English word on average. Your prompt, any documents you paste, the conversation so far, and the model's own answer all share this one budget.

Modern models accept very large windows, from tens of thousands to hundreds of thousands of tokens, and some accept more. A large window lets you paste a whole report and ask questions about it. This is the simplest way to give a model knowledge, and it is a good first choice when the material is small.

Large windows still have limits. Every token you send costs money and adds waiting time, so pasting a hundred pages for every question gets expensive. Research such as the 'Lost in the Middle' paper found that models often use information near the start and end of a long input better than information buried in the middle. More context is not automatically better context.

A useful habit is to count before you paste. Divide the number of characters by four for a rough token estimate in English. A ten page document of about five thousand words is roughly seven thousand tokens, which fits comfortably in most windows.

Embeddings: Turning Meaning Into Numbers

Keyword search finds documents that contain the words you typed. It misses a document about 'canines' when you search for 'dogs'. Embeddings fix this by mapping text to a list of numbers called a vector, so that texts with similar meaning land close together.

An embedding model is a neural network (week 3) trained for this job. You send it a sentence and it returns a vector with several hundred or a few thousand numbers. You cannot read the individual numbers, but you can compare two vectors. The standard comparison is cosine similarity, which measures the angle between them. A value near 1 means very similar direction, and a value near 0 means unrelated.

Think of each vector as a point on a huge map of meaning. 'Refund policy' and 'how do I get my money back' sit in the same neighborhood, even though they share no words. This is why embeddings power modern search, recommendations, and clustering.

Same embedding model must embed both the documents and the question.
Vectors from different models are not comparable with each other.
Embeddings capture topic and meaning well, but exact names and numbers less well.

Vector Search

Vector search answers the question: given the vector of a query, which stored vectors are closest? For a few thousand chunks you can compare the query against every vector with a simple loop. This brute force approach is exact and easy to understand.

For millions of vectors, systems use approximate nearest neighbor indexes. These skip most comparisons and return very close matches quickly, trading a tiny amount of accuracy for large speed gains. Tools that offer this include FAISS, and databases such as PostgreSQL with the pgvector extension, plus dedicated vector databases.

Many production systems run hybrid search: a keyword method such as BM25 and a vector method together, then merge the ranked results. Hybrid search catches exact terms like product codes that embeddings can blur.

Retrieval-Augmented Generation (RAG)

RAG combines search with generation. The idea was named in a 2020 paper by Lewis and colleagues, and it is now a standard pattern. Instead of hoping the model remembers a fact, you look the fact up and hand it over.

The pipeline has two phases. In the indexing phase, which you run once, you split your documents into chunks, embed each chunk, and store the vectors. In the query phase, which runs on every question, you embed the question, find the top few closest chunks, and build a prompt that contains those chunks followed by the question. The model then answers using the supplied text.

Index: load documents, split into chunks, embed, store.
Retrieve: embed the question, take the top k chunks (k is often 3 to 8).
Augment: place the chunks in the prompt with clear labels.
Generate: instruct the model to answer only from the chunks and to say when the answer is missing.
Cite: ask the model to name which chunk supports each claim.

The RAG Pipeline

Split documents into chunks
Embed each chunk
Embed the question
Find nearest chunks
Add them to the prompt
Model answers

Retrieval first, then generation.

Chunking And Why RAG Fails

Chunking means cutting documents into pieces small enough to embed and retrieve precisely. A common starting point is 200 to 500 words per chunk with a small overlap between neighbors, so a sentence cut in half still appears whole in one chunk. Splitting along natural boundaries such as headings and paragraphs usually beats cutting at a fixed character count.

Most RAG failures come from retrieval, not generation. If the right chunk never reaches the prompt, the model cannot use it. Check retrieval first by printing the chunks for several test questions and reading them yourself.

Typical problems include chunks that are too large and dilute the topic, chunks too small to carry a full idea, questions phrased very differently from the documents, and answers that need several chunks combined. The fixes are matching: adjust chunk size, add overlap, retrieve more chunks, rewrite the query, or add a re-ranking step that reorders candidates with a stronger model.

Tool Use: Letting The Model Act

Retrieval pulls text from a fixed collection. Tool use goes further: you describe functions the model may request, and your program runs them. Examples are a calculator, a weather lookup, a database query, or a search engine.

The flow has four steps. First, you send the question together with a list of tool descriptions, each with a name, a plain-language purpose, and the inputs it takes. Second, the model replies either with an answer or with a request such as 'call get_weather with city equal to Lisbon'. Third, your code runs the function and sends the result back. Fourth, the model reads the result and writes the final answer.

The model never runs code itself. It only writes the request, and your program decides whether to execute it. This keeps you in control, and it is the basis for the agents you will meet in week 8. Tools solve problems that retrieval cannot: exact arithmetic, live data, and actions in the world such as creating a calendar event.

Tool Use: The Model Asks, You Run

Model sees question and tool list
Model requests a tool call
Your code runs the tool
Result goes back to the model
Model writes the answer

The model never runs code itself.

Choosing An Approach

Start with the simplest method that works. If the material fits in the context window and changes rarely, paste it. If the collection is large, grows over time, or needs per-question selection, use RAG. If the answer requires live data, exact computation, or an action, add tools. Many assistants use all three: a system prompt with rules, retrieved passages for knowledge, and tools for fresh data.

Fine-tuning, which means further training a model on your examples, is a fourth option. It is best for teaching style, format, or narrow skills. It is a weaker choice for teaching facts, because facts change and a retrieval index is easy to update.

Whichever you choose, keep a short list of test questions with known answers. You will use it in week 9 to measure whether changes help.

Key Terms

Context window: The maximum number of tokens a model can read and write in one request.
Training cutoff: The date after which no new text was included in the model's training data.
Embedding: A vector of numbers that represents the meaning of a piece of text.
Cosine similarity: A score from about 0 to 1 that compares the direction of two vectors, used to rate how related two texts are.
Vector search: Finding the stored vectors closest to the vector of a query.
Chunk: A small piece of a document, sized to be embedded and retrieved on its own.
RAG: Retrieval-augmented generation: look up relevant text first, then ask the model to answer using it.
Tool use: A pattern where the model requests a function call and your program runs it and returns the result.
Grounding: Basing an answer on supplied sources so that each claim can be checked.

Exercise: Build A Tiny RAG Pipeline

Collect five to ten short text files or paragraphs about one topic you know, such as your course notes or a product FAQ.
Split them into chunks of about 150 words each, with a one-sentence overlap.
Use a free embedding option (for example the sentence-transformers library with the all-MiniLM-L6-v2 model) to embed every chunk and keep the vectors in a Python list.
Write a function that embeds a question and returns the three chunks with the highest cosine similarity.
Paste the question and the three chunks into a chat model using a prompt that says: answer only from the sources, cite the chunk number, and say 'not found' otherwise.
Ask five questions, including two whose answers are not in your documents. Record which chunks were retrieved and whether the answer was correct.
Write three sentences on what failed and which change (chunk size, k, wording) you would try next.