What You Will Be Able To Do

• Explain why models miss private and recent facts.
• Estimate whether text fits a context window.
• Describe embeddings and cosine similarity.
• Build a small RAG pipeline.
• Choose between context, RAG, and tools.

Recap Of Week 6: Prompting

• A prompt is text the model reads top to bottom.
• Good prompts name task, role, examples, and format.
• Few-shot examples steer style and structure.
• Clear instructions beat clever tricks.
• Prompts shape answers but add no new facts.

The Knowledge Problem

• Training stops at a cutoff date.
• The model never saw your private files.
• Rare facts are remembered poorly.
• A confident answer can still be wrong.

Four Ways To Add Knowledge

• Context window: paste the facts.
• RAG: search, then paste the best pieces.
• Embeddings: search by meaning.
• Tools: let the model fetch or act.

The Context Window

• Measured in tokens, shared by prompt and answer.
• One token is about three quarters of an English word.
• Bigger windows cost more and run slower.
• Models use the middle of long input less well.

Embeddings Turn Meaning Into Numbers

• Text goes in, a vector comes out.
• Similar meaning gives nearby vectors.
• 'Refund' and 'get my money back' land close together.
• One embedding model for both documents and questions.

Cosine Similarity

• Measures the angle between two vectors.
• Near 1 means closely related.
• Near 0 means unrelated.
• Length of the vector does not matter.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(y * y for y in b))
    return dot / (na * nb)

print(cosine([1, 2, 3], [2, 4, 6]))  # 1.0

Vector Search

• Embed the query, rank stored vectors by similarity.
• Small data: compare against every vector.
• Large data: use an approximate index such as FAISS or pgvector.
• Hybrid search adds keyword matching.

RAG: The Pipeline

• Index: split, embed, store (done once).
• Retrieve: embed the question, take top k chunks.
• Augment: put chunks in the prompt.
• Generate: answer only from the sources.

question -> embed -> top 3 chunks
   |
   v
prompt = rules + chunks + question
   |
   v
model -> answer with citations

A Grounded Prompt

• Label each source with a number.
• Tell the model to use only the sources.
• Ask for citations such as [2].
• Give it permission to say 'not found'.

Answer using only the sources below.
Cite sources like [1]. If the answer is
not in the sources, reply: not found.

[1] Refunds are accepted within 30 days.
[2] Shipping takes 3 to 5 days.

Question: How long do I have to return an item?

Chunking Choices

• Start with 200 to 500 words per chunk.
• Overlap neighbors by a sentence or two.
• Split on headings and paragraphs when possible.
• Too big dilutes topic, too small loses context.

Why RAG Fails

• The right chunk never reached the prompt.
• The question and document use different wording.
• The answer needs several chunks combined.
• Fixes: tune size, raise k, rewrite query, re-rank.

Tool Use: The Model Asks, You Run

• You describe tools by name, purpose, and inputs.
• The model replies with a request to call one.
• Your code runs it and returns the result.
• The model reads the result and answers.

user: What is 1843 * 27?
model: call calculator(expr='1843*27')
app:   result = 49761
model: 1843 times 27 is 49,761.

Choosing An Approach

• Small and stable material: paste it in context.
• Large or growing library: use RAG.
• Live data, exact math, or actions: use tools.
• Style or format problems: consider fine-tuning.

Recap

• Models stop learning at their cutoff.
• Context windows are working memory with a price.
• Embeddings let search match meaning.
• RAG retrieves then answers from sources.
• Tools let the model fetch facts and act.

Next Week: Agents And Workflows

• Loops that combine models and tools.
• Planning, memory, and human approval.
• Where agents go wrong and how to limit damage.

Questions And Exercise

• Build the tiny RAG pipeline from the handout.
• Ask five questions, two with no answer in the text.
• Note which retrieved chunks helped or failed.