Week 10 of 12 · Intermediate

Building With An LLM API

You write real Python that calls a language model API, keeps a conversation going, returns structured data, and tracks cost and speed. You finish with a working command-line chat app you can extend in the capstone.

3 hours18 slides6 min read

Open slides Read study notes

From Concepts To Code Setup And Your First Call Building A Chat App Structured Output Cost Basics Latency Basics Putting The Pieces Together

You Will Be Able To

Make a first API call from Python and read the response and token usage.
Build a chat loop that keeps conversation history.
Request structured JSON output, validate it, and retry when it is invalid.
Estimate the cost of a request and reduce latency with streaming and shorter prompts.
Handle errors and protect an API key.

From Concepts To Code

Weeks 7 to 9 gave you the ideas: retrieval, tools, agents, and evaluation. This week you write the code that connects them. A large language model API is a web service. Your program sends text over the internet, a model on the provider's servers produces a reply, and the service sends the reply back. You pay per use, measured in tokens (week 7).

The examples use Python and the official Anthropic SDK, a library that wraps the web calls in simple functions. Other providers offer very similar SDKs, with the same ideas under different names. Learn the pattern once and you can move between providers. Always check the provider's current documentation for model names and prices, because both change often.

Setup And Your First Call

You need Python 3.9 or newer, an account with a provider, and an API key. An API key is a secret string that identifies and bills your account. Treat it like a password. Never paste it into source code, a public repository, a screenshot, or a chat. Store it in an environment variable and let the code read it from there.

In a terminal, create a virtual environment, install the SDK with pip install anthropic, and export the key as ANTHROPIC_API_KEY. The SDK reads that variable automatically. The first call needs three things: a model name, a list of messages, and a maximum number of output tokens. The reply contains a list of content blocks, and the text lives in the first block. The reply also reports how many tokens went in and out, which you will use for cost tracking.

A system prompt sets the assistant's role and rules for the whole conversation, and it is passed separately from the messages. Use it for the instructions you learned to write in week 6.

One API Call

Your code builds messages
Send over HTTPS with key
Model generates
Response returns
Your code uses it

The model runs on the provider's servers.

Building A Chat App

The API is stateless. It remembers nothing between calls, so your program must keep the history and send it every time. A chat app is a loop with a list. Read a line from the user, append it to the list as a message with role user, call the API with the whole list, append the reply with role assistant, and print it.

Because the full history is sent each turn, long chats grow expensive and eventually exceed the context window. Common fixes are to keep only the last N messages, or to ask the model to summarize older turns into a short note and continue from that note, the same memory technique you saw in week 8.

Streaming improves the feel of the app. Instead of waiting for the full reply, the SDK delivers text in small pieces, and you print each piece as it arrives. The total time is similar, but the user sees words appear within a moment, which feels much faster.

Test the app with an unexpected input, such as an empty line or a very long paste. A robust program prints a clear message and keeps running instead of crashing.

Input Tokens Per Turn If You Resend The History

tokens

Illustrative numbers. Because the API has no memory, each call carries the whole conversation, so cost grows with every turn. Source: Concept illustration

Structured Output

Programs need data, not prose. If you want a list of tasks extracted from an email, you want JSON your code can read, not a friendly paragraph. The simplest method is to ask for it: describe the exact keys and types in the system prompt, give one example, and say to reply with JSON only.

Then parse the reply with json.loads and validate it. Check that required keys exist and values have the right type. Models usually follow the format, but not always, so wrap the call in a retry: if parsing fails, send the error message back to the model and ask it to fix its answer. Two or three attempts handle nearly all cases.

Many providers also offer stricter options. Tool use (week 7) can force the model to fill in a schema, and some APIs support constrained JSON output. These raise reliability further, but the habit of validating everything stays valuable.

Here is a small example. Given the email 'Please send the budget by Friday and book the room for Monday', the extractor should return two tasks: one titled send the budget with due Friday, and one titled book the room with due Monday. Run a few messy emails through your extractor and count how often the first reply is already valid. That count tells you whether retries are a safety net or a daily need.

Cost Basics

Providers price per million tokens, with one price for input tokens and a usually higher price for output tokens. The cost of a request is input tokens times the input price plus output tokens times the output price, each divided by one million. Larger and more capable models cost more per token than smaller, faster ones.

Cost grows in ways beginners miss. In a chat, every turn resends the history, so the input grows each turn. In RAG, every retrieved chunk adds input tokens. In an agent, every loop turn adds more. The practical rule is to measure instead of guess: read the usage numbers from each response, log them, and add them up.

Pick the smallest model that passes your eval set.
Trim history and retrieve fewer, better chunks.
Cap the maximum output tokens.
Use prompt caching if your provider offers it, since a long fixed prefix is then billed at a reduced rate on repeat calls.
Use batch processing for work that can wait, which providers often discount.

Latency Basics

Latency is the time a user waits. Two parts matter: the time to the first token, and the speed of generation after that. Shorter prompts start sooner. Shorter answers finish sooner, because the model produces tokens one after another, so asking for a concise answer helps directly. Smaller models respond faster.

Network calls fail sometimes. Plan for timeouts, rate limit errors (too many requests per minute), and temporary server errors. A good pattern is retry with exponential backoff: wait one second, then two, then four, and give up after a few tries. The official SDKs retry some errors for you, and you can configure them. Show the user a clear message when all retries fail.

Putting The Pieces Together

The code in the slides gives you four building blocks: a single call, a chat loop, a JSON extractor with retry, and a usage logger. Together they make the core of the capstone assistant. Add retrieval from week 7 by inserting retrieved chunks into the system prompt or the latest user message. Add a tool by passing a tools list and running the function when the model asks.

Keep your code organized as you grow. Put the model name and prompts in one config area at the top of the file, so you can change them without hunting through the code. Save a log of requests and responses to a file while developing, because reading real examples is the best way to find problems. Remove personal data from those logs, following the privacy habits of week 9.

Finally, run your eval set from week 9 with the new code. A chat app that feels good in a quick test can still fail on the harder questions, and the eval will tell you.

Common API Errors

Status	Meaning	What to do
400	Bad request	Fix the input
401	Not authorised	Check the API key
429	Too many requests	Wait and retry
500	Server error	Retry with backoff

Key Terms

API: A defined way for one program to send requests to another program, often over the internet.
API key: A secret string that identifies and bills your account when you call an API.
System prompt: Standing instructions that set the assistant's role and rules for a conversation.
Stateless: Remembering nothing between calls, so the caller must resend any needed history.
Streaming: Receiving a reply in small pieces as it is generated instead of all at once.
Structured output: Model output in a fixed machine-readable format such as JSON.
Input and output tokens: The tokens you send and the tokens the model writes, priced separately.
Latency: The time a user waits between sending a request and seeing the reply.
Backoff: Waiting longer between each retry after a failed request.

Exercise: Build A Command-Line Study Assistant

Create a folder, a virtual environment, and install the SDK with pip install anthropic. Export your API key as an environment variable and check that echo shows it is set.
Run the first-call script from the slides and print the reply text and the token usage.
Extend it into the chat loop with a system prompt for a study helper. Type quit to exit.
Add streaming so that text appears as it is generated.
Add a command /extract that sends the last user text to the JSON extractor and prints a validated list of tasks or key facts.
Log input and output tokens for each call and print the total cost using the prices from your provider's current price page at exit.
Run your 20 week 9 questions through the app and compare the score with your earlier result.