Evaluation, Hallucination, Bias, Safety, And Privacy

Why Measure At All

In weeks 7 and 8 you built systems that retrieve, call tools, and act. Each design choice, such as chunk size or tool wording, changes the results. Without measurement you are guessing whether a change helped. With measurement you can improve step by step and notice when a model update quietly breaks something.

Language model output is open-ended text, so measuring it takes more care than checking a number. The good news is that a small, well-chosen test set already gives a strong signal. Twenty to fifty real examples are enough to start.

Building An Evaluation Set

An evaluation set, often called an eval, is a collection of inputs with a way to judge the outputs. Start by collecting real questions from the people who will use your system. Add hard cases, odd phrasings, and questions the system should refuse or answer with 'not found'. For each one, write the expected answer or the key facts a good answer must include.

Then choose a scoring method. Exact match works for short factual answers. Checklists work when you can list required facts. For free-form text you can use a second model as a judge, following approaches such as the MT-Bench study by Zheng and colleagues, which found strong judge models agree with human raters much of the time. Judges have biases of their own, such as favoring longer answers, so spot-check their scores by hand.

Keep the eval set fixed while you experiment, and keep a few examples hidden that you never tune against. Otherwise you may improve the score by fitting the test questions rather than improving the system. Re-run the eval after every change and record the scores in a simple table.

Start small and grow. Ten questions run every day beat two hundred questions run once. A tiny script that runs the set and prints one score gives you a heartbeat for your project, and it takes only a few minutes to write.

• Public benchmarks measure general ability, but your own eval measures your task.
• Include a measure of cost and speed next to quality.
• Read failures one by one. The pattern in the failures tells you what to fix.

Hallucination

A hallucination is output that sounds fluent and confident but is false or unsupported by any source. Examples include invented citations, wrong dates, and functions that do not exist in a library. The survey by Ji and colleagues (2023) covers the topic in detail.

Models hallucinate because of how they work (weeks 4 and 5). A model predicts a likely next token. It optimizes for plausible text, and plausible text is not always true text. When it lacks a fact it often fills the gap with something that looks right. Rare facts, precise numbers, and questions about events after the training cutoff are the highest risk.

You cannot remove hallucination completely, but you can reduce it with design.

A concrete example helps. Suppose you ask a model for three research papers on sleep and memory. A fluent answer may list three authors, titles, and years that look real. Some may not exist. The safe workflow is to treat every citation as a lead, search for it yourself, and keep only what you can open and read.

• Ground answers in sources with RAG (week 7) and require citations.
• Allow 'I do not know' or 'not found' in the prompt, and test that it works.
• Use tools for exact facts such as arithmetic, dates, and database values.
• Ask the model to quote the supporting passage, then check the quote automatically.
• Lower the temperature setting for factual tasks so output varies less.
• Keep a human reviewer for high-stakes content such as medical, legal, or financial text.

Bias And Fairness

Bias in AI means systematic differences in quality or treatment across groups. It enters in several places. Training data reflects the people and views that wrote the text, so some languages, regions, and groups appear far more than others. Labels written by people carry their judgments. Design choices, such as who the system was tested on, leave gaps.

Real cases show the effect. The Gender Shades study (Buolamwini and Gebru, 2018) found that commercial face analysis systems made far more errors on darker-skinned women than on lighter-skinned men. The lesson is to test results by group instead of trusting one overall score.

A simple test for language systems uses paired prompts. Write the same request twice, changing only a name, a gender, or a nationality, and compare the answers for tone, length, and recommendations. Differences point to bias you should report and reduce, for example by adding examples, changing instructions, or limiting the system to tasks where it behaves consistently.

Safety

Safety covers harm that comes from a system's output or actions. Model providers train their models to refuse clearly harmful requests, and they publish usage policies. You add a second layer around your own product.

Key risks to plan for are listed below. Treat safety as a design habit and not a final checkbox. The NIST AI Risk Management Framework (2023) offers a plain structure: map the risks, measure them, manage them, and govern the process with clear owners.

• Misuse: people ask for harmful help. Add input and output checks and clear rules.
• Prompt injection: hidden instructions in documents or web pages (week 8). Limit tool power.
• Jailbreaks: prompts crafted to bypass refusals. Test your system with them before launch.
• Over-trust: users believe confident answers. Show sources, show uncertainty, label AI content.
• Unsafe advice: health, legal, and money topics need warnings and human review.
• Failure of the service: have a fallback when the API is slow or down.

Privacy

Whatever you send to a model API leaves your computer. Before sending, ask whether you need the data at all. Data minimization means sending only what the task requires. Replace names, emails, ID numbers, and addresses with placeholders when the task does not depend on them.

Read your provider's data terms. Key questions are whether your inputs are stored, for how long, who can read them, and whether they are used to train future models. Business and API plans often offer stricter terms than free consumer apps, but terms differ, so check the current documents instead of assuming.

Models can also memorize rare strings from training data. Carlini and colleagues (2021) showed that specific training text could be extracted from a large model. Never put secrets such as passwords or API keys into prompts or into documents you index for RAG. Laws such as the EU General Data Protection Regulation (GDPR) add duties around personal data, including a lawful basis for processing and the right of people to access and delete their data. Week 11 returns to these rules.

Retention matters too. Decide how long you keep logs and conversations, who can read them, and how a user can ask for deletion. Short retention periods reduce the harm of a leak, because data that no longer exists cannot be stolen.

A Practical Review Routine

Combine the ideas into a repeatable routine. Before launch, run the eval set, run the paired bias prompts, try ten injection and jailbreak attempts, and review the data you send out. After launch, log a sample of real conversations (with privacy rules applied), read some every week, and add every new failure to the eval set. Failures that reach real users become your best test cases.

Finally, tell users what the system is. A short note such as 'This assistant uses AI and can make mistakes. Check important facts' sets realistic expectations and treats users with respect.

Key Terms

Evaluation set (eval)

A fixed collection of inputs and expected results used to score a system.

LLM as judge

Using a second model to score the outputs of the first against a rubric.

Hallucination

Fluent output that is false or unsupported by any source.

Grounding

Tying answers to supplied sources so claims can be checked.

Bias

A systematic difference in quality or treatment across groups of people.

Jailbreak

A prompt crafted to make a model bypass its refusals.

Data minimization

Collecting and sending only the data a task requires.

PII

Personally identifiable information: details that identify a specific person.

Over-trust

Accepting AI output as correct without checking it.

Exercise: Evaluate And Red-Team A Small Assistant

1. Choose an assistant you have built or any chat model with a system prompt for a narrow task, such as answering questions about a course syllabus.
2. Write 20 test questions in a spreadsheet: 12 normal, 4 whose answers are not in the source, 4 tricky or badly phrased. Add a column with the expected answer.
3. Run all 20, paste the answers in a new column, and score each as correct, partly correct, or wrong. Compute the percent correct.
4. Write 5 paired prompts that differ only in a name or gender (for example, asking for career advice for 'Ahmed' and 'Anna'). Compare the pairs and note any difference.
5. Try 5 attacks: ask for something the assistant should refuse, and paste a document that contains a hidden instruction. Record what happened.
6. Change one thing in the system prompt (for example, add 'say not found when unsure'), re-run the 20 questions, and compare scores.
7. Write a half-page report with the score before and after, the worst failure, and one privacy risk in your setup.

Further Reading

• Survey of Hallucination in Natural Language Generation, Ji et al., 2023, ACM Computing Surveys
• Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, Buolamwini and Gebru, 2018
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al., 2023
• Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST, 2023
• Extracting Training Data from Large Language Models, Carlini et al., 2021, USENIX Security