Designing A Pilot
You learn to turn an approved use case into a small, measured experiment with a hypothesis, a baseline, clear metrics, a tight scope and a go or no-go rule. You finish with a one-page pilot plan ready for sign-off.
You Will Be Able To
- Write a pilot hypothesis that names the users, the task, the metric and the expected change.
- Capture a baseline before the pilot starts and choose a comparison method.
- Select outcome, quality, adoption and safety metrics with agreed targets.
- Set scope, roles, timeline and guardrails for a pilot of six to eight weeks.
- Define go, adjust and stop criteria before the pilot begins.
What A Pilot Is For
A pilot is a small, time-limited trial of a solution with real users and real work. Its purpose is to learn whether the idea delivers value in your organization, at acceptable risk and cost, before you commit larger budgets. It is not a demo, which only shows that a tool can work, and it is not a full rollout, which assumes that it does. Many AI projects stall in the gap between a good demo and a decision, because nobody defined what the trial should prove.
You now have the ingredients. Week 2 gave you a ranked use case and a one-page brief. Week 3 gave you a data inventory. Week 4 gave you a route and a cost view. Week 5 gave you rules and a risk tier. This week combines them into one plan that a sponsor can approve in a single meeting.
A pilot also builds skill and trust, which matter as much as the numbers.
Start With A Hypothesis
A hypothesis is a testable statement of what you expect. A useful template is: if we give this tool to these users for this task, then this metric will change by this amount, because of this reason. For example: if we give twelve support agents an AI reply assistant for billing emails, then the average time to first reply will fall from six hours to four, because agents start from a draft instead of a blank page.
Writing it down forces precision. It names the people, the task and the measure. It states an amount, so you can later say whether you reached it. And the 'because' clause records your reasoning, so that if the result misses you can ask which belief was wrong. A pilot with no hypothesis can only produce opinions.
- Hypothesis
- Baseline
- Run with guardrails
- Compare with targets
- Go, adjust or stop
Decision rules are written at the first step, not the last.
Measure The Baseline
A baseline is how the work performs today, measured the same way you will measure during the pilot. Without it, any result is a story. Gather it before the tool is switched on. Use existing system data where it exists, such as ticket timestamps, and short manual samples where it does not, such as timing twenty cases with a stopwatch. Record volume, time per item, error or rework rate, and the quality score if you have one.
Decide how to compare. The simplest design is before and after for the same people, but results may shift for other reasons, such as seasonal volume. A stronger design uses a control group: a similar team or a random half of the users who continue as normal. Even a small control group makes your conclusions more trustworthy. When you cannot create a control, state that limit in the report. Honest limits make results more credible, not less.
Choose Success Metrics
Use a small set of metrics in four groups. Outcome metrics measure the business result: time saved, cost per case, revenue, cycle time. Quality metrics measure how good the work is: error rate, customer satisfaction, review scores, how often a person must correct the output. Adoption metrics measure whether people use the tool: share of eligible users active each week, share of tasks started with the tool. Safety metrics, sometimes called guardrails, measure harm: data incidents, wrong answers reaching customers, complaints.
Set a target and a minimum for each before starting. A good tool that saves time but raises errors is not a success, so pair every speed metric with a quality metric. Be careful with measures that are easy to game. Ten drafted replies are worthless if staff rewrite all of them. Keep the list to five or six metrics so the team can follow them.
| Group | Metric | Baseline | Target |
|---|---|---|---|
| Outcome | Time to first reply | 6 hours | 4 hours |
| Quality | Customer satisfaction score | Measure first | Not lower than baseline |
| Quality | Share of drafts heavily rewritten | Not applicable | Below 30 percent |
| Adoption | Agents using tool weekly | Not applicable | At least 70 percent |
| Safety | Wrong answers reaching customers | Measure first | Zero serious cases |
Illustrative numbers for the hypothesis in the text. Set your own from your baseline.
Set The Scope And Roles
Keep the scope narrow. Pick one use case, one team of roughly ten to thirty users, one or two data sources, and a period of six to eight weeks. Write what is in scope and, just as important, what is out. For example, the assistant may draft billing replies but may not decide refunds. Specify human review: who checks output and when. Confirm the data, access and legal checks from weeks 3 and 5 before day one.
Name the roles. A sponsor is a senior leader who funds the pilot and makes the final decision. A business owner runs the process being changed. A champion is a respected user who helps colleagues and gathers feedback. A technical lead sets up and monitors the tool. A risk reviewer from legal, security or data protection signs off on guardrails. One person may hold two roles in a small company, but every role needs a name.
Plan The Timeline
A typical pilot has five phases. Preparation covers approvals, baseline and access, and takes about two weeks. Setup covers configuration, test cases and user training, and takes about two weeks. The run is the live period of about six weeks, with a short check-in every week to review metrics, collect feedback and fix issues. Analysis takes a week to compare results with the baseline and the targets. The decision meeting closes the pilot.
Build in learning loops. In the first two weeks of the run, expect confusion and low use, which is normal. Gather examples of good and bad outputs. Adjust instructions or training, but do not change the metrics or the targets during the run. If you must change scope for a safety reason, record when and why. At the end, the story of what happened should be clear from your notes without anyone needing to remember.
Illustrative plan of about twelve weeks end to end. Adjust to your organization.
Decide: Go, Adjust Or Stop
Agree the decision rules before the pilot starts. Go means the targets are met, safety metrics are clean and users want to continue: plan the scale-up, including cost, training and support. Adjust means results are promising but incomplete: change something specific and run a short second round. Stop means the benefit is too small, the risk too high or adoption too low: record what you learned and move to the next use case.
Stopping is a legitimate result. A pilot that ends in a clear no, cheaply and quickly, has done its job. Write a short report for any outcome: hypothesis, baseline, results against targets, user feedback, incidents, costs and the decision. Share it widely, including the failures. Weeks 7 to 12 build on your pilot: workflows and agents by department, change management, return on investment, the operating model, scaling and your final roadmap.
Communicate the pilot to the people affected. Tell the pilot team why it exists, what will be measured, and that the aim is to test the tool, not to judge individuals. Tell everyone else what is happening and where to ask questions. Trust in the process makes feedback more honest, and honest feedback is the main product of a pilot.
Close the loop with everyone who took part.
Key Terms
- Pilot
- A small, time-limited trial of a solution with real users and real work.
- Hypothesis
- A testable statement of the expected change, naming users, task, metric and reason.
- Baseline
- The measured performance of the process before the tool is introduced.
- Control group
- A comparable group that continues to work as before, used for comparison.
- Outcome metric
- A measure of business result such as time saved or cost per case.
- Guardrail metric
- A safety measure such as incidents or wrong answers reaching customers.
- Adoption
- The share of eligible people who actually use the tool in their work.
- Sponsor
- The senior leader who funds the pilot and decides what happens next.
- Go or no-go criteria
- Decision rules agreed before the pilot that define when to scale, adjust or stop.
Exercise: Write A One-Page Pilot Plan
- Take your top use case and write the hypothesis using the template: if we give this tool to these users for this task, then this metric changes by this amount, because of this reason.
- Collect the baseline: volume, time per item and error or rework rate, using system data or a sample of at least twenty cases.
- Choose five or six metrics across outcome, quality, adoption and safety, each with a target and a minimum.
- Write the scope: users, data, tasks in, tasks out, and who reviews output.
- Name the sponsor, business owner, champion, technical lead and risk reviewer.
- Draw the timeline from preparation to decision meeting with weekly check-ins.
- Write go, adjust and stop criteria in numbers, and ask your sponsor to initial the page.
Further Reading
- Trustworthy Online Controlled Experiments, Ron Kohavi, Diane Tang and Ya Xu, Cambridge University Press
- How to Measure Anything: Finding the Value of Intangibles in Business, Douglas W. Hubbard, Wiley
- The Lean Startup, Eric Ries, Crown Business
- Generative AI at Work, Erik Brynjolfsson, Danielle Li and Lindsey Raymond, NBER Working Paper 31161