Promise: you will write a one-page pilot plan a sponsor can approve.
Outcomes
What You Will Be Able To Do
Write a testable hypothesis
Capture a baseline and choose a comparison
Select outcome, quality, adoption, safety metrics
Set scope, roles, timeline and guardrails
Define go, adjust, stop criteria
Purpose
What A Pilot Is For
A small, time-limited trial with real users and real work
Learn about value, risk and cost before scaling
Not a demo and not a rollout
Many projects stall between demo and decision
Step 1
Start With A Hypothesis
If we give this tool to these users for this task
then this metric changes by this amount
because of this reason
Without it, a pilot produces only opinions
If we give 12 support agents an AI reply assistant
for billing emails,
then time to first reply falls from 6 hours to 4,
because agents start from a draft, not a blank page.
Step 2
Why The Baseline Matters
Baseline: performance today, measured the same way
Collect it before the tool is switched on
Use system data or short manual samples
Without it, any result is a story
Step 2
Comparison Designs
Before and after for the same people: simple, but other changes mix in
Control group working as normal: more trustworthy
Even a small control group helps
If none is possible, state the limit
Step 3
Four Groups Of Metrics
Outcome: time saved, cost per case, cycle time
Quality: errors, satisfaction, share of output corrected
Adoption: weekly active share of eligible users
Safety: incidents and wrong answers reaching customers
Group
Metric
Target
Outcome
Time to first reply
6h to 4h
Quality
Heavily rewritten drafts
Below 30%
Adoption
Agents using weekly
70% or more
Safety
Serious wrong answers
Zero
Metric hygiene
Pair Speed With Quality
Time saved with more errors is not success
Set a target and a minimum for each metric
Watch for easily gamed numbers
Keep the list short enough to follow
Story
Case: A Chatbot Answer The Company Had To Honor
2024: airline chatbot gave wrong bereavement fare advice
Airline argued the chatbot was responsible for its own words
Tribunal rejected this and ordered payment
Define what a tool may say and who is accountable
Step 4
Set The Scope
One use case, one team of 10 to 30 users
One or two data sources, six to eight weeks
Write what is in and what is out
Name who reviews output, and when
Step 4
Name The Roles
Sponsor: funds the pilot, makes the final decision
Business owner: runs the process being changed
Champion: respected user who helps and gathers feedback
Technical lead and risk reviewer
Step 5
Timeline
Preparation 2 weeks: approvals, baseline, access
Setup 2 weeks: configuration, tests, training
Live run 6 weeks with weekly check-ins
Analysis 1 week, decision meeting 1 week
During the pilot
Run With Learning Loops
Expect confusion and low use in the first two weeks
Collect examples of good and bad outputs
Adjust training or instructions, not targets
Record any scope change and why
Step 6
Go, Adjust Or Stop
Go: targets met, safety clean, users want to continue
Adjust: promising but incomplete, run a short second round
Stop: benefit small, risk high or adoption low
Agree these rules before the pilot starts
Decision
Rule (example)
Go
Target met, zero serious incidents
Adjust
Between minimum and target
Stop
Below minimum or any serious incident
Summary
Recap
A pilot is a measured experiment, not a demo
Hypothesis, baseline and metrics come first
Narrow scope, named roles, honest timeline
Decision rules are set before the run
Weeks 7 to 12
Next Week
Week 7: workflows and agents by department
Week 8: change management and AI literacy
Week 9: ROI and running costs
Weeks 10 to 12: operating model, scaling and risk, capstone roadmap