WORKED EXAMPLE · AI INTEGRATION
A support assistant with a human checkpoint
An AI assistant for a support team, built with a source register, an evaluation set and a human checkpoint before any customer sees a draft.
THE SITUATION
The company shape
A company sells subscription software to other businesses. Its support team answers product and billing questions from a help centre, a set of internal runbooks and an older wiki, and they disagree in places nobody has listed.
Leadership wants an assistant. Nobody owns an incorrect answer today, and the only access rule on the documents is that everyone in support can read everything.
DISCOVERY
Weeks 1 to 2
We would take read access to the help centre, the wiki and a sample of closed tickets, then sit with the support lead and the engineer who maintains the wiki.
We would write a source register that says which document wins when sources disagree, and pick one family of questions, billing, as the first scope.
With the support lead we would write the evaluation set: representative billing questions with the answers the team would accept, reviewed before any model is chosen.
We would name the answer owner: the person who decides what counts as wrong and approves corrections.
BUILD
Weeks 3 to 6
We would build retrieval over the agreed sources only, with the permission rules from the register applied at query time.
The assistant would draft answers inside the ticket tool, and an agent would approve every draft before it reaches a customer.
Every change to prompts, sources or model would be scored against the evaluation set before release, with inputs, retrieved passages and outputs logged.
Cost and latency limits would be written down and enforced, and the support lead would accept the increment against the evaluation results.
AT THE END
What the client would hold
AI assistants and agents
A working assistant, an access matrix and an operating runbook.
Retrieval over your documents
A retrieval pipeline, a source register and a retrieval test report.
The evaluation set
A versioned evaluation dataset, a scoring rubric and a results report.
LIMITS
What we would have said no to
Answers sent without an agent
In the first increment no draft goes to a customer without a person approving it. That changes only when the evaluation results justify it, in writing.
Refunds and account changes
Connecting the assistant to actions that move money or change accounts is a separate scope with its own review points.
The whole wiki on day one
Ingesting every page before the source register settles which ones are authoritative would teach the assistant the contradictions.
A promised deflection rate
We would not commit to a share of tickets answered automatically. We commit to a measured assistant and a record of how it behaves.
Bring your own situation
Describe what you run and where it hurts. We reply with how the first two weeks of AI integration work would look for you.