GUIDE / AI PRODUCTION READINESS
Taking an AI proof of concept to production
An AI proof of concept can show that a model performs a useful task on selected examples. Production requires evidence that the whole workflow can operate with real inputs, appropriate access, understandable failures, and a team responsible for it.
4 min read
Before expanding the pilot, write down what “ready” means for the people who will use and maintain it. That definition should cover the application around the model as well as the model's output.
Define the result in the workflow
Identify the user, the trigger, and the result that should become available. Decide whether the AI output is a draft, a recommendation, or an input to a controlled action.
Consider an illustrative workflow that extracts information from incoming documents for a staff member to review. Success includes presenting the correct source, identifying missing information, and making corrections straightforward. Producing well-formed text alone does not establish that the workflow works.
Write acceptance criteria in terms the operational owner can evaluate. These might cover the required fields, the exceptions that must be escalated, and the time the workflow can take before it disrupts the work. Choose thresholds for the actual task rather than borrowing a generic accuracy target.
Evaluate examples that challenge the workflow
A demonstration often uses a small set of convenient inputs. Extend evaluation to routine cases, ambiguous documents, missing data, and inputs that should not be processed.
Keep an evaluation set separate from the examples used to tune the system. Review failures with domain experts and rerun the evaluation when prompts, models, or retrieval behavior change. OpenAI's evaluation guidance recommends task-specific tests, representative data, and continued evaluation as the system evolves. Evaluation best practices ↗
Record enough context to understand a failure: the version under test, the expected behavior, the observed result, and the reviewer decision. Follow the data handling requirements of the environment when storing examples or diagnostic information.
Connect approved data and enforce access
A prototype may rely on a manually uploaded file. A production workflow needs a defined source, an owner, and an agreed method for receiving updates.
Establish which records a user or service can access. Enforce those permissions outside the prompt. Decide what happens when the source is unavailable, stale, or incomplete.
If the application retrieves supporting material, preserve the relationship between the result and its sources where the task requires verification. Users need a practical way to inspect the information behind a recommendation.
Make human review usable
“Human in the loop” needs an actual interface and a responsible person. Define when review is required, what the reviewer sees, and what happens if nobody responds.
In the document example, a reviewer could see the extracted fields beside the source, correct a value, and approve the next step. The workflow should distinguish a model suggestion from an approved record.
Include a route for uncertainty. A user should be able to flag an exception without inventing a workaround outside the system.
Test the surrounding application
Evaluate failures across the full path: input, model call, validation, review, and destination. Test denied access, malformed results, timeouts, and duplicate submissions.
Where the workflow changes another system, define how to verify whether a write succeeded before repeating it. Also identify the manual fallback when an external dependency is unavailable.
These checks are separate from judging the quality of the model's answer. Both are needed to understand whether the workflow can be relied on.
Assign ownership before expanding use
Name the people responsible for operating the workflow, reviewing exceptions, and approving changes. Agree on monitoring and support boundaries.
The handover should explain how to identify failures, stop or limit the workflow, restore the agreed operating path, and evaluate a proposed update. Credentials and deployment knowledge should not exist only in one person's memory.
Begin with an agreed rollout scope. Expand when the operational owner accepts the evidence, including the effort needed to handle exceptions.
A practical readiness review
| Area | Evidence to review |
|---|---|
| Workflow | Named user, owner, trigger, and expected result |
| Data and access | Approved sources, permissions, freshness requirements |
| Evaluation | Representative cases, acceptance criteria, known limitations |
| Actions and review | Validation rules, approval steps, exception route |
| Reliability | Tested failure paths, duplicate handling, manual fallback |
| Ownership | Deployment documentation, monitoring, handover, support scope |
An unresolved item does not automatically mean the concept should be abandoned. It identifies work that needs to be scoped before the deployment expands.
Common questions
What is the difference between an AI proof of concept and production?
A proof of concept shows that a model can do a task on selected examples. Production means the whole workflow runs on real inputs, with approved data, enforced access, visible failures and a named team responsible for it.
Why do AI pilots stall before production?
Usually because the work around the model was never built. There is no approved data source, no interface for the reviewer, no rule for stale or missing inputs and no owner. The model is often the part that works.
How do we know the model is good enough?
Test it against an evaluation set built from your own work: routine cases, ambiguous inputs, missing data and inputs that should be refused. Keep that set separate from the examples used for tuning, set acceptance criteria the operational owner can check, and rerun it when prompts, models or retrieval change.
Where does human review belong?
At the steps where a wrong result has consequences. Review needs a real interface and a responsible person: the reviewer sees the source next to the proposed result, can correct or reject it, and there is a rule for what happens if nobody responds.
Should we drop a pilot that fails the readiness review?
Not automatically. An open item marks work to scope before you expand: a data source to approve, a failure path to test, an owner to name. Expand when the operational owner accepts the evidence.
DEFINE THE NEXT PRODUCTION STEP
Bring the pilot and what still stands between it and daily use.
Tell us what the proof of concept does today, which systems it needs to reach and what is still manual. We work alongside your team to put it into production, and your team owns the result.