Taking an AI Prototype to Production: The Engineering Work Nobody Budgets For

Failure figures are often quoted roughly, so it's important to be precise about what the evidence indicates.

In 2024, Gartner predicted that by the end of 2025 at least 50% of generative AI projects would be given up on after the proof of concept; the figure was reported by Gartner in January 2026 as having reached at least 50%, the reason cited being poor data quality, weak risk controls, increasing costs, and an unclear business value (this is a summary of Gartner's findings).

You have probably also come across the 95% figure from MIT's NANDA initiative. This study discovered that about 95% of the enterprise generative AI pilots it examined had no measurable impact on profits. The study was preliminary, not peer-reviewed, and has attracted methodological criticism, so it should be regarded as indicating a general trend rather than giving an exact figure.

The Omdia 2025 AI Market Maturity Survey provides a more detailed view of the situation; it showed that 31% of enterprises had fewer than 5% of their proof of concepts made it into production, whereas just 9% had more than half of theirs accepted (The Register’s coverage).

The methods are different, but the pattern remains the same. A considerable amount of money spent on AI fails to reach the end users.

The Demo Trap: Why “It Worked in Testing” Isn’t Enough

The role of a prototype is to demonstrate that the idea can be carried out.

In order to carry out that task, it eliminates variability. The dataset is clean and well known. The workflow is the one that was designed by the team. The person going through it knows precisely what the system expects.

Actual users don't carry out any of those actions. They omit steps, leave fields empty, paste in formats no one anticipated, and use the product in ways the team never imagined.

A demo shows that the happy path is possible, while production also involves dealing with the unhappy paths, and there are many more of these.

What Changes Between a Prototype and a Live System

The jump seems harder than it is because three things change at once.

• The calculations change in this case: a prompt that costs only a fraction of a cent becomes free when ten requests are made each day and is charged as a separate line item when ten thousand are made.
• The meaning changes. In a demo a poor output is just a funny screenshot, but in production it becomes a support ticket, a compliance question, or a customer who no longer trusts the feature.
• Change is continuous. Models become obsolete, prompts change, data shifts, and requirements evolve. A prototype is merely a snapshot, whereas a production system must endure modification.

As one rule of thumb suggested by various teams that have been through the process, the time required to turn a prototype into a production system is equivalent to the time it took to build the prototype, or in some cases even longer (source). Teams that fail to plan for this end up meeting deadlines with a product that works well in demonstration but breaks under actual load.

Before any engineering work begins, answer one question: Is this codebase acting as a foundation or a specification?

It doesn't matter which option you choose; getting it wrong is costly. If you make unsalvageable code more robust, you end up paying twice over. If you instead rebuild good code, then you waste weeks that you had no need to spend.

What Makes a Prototype Salvageable

Check for these signs; if most of them are present, taking the hardening route is generally the cheaper one.

• The data flow is genuine; it draws its information from real sources rather than from a fixture file or a manually cleaned CSV.
• The prompts are separate from the application code and can be edited without redeploying.
• There is some separation of concerns because the model calls, the business logic, and the interface are not all in a single file.
• Someone can explain why it works, but if the behavior came from trial and error and wasn't recorded, there is no basis for it.
• The main task has in fact been solved; the model carries out the difficult portion correctly, and this can be measured.

When Rebuilding Is Cheaper Than Fixing

Rebuild the system when the prototype shows the idea works, but nothing about the implementation can be reused; this is typical of tools that can produce an application rapidly from a prompt.

The good point is that a working prototype is the best specification you can have, since it already addresses the questions that often slow projects down; for example, what the interface should do, what the model needs to handle, and which edge cases matter.

Many firms structure the work in phases for this reason. One published model runs a 2-4 week validation build, then a 6-12 week productionization phase, then continuous iteration (Last Rev’s phase model). The validation build is designed to inform the production architecture, not to become it.

Use this before launch and again whenever your model, tools, or risk profile changes. Every item should have a named owner.

Evaluation

• A fixed eval dataset exists, with expected outputs agreed upon by the business.
• Every prompt or model change runs against it before merge.
• Pass thresholds are defined, not judged by eye.

Reliability

• Timeouts and retries are set on every model and tool call.
• A fallback path exists for provider outages and rate limits.
• The system degrades to something useful, not a blank screen.

Data and integration

• The system reads from production data sources, not copies.
• Retrieval quality is measured separately from output quality.
• Integration failures are handled distinctly from model failures.

Observability

• The system traces every model call, tool call, and retrieval step.
• Traces link back to a request ID a support agent can search.
• You can see cost and delay for each request, not just for the whole month.

Cost

• The code limits the number of tokens you can use for each request.
• Spend alerts fire before the invoice does.
• Caching has been set up for repeated or nearly identical requests.

Security and governance

• A data flow diagram illustrates the various items that leave your infrastructure and shows where they go.
• Agent tool permissions are specific, limited in scope, and reviewable.
• A kill switch exists and has been tested.

Release

• The prompts and the various model versions are kept in version control.
• Rollback takes minutes, not a redeploy cycle.
• Changes are first applied to a small portion of the traffic.

Ownership

• One person is responsible for how the system behaves.
• On-call knows what an AI failure looks like and who to wake.
• The review cadence isn't improvised; it's scheduled.

This is where most productionization projects lose their schedule. It is also the part teams consistently underestimate, because the prototype never touched it.

Analysis of the failure data keeps landing in the same place. Pilots get built on data infrastructure that was never designed to support production AI, and the problem only surfaces when the system meets real sources (analysis).

Wiring AI into CRM, ERP, and Legacy Systems

A pilot running on a sandboxed dataset is a different engineering problem than a system wired into the CRM, the ERP, and whatever internal tool holds the actual source of truth (discussion of this gap). Most pilots never test that path, because the sandbox was faster to build.

Budget real time for the connection work. Expect authentication quirks, undocumented field behavior, rate limits on the systems you are reading from, and at least one source that is technically live, but nobody maintains.

Plan for partial failure too. If the CRM is slow, your AI feature should not go down with it.

Retrieval Quality and Data Readiness

If your system retrieves context before generating, retrieval is a separate thing to test. A correct model given the wrong documents produces a confident wrong answer.

Measure retrieval on its own. Given a known question, does the right source come back in the top results? Track that number the way you would track any other service metric.

Data readiness matters just as much. Duplicate records, stale documents, and inconsistent formatting all degrade output quality, and no prompt engineering fixes them. Clean the source or accept the ceiling it sets.

Conventional software, given the same input, returns the same output. An LLM-based system given the same input can return something different every time.

This is not a defect. That is how the technology works, and it changes how testing, monitoring, and quality definitions function (more on this).

You cannot assert equality. You have to measure behavior across a set of cases and watch the distribution.

Building an Eval Baseline Before You Ship

Start by collecting real inputs. Pull them from the prototype’s usage, from support tickets, and from whatever the team was manually testing with.

Aim for 50 to 200 cases covering common paths, known edge cases, and failure modes you have already seen. For each one, write down what a good answer looks like. Get the business owner to agree, not just the engineers.

Then score. Some checks are rules, such as valid JSON, required fields present, and no forbidden content. Others need a model to judge, or a human to spot-check a sample.

The point is to have a number before you ship. Without a baseline, you cannot tell improvement from noise.

Catching Regressions When Prompts or Models Change

Once the baseline exists, it becomes a gate. Any change to a prompt, model version, retrieval setting, or tool definition runs against it first.

Set a threshold and enforce it in CI. If the score drops below the line, the change does not merge.

Continuous evaluation on live traffic catches what the fixed set misses. Sampling real production requests and scoring them surfaces failures the moment they appear, rather than when a customer reports them (practitioner guidance).

Treat the model as an unreliable external dependency, because it is. You don't control its uptime, latency, or output.

Every pattern you would apply to a third-party API applies here, plus a few that are specific to AI.

Fallbacks, Rate Limits, and Graceful Degradation

Set a timeout on every model call. Without one, a slow provider becomes a hung request, and hung requests become an outage.

Decide what happens when the call fails. Retry with backoff, fall back to a second provider, serve a cached response, or return a clear message and a human handoff. Any of these beats a spinner that never resolves.

Rate limits deserve their own handling. Queue the work, shed low-priority requests, and make sure a burst from one customer cannot starve everyone else.

Designing for Hallucination, Drift, and Retrieval Failure

These three are distinct failure modes, and each needs its own control.

• Hallucination is the model asserting something false with confidence. Constrain it by grounding answers in retrieved sources, validating structured output against a schema, and checking claims against the source before display.
• Drift is quality changing over time without your code changing. It happens when providers update models, your data shifts, or user behavior changes. Continuous evals catch it if you are watching.
• Retrieval failure happens when the system finds nothing relevant and answers anyway. The fix is to detect the empty or low-confidence case and say so, rather than generating something to fill the gap.

Deciding not to answer is a valid production behavior. Build the path for it.

Cost and latency are the same engineering problem. Both are driven by how many tokens move through how many calls, and both get solved with the same levers.

Token Budgets and Per-Request Cost Visibility

You need cost per request, not cost per month. A monthly bill tells you that spending went up. Per-request data tells you which feature did it.

Instrument input tokens, output tokens, model used, and latency on every call, tagged by feature and customer. Then set budgets in code. A single request should have a ceiling it cannot exceed, and hitting that ceiling should be a logged event, not a surprise.

Alert on rate of change, not just totals. A 5x jump in a day is worth knowing about before month-end.

Caching, Routing, and Architectural Choices

A few decisions shape the cost curve more than any amount of prompt trimming.

• Route by difficulty. Send routine requests to a smaller, cheaper model and reserve the larger one for cases that need it. Most workloads are not uniformly hard.
• Cache aggressively. Exact-match caching handles repeats. Semantic caching handles near-repeats, which are more common than teams expect.
• Shorten the context. Retrieving five relevant chunks beats retrieving fifty and hoping. It costs less and usually produces better answers.
• Question the loop. Agent architectures that call the model repeatedly can be right for the task, or they can be a way to spend ten times as much for a marginally better result; measure before you commit to the pattern.

If you operate in fintech or healthtech, this section moves from important to gating. Getting it wrong does not just cost money; it stops the launch.

Data Handling and Access Review

Start with a map. Map what data enters the system, what leaves your infrastructure, where it goes, and how long it is retained.

Check your provider's terms regarding training and data retention. Verify the region in which your requests are processed. If the data is subject to HIPAA regulations, ensure a business associate agreement is in place before any request includes protected health information.

Next, deal with the data. Before the data reaches the model, mask or tokenize sensitive fields if the task permits. Use the user's current permissions when retrieving data so the AI feature doesn't display a document the person couldn't otherwise open.

Record logs the same way you do everywhere else. If an audit can't reconstruct who saw what, then you'll fail the audit.

Guardrails for Agents and Tool Use

An agent that has access to tools is software that carries out actions. Define the scope this way.

Grant each tool only the most limited permission necessary for it to function; grant read-only access where reading is sufficient, and provide write access only when writing is the main requirement, including setting limits on what may be written.

Actions that are costly, irreversible, or noticeable to outsiders should require human approval; for example, sending an email to a customer, issuing a refund and altering a production record all fall into that category.

Check inputs before they reach the model, and check outputs before you pass them to a tool, since both directions represent potential attack surfaces. Prompt injection via retrieved content is a real risk when your agent reads documents it didn't write.

Create a kill switch and test it. Being able to shut the system down in under a minute makes people more comfortable releasing it.

The main reason most AI systems fail in practice is that someone quickly altered a prompt and no one knew what caused the failure.

Treating Prompts and Models Like Code

Prompts are logical; store them in a version control system, review them in pull requests, and tag each deployed version.

Make sure prompts aren't included in the application code so non-engineers can suggest changes and iterations can happen without a full redeploy (as practitioners recommend). Additionally, having the prompts externalized makes it easy to roll back, a feature that is more important than it may at first sound.

Be sure to specify the pinned model version explicitly. Always avoid directing production to a floating alias that a provider might move beneath you.

Record the prompt version and the model version that produced each output. To reconstruct precisely what ran when a customer reports a bad answer from three weeks ago, you need to do so.

Canary Releases and Handling Provider Changes

The ship should start as a small proportion of the traffic. Before increasing it, compare the evaluation scores, cost, latency, and error rate with those of the present version.

Providers need a separate procedure. Deprecation notices include firm deadlines, and any model you rely on will eventually be withdrawn.

As soon as the new version is available, carry out your evaluation using the eval suite. You should anticipate behavior changes even with a minor version upgrade, so you need to plan a migration window rather than swapping within the same week.

If the system is business-critical, it's worth integrating and testing a second provider, since this turns a provider outage from an incident into a configuration change.

This question never comes up in the planning meeting, but everyone asks it after the first incident.

Named Ownership and On-Call for AI Failures

Each AI system should have one named individual who is responsible for its behavior. The absence of an owner is a major warning sign during any compliance examination, which is the reason why the quality of the system gradually deteriorates (source).

The idea of ownership involves a number of definite responsibilities. A person checks the eval dashboard on a weekly basis. A person makes the decision as to whether a deterioration in quality justifies a rollback. A person responds when legal asks how the system handles a data subject request.

We now need a new set of runbooks. AI failures usually look very different from conventional outages because the system stays up, keeps responding, and generates plausible garbage. The alerting system must detect a decline in quality, not just an increase in error rates, and responders must know what action to take when an alert triggers.

What an Ongoing Maintenance Cadence Looks Like

A workable rhythm for most teams:

• Weekly, review eval scores, cost per request, and latency, and review a sample of the actual outputs.
• Monthly, add new failure cases to the evaluation set and review flagged conversations alongside support escalations.
• Quarterly, reevaluate model selection based on currently available options, check permissions and access for each tool, and rerun the readiness checklist.
• When there is a change, whether it be a new tool, a new data source, or a new version of a model, a complete evaluation run and a permissions review are carried out.

Plan for engineering capacity from the start, because a production AI system is not a project with an end, and treating it as one turns good systems unreliable.

Most teams do not need help proving an AI idea works. They need help with the engineering that sits between a working demo and a system their customers can depend on.

Cypherox works with SaaS, fintech, and healthtech teams on exactly that gap. Our AI app development services cover evaluation, observability, and deployment discipline for systems already in flight. In contrast, our AI agent development services handle tool permissions, guardrails, and the orchestration work that agent systems need before they touch production data.

If your prototype works but you are not sure what it would take to ship it, start with an assessment against the checklist above. If the constraint is capacity rather than direction, you can also hire AI developers to work alongside your existing team.