Have you ever asked a co-worker a question and told them how much thought they should put into the answer? Why do we expect this with AI?
Most people have seen the pattern. They ask an AI a hard question. The default model gives them an answer that is technically plausible and practically useless. Then the user starts operating the cockpit: choose a bigger model, turn on extended thinking, increase reasoning, paste the prompt again, and wait.
For power users, those controls matter. They know when speed is the priority, when a task needs deeper reasoning, and when the better answer is worth the additional cost. But most employees are not trying to become model operators. They are trying to understand a customer issue, prepare for a meeting, resolve an exception, or complete a piece of work.
They should not need to know whether the request needs a fast model, a flagship model, more test-time compute, company context, or a tool call. They should be able to ask for help and get the right level of intelligence for the job.
The product goal is not one model for everything
It is also not a permanent model picker with a dozen settings. The better experience is an AI system that understands the request, uses the least expensive capability that can do the job well, and brings in more intelligence only when it is needed.
That changes the user experience in obvious ways:
- Lower cost because routine work does not automatically consume premium inference.
- Faster answers because simple work does not wait for maximum reasoning.
- Less frustration because users are not asked to diagnose the AI before they can use it.
- Better adoption because the system behaves like a useful colleague instead of a collection of controls.
Of course, the problem does not disappear. It moves from the user to the solution owner. That is where harness engineering begins.
Three ways to make the routing decision
1. Semantic routing
A lightweight classifier evaluates the request before it reaches the model. Is it simple, complex, ambiguous, risky, or likely to require tools and deeper reasoning? A straightforward policy question might use a fast, lower-cost model with retrieval. A financial analysis, multi-step technical investigation, or high-stakes customer response may be routed to a stronger model with more context and review.
The advantage is predictability. The risk is ambiguity blindness: a prompt that appears simple can conceal a complicated business problem. “Why did this customer churn?” may require far more than a neat summary if the answer depends on CRM history, support conversations, contract terms, product telemetry, and a judgment call about the evidence.
2. Escalation
Start with a lower-cost capability and escalate when the interaction signals that it is not enough. The system can look for repeated regenerations, rapid prompt edits, increasingly specific follow-ups, failed tool calls, or explicit user dissatisfaction.
This can prevent expensive models from handling every request. It can also create a poor experience if escalation happens too late or too often. A user should not have to fail through several weak answers before the system decides the work deserved more intelligence.
3. Speculative generation
A fast model begins producing an answer while another process checks whether the response meets important constraints. If it does not, the system escalates behind the scenes. It is an appealing performance pattern, especially for high-volume experiences, but it has to be designed carefully. Nobody enjoys watching an AI confidently start an answer, freeze halfway through a sentence, and return twenty seconds later with a completely different opinion.
Design for normal users; preserve control for experts
There is no single routing architecture that works everywhere. The right design depends on the workflow, consequence of error, tool access, data sensitivity, latency expectations, and unit economics. A customer-facing agent that can issue refunds needs a very different policy from an internal assistant that summarizes meeting notes.
What matters is the operating principle: make the default smart enough for normal users, and keep meaningful overrides available for power users. The user should be able to ask for speed, deeper analysis, or a specific model when they genuinely need it. They should not have to do so just to receive a competent answer.
Users do not care how elegant the routing architecture is. They care that the answer is useful, fast enough, and did not require them to manually manage the intelligence behind it. That is the real standard for enterprise AI adoption.
