Your enterprise AI architecture is wasting money every time an agent makes a decision. Because you’re using a racecar to pickup groceries.
Enter decision models …
Imagine hiring a PhD to stand at the entrance of a parking garage. Every time a car arrives, you ask, “Is this vehicle authorized to enter?” The PhD considers the question, constructs a thoughtful response, explains the reasoning, and concludes that yes, the vehicle is authorized.
Congratulations. You’ve built a very expensive gate.
That is the architectural habit I am interested in: using a general-purpose language model for every step of an agentic workflow, including the steps that only need a narrow, structured answer.
Sometimes that flexibility is valuable. Sometimes we are paying for capabilities the task does not need, then paying again in latency while the next step waits.
What is a decision model?
In this context, a decision model evaluates a situation against a defined set of possible answers. Instead of asking it to write a paragraph explaining what to do, you ask it to score or select among the choices your application can actually use.
- Is this a duplicate: yes or no?
- Which team should receive this request: Sales, Support, or Billing?
- Does this case need human review?
- Which of these options best meets the supplied criteria?
Some models return a probability for each option, which gives the workflow more information than a bare label. It can use those scores to decide whether to proceed, defer, or send the case for review … once those scores have been validated for the work at hand.
For a concrete example, Microsoft describes Microsoft-Decision-1 as a model for fast decision-scoring across tasks such as routing, classification, prioritization, verification, and workflow control. Its interface accepts fixed answer options and returns probability scores rather than generated text.
The distinction is about the task and interface, not a claim that these systems have nothing in common with language models. Microsoft describes post-training a language-model base for its decision-scoring workload. The useful question is what the resulting system is optimized to do.
A workflow contains more than one kind of intelligence
Take an enterprise processing 100,000 invoices. This is an example of the workload, not a claim about a particular client or a measured saving.
The system needs to extract information, identify possible duplicates, check policies, route exceptions, and determine which cases need approval. Those activities do not all require the same approach.
A database lookup can find an exact match for a known invoice identifier. A deterministic rule can check an explicit approval threshold. A decision model might classify an ambiguous exception against a fixed set of categories. A more capable language model might help interpret an unusual supplier explanation or assemble context for a reviewer.
Giving every one of those steps to the same LLM is convenient. It is not automatically the best architecture.
The opportunity is to separate the jobs, then use the simplest mechanism that meets each job’s requirements. Sometimes that is a decision model. Sometimes it is ordinary software. Sometimes you really do need the PhD.
Where I am looking for opportunities
I have been playing around with concepts for incorporating these models into larger AI solutions, particularly around optimization and performance.
Rather than defaulting every step to the same LLM, I am looking at routing, classification, validation, and escalation. These are places where a workflow often asks bounded questions repeatedly before completing the task the user actually requested.
Think about a support agent. It may need to identify the type of request, choose a retrieval path, assess whether it has enough information, and decide whether to escalate. Only some of that work involves composing the final response.
If those decisions happen sequentially, their delays accumulate. If the agent revisits them during retries, their costs accumulate too. Looking only at the model call that produced the final answer misses the work performed on the way there.
That does not establish that a smaller model will always be cheaper or faster. It establishes where to measure. Include preprocessing, calls, retries, human review, and the consequences of a wrong route when comparing the alternatives.
A confidence score is not permission to act
The probability attached to a decision is useful only if you understand how it behaves on representative cases.
A high score does not make an incorrect answer safe. Nor does a model’s benchmark result establish that it understands your company’s policy, unusual documents, or exceptions. Confidence needs to be checked against outcomes in your own workload.
Start with cases where the expected result is known, including ambiguous examples and cases where none of the offered answers is appropriate. Check whether the system can abstain or escalate rather than force every situation into a convenient label.
Choose thresholds according to the consequence of being wrong. A mistaken routing decision and an incorrect payment approval should not automatically use the same acceptance threshold.
And keep authority outside the score. If policy requires a human to approve a payment, a model returning “approve” does not remove that requirement. The workflow must enforce the boundary.
How to evaluate the architecture without rebuilding everything
Choose one repetitive decision in an existing workflow. Define the allowed answers, the information needed, and what happens when the answer is uncertain. Then compare a rule or lookup where applicable, a decision model, and the current LLM approach on the same representative cases.
Measure decision quality, latency, and cost per successfully completed task. Include false positives, false negatives, escalations, and recovery work. An inexpensive wrong decision can create a very expensive process downstream.
Before letting a new approach control actions, compare its recommendations alongside the existing process. Keep enough of a record to explain which model or rule produced each decision and which version was in use. If it underperforms, you need a way back.
That is a much more useful exercise than replacing a model because its price looks attractive on a comparison chart.
Optimize the decisions, not just the final answer
I suspect a meaningful amount of cost and latency is buried in enterprise AI architectures because we are using powerful models for tasks that do not need all that power.
Decision models are worth evaluating because they give us another option for those tasks. They are not a replacement for every LLM, and they do not eliminate the need for trusted data, approved rules, validation, or human judgment.
The practical question is straightforward: which decisions genuinely need open-ended reasoning, and which need a reliable answer from a defined set of choices?
Sometimes you need the PhD. Sometimes you just need the parking gate to open when you swipe the pass.
