In short
- Evaluating a situation and explaining the result are different responsibilities.
- Company-approved logic should govern the evaluation. A generative model can explain its output.
- Repeatability makes an evaluation inspectable. Its assumptions and outcomes still need scrutiny.
A pilot looks promising. Leadership approves it. The team ships.
Then an account gets flagged as urgent without a clear reason. Another gets missed. The same question produces a different conclusion, and nobody can explain what changed.
The team improves the prompt.
Adds context.
Tries another model.
Those changes may improve the output. The architecture still needs to answer a separate question: which part of the system is allowed to determine the result?
Two jobs
An operational AI response often combines evaluation and explanation.
Evaluation asks what changed, which rule applies and what requires attention. Some answers are directly verifiable: a metric crossed a threshold, or a required event failed to occur. Others involve forecasts or trade-offs. Each needs a defined basis for its conclusion.
Explanation asks how to communicate that result. Which details does the reader need? What uncertainty needs to remain visible? What next step follows?
An operator may perform both jobs. The responsibilities remain distinct.
The analyst can explain the model. The executive can question its assumptions. A clear boundary lets both inspect how the conclusion was reached.
Repeatability and correctness
An evaluation needs to distinguish the answer from what happened.
Anthropic’s January 2026 evaluation guide separates the task, the trial, the transcript and the outcome. It uses multiple trials because model outputs vary across runs. It also distinguishes what an agent says it did from the final state of the environment.
That gives the evaluator something to check.
The conclusion still needs to be right.
The evaluation needs an inspectable basis: evidence, approved logic, model version and evaluation context. The explanation needs to remain faithful to that result.
The judge
The judge evaluates evidence against defined logic.
- Did utilization cross the approved threshold?
- Did a renewal fall inside an uncovered week?
- Did a required delivery milestone slip?
For a forecast, which model, assumptions and uncertainty produced the estimate?
With the same evidence, approved model version and evaluation context, the result should be reproducible. A changed date, policy, input or permission context can legitimately change the result. The system should preserve that basis.
Missing evidence needs to remain visible. An unevaluated condition needs a different state from a condition that was evaluated and passed.
Consistent application makes a rule inspectable. The company still has to decide whether the rule is useful and revise it when the evidence warrants.
The narrator
The narrator takes the evaluation and explains it for a person.
An account owner may need the immediate next step. A manager may need the affected book and available cover. An executive may need the exposure and the decision that requires their authority.
The language can change for the audience. The underlying evidence and conclusion need to stay intact.
That requires enforcement beyond a prompt.
Keep the approved result separate from the generated prose. Check the explanation against it. Reject unsupported claims or return the structured result when a faithful explanation cannot be produced.
A read-only input protects the stored verdict. Checking the output protects the reader from an explanation that misrepresents it.
Consider a renewal summary that says “Coverage has been approved” when the stored result only requests a coverage review.
The record retains the right state.
The reader receives the wrong commitment.
A person may act on that sentence before opening the underlying record. The explanation needs the same care as the evaluation it represents.
Where the boundary breaks
Feed raw records into a model and ask which accounts need attention. The answer may combine several operations in one pass: infer a relationship, choose a threshold, assign importance and write the summary.
An inference can be useful. A proposed threshold can be useful. A draft recommendation can be useful.
The company needs to know which of those became the basis for the result.
Consider a manager asking why an account is at risk. The summary mentions a delayed implementation and an overloaded owner. The manager needs to trace both claims to their source records and see which approved rule made them significant.
Without that path, the manager has to reconstruct the evaluation before deciding whether to use it.
The cost appears in the next step: more checking, another meeting, a recommendation left unused.
Building both
A model can help build the evaluator. It can propose relationships, draft rules and surface trade-offs for people to review.
Approval establishes which logic governs the evaluation. A later change needs to be explicit and recorded.
The first rule can be simple: flag modeled utilization above an approved threshold. The same rule should identify the same condition when its evidence and context are unchanged.
Over time, evaluations can span more records and functions. A delivery delay can be considered alongside a renewal date, workload and the authority to respond.
Each added rule needs:
- A reason to exist.
- An owner.
- Evidence that it detects something useful.
- A way to challenge and revise it.
The narrator then has better material to explain: an evaluated situation with its basis intact.
Why it matters now
In McKinsey’s August 2026 State of AI survey, 37% of respondents reported that AI had contributed to their organizations’ EBIT.
Deloitte’s June 2026 research found that nearly 75% of surveyed technology executives expected their operating model to need change within 12 to 18 months.
Microsoft’s May 2026 Work Trend Index asks who reviews agent performance and who has authority to change the workflows agents run. Those are ownership questions for the evaluation itself.
My concern comes from years of building operating models by hand: a recommendation needs a basis people can inspect before they commit resources or make a promise.
Separating evaluation from explanation makes that basis easier to preserve. Whether it improves decisions still has to be tested in the work.
What this means for your team
Four questions to ask of a system that tells you what matters:
- Can you reproduce the evaluation? Hold the evidence, approved model version and evaluation context constant. Can you explain any difference in the result?
- Can you trace it? Each flag should lead back to the rule and source records that produced it.
- Can the language layer change the conclusion? Check whether generated prose can introduce a claim, alter severity or trigger an action beyond the approved result.
- Who owns the rule? Identify who approved the threshold, who can revise it and where that decision is recorded.
An answer that changes should lead to a recorded change in evidence, logic or context. An answer that repeats still needs its assumptions examined.
How OpX is built
This is the boundary OpX is built on. OpX is the decision runtime for customer operations, starting in Customer Success.
The governed situation evaluator applies company-approved rules without LLM calls. Language models work around it, helping map data, draft and explain results. Their suggestions remain separate from the approved logic used in an evaluation.
The judge preserves the basis for the result.
The narrator makes it understandable.
People retain authority over the decision.
Sources
- Demystifying evals for AI agents, Anthropic, January 9, 2026.
- The state of AI in 2026: On the road to ROI, McKinsey, August 25, 2026.
- Rewiring the enterprise operating model for AI scale, Deloitte Insights, June 29, 2026.
- Agents, human agency, and the opportunity for every organization, Microsoft 2026 Work Trend Index, May 5, 2026.
