AI Is Often Wrong, Never Unsure — Manage Risk with a Matrix

Read this article on LinkedIn

Agentic AI projects are failing in no small part because of how large language models communicate. They've earned the label "often wrong, never unsure" for their habit of responding like your most confident junior employee. Ask an agent a question and it will give you a verbose, well-structured answer even if it’s guessing or responding from outdated information.

Of course not all questions carry the same weight. A question like, “What’s the subject line for this email?” poses no existential risk whereas a question like, “Should we approve a $5 million dollar loan?” does. Moreover, assessing an answer’s accuracy isn’t always evident. Each organization has answers that are not universally right, but right for the organization. This is what makes each business unique. The point is: riskier questions require more trustworthy answers.

These two questions–“How consequential is the question?” and “How confident am I in the AI-generated answer?”– form what we call the Consequence-Confidence matrix. This matrix helps leaders think about how to implement AI in their organizations. It’s a high-level strategy tool that can be used for mapping out which tasks can handle answers that are “good enough”, which ones require more precision, and what level of guardrails are required in each circumstance.

The Consequence-Confidence Matrix

The matrix maps out four quadrants:

  • Flag and Move On: Low Consequence/Low Confidence

  • Automate Freely: Low Consequence/High Confidence

  • Escalate Immediately / Pre-determined Answers: High Consequence/Low Confidence

  • Proceed Carefully: High Consequence/High Confidence


Article content

Here is an example of queries that an employee might ask a human resources agent.

In the graphic above, low and high consequence refers to the level of risk a type of question poses. Confidence refers to the type of answer you can rely on the model for.

Similarly, low and high confidence in AI output is determined primarily by how available and verifiable the knowledge behind it is. Some answers consist of widely available information (e.g. "What's the population of Guatemala?") while others are niche or context-dependent (e.g. “What is the best course of action for this specific client?”). Similarly, not all questions carry the same consequences. Some wrong answers are trivial, while others carry real weight: legal exposure, safety risks, financial liability.

Consequence: Which questions carry the greatest risk?

Consequence has to be defined by the people who understand the business—not data scientists, marketing teams, or your AI guy. That means identifying, domain by domain, which activities and decisions carry real weight. Without being provided with extensive context, a language model doesn’t automatically distinguish between a question about shoe policies and a question about workplace harassment. Both appear in the HR knowledge base and, therefore, from the model's perspective, they're roughly equivalent. From your organization's perspective, they're miles apart.

The consequential categories tend to cluster around revenue drivers, cost drivers, employee safety, employee experience, customer experience, and legal or compliance exposure. You don't need to tune every question with equal care. You need to identify the small number of high-consequence interactions and build appropriate handling for those first. Moreover, segmenting questions/answers based on consequence can be used to decide where cheaper, simpler modeling will suffice and where the most sophisticated and expensive modeling is required.

Confidence: Which answers can I trust?

Being confident in an AI response can be challenging, namely because most AI-responses don’t come with a self-determined accuracy score. And, if you ask them directly how confident they are, what they report is how well they've predicted the next word. That's not the same as reporting whether they've assessed all the relevant documentation and produced a response that correctly captures company policy.

That’s why when determining confidence in an AI-derived response, the first step is for companies to define a framework for what “accurate” means. RAC (Relevant to the actual question, Factually accurate, and Comprehensive–no missing details) is one that usefully expands the notion of accuracy, but a company can come up with its own. The point is to have one.

Here are some practical ways to be able to evaluate your level of confidence in an AI- generated answer:


  1. Assess LLM sourcing: Surface and score the quality of the source data on which the answer is based: is it deep, authoritative and reliable?

  2. LLM as Judge: Use your LLM model as a judge and ask it to score its confidence in its own answer. This can be expanded upon by designing personas that view confidence through a specific lens (e.g. a customer experience persona will test whether from a CX standpoint the answer is likely to be relevant, accurate and comprehensive for the customer or not).

  3. Challenger LLM as Judge: Extend this exercise by having other models score the output and explain why.

  4. Human Evaluation: Lastly, include a human domain expert or a user in the loop. This is useful when tuning the model and in inference operations. This is particularly important for addressing questions where consequence is high and confidence is low.


Confidence in the answer, combined with the consequence of the question, determines what guardrails you need. A low-confidence response to a low-consequence question should be flagged, but not blocked. A low-confidence response to a high-consequence question should be escalated to a human where possible, or answered with a rules-based response instead of a generated one. Only a small share of the questions an agent handles are high-consequence enough to need a verified, human-confirmed answer attached. But for those questions, nothing else will do. When an internal expert has reviewed the question, written the response, and confirmed it's accurate, that creates a different and more defensible kind of confidence than anything an AI system generates on its own.

Using the Matrix Effectively

Article content

This framework helps you identify which quadrant a task belongs in

Implementing AI in your organization comes down to two questions: "What exactly do we want the AI to do?" and "What happens if it gets that wrong?" This matrix gives you a way to address both. First, select the business unit where you’d like to implement an AI system. Then, select the user who will be engaging with that system. For example, an employee and an HR manager wouldn’t use an HR chatbot for the same things.

Then, list out the tasks the AI system would assist with. For each task, produce a sample question and provide an answer that this unit would deem correct, meaning complete and comprehensive. Then, feed that question into an LLM—the one you are currently using or one of your choice.

Compare the LLM output with your correct answer. Give a consequence score from 1 to 5. What does a wrong answer cost? Score against revenue, cost, employee safety & experience, customer experience, legal/compliance. Any question that could create a legal record, such as harassment, misconduct, or personal/sensitive data is automatically high consequence.

Then, give a confidence score according to how many verification steps (LLM as judge, human review, source quality) would be required to make the answer more accurate. More steps, less confidence.

After scoring, determine the quality of guardrail–light, moderate, strong, hard stop–needed. Here is a starter list of guardrails you could implement:


Light

  • Sample review: spot-check a batch of responses on a cadence.

  • Confidence surfacing: show the user how sure the system is.

  • Citation requirement: make the answer name its sources.

Moderate

  • LLM-as-judge: the model scores its own answer.

  • Challenger LLM: a second model scores and explains.

Strong

  • Human verification: a domain expert confirms the answer.

Hard Stop

  • Human routing: hand the question to a person instead of answering.

  • Rules-based answer: serve a pre-written, verified response.


You don’t have to fully trust AI output to start implementing AI in your organization. This matrix is how to move forward in the uncertainty. Remember, your most junior employee is good for some things but needs micromanaging for others. What matters is comprehensively mapping the consequences, shrewdly assessing the confidence in the tools available, and ensuring you have the right guardrails to cover the gaps.

Knowledge layers, like Trovia, which sit between your knowledge base and the foundation model, make this work substantially easier. Trovia both scores questions according to criticality and rates answers according to confidence. The system automatically seeks input from designated experts in your organization and subsequently tunes itself. Learn more here.