VOCABULARY / ENTRY
Explainability
explainability
The system shows why it produced a given result using evidence the user can check, and states just as clearly when that result is unreliable; this covers only what a user can see and understand about a result, not model interpretability as an engineering discipline.
Recognizing it
- The rationale points to conditions actually present in the input, a matched rule, or a cited source the user can check line by line, rather than a vague 'the system decided'
- Right alongside the rationale, the system states just as clearly where it does not apply or tends to be wrong, instead of only arguing its case
- The explanation's presentation never implies or claims that giving a reason makes the result correct; a separate path for verification sits alongside it
Concept boundary
Explainability is about whether the system's basis and known failure modes are both stated clearly. That differs from Recognition Rather Than Recall, which is about whether options and actions are visible so people don't need to remember them, not about a result's underlying basis. It also differs from Trust Calibration, which measures whether confidence signals match real accuracy; explainability supplies one raw material for that calibration, but an explanation on its own, however coherent, does not prove a result is correct: an explanation can raise a user's confidence without raising accuracy, and the two must be judged separately.
When to consider it
- The result will be acted on directly for a consequential decision and the user needs to judge whether to check further
- The user is surprised by or doubts the system's basis and needs a specific reason, not just a re-run
- The same system performs very differently across inputs or situations, so a generic description of how it works cannot cover this particular result
When to be cautious
- Treating the generated explanation text as proof the result is correct, using it to replace actual verification or skip error handling
- Showing only the parts of the reasoning that look tidy and reasonable while avoiding known failure cases or input types
- Requiring a full explanation for every low-risk, easily reversible result, adding needless interface load
Tradeoff
Presenting basis and limits together helps users make better-grounded decisions about accepting or checking a result, but it costs interface space and the effort to generate and maintain that content; an explanation that only sounds reasonable, without accepting a limits disclosure, can make it faster for users to accept a wrong result.
Conditional rules · Editorial advice
Related entries
Explainability supplies one raw material for trust calibration — stating basis and limits — but an explanation alone cannot substitute for checking whether confidence signals actually match real accuracy.
Conversational interaction naturally offers an entry point for follow-up questions that unpack the basis layer by layer, making it a common vehicle for checkable explanations.
Both are about what happens inside the system that should be visible to the user, but status visibility covers what is happening now and how far along it is, while explainability covers the basis and limits of a result already produced.
Sources and what they support
HAX guideline G11 requires that users be able to access an explanation of why the system behaved as it did; guidelines G1 and G2 require making clear what the system can do and roughly how often it errs at that task — together forming a minimum disclosure of basis plus limits.
Retrieved: 2026-09-15 · Archived 2026-09-16NIST's 'Explanation Accuracy' principle requires that an explanation faithfully reflect the system's actual process for producing the output, rather than any plausible-sounding reason; the 'Knowledge Limits' principle requires the system to operate only within its designed conditions or once it reaches sufficient confidence — together showing that whether an explanation is faithful and whether a result is correct are two separate things to verify.
Retrieved: 2026-09-15 · Archived 2026-09-16This study found that attaching explanations to AI recommendations increased how often people accepted them, and that increase occurred 'regardless of its correctness' — explanations raised acceptance without a matching rise in the human-AI team's actual accuracy, revealing a gap between the two.
Retrieved: 2026-09-15 · Archived 2026-09-16Definitions reference these sources. Digital specimens, selection advice, relationships, and recipes are editorial work and have not been validated through user research.
Type extensions and implementation notes
This entry has no working specimen yet; the illustration is editorial. Applying it to a product still requires verifying: whether the shown basis truly corresponds to the input and rules the system actually used, whether the limits disclosure covers known failure cases, and whether any wording implies that an explanation equals correctness.
{
"evidenceType": "An editorial interpretation of human-AI interaction guidelines and explainable-AI principles",
"limit": "This entry does not measure how much any particular explanation presentation changes acceptance or verification rates, and does not assume that 'explanation accuracy' (faithfulness to the system's actual process) is the same thing as the result being correct; the study read here shows that adding explanations raises acceptance regardless of whether the recommendation was right, so this entry does not restate other effect sizes it has not read, nor prescribe a universal amount of explanation that is 'enough'."
}Open the Agent JSON file for this entryThis file repeats every field of this entry verbatim and adds its sources, archive status, and editorial scope.