Explainability

This entry has no working specimen yet; the illustration is editorial. Applying it to a product still requires verifying: whether the shown basis truly corresponds to the input and rules the system actually used, whether the limits disclosure covers known failure cases, and whether any wording implies that an explanation equals correctness.

{
  "evidenceType": "An editorial interpretation of human-AI interaction guidelines and explainable-AI principles",
  "limit": "This entry does not measure how much any particular explanation presentation changes acceptance or verification rates, and does not assume that 'explanation accuracy' (faithfulness to the system's actual process) is the same thing as the result being correct; the study read here shows that adding explanations raises acceptance regardless of whether the recommendation was right, so this entry does not restate other effect sizes it has not read, nor prescribe a universal amount of explanation that is 'enough'."
}

{
  "id": "explainability",
  "type": "principle",
  "name": {
    "zh": "可解释性",
    "en": "Explainability"
  },
  "aliases": [
    "Explainable Results",
    "结果说明",
    "为何如此"
  ],
  "granularity": "experience",
  "intents": [
    "calibrate-trust",
    "explain-result"
  ],
  "tags": [
    "explainability",
    "result rationale",
    "limits disclosure",
    "checkable evidence",
    "not proof of correctness"
  ],
  "definition": "The system shows why it produced a given result using evidence the user can check, and states just as clearly when that result is unreliable; this covers only what a user can see and understand about a result, not model interpretability as an engineering discipline.",
  "boundary": "Explainability is about whether the system's basis and known failure modes are both stated clearly. That differs from Recognition Rather Than Recall, which is about whether options and actions are visible so people don't need to remember them, not about a result's underlying basis. It also differs from Trust Calibration, which measures whether confidence signals match real accuracy; explainability supplies one raw material for that calibration, but an explanation on its own, however coherent, does not prove a result is correct: an explanation can raise a user's confidence without raising accuracy, and the two must be judged separately.",
  "signature": [
    "The rationale points to conditions actually present in the input, a matched rule, or a cited source the user can check line by line, rather than a vague 'the system decided'",
    "Right alongside the rationale, the system states just as clearly where it does not apply or tends to be wrong, instead of only arguing its case",
    "The explanation's presentation never implies or claims that giving a reason makes the result correct; a separate path for verification sits alongside it"
  ],
  "when": [
    "The result will be acted on directly for a consequential decision and the user needs to judge whether to check further",
    "The user is surprised by or doubts the system's basis and needs a specific reason, not just a re-run",
    "The same system performs very differently across inputs or situations, so a generic description of how it works cannot cover this particular result"
  ],
  "when_not": [
    "Treating the generated explanation text as proof the result is correct, using it to replace actual verification or skip error handling",
    "Showing only the parts of the reasoning that look tidy and reasonable while avoiding known failure cases or input types",
    "Requiring a full explanation for every low-risk, easily reversible result, adding needless interface load"
  ],
  "tradeoff": "Presenting basis and limits together helps users make better-grounded decisions about accepting or checking a result, but it costs interface space and the effort to generate and maintain that content; an explanation that only sounds reasonable, without accepting a limits disclosure, can make it faster for users to accept a wrong result.",
  "comparison": {
    "focus": "What the system used to produce this result and where that result is unreliable, not whether the system 'seems trustworthy' overall",
    "mechanism": "Checkable evidence and known limits are shown side by side so users can judge for themselves against the original input",
    "cost": "The cost of generating and maintaining explanations and limits disclosures, and the discipline not to dress up an explanation as proof of correctness"
  },
  "sources": [
    {
      "id": "msr-hax-guidelines-g11",
      "title": "Microsoft Research · Guidelines for Human-AI Interaction",
      "url": "https://www.microsoft.com/en-us/haxtoolkit/library/",
      "claim": "HAX guideline G11 requires that users be able to access an explanation of why the system behaved as it did; guidelines G1 and G2 require making clear what the system can do and roughly how often it errs at that task — together forming a minimum disclosure of basis plus limits.",
      "checkedAt": "2026-09-15",
      "archive": {
        "lookupUrl": "https://web.archive.org/web/20260915*/https://www.microsoft.com/en-us/haxtoolkit/library/",
        "status": "available",
        "checkedAt": "2026-09-16T05:43:24.829Z",
        "jobId": "spn2-7cb34684fb905780ab4c35f6e41f0ec7680b9b7e",
        "url": "https://web.archive.org/web/20260916054347/https://www.microsoft.com/en-us/haxtoolkit/library/",
        "timestamp": "20260916054347"
      }
    },
    {
      "id": "nist-four-principles-explainable-ai",
      "title": "NIST · NIST Asks A.I. to Explain Itself (Four Principles of Explainable AI)",
      "url": "https://www.nist.gov/news-events/news/2020/08/nist-asks-ai-explain-itself",
      "claim": "NIST's 'Explanation Accuracy' principle requires that an explanation faithfully reflect the system's actual process for producing the output, rather than any plausible-sounding reason; the 'Knowledge Limits' principle requires the system to operate only within its designed conditions or once it reaches sufficient confidence — together showing that whether an explanation is faithful and whether a result is correct are two separate things to verify.",
      "checkedAt": "2026-09-15",
      "archive": {
        "lookupUrl": "https://web.archive.org/web/20260915*/https://www.nist.gov/news-events/news/2020/08/nist-asks-ai-explain-itself",
        "status": "pending",
        "checkedAt": "2026-09-16T05:52:07.543Z",
        "error": "status-request-failed",
        "jobId": "spn2-3b30129de8757a6f84c79f718b0f7aebf58e4aa2"
      }
    },
    {
      "id": "bansal-2021-explanations-overreliance",
      "title": "Bansal et al. · Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (CHI 2021)",
      "url": "https://arxiv.org/abs/2006.14779",
      "claim": "This study found that attaching explanations to AI recommendations increased how often people accepted them, and that increase occurred 'regardless of its correctness' — explanations raised acceptance without a matching rise in the human-AI team's actual accuracy, revealing a gap between the two.",
      "checkedAt": "2026-09-15",
      "archive": {
        "lookupUrl": "https://web.archive.org/web/20260915*/https://arxiv.org/abs/2006.14779",
        "status": "available",
        "checkedAt": "2026-09-16T05:52:08.058Z",
        "jobId": "spn2-ca81f214340e0a259c2bab721d3c6d122126dea2",
        "url": "https://web.archive.org/web/20260916055231/https://arxiv.org/abs/2006.14779",
        "timestamp": "20260916055231"
      }
    }
  ],
  "relations": [
    {
      "target": "trust-calibration",
      "kind": "composes_with",
      "condition": "The system needs to convey confidence signals and decision basis at the same time",
      "reason": "Explainability supplies one raw material for trust calibration — stating basis and limits — but an explanation alone cannot substitute for checking whether confidence signals actually match real accuracy.",
      "basis": "editorial"
    },
    {
      "target": "conversational-interface",
      "kind": "informs",
      "condition": "The user can keep asking 'why' about a particular result",
      "reason": "Conversational interaction naturally offers an entry point for follow-up questions that unpack the basis layer by layer, making it a common vehicle for checkable explanations.",
      "basis": "editorial"
    },
    {
      "target": "visibility-of-system-status",
      "kind": "related",
      "condition": "Both system status and result basis need to be shown",
      "reason": "Both are about what happens inside the system that should be visible to the user, but status visibility covers what is happening now and how far along it is, while explainability covers the basis and limits of a result already produced.",
      "basis": "editorial"
    }
  ],
  "rules": [
    {
      "when": "The system shows a result or recommendation",
      "instruction": "Must state the result's basis using evidence the user can check against the original input, not just an unverifiable general judgment.",
      "strength": "must",
      "basis": "editorial"
    },
    {
      "when": "In the same place the result's basis is shown",
      "instruction": "Must state just as clearly the situations where this kind of result is known to fail or be unreliable, not only the reasoning behind it.",
      "strength": "must",
      "basis": "editorial"
    },
    {
      "when": "The explanation text itself reads as coherent and reasonable",
      "instruction": "Must not treat the generated explanation as proof the result is correct; a verification path independent of the explanation is still required.",
      "strength": "must",
      "basis": "editorial"
    }
  ],
  "principle": {
    "evidenceType": "An editorial interpretation of human-AI interaction guidelines and explainable-AI principles",
    "limit": "This entry does not measure how much any particular explanation presentation changes acceptance or verification rates, and does not assume that 'explanation accuracy' (faithfulness to the system's actual process) is the same thing as the result being correct; the study read here shows that adding explanations raises acceptance regardless of whether the recommendation was right, so this entry does not restate other effect sizes it has not read, nor prescribe a universal amount of explanation that is 'enough'."
  },
  "demo": null,
  "version": "0.1.0",
  "editorialStatus": "drafted",
  "requirements": [],
  "demoCapabilities": [],
  "implementationNote": "This entry has no working specimen yet; the illustration is editorial. Applying it to a product still requires verifying: whether the shown basis truly corresponds to the input and rules the system actually used, whether the limits disclosure covers known failure cases, and whether any wording implies that an explanation equals correctness.",
  "image": {
    "src": "/images/explainability.webp",
    "alt": "A fictional FIELDNOTES result card with an expanded 'why' panel beside it: three short reason bars next to one limits bar in a visibly different tone, on a warm background with no legible text."
  },
  "searchTerms": [
    "可解释性",
    "结果说明",
    "局限说明",
    "可核对依据",
    "非正确性证明"
  ]
}