Trust Calibration

This entry has no operable specimen yet; the illustration is an editorial sketch. Applying it to a product still requires verifying that confidence or certainty signals match measured accuracy, that the verification flow is actually workable in the real setting, and that differences in system capability across tasks or groups are honestly communicated.

{
  "evidenceType": "A design interpretation of human-AI interaction guidelines and automation-trust research",
  "limit": "This entry does not measure how much any specific calibration technique reduces over-acceptance or under-use, nor does it prescribe a universal \"right\" confidence threshold; trust calibration remains an active research area, and this entry cites only verifiable definitional claims, not experimental effect sizes it has not read."
}

{
  "id": "trust-calibration",
  "type": "principle",
  "name": {
    "zh": "信任校准",
    "en": "Trust Calibration"
  },
  "aliases": [
    "Calibrated Trust",
    "适当信任",
    "Appropriate Reliance"
  ],
  "granularity": "experience",
  "intents": [
    "calibrate-trust"
  ],
  "tags": [
    "trust calibration",
    "perceived reliability",
    "over-reliance",
    "confidence signals",
    "verification"
  ],
  "definition": "When a system's signals about its own capability and limits match its actual reliability, the user's trust matches it too: they neither accept output blindly where the system tends to be wrong, nor doubt it needlessly where it tends to be right. Trust calibration is about the correspondence between confidence signals and real performance, not about making people trust the system more overall.",
  "boundary": "The goal of trust calibration is to make trust match the system's actual capability, not to maximize trust; over-trust (accepting a wrong output that should have been questioned) and under-trust (ignoring a correct output that could have been used) are both calibration failures, and neither is worse than the other. It concerns the correspondence between confidence signals and real accuracy, distinct from \"explainability\" — which concerns why the system produced a given result and what reasoning it used, a separate entry not yet written; this entry only mentions it without drawing out the comparison.",
  "signature": [
    "The system proactively states which situations or tasks it tends to get wrong, not only where it performs well",
    "Confidence or certainty cues roughly track the actual probability that an output is correct, rather than a fixed, flattering score",
    "A way to verify an output before acting on it is available, sized to the consequence"
  ],
  "when": [
    "Output is adopted directly for decisions, and the system's accuracy is uneven across tasks, inputs, or groups",
    "Users lack first-hand experience with the system's capability and are prone to judging trustworthiness by tone, personality, or interface polish",
    "System capability shifts with task type, data freshness, or version updates, so a past impression may already be outdated"
  ],
  "when_not": [
    "Treating \"raising trust\" itself as the design goal, without checking whether trust actually matches measured accuracy",
    "Using confident phrasing, personality, or a polished interface to mask the system's real error rate in a given situation",
    "Requiring the same level of verification for every low-stakes, easily corrected output, creating needless friction"
  ],
  "tradeoff": "Clearly communicating weak spots and providing a way to verify reduces both over-acceptance and under-use, but it adds interface information and interaction steps; requiring the same verification for every output regardless of consequence undercuts the judgment cost that calibration is meant to save.",
  "comparison": {
    "focus": "Whether the user's trust matches the system's real reliability, not how high or low that trust is",
    "mechanism": "Keeping accuracy signals (when it tends to be wrong, confidence cues) and an actionable way to verify aligned with the system's actual performance",
    "cost": "Requires continually measuring real accuracy, designing a workable verification flow, and resisting the temptation to mask weaknesses with confident tone or polished presentation"
  },
  "sources": [
    {
      "id": "msr-hax-guidelines",
      "title": "Microsoft Research · Guidelines for Human-AI Interaction",
      "url": "https://www.microsoft.com/en-us/haxtoolkit/library/",
      "claim": "Guideline 2 calls for helping users understand \"how well the system can do what it can do,\" i.e. how often it may make mistakes; Guideline 10 calls for the system to disambiguate or gracefully degrade — scoping down the services it offers — when it cannot determine the user's goal.",
      "checkedAt": "2026-09-15",
      "archive": {
        "lookupUrl": "https://web.archive.org/web/20260915*/https://www.microsoft.com/en-us/haxtoolkit/library/",
        "status": "available",
        "checkedAt": "2026-09-16T05:43:24.829Z",
        "jobId": "spn2-7cb34684fb905780ab4c35f6e41f0ec7680b9b7e",
        "url": "https://web.archive.org/web/20260916054347/https://www.microsoft.com/en-us/haxtoolkit/library/",
        "timestamp": "20260916054347"
      }
    },
    {
      "id": "lee-see-2004-trust-automation",
      "title": "Lee & See · Trust in Automation: Designing for Appropriate Reliance (Human Factors, 2004)",
      "url": "https://journals.sagepub.com/doi/10.1518/hfes.46.1.50_30392",
      "claim": "Automation is often problematic because people fail to rely on it appropriately; because people respond to technology socially, trust shapes reliance on automation, especially when complexity and unanticipated situations make full understanding of the system impractical, so trust ends up guiding reliance instead.",
      "checkedAt": "2026-09-15",
      "archive": {
        "lookupUrl": "https://web.archive.org/web/20260915*/https://journals.sagepub.com/doi/10.1518/hfes.46.1.50_30392",
        "status": "unconfirmed",
        "checkedAt": "2026-09-16T05:17:59.476Z",
        "error": "status-request-failed",
        "jobId": "spn2-905c8fc68f68c029bfa7acd26e82e2eba3a958ed"
      }
    },
    {
      "id": "wikipedia-automation-bias",
      "title": "Wikipedia · Automation bias",
      "url": "https://en.wikipedia.org/wiki/Automation_bias",
      "claim": "Automation bias is the tendency to favor an automated system's suggestions and ignore other correct information, an \"error of commission\" where over-trust leads to accepting a suggestion that should have been questioned; there is also an \"error of omission,\" where a user skips proper review and misses a problem the system failed to catch, corresponding to under-trust or disuse of the automation.",
      "checkedAt": "2026-09-15",
      "archive": {
        "lookupUrl": "https://web.archive.org/web/20260915*/https://en.wikipedia.org/wiki/Automation_bias",
        "status": "available",
        "checkedAt": "2026-09-16T05:49:42.639Z",
        "jobId": "spn2-f1a0b9ee18dca813e937f9658525efa07ea33c0c",
        "url": "https://web.archive.org/web/20260916055006/https://en.wikipedia.org/wiki/Automation_bias",
        "timestamp": "20260916055006"
      }
    }
  ],
  "relations": [
    {
      "target": "visibility-of-system-status",
      "kind": "composes_with",
      "condition": "The system needs to continuously show its own status and confidence level",
      "reason": "Calibrating trust depends on making the system's current status and uncertainty clear; visibility is the precondition for calibration signals to be communicated at all.",
      "basis": "editorial"
    },
    {
      "target": "human-centered-design",
      "kind": "informs",
      "condition": "A design team is shaping a human-AI collaboration flow or how the system's capability is presented",
      "reason": "Human-centered design requires understanding how people actually interpret and use system output, which reinforces the goal of calibrating trust and avoiding over- or under-reliance.",
      "basis": "editorial"
    },
    {
      "target": "error-prevention",
      "kind": "composes_with",
      "condition": "System output feeds directly into a high-stakes decision",
      "reason": "Offering a verification step before a costly adoption is the same kind of design move as error prevention's \"check or confirm before submitting\" — only here the thing being checked is the system's judgment rather than the user's own input.",
      "basis": "editorial"
    }
  ],
  "rules": [
    {
      "when": "The system's accuracy is noticeably lower for certain tasks, inputs, or groups",
      "instruction": "Communicate where the system performs poorly just as clearly as where it performs well; do not showcase only the favorable cases.",
      "strength": "must",
      "basis": "editorial"
    },
    {
      "when": "Deciding the tone, personality, or interface presentation of output",
      "instruction": "Do not let confident phrasing, personality, or visual polish stand in for an actual accuracy signal.",
      "strength": "must",
      "basis": "editorial"
    },
    {
      "when": "Output will be adopted directly for a decision with real consequences",
      "instruction": "Provide a way to verify the output before acting on it, sized to the consequence, rather than requiring the same verification for everything.",
      "strength": "recommend",
      "basis": "editorial"
    }
  ],
  "principle": {
    "evidenceType": "A design interpretation of human-AI interaction guidelines and automation-trust research",
    "limit": "This entry does not measure how much any specific calibration technique reduces over-acceptance or under-use, nor does it prescribe a universal \"right\" confidence threshold; trust calibration remains an active research area, and this entry cites only verifiable definitional claims, not experimental effect sizes it has not read."
  },
  "demo": null,
  "version": "0.1.0",
  "editorialStatus": "drafted",
  "requirements": [],
  "demoCapabilities": [],
  "implementationNote": "This entry has no operable specimen yet; the illustration is an editorial sketch. Applying it to a product still requires verifying that confidence or certainty signals match measured accuracy, that the verification flow is actually workable in the real setting, and that differences in system capability across tasks or groups are honestly communicated.",
  "image": {
    "src": "/images/trust-calibration.webp",
    "alt": "A balance scale with a confidence meter on one pan and a verification checkmark on the other, over a faintly visible fictional FIELDNOTES panel, with no legible numbers anywhere in the image."
  },
  "searchTerms": [
    "信任校准",
    "可靠性感知",
    "过度依赖",
    "置信信号",
    "核查"
  ]
}