ACE Journal

Designing Trust Cues for AI Copilots

Who this is for: product engineers and UX designers building features on top of a large language model or other probabilistic system, who have a working copilot but are watching users either rubber-stamp its output or ignore it entirely. The assumption is that you can instrument your own UI and have some access to the model’s output distribution or a retrieval layer, but you are not retraining the model and you are not asking users to read logits. If your honest internal metric is “people click accept without reading,” this is about the signals that change that behavior.

The accept button nobody reads

Watch someone use a code copilot for an hour and you see the same motion repeated: a gray suggestion appears, the developer glances at it, and they press Tab. Sometimes they read it. Often they do not. The suggestion that deletes a migration and the suggestion that renames a local variable arrive through the same channel, in the same font, with the same ghosted styling, and they get the same reflexive Tab. The interface has flattened a spectrum of risk into a single uniform gesture.

The mirror image is just as common and just as costly. A clinician using a diagnostic dashboard dismisses an AI-suggested differential on principle, because the last three suggestions were obvious and one was wrong, so the tool has been filed under “noise.” The good suggestion and the bad one are again indistinguishable at the surface, so the safe move is to ignore all of them.

These are not two problems but one. The human-factors literature has shown for decades that operators calibrate their reliance on automated aids from the cues the system gives them, and that miscalibration runs both ways: over-reliance when the system looks more trustworthy than it is, disuse when it looks less [1], [2]. The job of a trust cue is not to make the user trust the AI more. It is to make their trust track the AI’s actual reliability, suggestion by suggestion. The literature calls that target appropriate reliance, and it is the thing we are designing for.

Figure 1 shows the loop a copilot interface is really mediating. The model produces an output with some internal uncertainty; the interface translates that into cues; the user forms a level of trust and decides to accept, edit, or reject; and that decision, plus the eventual outcome, is the signal we want to feed back. Every section below is a different intervention point on this loop.

flowchart LR
    Model[model output<br/>+ uncertainty] --> Cues[interface trust cues]
    Cues --> Trust[user's calibrated trust]
    Trust --> Action[accept / edit / reject]
    Action --> Outcome[outcome observed]
    Outcome -. feedback signal .-> Model
    Outcome -. feedback signal .-> Cues

Figure 1. The reliance loop a copilot interface mediates, from model uncertainty through displayed cues to the user’s accept-or-reject decision. Look at the dotted return arrows: the outcome of a decision is a signal, and a design that drops it on the floor is leaving its best trust cue unused.

Confidence signals that do not lie

The most common mistake in copilot design is the most natural one: take the model’s confidence score and render it as a progress bar or a percentage. It feels honest. It is usually misleading.

The problem is calibration. A softmax output is a probability distribution, but a probability is only meaningful if it is calibrated, that is, if the things the model calls “90% likely” are right about 90% of the time. Modern neural networks are systematically overconfident, and that overconfidence is worst on inputs that differ from the training distribution [3], [4]. So the number behaves backward relative to what the user needs. On easy in-distribution inputs where the user hardly needs help, the score is roughly right. On the strange off-distribution input where a warning would be most valuable, the model is confidently wrong, and the UI faithfully renders that false confidence as a reassuring 92%.

There are more honest signals, and they share a property: they expose the structure of the model’s uncertainty instead of compressing it into one lossy number.

Show the spread, not a point. When the model is genuinely torn between two outputs, surface both rather than picking one and hiding the conflict. The disagreement is information. A code copilot offering several completions to cycle through, rather than asserting one answer, is a working instance: the presence of plausible alternatives is itself a signal that the model is not certain, and it invites the read-before-accept behavior a single ghosted line suppresses.

Flag the off-distribution input. A lightweight anomaly check on the input, separate from the main model, can catch the case the confidence score will lie about. A detector that notices the current input sits far from a reference set of typical inputs is enough to attach a quiet “this is unusual, double-check” rather than a confident answer dressed as routine.

Prefer hedged language to numbers. For generated text, calibrated natural-language hedging often communicates uncertainty better than a score, because users read “this may not handle the empty-input case” as an actionable caution, while they read “78%” as a grade and round it up.

Here is the anti-pattern and the fix side by side. First, the version that ships the raw score straight to the UI:

// Anti-pattern: raw softmax rendered as authoritative confidence.
function SuggestionBadge({ logits }: { logits: number[] }) {
  const probs = softmax(logits);
  const top = Math.max(...probs);
  // This reads as precision the model does not have on edge cases.
  return <Badge>{`${Math.round(top * 100)}% confident`}</Badge>;
}

The numeric badge is most wrong precisely when the input is unusual. The fix keeps the same data but changes what the interface asserts: it checks distance from the training distribution, and on the margin it shows alternatives and a hedge instead of a single confident claim.

type Cue =
  | { kind: "answer"; text: string }
  | { kind: "alternatives"; options: string[] }
  | { kind: "ood"; text: string };

function deriveCue(probs: number[], outputs: string[], oodScore: number): Cue {
  // oodScore: distance from the reference input distribution, 0 = typical.
  if (oodScore > OOD_THRESHOLD) {
    return { kind: "ood", text: outputs[argmax(probs)] };
  }
  const ranked = rankByProb(outputs, probs);
  // Margin between top two: small margin means the model is split.
  if (ranked[0].prob - ranked[1].prob < SPLIT_MARGIN) {
    return { kind: "alternatives", options: ranked.slice(0, 3).map(r => r.text) };
  }
  return { kind: "answer", text: ranked[0].text };
}

The answer branch is the only one that lets the UI speak with a single voice, and it is gated on the input being typical and the model being decisive. Everything else degrades gracefully into “here are the options” or “this looks unusual.” The model’s raw confidence still informs the decision; it just never reaches the user as a bare percentage.

Contextual disclosure over global warnings

The second insight is about where the caution lives, not how confident it is. The default move, a banner reading “AI can make mistakes” pinned to the top of the interface, is the least effective option available, and it is the one almost every product ships first.

The reason it fails is well understood under the name warning habituation: a warning that fires constantly and identically becomes invisible, because the user’s attention adapts to it as background [5]. A persistent banner is the strongest possible habituation trigger. It is on every screen, says the same thing every time, and is decoupled from any specific risk, so within days of onboarding the eye stops registering it. Worse, it is technically present, which lets a team believe the user was warned while the user, in practice, was not.

The alternative is contextual disclosure: attach the caution to the specific high-stakes moment, not to the tool as a whole. When a copilot is about to suggest deleting data, making an irreversible API call, or recommending a medication dose, that is the moment to surface a caution, placed adjacent to the action, phrased in terms of what to verify. A single inline note, “verify before applying to production,” next to the suggestion it concerns does more to trigger review than a banner the user scrolled past at signup ever will.

Figure 2 contrasts the two models. The global warning fires once, far from the action, and habituates. The contextual cue stays silent on low-risk actions and fires only at the moment of a consequential one, which both preserves its salience and ties it to something the user can act on.

flowchart TB
    subgraph Global["global warning (habituates)"]
        A[onboarding] --> B[banner: 'AI can make mistakes']
        B --> C[every later action]
        C -.->|ignored| C
    end
    subgraph Contextual["contextual disclosure (stays salient)"]
        D[low-risk action] -->|no cue| E[apply silently]
        F[high-risk action] -->|inline caution<br/>at the moment| G[user verifies]
    end

Figure 2. Two models for surfacing caution. The global banner fires once and decays into background; the contextual cue stays silent on routine actions and appears only at the high-risk moment. Look at the right path: the cue is co-located with the consequential action rather than the tool.

This pushes work onto the engineer, because someone has to classify which actions are high stakes, and the classification is domain-specific. A reversible config edit is low stakes; a schema migration is not. But that classification is exactly the design judgment a trust cue encodes, and it is cheap relative to its payoff. A minimal version attaches a risk tier to each action type:

RISK = {
    "rename_local": "low",
    "format_file": "low",
    "drop_table": "high",
    "delete_resource": "high",
    "charge_card": "high",
}

def disclosure_for(action: str, target_env: str) -> str | None:
    tier = RISK.get(action, "medium")
    if tier == "high":
        return f"This {action} is hard to undo. Verify before applying."
    if tier == "medium" and target_env == "production":
        return "Applying to production. Confirm this is intended."
    return None  # low risk: stay silent, do not train the user to ignore you

Returning None is the load-bearing case. The discipline that keeps contextual disclosure effective is staying quiet on everything that does not warrant a warning, because every unnecessary caution spends down the credibility of the next necessary one. A system that warns about everything has, functionally, warned about nothing.

Audit trails as trust infrastructure

The third insight is that inspectability changes behavior. Users who can see why a suggestion was made trust it more when it is right and override it more readily when it is wrong, which is exactly the two-way calibration we are after. This is the practical, UI-level descendant of the explainability literature: the point is not a faithful mathematical account of the model’s internals, which users cannot use, but a usable mental model of where this particular answer came from [6], [7].

The cues that work here expose provenance, not parameters. For a retrieval-augmented system, showing which retrieved chunks fed an answer is high value and cheap, because the chunks already exist in the pipeline. The user does not need the embedding math; they need to see that the answer about refund policy was grounded in the refund policy document and not a tangential FAQ. When the provenance is wrong, the user catches a hallucination before acting on it. When it is right, the citation does the trust-building work a confidence score cannot.

This matters most in regulated settings, where a compliance or clinical reviewer has to reconstruct why an AI-assisted decision was made. A provenance UI that ties each response to its inputs is the difference between an auditable system and an opaque one. Tooling has moved this way: tracing systems for LLM applications, such as LangSmith and Weights and Biases Prompts, provide per-response trace views capturing the prompt, the retrieved context, and the output as one inspectable record [8], [9]. The same trace an engineer uses to debug a bad answer is, with a lighter presentation, the provenance the end user needs.

The engineering ask is modest: thread an identifier through the call and persist the retrieved context and final prompt alongside the response, so any answer can be expanded into its sources on demand.

def answer_with_provenance(query: str, store) -> dict:
    chunks = store.retrieve(query, k=4)         # the evidence
    prompt = build_prompt(query, chunks)
    response = model.generate(prompt)
    return {
        "response": response,
        "provenance": [
            {"source": c.source, "snippet": c.text[:200], "score": c.score}
            for c in chunks
        ],
        "trace_id": new_trace_id(),             # join key into the trace store
    }

The UI then renders the answer with an expandable “based on” section listing each source. The cost is one extra field on the response object and a disclosure widget. The payoff is that the user can check the work, the single most reliable way to let trust track reliability instead of floating free of it.

Closing the feedback loop

The fourth insight is that the feedback affordance is itself a trust cue, and most products break it. Thumbs-up and thumbs-down sit next to every response, the user occasionally clicks one, and nothing observable happens. The signal disappears into a metrics dashboard the user never sees, and they learn, correctly, that their input changed nothing. After that they stop giving it, and the dead loop becomes its own quiet message: this system does not listen.

Closing the loop does not require retraining the model on every correction. That framing is what makes teams treat feedback as a batch input to some future fine-tune rather than a live element of the interaction. A correction can be honored immediately and cheaply within the session. A thumbs-down on a retrieved answer can trigger a re-rank that demotes the rejected source and surfaces a different one. A “not what I meant” can adjust the prompt for the next turn. None of these touch model weights, and all give the user the thing that sustains engagement: visible evidence that the signal landed.

Figure 3 shows the difference. In the open loop, feedback exits to a store the user never sees again. In the closed loop, the same click produces an immediate, observable change in the next response, and that observability is the trust cue.

sequenceDiagram
    participant U as user
    participant C as copilot
    participant R as retrieval / prompt
    U->>C: thumbs-down on answer
    Note over C: open loop ends here
    C-->>U: (nothing visible changes)
    U->>C: thumbs-down on answer
    C->>R: re-rank, demote rejected source
    R-->>C: revised context
    C-->>U: new answer, visibly different
    Note over U,C: closed loop: the signal landed

Figure 3. Open versus closed feedback loops on the same thumbs-down gesture. In the open path the click vanishes; in the closed path it re-ranks retrieval and produces a visibly different answer. Look at the final message: the observable change is the trust cue, not the button itself.

A within-session implementation can be small. Keep a per-session set of demoted sources and apply it on the next retrieval, so a rejection takes effect on the very next turn:

class SessionFeedback:
    def __init__(self):
        self.demoted: set[str] = set()

    def reject(self, source_id: str) -> None:
        self.demoted.add(source_id)          # immediate, session-scoped

    def rerank(self, chunks: list) -> list:
        # Push rejected sources down without dropping them entirely.
        return sorted(
            chunks,
            key=lambda c: (c.source in self.demoted, -c.score),
        )

The correction never leaves the session and never touches a model, but the next answer is observably different, which is the entire point. Persisting these signals for later analysis is fine, but it is a separate concern from the live loop. The trust cue is the immediate, visible response; the analytics are a bonus the user cannot see and therefore cannot be reassured by.

How the cues fit together in a real surface

In a shipping copilot these four cues are not separate features; they layer onto the same suggestion, in roughly the order of the reliance loop. First the confidence logic decides whether to assert one answer, show alternatives, or flag the input as unusual. Then the disclosure logic decides whether the specific action warrants an inline caution. The provenance affordance hangs off the rendered answer, on demand. And the feedback control sits adjacent, wired to do something observable on the next turn.

The integration discipline is restraint. Each cue competes for the same scarce attention, and a suggestion festooned with a percentage, a banner, a sources panel, and three feedback widgets has communicated nothing because it tried to communicate everything at once. The strong default is to show the answer cleanly, escalate to alternatives or a caution only when warranted, and keep provenance and feedback one interaction away rather than always on screen. Salience is a budget, and spending it on a routine suggestion leaves nothing for the consequential one. The design reduces to one rule: make the unusual case look unusual and let the routine case stay quiet.

The space of approaches to AI-output trust sorts into a few families, and the cue-based UX design here is one corner of it.

Model calibration attacks the problem upstream, at the score itself. Techniques such as temperature scaling and other post-hoc methods aim to make a model’s stated confidence match its real accuracy [3]. This is genuinely worth doing, and a well-calibrated score is a better input to the cue logic above. But calibration is a complement, not a substitute. Even a perfectly calibrated number still has to be communicated, and a calibrated 70% shown as a progress bar invites the same misreading as an uncalibrated one. Calibration improves the signal; it does not solve the display problem.

Explainable AI in its fuller form, feature attributions, saliency maps, surrogate models, aims to explain the model’s reasoning rather than its provenance [6], [7]. For end users in a copilot, most of this is too heavy: a developer accepting a completion will not read a SHAP plot. The provenance approach here is the pragmatic subset, showing where an answer came from rather than why the weights produced it, because source attribution is cheaper and more actionable for a non-expert at the moment of decision.

Guardrails and hard constraints take the opposite stance: rather than helping the user judge, they prevent certain outputs outright. These matter for genuinely unacceptable outcomes, but they are blunt, and a system that only blocks teaches the user nothing about calibration. Trust cues and guardrails are layers: the guardrail stops the catastrophic action, the cue helps the user navigate the large remaining space of merely risky ones. None of these replaces the others. Calibrate the score, explain what you usefully can, guard the unacceptable, and use cues to calibrate reliance on everything in between.

Where it breaks

Trust-cue designs fail in recognizable ways, and most of the failures are about credibility rather than mechanics.

The cry-wolf problem. A caution or anomaly flag that fires too often trains the user to dismiss it, at which point it is worse than absent, because it consumed attention and credibility before going silent in the user’s mind. This is the warning-habituation failure [5] reappearing one level down: contextual disclosure is only as good as its precision, and a noisy anomaly detector poisons the very cue it powers. Tuning the threshold to favor silence is usually right.

Trust cues as a license to over-trust. A provenance panel or a confidence display can backfire by making a wrong answer look authoritative. Users have been shown to over-rely on systems that merely appear transparent, treating the presence of an explanation as evidence of correctness regardless of its content [10]. A sources panel listing plausible but irrelevant chunks can increase misplaced trust. The cue has to be faithful, not just present, or it manufactures the over-reliance it was meant to prevent.

Calibration drift over time. The off-distribution detector and the risk tiers are built against an assumption about what normal input looks like, and that assumption decays as usage shifts. A detector tuned at launch silently degrades, and a risk classification written for last year’s feature set misses this year’s high-stakes action. These need the same monitoring as any model in production.

The cold-start feedback gap. The closed loop needs signal to act on, and early in a session, or for a new user, there is none. A re-rank with nothing to re-rank does nothing, and a loop that visibly does nothing reads as a dead one. The honest design degrades gracefully rather than promising responsiveness it cannot yet deliver.

Cue overload. Layering all four cue families onto every suggestion defeats all of them, because attention is finite and a uniformly loud interface is a uniformly ignored one. The cues are individually sound and collectively counterproductive when shown at once; the discipline of staying quiet on the routine case is what keeps any single cue legible.

Domain mismatch in stakes. A risk taxonomy is domain-specific. A caution model tuned for a code editor, where most actions are reversible in version control, is dangerously permissive transplanted into a clinical or financial tool, where the same nominal action carries far higher stakes. The classification of what is high-stakes has to be redone per domain; it does not transfer.

Further reading

Start with the foundational human-factors work on reliance: Lee and See on trust in automation [1] and Parasuraman and Riley on use, misuse, and disuse [2] are the frame everything here sits inside. For the calibration problem behind confidence displays, Guo and colleagues on the calibration of modern neural networks [3] and Hendrycks and Gimpel on detecting out-of-distribution inputs [4] are the canonical sources. On warning habituation, Wogalter’s handbook treatment [5] explains why the global banner fails. For the explainability and provenance angle, the DARPA XAI program framing [6] and Doshi-Velez and Kim on interpretability rigor [7] set the terms, while the LangSmith [8] and Weights and Biases Prompts [9] docs show the tracing infrastructure in practice. Finally, Bansal and colleagues on whether explanations can cause over-reliance [10] is the necessary corrective to assuming more transparency is always better.

Written with AI assistance, editorially reviewed.

References

[1] J. D. Lee and K. A. See, “Trust in Automation: Designing for Appropriate Reliance,” Human Factors, vol. 46, no. 1, 2004. https://journals.sagepub.com/doi/10.1518/hfes.46.1.50.30392

[2] R. Parasuraman and V. Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse,” Human Factors, vol. 39, no. 2, 1997. https://journals.sagepub.com/doi/10.1518/001872097778543886

[3] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” ICML, 2017. https://arxiv.org/abs/1706.04599

[4] D. Hendrycks and K. Gimpel, “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks,” ICLR, 2017. https://arxiv.org/abs/1610.02136

[5] M. S. Wogalter, ed., “Handbook of Warnings,” Human Factors and Ergonomics, 2006. https://www.taylorfrancis.com/books/edit/10.1201/9781482289688/handbook-warnings-michael-wogalter

[6] D. Gunning and D. Aha, “DARPA’s Explainable Artificial Intelligence (XAI) Program,” AI Magazine, vol. 40, no. 2, 2019. https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2850

[7] F. Doshi-Velez and B. Kim, “Towards a Rigorous Science of Interpretable Machine Learning,” 2017. https://arxiv.org/abs/1702.08608

[8] LangChain, “LangSmith Tracing Documentation.” https://docs.smith.langchain.com/

[9] Weights and Biases, “W&B Prompts and Tracing.” https://docs.wandb.ai/guides/prompts/

[10] G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro, and D. S. Weld, “Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance,” CHI, 2021. https://dl.acm.org/doi/10.1145/3411764.3445717

[11] A. Vasconcelos et al., “Explanations Can Reduce Overreliance on AI Systems During Decision-Making,” Proc. ACM HCI (CSCW), 2023. https://dl.acm.org/doi/10.1145/3579605

[12] GitHub, “GitHub Copilot Documentation.” https://docs.github.com/en/copilot