PINOCCHIO // BLACK-BOX UNCERTAINTY INSTRUMENT

Fast uncertainty estimates for black-box language models  /  P(correct) for any black-box API

Pinocchio

One external calibrator reads only the text of a response and returns the probability it is correct, in a single forward pass. No log-probabilities, no sampling, no weights, no fine-tuning of the target. It scores closed models it never saw.

Kevin David Hayes, Arka Pal, Haosong Zhang, Tom Goldstein, Micah Goldblum

University of Maryland · Ritual AI · Fudan University · Columbia University

Accepted to COLM 2026

0.863
held-out AUROC predicting correctness, against 0.649 for the best single-pass baseline
0.814
mean AUROC, zero-shot, on 13 unseen models across 8 organizations
1 pass
single forward pass, no logits, weights, samples, or fine-tuning

The models doing the most consequential work are closed. They do not return log-probabilities and cannot be fine-tuned, which rules out most ways of estimating uncertainty, and their own stated confidence is poorly calibrated.

Pinocchio sits outside the target model. It reads the question and the answer, and predicts whether that answer is correct. It handles vision-language inputs and transfers to models it was never trained on. Full method, baselines and ablations are in the paper.

01

Wiring it in

quickstart.pypython
# pip install pinocchio-uq
from pinocchio import Pinocchio

judge = Pinocchio()  # load once

response = client.chat.completions.create(model="gpt-5", messages=messages)
p_correct = judge.score(response, messages=messages)

if p_correct < 0.5:
    escalate(response)

Same interface across model families: swap model="gpt-5" for a Claude, Gemini, or local response and call judge.score unchanged. The package ships a lightweight text-only 0.8B checkpoint.

02

Cite

bibtex
@inproceedings{hayes2026pinocchio,
  title     = {Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models},
  author    = {Hayes, Kevin David and Pal, Arka and Zhang, Haosong
               and Goldstein, Tom and Goldblum, Micah},
  booktitle = {Conference on Language Modeling (COLM)},
  year      = {2026},
}