Fast uncertainty estimates for black-box language models / P(correct) for any black-box API
One external calibrator reads only the text of a response and returns the probability it is correct, in a single forward pass. No log-probabilities, no sampling, no weights, no fine-tuning of the target. It scores closed models it never saw.
University of Maryland · Ritual AI · Fudan University · Columbia University
Accepted to COLM 2026
The models doing the most consequential work are closed. They do not return log-probabilities and cannot be fine-tuned, which rules out most ways of estimating uncertainty, and their own stated confidence is poorly calibrated.
Pinocchio sits outside the target model. It reads the question and the answer, and predicts whether that answer is correct. It handles vision-language inputs and transfers to models it was never trained on. Full method, baselines and ablations are in the paper.
# pip install pinocchio-uq
from pinocchio import Pinocchio
judge = Pinocchio() # load once
response = client.chat.completions.create(model="gpt-5", messages=messages)
p_correct = judge.score(response, messages=messages)
if p_correct < 0.5:
escalate(response)
Same interface across model families: swap model="gpt-5" for a Claude, Gemini, or local response and call judge.score unchanged. The package ships a lightweight text-only 0.8B checkpoint.
@inproceedings{hayes2026pinocchio,
title = {Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models},
author = {Hayes, Kevin David and Pal, Arka and Zhang, Haosong
and Goldstein, Tom and Goldblum, Micah},
booktitle = {Conference on Language Modeling (COLM)},
year = {2026},
}