Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibration

Is verbalized confidence better calibrated than token likelihood? The answer depends on how the token likelihood is measured: which answer is scored, and under which prompt. Published comparisons diverge on this, and in a twelve-study audit five never state the choice. We fix one prediction event per question, the model's own answer together with its correctness label, and score that same answer under a plain query and inside the confidence prompt, holding the answer and its label fixed. Across four QA datasets and three 7-8B Instruct models this changes which signal performs better, by point estimate, in 4 of 12 settings under ECE and 9 of 12 under AUROC. The AUROC result cannot come fro...

<iframe src="https://silytics.si/embed/item/987d0d72-ef32-44aa-8d39-7f23ad9e7545" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.