When Counterbalancing Hides the Bias: Access-Conditioned Position Lock in Forced-Choice LLM Evaluation
Forced-choice probes with counterbalanced orientations are a standard tool for measuring language-model "value dispositions," and a concentration/extremity index over repeated draws is read as how sharply a model commits. We show this estimator is not identifiable at its low end: counterbalancing, meant to remove position bias, instead maps a position-lock (a model returning the same answer letter regardless of content) onto the same near-0.5 signature as genuine neutrality, so "softness" and "non-engagement" cannot be distinguished from a content-independent letter-bias. Across nine models the fraction of position-locked items tracks the index almost perfectly (r=-0.986) - a structural c...
| Authors | Hong-In Won, Jinseok Jang, Hyoseop Kim |
|---|---|
| Primary category | cs.LG |
| Published | 11 July 2026 |
| Last updated at source | 8 August 2026 |
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.