Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models
Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results. Reliable evaluation is particularly challenging for base models, whose limited instruction-following ability complicates task-based assessment. We introduce Uncheatable Eval, a dynamic benchmark that regularly collects newly published text to evaluate base language models and reduce the risk of data contamination. Drawing on the relationship between a model's predictive ability and its ability to compress data losslessly, we use compression rate to evaluate how well models predict new text. W...
| Authors | Kaifeng Tan, Yudong Li, Linlin Shen |
|---|---|
| Primary category | cs.CL |
| Published | 23 September 2026 |
| Last updated at source | 23 September 2026 |
Popularity, last 5 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Models and datasets referencing this paper
- udgamlabs/fresh-text-2026-10Dataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 21 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.