sam-paech/mmlu-pro-irt-1-0
MMLU-Pro-IRT This is a small subset of MMLU-Pro, selected with Item Response Theory for better separation of scores across the ability range. It contains 2059 items (compared to 12000 in the full MMLU-Pro), so it's faster to run. It takes ~6 mins to evaluate gemma-2-9b on a RTX-4090 using Eleuther LM-Eval. Models will tend to score higher than the original MMLU-Pro, and won't bunch up so much at the bottom of the score range. Why do this? MMLU-Pro is great, but it… See the full description on the dataset page: https://huggingface.co/datasets/sam-paech/mmlu-pro-irt-1-0.
| Author / organization | sam-paech |
|---|---|
| License (as declared) | mit |
| Downloads | 171 (as reported by Hugging Face) |
| Likes | 6 |
| Created | 3 July 2024 |
| Last updated at source | 5 July 2024 |
Popularity, last 4 days
171 now. Counts as reported by the source, snapshotted daily by Silytics.
Recent changes
- 17 Oct 2026 Description: MMLU-Pro-IRT This is a small subset of MMLU-Pro, selected with Item Response Theory for better separation of scores across the ability range. It contains 2059 items (compared to 12000 in the full MMLU-Pro), so it's faster to run. It takes ~ → MMLU-Pro-IRT This is a small subset of MMLU-Pro, selected with Item Response Theory for better separation of scores across the ability range. It contains 2059 items (compared to 12000 in the full MMLU-Pro), so it's faster to run. It takes ~
Changes detected by Silytics between refreshes of the source.
More from sam-paech
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 19 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.