Multi-Legal-Bench: When the Answer Is in the Input. Label Leakage in Legal Benchmarks Built from Court Registries

Court registries publish millions of decisions with structured metadata, which makes them an attractive source of labelled legal benchmarks: the court type, the form of the decision and its subject area come for free. We show that this convenience has a cost that registry-derived benchmarks rarely measure. We build Multi-Legal-Bench, which evaluates identical tasks on native court decisions from five national registries (France, the Netherlands, Poland, the Czech Republic and Lithuania), extending the Ukrainian UA-Legal-Bench, and audit what its cells actually measure. A keyword scan that uses no model reaches 96% on Dutch judgment-form classification against a 50% majority baseline, 84% ...

<iframe src="https://silytics.si/embed/item/3c0aabd4-9605-44b3-ac9d-d874eaf8469c" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 18 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.