Dataset · Hugging Face

commoncrawl/CommonLID

Name: CommonLID
Creator: commoncrawl
Keywords: task_categories:text-classification, annotations_creators:expert-generated, annotations_creators:crowdsourced, language_creators:found, multilinguality:multilingual, language:ace, language:acf, language:aeb, language:afr, language:amh, language:apd, language:ara, language:arb, language:arg, language:ars

CommonLID: Language Identification for the Real Web CommonLID is a community-created language identification (LID) benchmark. CommonLID consists of web text manually annotated for the language that it is written in. CommonLID contains annotations for 109 languages, where 78 of those languages have at least 100 lines of data. The number of lines available for each language is provided in Appendix A of the paper. ➡️ CommonLID Leaderboard on Hugging Face 🤗 ➡️ Source Code on… See the full description on the dataset page: https://huggingface.co/datasets/commoncrawl/CommonLID.

View on Hugging FaceReport an issue

Author / organization	commoncrawl
Downloads	205 (as reported by Hugging Face)
Likes	57
Created	3 February 2026
Last updated at source	22 September 2026

Embed this entry

<iframe src="https://silytics.si/embed/item/678f088a-8500-408c-a734-dd35afc3d7a0" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

More from commoncrawl

commoncrawl/gneissweb-annotation-url-testing-v1Dataset
commoncrawl/host-index-testing-v2Dataset
commoncrawl/statisticsDataset
commoncrawl/commonlid-resultsDataset
commoncrawl/gneissweb-annotation-host-testing-v1Dataset
commoncrawl/llms.txtDataset

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 16 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.