Dataset · Hugging Face

BramVanroy/stack_md_lid

Name: stack_md_lid
Creator: BramVanroy
Keywords: language:multilingual, language:abk, language:ace, language:ady, language:afr, language:aka, language:als, language:amh, language:arb, language:arn, language:asm, language:ast, language:ayr, language:azb, language:azj

A copy of the deduplicated the stack Markdown files, annotated with fastText to include language labels (lid column) and probabilities (lid_prob). This was done using the NLLB fastText model facebook/fasttext-language-identification. The following languages were detected: abk_Cyrl ace_Arab ace_Latn ady_Cyrl afr_Latn aka_Latn als_Latn amh_Ethi arb_Arab arb_Latn arn_Latn asm_Beng ast_Latn ayr_Latn azb_Arab azj_Latn bak_Cyrl bam_Latn ban_Latn bel_Cyrl bem_Latn ben_Beng bho_Deva bis_Latn bjn_Arab… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/stack_md_lid

View on Hugging FaceReport an issue

Author / organization	BramVanroy
Downloads	816 (as reported by Hugging Face)
Likes	4
Created	5 June 2024
Last updated at source	22 August 2024

Embed this entry

<iframe src="https://silytics.si/embed/item/47c3bf2c-ee1e-4526-87a5-9b116e947896" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

More from BramVanroy

BramVanroy/wikipedia_culturax_dutchDataset
BramVanroy/fineweb-2-duckdbsDataset
BramVanroy/CommonCrawl-CreativeCommonsDataset
BramVanroy/CommonCrawl-CreativeCommons-strictDataset
BramVanroy/fineweb-duckdbsDataset
BramVanroy/CommonCrawl-CreativeCommons-fineDataset

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 16 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.