Dataset · Hugging Face

LocalDoc/azerbaijani-pretrain-corpus

Name: azerbaijani-pretrain-corpus
Creator: LocalDoc
License: cc-by-4.0
Keywords: task_categories:text-generation, language:az, license:cc-by-4.0, size_categories:1M<n<10M, format:parquet, modality:text, library:datasets, library:dask, library:polars, library:mlcroissant, region:us, azerbaijani, pretraining, deduplicated

Azerbaijani Pretraining Corpus (merged & deduplicated) A cleaned Azerbaijani text corpus assembled for language-model pretraining, merging two curated sources and removing exact duplicates. Contents Documents: 6,931,898 Tokens: ~5.36B (measured with the o200k_base tokenizer; an Azerbaijani-specific tokenizer will yield fewer tokens, as o200k_base segments agglutinative Azerbaijani inefficiently) Avg tokens/document: ~773 Fields text — the… See the full description on the dataset page: https://huggingface.co/datasets/LocalDoc/azerbaijani-pretrain-corpus.

View on Hugging FaceReport an issue

Author / organization	LocalDoc
License (as declared)	cc-by-4.0
Downloads	151 (as reported by Hugging Face)
Likes	0
Created	5 June 2026
Last updated at source	5 June 2026

azerbaijanipretrainingdeduplicated

Embed this entry

<iframe src="https://silytics.si/embed/item/9b24c629-fe8c-46a2-9dc4-eaf4cb3dc703" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

More from LocalDoc

LocalDoc/azerbaijani_asrDataset
LocalDoc/climbmix-40b-azDataset
LocalDoc/azerbaijani_retriever_corpusDataset
LocalDoc/AzTCDataset
LocalDoc/AzTC-fullDataset
LocalDoc/YOXLA-BenchmarkDataset

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 16 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.