Dataset · Hugging Face

Polygl0t/tokenizers

Name: tokenizers
Creator: Polygl0t
License: other
Keywords: task_categories:text-generation, language:hi, language:bn, language:en, language:pt, license:other, size_categories:1M<n<10M, format:parquet, modality:text, library:datasets, library:dask, library:mlcroissant, library:polars, arxiv:2603.03543, arxiv:2603.03508

Polygl0t Tokenizers Dataset Summary This dataset contains several subsets for training multilingual tokenizers. Every subset possesses a collection of curated text samples in different languages. Supported Tasks and Leaderboards This dataset can be used for the task of text generation, specifically for training and evaluating tokenizers in multiple languages. Languages Hindi, Bengali, English, Portuguese, and Code (a mixture of 36… See the full description on the dataset page: https://huggingface.co/datasets/Polygl0t/tokenizers.

View on Hugging FaceReport an issue

Author / organization	Polygl0t
License (as declared)	other
Downloads	162 (as reported by Hugging Face)
Likes	0
Created	24 September 2025
Last updated at source	5 March 2026

tokenizertokenizationenglishcodebengalihindiportuguese

Embed this entry

<iframe src="https://silytics.si/embed/item/f3d565b5-0d0d-47bd-998a-b6f0e727db44" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

Recent changes

16 Oct 2026 Description: Polygl0t Tokenizers Dataset Summary This dataset contains several subsets for training multilingual tokenizers. Every subset possesses a collection of curated text samples in different languages. Supported Tasks and Leaderboards This datase → Polygl0t Tokenizers Dataset Summary This dataset contains several subsets for training multilingual tokenizers. Every subset possesses a collection of curated text samples in different languages. Supported Tasks and Leaderboards This datase

Changes detected by Silytics between refreshes of the source.

More from Polygl0t

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 16 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.