Dataset · Hugging Face

Abzalbek89/kk-tokenizer-fertility-baseline

Name: kk-tokenizer-fertility-baseline
Creator: Abzalbek89
License: apache-2.0
Keywords: task_categories:text-classification, task_categories:fill-mask, language:kk, license:apache-2.0, size_categories:n<1K, region:us, kazakh, tokenizer, fertility, benchmark, low-resource, agglutinative, morphology-aware, nlp

Kazakh Tokenizer Fertility Baseline Reproducible fertility benchmark of subword tokenizers on the Kazakh language. Companion artifact for the paper "Tokenizer Optimization for Kazakh Small Language Models" (in preparation, target: ACM TALLIP). Headline numbers Tokenizer Fertility 🥇 Best overall kk-bpe-32k 1.679 🚨 Worst GPT-4 (cl100k) 5.895 GPT-4 penalty GPT-4 (cl100k) is 3.51× worse than the best Kazakh-trained tokenizer → The custom Kazakh… See the full description on the dataset page: https://huggingface.co/datasets/Abzalbek89/kk-tokenizer-fertility-baseline.

View on Hugging FaceReport an issue

Author / organization	Abzalbek89
License (as declared)	apache-2.0
Downloads	480 (as reported by Hugging Face)
Likes	0
Created	29 April 2026
Last updated at source	11 July 2026

kazakhtokenizerfertilitybenchmarklow-resourceagglutinativemorphology-awarenlp

Embed this entry

<iframe src="https://silytics.si/embed/item/ed09ad55-8101-4c18-82f3-68a3c3ffb96f" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

Popularity, last 3 days

480 now (-35 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 18 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.