Repository · GitHub

huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Owner	huggingface
License (as declared)	Apache-2.0
Stars	11,170
Forks	1,224
Main language	Rust
Created	1 November 2019
Last updated at source	14 October 2026

rustbertgptlanguage-modelnatural-language-processingnatural-language-understandingnlptransformers

Embed this entry

<iframe src="https://silytics.si/embed/item/3e11758c-f036-4111-a034-a2baa21a47a9" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

More from huggingface

transformersRepository
pytorch-image-modelsRepository
diffusersRepository
datasetsRepository
peftRepository
transformers.jsRepository

Silytics shows public metadata from GitHub and links to the original. Information may be incomplete or out of date. Last refreshed 15 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.