Dataset · Hugging Face

ASTERIZER/Ezaris-Tokenizer

Name: Ezaris-Tokenizer
Creator: ASTERIZER
License: cc-by-nc-nd-4.0
Keywords: license:cc-by-nc-nd-4.0, region:us, tokenizer, multilingual, bpe, corpus

Ezaris Tokenizer — Program & Corpus The single home of the Asterizer tokenizer for the Ezaris program: one byte-level BPE tokenizer, frozen once and reused across every ASTERIZER model from 100M → 1T params. South-Indian-first (Kannada / Tamil / Telugu / Malayalam), plus code, math, and broad multilingual coverage. Built only from open, license-audited data. This repo consolidates the former tokeniser / LUNA-Tokenizer-Corpus, LUNA-Tokenizer-Gap-V2, LUNA-1B-Tokenizer, and the… See the full description on the dataset page: https://huggingface.co/datasets/ASTERIZER/Ezaris-Tokenizer.

View on Hugging FaceReport an issue

Author / organization	ASTERIZER
License (as declared)	cc-by-nc-nd-4.0
Downloads	814 (as reported by Hugging Face)
Likes	0
Created	28 June 2026
Last updated at source	28 August 2026

tokenizermultilingualbpecorpus

Embed this entry

<iframe src="https://silytics.si/embed/item/a6d4a143-e3ab-4162-9844-f0b3b3385b98" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

Popularity, last 3 days

814 now (-150 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.

More from ASTERIZER

ASTERIZER/Ezaris-Training-SetsDataset
ASTERIZER/LUNA_PreTrainDataset

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 18 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.