Dataset · Hugging Face

anisoleai/fineweb-tokenized

Name: fineweb-tokenized
Creator: anisoleai
License: odc-by
Keywords: task_categories:text-generation, language:en, license:odc-by, size_categories:n>1T, format:parquet, modality:tabular, modality:text, library:datasets, library:dask, library:polars, library:mlcroissant, arxiv:2406.17557, region:us, tabular, text

FineWeb Tokenized > 4 trillion tokens of the pre-tokenized data the 🌐 web has to offer What is it? This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards. By distributing the pre-tokenized corpus, we eliminate… See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.

View on Hugging FaceReport an issue

Author / organization	anisoleai
License (as declared)	odc-by
Downloads	45.3K (as reported by Hugging Face)
Likes	34
Created	26 May 2026
Last updated at source	29 May 2026

tabulartextpre-training

Embed this entry

<iframe src="https://silytics.si/embed/item/b4503658-3836-421d-9ec7-b80d22837910" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 15 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.