jiezhueval/fineweb_edu_llama3
fineweb_edu_llama3 Pre-tokenized subset of HuggingFaceFW/fineweb-edu (sample/100BT configuration), tokenized with the Llama 3 tokenizer and stored as flat binary token streams for direct memory-mapped consumption during LM pretraining. Contents train.bin.part00 … train.bin.part08 — shards of train.bin (43,913,348,103 tokens) val.bin — validation split (71,512,480 tokens) meta.json — tokenizer metadata and checksums Format Each .bin file is a raw… See the full description on the dataset page: https://huggingface.co/datasets/jiezhueval/fineweb_edu_llama3.
| Author / organization | jiezhueval |
|---|---|
| License (as declared) | odc-by |
| Downloads | 0 (as reported by Hugging Face) |
| Likes | 0 |
| Created | 22 October 2026 |
| Last updated at source | 22 October 2026 |
Recent changes
- 22 Oct 2026 License: none → odc-by
- 22 Oct 2026 Description: none → fineweb_edu_llama3 Pre-tokenized subset of HuggingFaceFW/fineweb-edu (sample/100BT configuration), tokenized with the Llama 3 tokenizer and stored as flat binary token streams for direct memory-mapped consumption during LM pretraining. Cont
- 22 Oct 2026 Topics added task_categories:text-generation, pretraining, tokenized, fineweb-edu, llama3
Changes detected by Silytics between refreshes of the source.
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 22 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.