jalenluorion/prefetchable-store-corpus-v4
Pre-tokenised language-model training corpus, v4 The v4 build of the v3 corpus: a byte-identical superset of v3 for longer runs. 64,600,000,000 tokens of FineWeb-Edu, tokenised with the Llama-2 tokenizer (vocabulary 32,000, uint16, no BOS/EOS, documents concatenated), plus the same three held-out evaluation slices as v3. Sequences are 2,048 tokens; seq_permutation.npy fixes the training order so that a run on the first N sequences is a byte-identical prefix of any longer run.… See the full description on the dataset page: https://huggingface.co/datasets/jalenluorion/prefetchable-store-corpus-v
| Author / organization | jalenluorion |
|---|---|
| License (as declared) | odc-by |
| Downloads | 0 (as reported by Hugging Face) |
| Likes | 0 |
| Created | 24 October 2026 |
| Last updated at source | 24 October 2026 |
Recent changes
- 24 Oct 2026 License: none → odc-by
- 24 Oct 2026 Description: none → Pre-tokenised language-model training corpus, v4 The v4 build of the v3 corpus: a byte-identical superset of v3 for longer runs. 64,600,000,000 tokens of FineWeb-Edu, tokenised with the Llama-2 tokenizer (vocabulary 32,000, uint16, no BOS/E
- 24 Oct 2026 Topics added task_categories:text-generation
Changes detected by Silytics between refreshes of the source.
More from jalenluorion
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 24 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.