pico-lm/pretokenized-dolma
The Pretokenized Dolma Dataset A pre-tokenized, pre-shuffled version of Dolma, the high-quality text corpus from AI2. This dataset is designed to be plug-and-play with the pico-train library. Overview Key Features: Tokenized with allenai/OLMo-7B-0724-hf, a BPE-tokenized with a vocabulary size of 50280 Sequence length: 2049 tokens (2048 + 1 for next-token prediction) Sharded into 10,000 Parquet files (~78MB each) 420B tokens total size (perfect for training a… See the full description on the dataset page: https://huggingface.co/datasets/pico-lm/pretokenized-dolma.
| Author / organization | pico-lm |
|---|---|
| License (as declared) | apache-2.0 |
| Downloads | 180 (as reported by Hugging Face) |
| Likes | 4 |
| Created | 4 February 2025 |
| Last updated at source | 16 April 2025 |
Popularity, last 2 days
180 now (-18 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.
Connected in the knowledge graph
- pico-lm/pico-decoder-mediumModel · used to train
- pico-lm/pico-decoder-largeModel · used to train
- pico-lm/pico-decoder-smallModel · used to train
- pico-lm/pico-decoder-tinyModel · used to train
Recent changes
- 17 Oct 2026 Description: The Pretokenized Dolma Dataset A pre-tokenized, pre-shuffled version of Dolma, the high-quality text corpus from AI2. This dataset is designed to be plug-and-play with the pico-train library. Overview Key Features: Tokenized with allenai/OL → The Pretokenized Dolma Dataset A pre-tokenized, pre-shuffled version of Dolma, the high-quality text corpus from AI2. This dataset is designed to be plug-and-play with the pico-train library. Overview Key Features: Tokenized with allenai/OL
Changes detected by Silytics between refreshes of the source.
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.