Dataset · Hugging Face

emozilla/pg_books-tokenized-bos-eos-chunked-65536

Name: pg_books-tokenized-bos-eos-chunked-65536
Creator: emozilla
Keywords: size_categories:10K<n<100K, format:parquet, library:datasets, library:dask, library:mlcroissant, library:polars, region:us

Dataset Card for "pg_books-tokenized-bos-eos-chunked-65536" The pg19 dataset tokenized under LLaMA into 64k chunks, bookended with BOS and EOS

View on Hugging FaceReport an issue

Author / organization	emozilla
Downloads	942 (as reported by Hugging Face)
Likes	7
Created	31 August 2023
Last updated at source	7 October 2023

Embed this entry

<iframe src="https://silytics.si/embed/item/05cc3754-26d7-4336-9354-630596be4f08" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

Popularity, last 5 days

942 now (-66 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.

More from emozilla

emozilla/pg19Dataset
emozilla/pg19-testDataset
emozilla/qualityDataset
emozilla/yarn-train-tokenized-16k-mistralDataset
emozilla/Long-Data-Collections-Pretrain-Without-BooksDataset
emozilla/dolma-v1_7-305B-tokenized-llama3-nanosetDataset

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 20 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.