AndrewK101/cs336-owt-32k-bpe
CS336 OWT 32k BPE tokenized Pre-tokenized OpenWebText sample (Stanford CS336 assignment 1 source) using a custom 32k byte-level BPE tokenizer. Files owt_vocab_32k.pkl — vocab dict[int, bytes] (32000 entries; ID 0 = <|endoftext|>, 1..256 = raw bytes, 257..31999 = merges) owt_merges_32k.pkl — list[tuple[bytes, bytes]] in merge order owt_train.npy — uint16 token IDs of 11 GB owt_train.txt (2,729,574,028 tokens) owt_valid.npy — uint16 token IDs of 277 MB… See the full description on the dataset page: https://huggingface.co/datasets/AndrewK101/cs336-owt-32k-bpe.
| Author / organization | AndrewK101 |
|---|---|
| License (as declared) | mit |
| Downloads | 324 (as reported by Hugging Face) |
| Likes | 1 |
| Created | 27 April 2026 |
| Last updated at source | 27 April 2026 |
Popularity, last 4 days
324 now (+10 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.
Recent changes
- 16 Oct 2026 Description: CS336 OWT 32k BPE tokenized Pre-tokenized OpenWebText sample (Stanford CS336 assignment 1 source) using a custom 32k byte-level BPE tokenizer. Files owt_vocab_32k.pkl — vocab dict[int, bytes] (32000 entries; ID 0 = <|endoftext|>, 1..256 = r → CS336 OWT 32k BPE tokenized Pre-tokenized OpenWebText sample (Stanford CS336 assignment 1 source) using a custom 32k byte-level BPE tokenizer. Files owt_vocab_32k.pkl — vocab dict[int, bytes] (32000 entries; ID 0 = <|endoftext|>, 1..256 = r
Changes detected by Silytics between refreshes of the source.
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 19 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.