procmarco/ita-en-code-tokens-48k
ita-en-code-tokens-48k Pretokenized 40% Italian / 40% English / 20% code pretraining tokens (~40B), tokenized with procmarco/ita-en-code-bpe-48k (vocab 49152). Documents packed <|bos|> … <|eos|>. Sources: FineWeb-2 ita_Latn (filtered), FineWeb-Edu sample/100BT (int_score≥3), codeparrot/github-code-clean. The three streams are interleaved to the 40/40/20 token ratio. Format tokens_000000.bin, … — fixed 268,435,456 tokens each (uint16 LE, 512 MiB). tokens_eval.bin… See the full description on the dataset page: https://huggingface.co/datasets/procmarco/ita-en-code-tokens-48k.
| Author / organization | procmarco |
|---|---|
| License (as declared) | odc-by |
| Downloads | 391 (as reported by Hugging Face) |
| Likes | 0 |
| Created | 30 June 2026 |
| Last updated at source | 30 June 2026 |
Popularity, last 2 days
391 now (-55 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.