malaysia-ai/mosaic-dedup-text-dataset-filtered
Mosaic format for filtered dedup text dataset to train Malaysian LLM This repository is to store dataset shards using mosaic format. prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-dedup-text-dataset-filtered-4096.ipynb using tokenizer https://huggingface.co/malaysia-ai/bpe-tokenizer 4096 context length. how-to git clone, git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-dedup-text-dataset load it… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/mosaic-dedup-text-dataset-filtered.
| Author / organization | malaysia-ai |
|---|---|
| Downloads | 509 (as reported by Hugging Face) |
| Likes | 0 |
| Created | 30 November 2023 |
| Last updated at source | 1 December 2023 |
Popularity, last 5 days
509 now (+29 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.
More from malaysia-ai
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 20 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.