malaysia-ai/mosaic-dedup-text-dataset
Mosaic format for dedup text dataset to train Malaysian LLM This repository is to store dataset shards using mosaic format. prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-dedup-text-dataset-4096.ipynb using tokenizer https://huggingface.co/malaysia-ai/bpe-tokenizer 4096 context length. how-to git clone, git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-dedup-text-dataset load it, from streaming… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/mosaic-dedup-text-dataset.
| Author / organization | malaysia-ai |
|---|---|
| Downloads | 431 (as reported by Hugging Face) |
| Likes | 0 |
| Created | 27 November 2023 |
| Last updated at source | 1 December 2023 |
Popularity, last 3 days
431 now (+2 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.
Recent changes
- 17 Oct 2026 Description: Mosaic format for dedup text dataset to train Malaysian LLM This repository is to store dataset shards using mosaic format. prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-dedup-text-dataset-4096 → Mosaic format for dedup text dataset to train Malaysian LLM This repository is to store dataset shards using mosaic format. prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-dedup-text-dataset-4096
Changes detected by Silytics between refreshes of the source.
More from malaysia-ai
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 18 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.