jingyaogong/minimind_dataset
📌 数据介绍 Ⅰ Tokenizer 分词器可以粗略理解成 LLM 使用的一本“词典”，负责把自然语言映射成 token id，再把 token id 解码回文本；项目中也提供了train_tokenizer.py作为词表训练示例。不建议重新训练 tokenizer，因为词表和切分规则一旦变化，模型权重、数据格式、推理接口与社区生态的兼容性都会下降，也会削弱模型的传播性。同时，tokenizer 还会影响 PPL 这类按 token 统计的指标，因此跨 tokenizer 比较时，BPB（Bits Per Byte）往往更有参考价值，可参考这篇。 对 MiniMind 这类小模型来说，词表大小还会直接影响 embedding 层和输出层的参数占比，因此保持词表精简通常是更合适的取舍。 Tokenizer介绍 第三方强大的开源模型例如 Yi、Qwen2、ChatGLM、Mistral、Llama 3 的 tokenizer 词表长度如下：… See the full description on the dataset page: https://huggingface.co/datasets/jingyaogong/minimind_dataset.
| Author / organization | jingyaogong |
|---|---|
| License (as declared) | apache-2.0 |
| Downloads | 3.9K (as reported by Hugging Face) |
| Likes | 120 |
| Created | 9 September 2024 |
| Last updated at source | 9 April 2026 |
Popularity, last 3 days
3,902 now (+6 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.
Connected in the knowledge graph
- zhoumiaosen/minimind-64m-sftModel · used to train
- jingyaogong/minimind-3-ggufModel · used to train
- SnifferCaptain/YModel2-s0Model · used to train
- SnifferCaptain/YModel2-s-2Model · used to train
- jingyaogong/minimind-3Model · used to train
- PP12546/MiniPolaris-64M-DenseModel · used to train
- zhoumiaosen/minimind-64m-pretrainModel · used to train
More from jingyaogong
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 18 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.