Improved Large Language Diffusion Models
Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present \emph{iLLaDA}, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21...
| Authors | Shen Nie, Qiyang Min, Shaoxuan Xu et al. |
|---|---|
| Primary category | cs.CL |
| Published | 24 June 2026 |
| Last updated at source | 24 June 2026 |
Connected in the knowledge graph
- GSAI-ML/iLLaDA-8B-BaseModel · cited by
- GSAI-ML/iLLaDA-8B-InstructModel · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.