Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context
Diffusion language models offer a promising alternative to autoregressive models due to their potential for parallel and iterative generation. However, existing approaches use a single network for both context representation and iterative denoising, forcing one model to serve both roles and limiting its capacity for either role. We propose TwoTower, a block-wise autoregressive diffusion model that decouples these roles into two towers: a frozen AR context tower that causally processes clean tokens, and a trainable diffusion denoiser tower with bidirectional block attention that refines noisy blocks via cross-attention to the context. Built on Nemotron-3-Nano-30B-A3B, an open-weight 30B hy...
| Authors | Fitsum Reda, John Kamalu, Roger Waleffe et al. |
|---|---|
| Primary category | cs.CL |
| Published | 25 June 2026 |
| Last updated at source | 29 June 2026 |
Connected in the knowledge graph
- nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16Model · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.