Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes

We study whether a decoder-only language model requires an independently trainable input vector for every token. For a vocabulary of size $V$, an injective fixed-length binary identifier requires $K=\lceil\log_2 V\rceil$ bits. We replace the usual trainable $V\times d_{\mathrm{model}}$ input table with fixed minimal binary token codes and a parameter-free tiled lift to model width. With $V=65{,}536$ and $d_{\mathrm{model}}=1024$, this supplies each token as a fixed 16-bit code and removes 67.1M trainable parameters, approximately 12.5% of the untied learned-input baseline. We also study a table-free implementation with one fixed invertible affine recoding over $\mathbb{F}_2^{16}$. Across ...

<iframe src="https://silytics.si/embed/item/66f4da65-8048-4255-89a2-9bc9c1121e40" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.