melephant/2l-bilinear-attn
pile Quadratic/bilinear attention causal language model trained with the tensor-mars research stack. This repository packages the final checkpoint, configuration, and reference model code. ## Training configuration ```yaml batch_size: 384 max_steps: 33333 warmup_steps: 200 lr: 0.0003 optimizer: Muon + AdamW dtype: bfloat16 grad_clip: 1.0 ``` ## Data + tokenizer - Context length: 512 | Vocab size: 4096 ## Metrics - **train_loss**: 3.9820… See the full description on the dataset page: https://huggingface.co/datasets/melephant/2l-bilinear-attn.
| Author / organization | melephant |
|---|---|
| Downloads | 136 (as reported by Hugging Face) |
| Likes | 0 |
| Created | 18 April 2026 |
| Last updated at source | 18 April 2026 |
Popularity, last 3 days
136 now (-19 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 18 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.