BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: they lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate the vast ecosystem of specialized generative models. In this work, through systematic ablations on the Gemma3 and Qwen3 families, we identify the key factors driving successful adaptation, highlighting the critical role of an often-omitted prior masking phase. To scale this process without original pre-training data, we introduce a dual strategy combining linear weight merging with a lig...
| Authors | Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef et al. |
|---|---|
| Primary category | cs.CL |
| Published | 2 April 2026 |
| Last updated at source | 2 April 2026 |
Models and datasets referencing this paper
- BidirLM/natcapDataset
- BidirLM/colpali_train_retrievalDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.