ColBERT-Zero: To Pre-train Or Not To Pre-train ColBERT models
Current state-of-the-art multi-vector models are obtained through a small Knowledge Distillation (KD) training step on top of strong single-vector models, leveraging the large-scale pre-training of these models. In this paper, we study the pre-training of multi-vector models and show that large-scale multi-vector pre-training yields much stronger multi-vector models. Notably, a fully ColBERT-pre-trained model, ColBERT-Zero, trained only on public data, outperforms GTE-ModernColBERT as well as its base model, GTE-ModernBERT, which leverages closed and much stronger data, setting new state-of-the-art for model this size. We also find that, although performing only a small KD step is not eno...
| Authors | Antoine Chaffin, Luca Arnaboldi, Amélie Chatelain et al. |
|---|---|
| Primary category | cs.CL |
| Published | 18 February 2026 |
| Last updated at source | 18 February 2026 |
Popularity, last 4 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Connected in the knowledge graph
- lightonai/ColBERT-ZeroModel · cited by
- lightonai/ColBERT-Zero-nopromptsModel · cited by
- lightonai/ColBERT-Zero-unsupervisedModel · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 21 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.