Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
Low-rank adaptation (LoRA) enables parameter-efficient specialization of foundation models, but the proliferation of task-specific adapters fragments capabilities across many adapters, complicating reuse and deployment. We study the problem of merging $T$ LoRAs into a single rank-$r$ LoRA, thereby preserving the benefits of low-rank structure. Existing Merge-then-Compress pipelines treat the rank constraint as an afterthought: they merge adapters in the full parameter space, then compress the merged result to rank $r$ via truncated SVD. However, full-parameter merging may destroy the low-rank structure, making it difficult for subsequent compression to recover an effective rank-$r$ LoRA. ...
| Authors | Zhengbao He, Ruiqi Ding, Zhehao Huang et al. |
|---|---|
| Primary category | cs.LG |
| Published | 2 June 2026 |
| Last updated at source | 2 June 2026 |
Popularity, last 25 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Connected in the knowledge graph
- akhaliq/viggle-remake-turboModel · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 11 November 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.