RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation
Published in International Conference on Machine Learning (ICML) 2024, 2024
Recommended citation: M. Nikdan†, S. Tabesh†, E. Crnčević, D. Alistarh. (2024). "RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation." International Conference on Machine Learning (ICML) 2024. https://arxiv.org/abs/2401.04679
RoSA blends low-rank adapters with a sparse “outlier” mask inspired by robust PCA, jointly optimised to approximate full-fine-tuning quality at a fraction of the parameter cost. Across math reasoning and SQL generation tasks, RoSA consistently beats LoRA and pure-sparse baselines at equal budgets and even recovers full-fine-tuned accuracy on several benchmarks; bespoke sparse GPU kernels keep memory footprints minimal.