LoRA has been adapting the wrong part of the weight matrix
Minor Component Adaptation targets low-variance singular subspaces rather than dominant ones, achieving up to 5.9x more knowledge acquisition than LoRA using a fraction of the parameters.
- research
- tools