Hey folks, I recently went down a rabbit hole with fine-tuning qlora models and wanted to share some findings and hopefully save someone else the headache. A lot of the tutorials and documentation for qlora suggest starting with a learning rate of 2e-4. But for those of us working with smaller datasets (in the 5k-10k range), this seems like a trap.
To paint the picture, it seems that 2e-4 is a recommendation with roots in the Alpaca dataset's 52k samples. However, when you're operating with a significantly smaller dataset, this learning rate can lead to overfitting as fast as within the first epoch. Over several attempts, I noticed my evaluation loss plateau or even rise while training loss kept decreasing—a classic sign of overfitting.
After spending three weeks puzzling over this, I re-evaluated my approach. I trimmed down my dataset to around 7,200 samples after a stringent review. Still no dice. Finally, I experimented with lowering the learning rate to 1e-4 and increased the epochs to 5. The results were night and day—the eval scores jumped significantly!
It was frustrating because the usual unsloth and Hugging Face docs casually suggest 2e-4 as a "starting point", but most shared notebooks don't comment on it, leading to a lot of copy-paste configurations. My advice: if you're working with datasets under 10k, start with 1e-4 or even lower and allow more epochs. For datasets above 30k, the initial recommendation might hold. For anything in between, spend the time tweaking—it’s worthwhile.
If there’s any substantial research that backs the default 2e-4 for small datasets, I'd genuinely love to see it. Also, if I’m late to the game and this was common knowledge, give me a heads-up next time!
Looking forward to hearing about your experiences and any tips you might have.
This is super helpful! I’m about to start working on a project with roughly 6k samples and was wondering about the same thing. When you changed to a 1e-4 learning rate and increased the epochs, did you notice any change in the convergence speed? I'm curious if this adjustment makes the overall training time significantly longer. Also, did you test using any kind of regularization techniques, like dropout, to further counteract the overfitting?
I totally agree with your observations! I faced the same issues when I first started with qlora. With a dataset around 8k samples, the default 2e-4 learning rate was definitely a trap for me too. I ended up reducing it to 5e-5, and it worked much better in terms of preventing overfitting. The tweak really improved my model's performance in a similar way to what you described. Thanks for sharing, it's great to know I'm not the only one finding my way through this maze!
