Fine tune LLAMA3 on million scale dataset in consumer GPU using QLora, DeepSpeed(medium.com)
medium.com
Fine tune LLAMA3 on million scale dataset in consumer GPU using QLora, DeepSpeed
https://medium.com/@sumandas0/fine-tune-llama3-on-million-scale-dataset-in-consumer-gpu-using-qlora-deepspeed-3ae8ad75299a
3 comments
This might be an unfair statement but it really feels like all of these blogs don't know why. They copy/paste each other (you often seem the same errors in multiple notebooks/blogs) and I have a feeling no one really deeply understands what they're doing.
Found my answer for why thanks to the issues in latest dolphin fine-tune. They do these types of fine tunes mainly to reduce refusal rates and increase intelligence. They did the knee-jerk rerun of the same old data this time, as I suspected, just for lols to see where open-source is at.
Spoiler alert, fine-tunes won't be better until the data quality is better than meta's instruction fine-tune. Give it some weeks.
Why does [doplin-l3-8B] perform substantially worse in some tests?
Essentially, it's trained like this:
Spoiler alert, fine-tunes won't be better until the data quality is better than meta's instruction fine-tune. Give it some weeks.
Why does [doplin-l3-8B] perform substantially worse in some tests?
Essentially, it's trained like this:
LLama-3-8B-base_model --> LLama-3-8B-Instruct
LLama-3-8B-base_model --> dolphin-2.9-llama3-8B
LLama-3-8B-Instruct --> dolphin-2.9-llama3-8B
Most of the entire field of machine learning is “try shit and see what works”. So it seems like they’re par for the course.
Same as software engineering field too.
It's even worse for AI given that nobody really understands why anything works.
I wonder what we don’t understand from the SE POV?
One additional problem with people who write breathless tutorials about doing things with AI is that they are more likely than average to have been written with ChatGPT. Which, given the knowledge cutoff for most models, is not where I'd personally turn for data on recent technical developments, but is par for the course for the kind of low-effort copy-paste bloggers doing it for attention.
This particular one seems to be from someone who is documenting their learning process, which is a valuable contribution but, obviously, not a source of great authority on the how's and why's.
This particular one seems to be from someone who is documenting their learning process, which is a valuable contribution but, obviously, not a source of great authority on the how's and why's.
Thank you for saying this! The number of people that would need to fine tune vs just using RAG is really small. People that are not familiar with the source often jump to fine tuning as an option
I am still unsure where to stand on this fine tuning vs rag. I feel that for live data, rag would be preferable but for daily/weekly updated one, then fine tuning.
Another aspect where I am unsure is multi user for a model e.g. can we have concurrency for a model or the queries have to be queued.
Another aspect where I am unsure is multi user for a model e.g. can we have concurrency for a model or the queries have to be queued.
Fine tuning doesn’t ’add content’ the way RAG does though. They’re not really comparable in that way.
So more to be optimized for specific tasks in a domain ?
Dataset may not be public. All large companies have millions of internal documents. Internal LLM can be trained on them.
Qlora won't work well to add knowledge via private data.
Parameter efficient methods are not useful for these cases at the 8b scale without a more complex training procedure that periodically merges back adapters. Maybe at the 70B scale.
Parameter efficient methods are not useful for these cases at the 8b scale without a more complex training procedure that periodically merges back adapters. Maybe at the 70B scale.
What scale of company do you need to be to actually be able afford and get return on investment on retraining base models with your own proprietary knowledge and docs? Considering also the implications of continually retraining?
I was under the impression that you wouldn't. If you want access to proprietary knowledge, you would use RAG + LLM.
The only experience I have is first hand, what my company is doing for our client base. We are doing continuous pretraining and the rest of the alignment stack training on about 10B private tokens + private customer data to produce private custom models for companies in the 500 to 3000 employee range. We built and operate a single rack cluster that cost mid 6 figures in order to be able to do this.
These models get combined with rag for highly specific technical doc authoring and other uses.
These models get combined with rag for highly specific technical doc authoring and other uses.
This is very helpful context on what works right now, thanks for sharing.
I don't think anyone has the answer to this question yet.
Can you point to any literature on this by any chance? I would be really interested to see some in depth analysis.
I don't have the arxiv link bookmarked, but there was a paper written on pretraining with Lora. It involved merging adapters back every n steps with good results.
With unsloth's optimizations you can do llama-3-8b's QLoRA fine-tuning on your 8GB card(mine's a 2070S) with 900MB to spare with BS of 4.
I don't understand why you would use an old dataset that worked for llama2 and just fine-tune llama3 on it. Isn't it most likely that the new model has covered off everything it missed last time around and now the last dataset is only valuable for the last gen.