I tried this myself using GPT-2J and nearly 15 years of IRC logs trimmed down to only contexts that include me. Each training item would be my message and the preceding 20 messages. Still ended up being an 80 MB log file of over 1M lines.
The results were...not good at all. I ran an IRC bot that would watch the channel, and any time someone sent a message, it would send the last 20 messages to the trained model to see what it would predict the next message to be. It generated mostly garbage. Generated chat messages were on-topic, but non-sensical. It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related.
At some point, I'll probably try again with a far better model. Likely once I do my next system overhaul so I can do it locally. The only thing I'm not sure of is how to properly train and generate. GPT3 and its predecessors are very much simple text-completion AIs, whereas it seems all the models today are designed for a conversation back-and-forth between the AI and a user.
My guess is you'd better use your message + a bunch of other messages with RLHF, yours being the positive one and the others negative. If you don't have other messages, just generate some. You can also do some supervised fine tuning to have the model memorize your facts.
When I read your comment I trained my own mental model on your words. How is that any different? When a human reads words they apply a sophisticated theory of mind to contextualize the writing and the mental state of the author. If anything, LLM fine tuning is far less invasive than having a person read your writing.
The idea that reading a piece of text constitutes copyright infringement is ridiculous. Copyright isn’t some infectious thing. Reading copyrighted text doesn’t give the copyright holder a claim to the future creative work of the reader.
You want to restrict model training, I get it. The debate is still ongoing, but I’m confident when these “copyright” claims work their way through court the AI companies will come out on top.
> The idea that reading a piece of text constitutes copyright infringement is ridiculous.
No man, it's not ridiculous. If I write a program that copies someone's book and try to sell it I'm infringing on that copyright. I cannot sell a zipped version of the Harry Potter books. I feel like there's so many people weighing in on this discussion who haven't actually done any real world copyright related stuff.
You're going to have to let a lot of scientists know that, because they're still publishing papers with that understanding. I guess they should have consulted you first.
> It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related.
An accurate depiction of what's going on!
I had terrible results with GPT2 but the newer models are much better. I recommend fine tuning mistral (I had better results than any llama based model)
I've got an RTX 3080, i9-9900K, and 32 GB of RAM right now. I eventually plan on upgrading to an RTX 5090 when those come out, as well as i9-15900K (Or whatever they're gonna call it) and 128 GB of RAM once the next Intel CPU comes out.
My understanding is that even with an RTX 5090 (Assuming 32 GB of VRAM, nothing has been announced yet), if I use fp32, I still won't have enough GPU memory to do inference on a 7B model, let alone training. Though if I quantize down to fp16, I could do inference, but still not training.
My naive intuition is that you'd want to train on the highest precision you have the horsepower for, and then after training, you might be able to get away with quantizing to a lower precision for inference.
I would also expect that fp32 might be overkill, but that anything less than fp16 would start to show a noticeable loss in quality, especially in training. I can't imagine int8 being good at all.
And I would think training with int5 would produce garbage. I didn't even know int5 was an option.
Do you have any published benchmarks comparing the quality of results between the different precisions, going all the way down to int5?
You are broadly better off running a larger, more quantized model, than a smaller model at a higher precision. A 65b parameter model with 2bit quantization still outperforms an fp16 30b parameter model in perplexity and qualitative testing. No one I know does fp32, there's no benefit.
People train LoRAs at low precision regularly. The folks on /r/localllama are really pushing the boundaries of what can be done on consumer hardware, and the boundaries are further out than you'd think.
Also, if you want to really blow-up your intuition, check out model merges...
The results were...not good at all. I ran an IRC bot that would watch the channel, and any time someone sent a message, it would send the last 20 messages to the trained model to see what it would predict the next message to be. It generated mostly garbage. Generated chat messages were on-topic, but non-sensical. It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related.
At some point, I'll probably try again with a far better model. Likely once I do my next system overhaul so I can do it locally. The only thing I'm not sure of is how to properly train and generate. GPT3 and its predecessors are very much simple text-completion AIs, whereas it seems all the models today are designed for a conversation back-and-forth between the AI and a user.