Could an open source/data GPT like chatbot be created?

12 pointsby sharemywin4 lata temu3 comments

The resulting InstructGPT models are much better at following instructions than GPT-3. They also make up facts less often, and show small decreases in toxic output generation. Our labelers prefer outputs from our 1.3B InstructGPT model over outputs from a 175B GPT-3 model, despite having more than 100x fewer parameters. At the same time, we show that we don’t have to compromise on GPT-3’s capabilities, as measured by our model’s performance on academic NLP evaluations.

https://openai.com/blog/instruction-following/

It seems like the a 1.3B parameter model wouldn't require massive resources.

I understand training data could be hard to come by but is it in the realm of possiblity

2 comments

cottonseed4 lata temu

Yes. EleutherAI is doing it, probably one of many:

https://www.eleuther.ai/projects/gpt-neox/ https://github.com/EleutherAI/gpt-neox https://arxiv.org/abs/2204.06745

They have a 20B parameter model. I think the primary dataset for these open models is The Pile: https://arxiv.org/abs/2101.00027 (web scrape, pubmed, arxiv, github, wikipedia, etc. There is a nice diagram on page 2 that summarizes the contents.)

sharemywin4 lata temu

from what I gather the pile is only a first step. it would require more steps. task oriented chats. as well as building something that can rate answers.

seydor4 lata temu

Question is when. In any case i would like to be able to run them locally and save all the chats we make to feed them to future versions. Over the years this would become a copy of my brain. And when i die we can keep on chatting