Stanford benchmarks and compares numerous Large Language Models(crfm.stanford.edu)
crfm.stanford.edu
Stanford benchmarks and compares numerous Large Language Models
https://crfm.stanford.edu/helm/latest/?group=core_scenarios
5 comments
One of OpenAI's engineers / cofounders (Greg Brockman) specifically said it in an interview that GPT-4 itself is quite calibrated, it goes away with the fine tuning process (I don't remember exactly which part of the interview). Of course they are working on fixing it, I guess GPT-4.5 will contain the fix.
https://www.youtube.com/watch?v=YtJEfTTD_Y4
https://www.youtube.com/watch?v=YtJEfTTD_Y4
That's great, now I guess we need a page that ranks those benchmarks to tell us what we're looking at...
It seems to be missing Llama and other Llama derived models, and GPT4.
> last updated 2023-03-19
Yep, anything that old is ancient in the AI world
Yep, anything that old is ancient in the AI world
Singularity when? Plot twist:..
- Not all these models are instruct aligned
- Cohere's suite is free for personal use
- anthropic uses reinforcement learning from ai feedback for instruct alignment, removing humans from the process almost entirely(humans still make the rules the language model checks the responses against).
- open ai's models are pretty poorly calibrated compared to its peers. Calibration is how well the model's confidence solving a problem tracks with how it actually performs that problem. It seems to be a consequence of the way they are performing the instruct tuning (or maybe it's a consequence of instruct tuning in general). The gpt-4 report actually shows this. Base gpt-4 was excellently calibrated but rlhf knocked it right out. The excellent calibration was also independently verified (in the medical domain) from the paper about gpt-4's medical prowess. https://arxiv.org/abs/2303.13375