And on creativity at least visually, Gemini 3.1 pro is somehow still up there. But its really hindered by its inability to use tool calls effectively or make a long term plan.
My personal benchmark for new models has been to compare video making skills with something like remotion. Usually reveals if they have any "taste" or outside the box thinking.
I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.
And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one.
Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.
Hadn't tested out any of the image models for training loras since Flux 1 dev, so I was curious which one is the current best. Results were pretty interesting
I'm on max $200, I had this idea for around a month since Fable got banned. For the usage, not exactly sure since I had like 4 agent heavy tasks running at the same time.
I think personally you can do first half of this with the direction with fable and then render + checking using sol since thats what I ended up doing. Guesstimate I'd say it takes around 10-20% of the 50% fable usage
Had some fable usage to waste yesterday cause my reset was gonna happen. Ended up burning all my usage so had to finish the last part with Sol itself.
Imo Remotion based videos feel better to watch than just actual GenAI videos, like Seedance.
the exact prompt for reference:
"I wanna make a plan for making good fireship videos automatically. so my thinking is, you need to first need to find a good way to insert memes like gifs, short videos, images stuff like that. another part of a good fireship video is the voice itself, so you need to find a way to voice clone fireship's audio(probably download a sample video and use our existing higgs audio api, check airoleplay and search "higgs" for the api endpoint details and the api keys and stuff if you need). another part of it is the animations and creative text animations and extra touches that he puts, maybe for this you could analyze a couple of his vids in general using gemini 3.1 pro with openrouter in detail and you can ask multiple questions abut a video to explain in detail exactly what happens.
now remember you're the orchestrator and this is likely going to be a very long task so you need to be using your context very carefully, and assign various research tasks to opus or fable subagents depending on the complexity.
the video itself should be about claude fable 5 and make another one for gpt 5.6 sol. some more reference for how we recreated antoher vidoe with a single prompt before: '/Users/test/Documents/openmotion/recreate-video-user-prompts.md'"
As long as they have the best model they can afford to lose goodwill.
People who don't wanna spend too much on LLMs and are trying to optimize whats subsidized even on the Max plans are customers Anthropic is honestly better off without.
Seems really good at frontend work, and as a result on remotion programmatic videos. Not the best yet, thats still Gemini 3.1 pro(trained on actual videos) or Fable, but often better than what Opus can come up with
Overall an improvement over Opus 4.8, but I'd still say Gemini 3.1 Pro has more of an artistic vision even tho it fails tool calls and writes buggy code sometimes.
Ik almost everyone is interested just in the SWE stuff, but this has been a good eval for me to think about how big the model is, how "creative" it is for generating new ideas etc.
Yep sorry was just pulling it out my rear, not like a market trend that nearly every enterprise uses Anthropic or Openai models for coding or that Anthropic has had such ridiculous growth that they're 10x-ing year over year
My point was that even openrouter, the one place people who are looking for open source SOTA models go to, doesn't definitively have opensource models at the top. Esp considering quite a lot of the closed models usage is through AWS, GCP , Azure etc, probably dwarfing the usage on openrouter by a huge factor
As long as closed source is 6 months ahead in terms of current difference. Although this is hard to figure out using simple percent based coding benchmarks, you def. notice it when you're actually trying to do a long task. Even simple things like UI "taste" is enough for me to use opus instead of 5.5 though even though 5.5 is strictly better for anything that doesn't have a UI, ie backend, scripts, making agent workflows etc
As long as closed models are 6 months ahead I won't be switching from them to prev. 6 month SOTA open source models. Maybe its just a different calculation if you're in a job, but as an indiehacker I'll take any edge I can get
Ofc again, can be convinced to switch if there's however a clear speed difference, like 5x+ for a open source sota even if it was SOTA for 6 months ago
Based on current market for LLMs I'd say my use of "you" in the general is fine. Even openrouter which doesn't capture all of the SOTA closed models but nearly all of opensource model usage has Opus as 1st(on last week) on "Programming" category and 3rd in overall rankings
Cost for the value delivered. Like if you offered the current SOTA open source models at $0.1/M, I still think I'd be using Opus or 5.5 at $30/M. Or say GPT 5 which was released Aug 25, I don't think I'd use it for coding for even $0.1. I'd def find other uses for it(translations, agentic workflows, prompt guards etc), but for coding I don't think I'd ever completely switch to a SOTA open model
Unless ofc there was an actual speed difference, only reason I'd be willing to go with a worse model couple of percent worse than current best model is if the speed was at least 5x higher. Looking forward to kimi k2.6 offered publicly by Cerebras
For coding you always want to go with the best model in the category, not something that would be the best model if we went 1 year back which GLM 5.1 is, and I'm saying that as a big fan of GLM cause I run a translation site where GLM is good enough for the price.
Most of the money right now is in coding. Openai and Anthropic just have to be 6 months ahead of SOTA open source models and they'll capture most of the enterprise and dev market