Yes I agree, but I actually did a lot more runs, with different prompts, different times ect... And each time /goal had a small or insignificant impact
It seems that it is very strong at Frontend, I wonder how good are its multimodal capabilities. This is very important for me and not well represented in benchmarks in my opinion.
The US Department of Commerce told Anthropic it had lifted the ban on foreigners accessing the company’s Mythos and Fable models on Tuesday evening, according to people with knowledge of the matter. The decision allows the AI group to re-release its latest model, Fable 5, to the general public.
TL;DR: GLM-5 tops coding benchmarks. I tested it on an unpublished NP-hard optimization problem (KIRO) and 89-task Terminal-Bench. Best case: competitive. Typical case: 30% invalid output, every trial timed out, and two identical runs could produce a valid solution or complete garbage. Zhipu AI reports 56% on Terminal-Bench; I got 40%.
I gave an unpublished fiber network optimization problem to Claude Code, Codex, Gemini CLI, and Mistral. The score is total fiber length (lower is better). A good human solution in 30 minutes: ~40,000. My best after days of C++: 34,123. Given one hour, Claude Code hit 34,061 — beating me by 62 points. A 7-word prompt hint improved every agent by 18-30%. About 15% of all trials produced completely invalid outputs.
At Jimmy, we're developing France's first Small Modular Reactor. Our engineers needed to quickly search and extract insights from thousands of complex scientific PDFs—nuclear research papers, regulatory documents, multilingual content filled with equations and diagrams.
Manual search wasn't cutting it. So we built a RAG system to give our team instant access to critical technical knowledge.
What worked:
- AWS Titan V2 crushed it (69.2% hit rate vs. 57.7% for Qwen, 39.1% for Mistral)
- Chunk size? Barely mattered (2K to 40K—no significant difference)
- Qdrant: Easy to use, solid performance, great for self-hosting
- Mistral OCR: Unmatched, the only tool that parsed our equations correctly
- Naive chunking beat context-aware (70.5% vs 63.8%)
- Dense-only search outperformed hybrid search (69.2% vs 63.5%)
Hard lessons:
- OpenSearch from AWS is ridiculously expensive for no reason and presented as the default option by AWS
- Mistral Embed works well in English but not in French