my unvalidated theory is that this comes down to the coding model’s training objective: Tetris is fundamentally an optimization problem with delayed rewards. Some models seem to aggressively over-optimize toward near term wins (clearing lines quickly), which looks good early but leads to brittle states and catastrophic failures later. Others appear to learn more stable heuristics like board smoothness, height control, long-term survivability even if that sacrifices short-term score
That difference in objective bias shows up very clearly in Tetris, but is much harder to notice in typical coding benchmarks. Just a theory though based on reviewing game results and logs
answered this in a comment above! It's not turn or visual layout based since LLMs are not trained that way. The representation is a JSON structure, but LLMs plug in algorithms and keeps optimizing it as the game state evolves
Thanks for all the questions! More details on how this works:
- Each model starts with an initial optimization function for evaluating Tetris moves.
- As the game progresses, the model sees the current board state and updates its algorithm—adapting its strategy based on how the game is evolving.
- The model continuously refines its optimizer. It decides when it needs to re-evaluate and when it should implement the next optimization function
- The model generates updated code, executes it to score all placements, and picks the best move.
- The reason I reframed this problem to a coding problem is Tetris is an optimization game in nature. At first I did try asking LLMs where to place each piece at every turn but models are just terrible at visual reasoning. What LLMs great at though is coding.
Before I began the test, I thought the agents would be much better at this task than most humans -- after all they should have better, more stateful memory than us. The results are intriguing.
Here are the scores from 10 attempts:
OpenAI operator: 5, 5, 6, 5, 5, 4, 6, 5, 5, 5
Anthropic computer use agent: 7, 9, 6 (rate limited), 12, 9, 7, 9, 11, 12, 6 (rate limited)
hey! thanks so much for the feedback. I'd actually love to keep updating / iterating these cartoons so they are more approachable. If you have time, I'd love to hear more on which pages are confusing & how I could have explained it better!
I _tried_ to give a definition to embeddings on page 11, but maybe that's not the most intuitive? Lmk! feel free to DM
Hi HN! Recently we made a starter kit for better learnings on how to get started building AI apps with multi modal models. It's been fun to build, but we also discovered there are many things we needed to take care of: caching, long video processing pipelines, model evaluation, etc
> each step is a code-level transaction backed by its own job in the queue. If the step fails, it retries automatically. Any data returned from the step is automatically captured into the function run's state and injected on each step call.
This is one thing I've seen so many companies spending tons of time implementing themselves, and happens _everywhere_ -- no code apps, finance software, hospitals, anything that deals with ordering system...the list goes on.