Here's the post with all the examples and analysis: anokas.substack.com/p/o3-and-arc-a… You might have seen this task today on x dot com as a failure! I actually think o3's answer is as valid as the ground truth hereIn several cases, we see o3 struggle to output an aligned grid at all. It seems that problems which require outputting repeated rows can make o3 struggle to keep track.And on a couple occasions - the model appears to give up on the 2nd attempt, in this case outputting a single black pixel? It's unclear if OpenAI feeds in the previous attempt and tells the model it was wrong, or if something else led to this behaviour.The full post goes through all these examples, and I'd love to hear your thoughts and theories:- For a deeper analysis of why o3 did so much better than previous models, and the caveats there might be in that evaluation, check out this thread!Why do pre-o3 LLMs struggle with generalization tasks like @arcprize? It's not what you might think. OpenAI o3 shattered the ARC-AGI benchmark. But the hardest puzzles didn’t stump it because of reasoning, and this has implications for the benchmark as a whole. Analysis below🧵
Excellent analysis, thanks. If anything this just makes me more impressed with o3 - in many cases where it failed-it was close, or we could imagine how it could be improved quite easily
