Settings tripled GPT-5.6's ARC-AGI-3 score
OpenAI found GPT-5.6 Sol's low ARC-AGI-3 score was a harness artifact: retained reasoning and compaction took it from 13.3% to 38.3% on the public set.
GPT-5.6 Sol scored just 7.8% on the official ARC-AGI-3 leaderboard, a benchmark of 2D puzzle games designed to measure how well agents learn and reason without instructions, even though the same model has solved open problems in mathematics and beaten games like Pokémon FireRed. OpenAI investigated and found the gap wasn’t capability, it was harness configuration: turning on two API settings already used in ChatGPT and Codex, retained reasoning and compaction, tripled the model’s score on the ARC-AGI-3 public set, from 13.3% with the official generic harness to 38.3%, while cutting output tokens by 6x. For context, OpenAI estimates the average human tester scores 48% on the same set using Relative Human Action Efficiency scoring. ARC-AGI-3 uses a deliberately generic harness precisely so model shortcomings are more visible and comparisons stay fair; OpenAI’s finding shows that fairness cuts both ways, since a harness with no memory of prior actions also hides real capability.
What it means for operators
Public leaderboard scores measure the model plus whatever harness the benchmark enforces, not the model in isolation. If you’re choosing between models based on a leaderboard number, check whether the harness that produced it resembles how you’ll actually deploy the model, retained context, compaction, and tool access all matter, or you may be comparing apples that were deliberately picked without stems.