Benchmarking coding agents

Each task was scored from 0 to 3 on correctness and on a second measure suited to that task, such as quality of insight or statistical validity. We also counted each time a person had to step in, and treated that as a cost. An agent that needs frequent correction is slower in practice than its final output suggests.

We based the design on published work on agent planning, tool use and verification, and framed the study as a comparison of the scaffolding around each model rather than of the models themselves.

Alongside the scores we logged how each tool failed. The failures included steps that failed silently, metrics that were reported but never computed, and state that was lost between tool calls. We finished with a recommendation for each agent.