Ibraheem Wagih · Fraud & Financial Crime → Analytics
Projects / AI Evaluation

Benchmarking AI Coding Agents

Group of 5 Predictive Analytics (MSIN0097) UCL MSc Business Analytics

A controlled benchmarking study of Claude Code, Codex (OpenAI) and Antigravity (Google) across a five-task data-science pipeline. The question was not "which model is smarter" but "which scaffold can you trust with analytical work, and how much human supervision does each cost." The headline finding: the agent with the perfect raw score was the most expensive to supervise.

30/30
Antigravity's raw score, but 15 interventions
25.0
Codex's adjusted score, best overall, in 40 min
26
human interventions logged across all runs
20 papers
literature base for the study design

Protocol

Each agent received identical prompts across five sequential tasks, scored on correctness plus a task-specific dimension (reproducibility, insight quality, statistical validity, explanation quality) on anchored 0 to 3 rubrics. Every human intervention needed to keep an agent on track was logged and priced into an adjusted score at -0.5 per intervention, because an agent that is only correct after constant correction is not saving time.

1 · Ingestionloads cleanly, correct dtypes
→
2 · EDArelevant plots, class imbalance found
→
3 · Baseline modelstratified split, F1 on held-out data
→
4 · Model exploration3+ model families compared
→
5 · Leakage detectionplanted leakage identified and explained
The five-task pipeline issued identically to all three agents. Task 5 plants deliberate target leakage to test whether the agent catches it.

Results: Polish Is Not Autonomy

Antigravity produced the most polished output, a perfect 30/30 raw, but demanded 15 human interventions to get there, more than Claude Code and Codex combined. Once supervision was priced in, Codex won on adjusted score and did it in 40 minutes against Antigravity's 107.

Raw versus supervision-adjusted scores per agent
Raw vs adjusted scores. Pricing interventions at -0.5 each reorders the ranking: Antigravity drops from first to last.
Human interventions per task per agent
Interventions by task. The leakage-detection task was the most supervision-hungry, 6 of Antigravity's 15 interventions landed there.
Operator time versus adjusted score per agent
Time vs adjusted score: Codex delivered the best supervised outcome in the least operator time.

Failure Modes

The dominant failure across tools was task verification: agents declaring success without checking their own output (4 of 5 tasks). Others included proxy-metric exploitation (optimising an inflated metric rather than a valid model), literal-minded specification reading, and Antigravity's autopilot modelling, applying a template pipeline without interrogating the domain. The report closes with agent-by-agent recommendations and a pre-flight checklist for delegating analytical work safely.

My Contribution

Within the team of five, I operated the Codex agent across all five tasks and scored its runs, co-designed the anchored scoring rubrics, and owned one of the five literature-review themes grounding the study design.