Integration Bench grades coding agents on the state they leave in a vendor system, not on their code

Integration Bench is our benchmark, public in preliminary form since August 20, 2026, that grades coding agents on the state an integration leaves in one of 14 synthetic recruiting-software vendor systems, not on the code that produced it. We publish it as the RL environments and benchmarks division of HeyMilo. Fifty tasks are public with a test harness (the scripts that run and score each task) anyone can rerun, and private evaluation is available on request.

Update, September 16, 2026. The benchmark now has its own site. Its leaderboard (revision 01, marked preliminary, data dated September 4, 2026) lists 20 models on the 50 public tasks; the top task score is 69.35 out of 100. Links in this post point at the new site.

What we grade

Most coding benchmarks look at the code. They check whether the patch applies and whether the unit tests go green, which is a fair proxy for a self-contained bug fix and a poor one for integration work. When you connect two systems, what matters is whether the data arrived, whether the webhook fired, and what state the other system was left in after something went wrong.

So we grade the system instead of the code. Every task runs against one of 14 synthetic vendor systems with a real API surface (StaffLine, TalentForge, Placemint and others) that model recruiting software: candidates, jobs, applications, placements. Each runs in a container, and the grader reads that vendor’s state and request log rather than the code that produced them. An agent has to find out what the live state is, because reading the documentation and pattern-matching against it will not get it through.

The four kinds of task

The fifty tasks fall into four groups:

• Build. The starter code does not work yet. The agent writes the connector against the vendor’s documentation.
• Fix. Find and repair planted defects in an integration without breaking what already worked.
• Harden. Keep a working connector correct under rate limits, injected faults, and forged webhooks.
• Migrate. Move an integration from one vendor to another, matching records across two different schemas without losing or duplicating any.

Every task is scored against written criteria for correctness, behavior under injected faults and rate limits, and data integrity, then normalized to 100.

Early results: models stall on harden and migrate tasks

Early runs have a consistent shape. Models scaffold quickly and confidently, then stall at the point where being correct means reconciling what the system contains against what the ticket asked for. Scores fall on the harden and migrate tasks, which is also where most of the real work sits in production. The current numbers are on the leaderboard. rlsupply, which publishes this benchmark, also sells RL environments and private evaluation, which is why the public harness exists: anyone can rerun our numbers.

Public tasks and private tasks

How is Integration Bench scored?

On the state the vendor system is left in after the agent’s integration runs: records present, webhooks fired, data intact under rate limits, injected faults, and forged webhooks. Criteria cover correctness, resilience, and data integrity, normalized to 100. Code quality is not scored.

Can my lab get held-out tasks?

Yes. The 50 public tasks exist so anyone can reproduce our results and build on them. A private (held-out) set is never published and never enters training data, so it scores a model nobody has tuned toward it. Request holdout access, sample packets, or the harness wired into your own evaluation stack.