IBM and Kaggle will continuously benchmark Java migration agents with ScarfBench
IBM Research and Kaggle launched a continuously updated ScarfBench leaderboard that evaluates whether coding agents can migrate enterprise Java applications across Jakarta EE, Quarkus and Spring while preserving working behavior.
AT A GLANCE
Key points
- The benchmark covers 204 Java framework migration tasks.
- Behavioral correctness remains harder than compilation for the leading systems.
- Kaggle plans to keep the leaderboard updated with new models.
What is changing?
IBM Research and Kaggle have turned ScarfBench into a continuously updated evaluation system. The open benchmark compares model-and-agent-harness combinations across 204 enterprise Java migrations between Jakarta EE, Quarkus and Spring, testing not only compilation but also application startup and preservation of expected behavior.
In IBM’s October 27 technical note, Gemini 4 Argon with Antigravity ranked first on the primary smoke-test metric at 75.2%, while Claude Opus 5.5 with Claude Code led the all-tests-pass rate at 72.1%. These are results from IBM and Kaggle’s own evaluation. About 72% of 745 classified failures were runtime or behavioral errors that appeared after code had compiled.
Kaggle advised on the evaluation design, independently reproduced the benchmark on its infrastructure and plans to maintain the leaderboard as new models arrive. Each score measures a model-and-agent-harness pair rather than a model in isolation, and public model submission is not available at launch.
Explore the original sources
Source date: 27 October 2026.
Published by AI Radar: 27 Oct 2026, 19:13 (Istanbul, UTC+3).
FOLLOW THE TOPIC
Topics and tags
Share your discovery.
Send this story to a friend.
Reader comments
What do you think about this development? Share your experience or question. Comments are published after editorial approval.
Your name/nickname and comment become public if approved. Read: Privacy notice.
Loading comments…