Hugging Face releases CWE-bench v2 across 91 vulnerability classes

What is changing?

Hugging Face released CWE-bench v2 on November 3 to evaluate how well coding agents repair security vulnerabilities. The public evaluation set contains 100 tasks drawn from real open-source repositories and spans 91 CWE weakness classes. For each task, an agent must inspect the repository and produce a patch that closes the weakness without breaking existing functionality.

Evaluation runs in isolated Harbor tasks with deterministic tests. Each system receives four attempts per task, producing 400 trials across the 100-task set. The official table reports 67.8% pass@1 for Gemini 4 Argon, 67.3% for Claude Opus 5.5 and 63.8% for GPT-6 Astra. Even the leading system failed to solve 19% of the tasks in any of its four attempts.

The public set, grader and execution harness have been released, while a held-out evaluation set remains. The results show that security-patching agents should be assessed across success rate, cost, reproducibility, regressions and human review rather than a single headline score.