Ai2 NoveltyBench stress-tests LLM judgments of research novelty

What is changing?

Ai2 released the NoveltyBench benchmark, dataset and code on October 27 alongside its “Old Ideas, Novel Problems” study. The system compares LLM “novelty judges” that decide whether a scientific idea is genuinely new relative to prior work, testing six models across different prompts, reasoning budgets and access to literature.

The benchmark separates 154 accepted ICLR 2026 ideas whose novelty reviewers explicitly praised from 145 rejected ideas whose originality reviewers questioned. A second setting pairs the same 154 validated human ideas with 154 ideas from a basic LLM generator; the authors explicitly describe the generated-idea labels as weak labels rather than verified judgments for every case.

The results reveal instability. GPT-5.4 fell from 93.5% to 40.9% accuracy on the same human-versus-generated idea pairs after only a prompt wording change, reversing more than half of its decisions. More search and reasoning did not consistently improve performance, leading Ai2 to argue that these judges should not act as standalone novelty gates for automated research ideation.