Fast but False Progress on Benchmarks with Richer Feedback

1. Best-model selection

Does the best-ranked model hold up on unseen data?

We measure model selection loss: the held-out score of the model chosen before the attack minus that of the model chosen afterward. Positive values mean following the new ranking made selection worse.

Below, we observe that following the public ranking lowers held-out performance on all six benchmarks at 128 leaderboard submissions. In fact, in all cases, 16 submissions suffice to approach the 1 pp loss mark. The largest losses are on LiveBench and HELM Lite.

Each submission budget produces its own final attack model, so its held-out performance can improve as well as deteriorate. The dip for Terminal-Bench-Science reflects such an improvement; selection loss stays positive because the new choice still performs worse, on average, than the original public winner.

2. Best-score generalization

Does the best score found through repeated submissions hold up on unseen data?

Now after each leaderboard submission, we compare the best score so far on the reused set with the best score so far on the held-out set. The largest absolute gap over this history is the score generalization gap. The two best scores may belong to different models. The baseline uses only the original benchmark models; it can be positive because their scores already differ between the two splits.

Below, we observe that the gap grows early on LiveBench and SWE-bench Verified, while HELM Lite and Open LLM continue to diverge with more submissions. Recall that these curves tell us how well the best reported score holds up; Figure 1 shows what happens to the model we choose.

Each budget ends with its own final attack model, so the final model from a shorter run is absent from the longer run’s history. The maximum is taken within each of these histories, so the plotted gap can decrease across budgets even though a running maximum cannot decrease within one run.

Metric, uncertainty and complete results

Let denote the reused set and the held-out set. Write for the smallest loss on set among the first submissions, where loss is one minus the score on a 0–1 scale. With original models and new submissions,

The history includes the original models and all new submissions, including the final one. Scores follow each benchmark’s evaluator; HELM Lite uses equal-category means and SWE-bench Verified weights instances equally. We compare attack and baseline errors on the same trials and resample those pairs to form confidence bounds. Statistical details.

The complete score-gap curves include all budgets and intervals.

3. How the attack works

To keep things simple and to avoid running expensive model inferences, each attack leaderboard submission is a router: a rule that sends each prompt to one of two existing models. The attack uses public task scores from repeated submissions to choose which model handles each prompt. We illustrate the steps with one selected HELM Lite trial using 32 leaderboard submissions. The attack is mostly an adaptation of the boosting attack, which we discuss at the end of the section.

A. Choosing the two routing endpoints

The two models should be strong, but different enough that routing between them can help. We first remove models for which another model does at least as well on every task and better on at least one. Among the five highest-ranked models left, we choose the pair with the largest average absolute difference between their task scores.

In this example trial, that gives us A: Gemma 2 27B IT and B: DeepSeek V3. Before any new submissions, GPT-4o (August 2024) leads the public ranking.

B. Turning task scores into weights

Each submission is simply an independent random routing rule to choose model A or B for each prompt. Only a hash of the input/prompt is required to make the choice repeatable, so identical prompts always go to the same model. These rules are fixed before any feedback arrives. Each submission returns ten category scores in this example; two are shown below.

For a given task , let be the two models’ public losses, where loss is one minus the score on a 0–1 scale. Their average is the reference loss for random routing. If candidate returns public loss , we give it the following weight:

Weight = reference loss − candidate loss

(1)

A candidate that beats the reference gets a positive weight; a candidate that does worse gets a negative weight. The farther its loss is from the reference, the more its choices count. In the example, candidate 1’s LegalBench feedback gives weight -0.01426.

C. Combining the routing choices

For prompt , write for candidate ’s choice: +1 selects and −1 selects . Add these choices with their task-specific weights:

Final choice = sign of the weighted sum

(2)

The final router selects when and when . Thus, a positive weight supports a candidate’s choice; a negative weight reverses it. Each task score weakly reflects which choices worked on that task. Combining many such scores strengthens the signal on the reused examples. For the reused LegalBench prompt with the smallest public hash, the weighted vote is +0.02201, so the final router chooses A.

The final router takes first place publicly but ranks fifth on the held-out split. Following it instead of GPT-4o lowers held-out performance from 74.10% to 70.14%, a 3.95 pp model-selection loss. This example illustrates the mechanism only; the earlier curves include every trial.

DeepSeek V3 has a slightly higher public mean score than GPT-4o in this figure, yet GPT-4o leads the original ranking. HELM Lite ranks models by their average win rate against other models within categories, which can order them differently from these mean scores.

Our work builds on the elegant line of research on adaptive data analysis, initiated by Dwork et al. (2015), discussed further in our previous paper. The attack draws on Blum and Hardt’s boosting construction (2015), Hardt’s generalized wacky boosting (2015), and the feedback-weighted aggregation of Feldman, Frostig, and Hardt (2019). Here, an important adaptation is that we select two strong models with contrasting strengths by maximizing task-profile separation among the top five models on the public Pareto frontier. We then combine random routers using public task scores, with the specific aim of studying how feedback richness accelerates overfitting and affects model selection across modern benchmark snapshots.

4. Controlling feedback richness

How much does richer feedback make the attack more efficient?

We want to understand how much the number of evaluation criteria accelerates the inflations we found. Consider LiveBench as an example where richer feedback brings selection harm fast. At 8 leaderboard submissions, mean selection loss is 1.43 pp with 1 returned score and 4.29 pp with 18.

We group the 18 LiveBench tasks into a varying number of super-categories, and return one average score per group, keeping the model pairs, random routing rules, and evaluation fixed. Each outcome is plotted against submissions and the total number of scores released.

Below to the right, indexing by the total number of released scores brings the curves quite close. This suggests that the amount of feedback---simply defined as the total number of unit measurements---helps explain how quickly the attack succeeds. By 128 submissions, mean loss is 4.59–4.85 pp across all six interfaces. Coarser feedback delays harm here, at the cost of hiding task differences that may matter for model selection.

5. Frequently asked questions

6. Takeaways

1. Stress-test your benchmark’s feedback interface. Run an adaptive attack using your benchmark’s scoring rules, released feedback, and realistic submission budgets. Keeping examples private does not prevent overfitting through repeated aggregate scores.

   To run our attack on your benchmark, connect its public scores and submission feedback through the adapter, or read the working example.

2. Develop scaling laws for benchmark lifespan. A useful scaling law would relate test-set size, feedback richness, and differences among models to the number of submissions before score inflation or selection loss exceeds a chosen tolerance. Our initial score-gap predictor is a starting point, though it does not currently transfer reliably across benchmarks. Better predictions could help maintainers decide when to refresh their data or revise their feedback policy.

3. It remains open whether ordinary development encounters this vulnerability. Our attacks leave open how often this happens in practice. Answering this may require tracing which comparisons guide later models and checking whether the resulting gains persist on fresh samples from the same population. If reuse remains reliable, are there properties of the model families or their use of feedback that explain why in rich-feedback setting?

   If reuse does not remain reliable in ordinary development, an open problem is to design feedback mechanisms that preserve utiltiy while controlling spurious progress at realistic data sizes and computational costs. Our previous paper's Frontier Ladder offers a starting point.

7. Citations

@misc{allouah_duchi_koyejo_benchmark_progress,
  author = {Allouah, Youssef and Duchi, John and Koyejo, Sanmi},
  title = {Fast but False Progress on Benchmarks with Richer Feedback},
  year = {2026},
  howpublished = {Research artifact and software, version 0.1.0},
  url = {https://false-benchmark-progress.com}
}

@misc{allouah_duchi_benchmark_reuse,
  author = {Allouah, Youssef and Duchi, John},
  title = {How Reusable Are Benchmarks with Richer Feedback?},
  year = {2026},
  howpublished = {arXiv preprint arXiv:2609.32109},
  eprint = {2609.32109},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.32109}
}