As EvalEval has grown, so has the range of people, institutions, and projects that make up the Coalition. We began as an initiative incubated by EleutherAI, Hugging Face, and the University of Edinburgh, and have since developed into a broader community (800 strong at the time of this announcement) working across AI evaluation research, infrastructure, events, and engagement with policy and standards. Our new bylaws are intended to make the Coalition’s governance reflect that evolution.

A central change is that we are moving away from the concept of institutional “hosts.” Our founding organizations will continue to be recognized for their role in building EvalEval, but they will no longer hold governance authority by virtue of that status. People participate in EvalEval in their individual capacities, and participation does not commit or imply endorsement from their employers. Leadership and decision-making will instead sit with a Steering Committee and Chairs whose responsibilities are defined in the bylaws, alongside explicit conflict-of-interest and recusal requirements.

We are also creating Project Leads as a new and important pathway for leadership, open to newly joined members. Any member can develop and propose new work, and approved projects will have a named Project Lead with primary responsibility for carrying the work forward. This is meant to make it easier for researchers across institutions, career stages, and geographies to originate and lead work within EvalEval, rather than requiring every initiative to flow through a small set of organizational leaders. All positions are open to all members of the Coalition and are determined democratically, as set out in the bylaws.

These changes reaffirm what we want EvalEval to be: an open, multi-stakeholder, independent research coalition, where people with different institutional affiliations can collaborate while the Coalition’s agenda and positions remain distinct from those of any one employer or partner. The new structure distributes responsibility among the Steering Committee, Chairs, and Project Leads, while requiring broader agreement for decisions that affect the Coalition’s strategic direction, independence, or public positions.

We are proud of how far EvalEval has come, and we see these bylaws as infrastructure for the next stage rather than an endpoint. There is much more we want to build, and many more people we hope will build it with us.

Meet our Steering Committee and Chairs on the About page. To get involved, email join@evalevalai.com.

@misc{evaleval2026governance,
  author       = {{EvalEval Coalition}},
  title        = {{EvalEval} Adopts a New Governance Structure},
  year         = {2026},
  month        = oct,
  howpublished = {EvalEval Coalition Blog},
  url          = {https://evalevalai.com/organization/2026/10/05/new-governance-structure/}
}

A push for independent assessment

On September 18, California Governor Gavin Newsom issued an executive order accelerating the state’s work on independent AI oversight. Among other things, the order asks state agencies to develop recommendations for embedding independent verification organizations within frontier AI companies to conduct regular audits and evaluations, verify safety frameworks and risk assessments, and maintain independent verification of emergency shutdown mechanisms. California has also recently created frameworks for independent verification organizations (SB 813) and a registry for AI auditors (AB 1405).

This builds on a broader shift toward technical evaluation and assurance around the world. The UK’s AI Security Institute (AISI) has been conducting independent evaluations of advanced AI systems since 2023, including pre- and post-deployment assessments of potentially harmful capabilities, and Parliament’s Joint Committee on Human Rights (JCHR) recently called for new statutory safeguards against the human rights risks posed by AI in its report on Human Rights and the Regulation of AI. In the EU, the AI Act requires providers of general-purpose AI models that pose systemic risk to conduct and document state-of-the-art model evaluations, including adversarial testing, while the GPAI Code of Practice further develops expectations for pre-deployment evaluation, post-market monitoring, and external evaluations. The European AI Office is also developing independence and qualification requirements for external evaluators of systemic-risk GPAI models. Similar efforts elsewhere, such as Singapore’s AI Verify, are pushing toward more structured approaches to testing, documentation, and independent scrutiny.

The evidence question

Across these approaches, a practical implementation question keeps coming up: once an evaluation is conducted, how should its evidence be recorded and communicated?

Policymakers need to know what was tested and how. Evaluators and auditors need enough methodological detail, including completed metadata, to interpret or reproduce results. Developers increasingly face requests for similar evidence in different formats. Shared reporting conventions can streamline that evidence across research, assurance, and governance processes.

Shared reporting can also reduce the burden on oversight bodies. When providers report evaluation evidence in different formats and at varying levels of detail, regulators have to spend additional time locating, interpreting, and comparing the information they need. Standardized, structured reporting can make that process easier and also enable more reliable automated parsing and analysis of evaluation evidence.

Such a common baseline can also support other governance processes. In California, for example, the state will need to evaluate and compare prospective Independent Verification Organizations as it develops its designation regime. SB 813 requires applicants to provide information about the benchmarks, technologies, metrics, and methodologies they propose using, and directs the state to develop the regime with attention to consistency, comparability, and avoiding duplicative requirements where practicable. A common format for describing evaluation evidence could make that information easier to assess across applicants and over time.

Mapping requirements to Evaluation Cards

This is where our work at EvalEval fits into a broader open evaluation ecosystem. Through Every Eval Ever and the wider Evaluation Cards effort, we are working on shared, open infrastructure for documenting evaluation results in ways that make them easier to find, compare, analyze, reproduce, and reuse.

We are currently working with public-sector evaluators to put this infrastructure into practice. In our collaboration with the UK AI Security Institute, AISI is making publicly reported evaluation methods and findings available through Evaluation Cards where appropriate, including verified results, context, and configuration information. Feedback from AISI has also helped shape the Every Eval Ever (EEE) schema itself. This gives us a concrete example of how open reporting infrastructure can support independent evaluation: results produced by an evaluator can be published in a shared structure and compared with evidence from the wider ecosystem.

To understand how far this infrastructure could support emerging governance requirements, we reviewed 40 requirements across the California (AB 1405, SB 813), EU (AI Act, GPAI Code of Practice), and UK (JCHR report) instruments in our crosswalk and broke each requirement into the discrete pieces of evidence needed to satisfy it. This produced 281 evidence elements in total.

We then classified each element according to its unit of record. We treated an element as evaluation-reporting evidence when it primarily describes an evaluation run, its methodology, configuration, benchmark, results, or the conditions under which it was conducted. We classify this as information that can reasonably be attached to an evaluation record. Evidence about an evaluator’s institutional independence or qualifications, a model as a whole, an incident, organizational governance, or a regulatory process was instead assigned to the corresponding evaluator-, model-, incident-, organization-, or regulator-level record. On this basis, 60 of the 281 elements fall within the scope of evaluation reporting, while the remaining 221 belong in other kinds of records.

For each of those 60 elements, we then asked whether EEE or AutoBenchmarkCards contains a field whose unit and meaning match the required evidence, and whether that information can be represented in a structured field, in free text only, or not at all. Evaluation Cards unifies evaluator-reported information from EEE with benchmark metadata automatically supplied by AutoBenchmarkCards, as shown in the visualization.¹

What the mapping found

Of the 60 evidence elements in scope for evaluation reporting, EEE can already represent 56. Fifty-one are captured through dedicated, typed fields; another five can currently be recorded through free-text fields supplied by the evaluator or other reporter.

For example, requirements in the EU’s General-Purpose AI (GPAI) Code of Practice to document an evaluation’s tooling and resource constraints map onto existing EEE fields for tools available during an evaluation and token limits, while requirements to record whether an evaluation was conducted internally or by an external evaluator map to source_metadata.evaluator_relationship.

Evaluation Cards can combine those submissions with benchmark-level information automatically pulled from AutoBenchmarkCards. In our crosswalk, AutoBenchmarkCards turns two of the five elements that EEE captures only in free text (benchmark data type and benchmark languages) into structured metadata. This increases the number represented through structured fields from 51 to 53.

Where other standards fit

Our research on evaluation reporting standards is one part of the broader push towards independent assessment. Questions about evaluator independence, conflicts of interest, access arrangements, credentials, and regulator-facing processes belong in standards focused on evaluators themselves. AEF-1, for example, addresses operating conditions for independent third-party evaluators. Model-level information has a natural home in model cards, while incident and flaw reporting is being developed through efforts such as FLARE-AI.

These efforts can work as complementary layers: evaluator standards establish expectations for who performs an assessment and under what conditions; model cards document the system being assessed; evaluation reporting standards record what was tested, how, and with what results; and incident-reporting standards capture failures that emerge after deployment. Making those layers interoperable can allow the same underlying evidence to travel more easily between evaluators, developers, researchers, and oversight bodies without asking each actor to reinvent its documentation from scratch.

Explore the map

The interactive visualization below presents our current mapping. It shows which pieces of governance evidence an Evaluation Card can address, with each field color-coded by where it comes from: reported fields that evaluators fill in and submit through Every Eval Ever, and auto-pulled benchmark metadata that Evaluation Cards brings in from AutoBenchmarkCards. What a reporter controls is what they submit through EEE. Open it in a new tab for the most room, and hover over any band for the source text and field definitions.

Evaluation Cards are designed to be living documents that grow with multi-stakeholder input over time. We therefore believe that the relationship between governance requirements and evaluation infrastructure should be bidirectional: new policy requirements can help identify where Evaluation Cards need to become more expressive, while existing schemas can give policymakers a concrete, machine-readable way to specify and compare the evaluation evidence they are asking for. Our mapping shows that the same reporting infrastructure can already support a substantial share of the evaluation-specific evidence appearing across California, the EU, and the UK, while making clear where complementary standards or new fields are still needed. We hope regulators, evaluators, developers, and researchers will use the crosswalk as a practical reference and share feedback from real-world use so that Evaluation Cards can continue to evolve alongside emerging governance requirements.

@misc{ghosh2026evalcardsgovernance,
  author       = {Ghosh, Avijit and Manheim, David and Tran, Andrew and Nelaturu, Sree Harsha and Ji, Jessica and Batzner, Jan and Kennedy, Wm. Matthew and Gohar, Usman and Reuel, Anka and Solaiman, Irene},
  title        = {Mapping {Evaluation Cards} to Emerging {AI} Governance Requirements in {California}, the {EU}, and the {UK}},
  year         = {2026},
  month        = sep,
  howpublished = {EvalEval Coalition Blog},
  url          = {https://evalevalai.com/infrastructure/2026/09/28/mapping-evaluation-cards-to-ai-governance-requirements/}
}

The EvalEval Coalition is thrilled to share that the UK AI Security Institute (AISI) is using EvalEval’s infrastructure to openly share evaluation results, supporting more reproducible and verifiable evaluation science.

AISI and EvalEval have previously collaborated on research that began at a joint workshop alongside NeurIPS 2025, and feedback from the Institute has helped shape the Every Eval Ever (EEE) schema. This next phase of the collaboration puts that shared infrastructure into practice.

Why reproducible evaluation reporting matters

As AI deployment accelerates, evaluations are becoming increasingly important sources of evidence about model and system performance. Yet results are reported across many formats, platforms, and outlets, often without enough information to reproduce them. Running the evaluations again may itself be prohibitively expensive.

EvalEval’s mission is to improve this ecosystem through a shared reporting schema, Every Eval Ever, and an open platform, Evaluation Cards, that brings evaluation results and the information needed to interpret them into a common structure.

This builds naturally on AISI’s work to make evaluation more efficient through OptStop, more statistically rigorous through HiBayES, and more standardised in areas including transcript analysis and capability elicitation. Together, AISI and EvalEval are working to diagnose gaps in evaluation reporting and build shared infrastructure to close them.

What AISI is sharing

Transcript-level transparency matters not only for reproducibility, but also for analysis and diagnosis. In this new phase of the collaboration, AISI is making publicly reported evaluation methods and findings available through Evaluation Cards where appropriate. The release includes verified results, context, and configuration information for the five benchmarks in the paper’s main experiment:

• HealthBench
• FrontierMath
• Humanity’s Last Exam
• SWE-Bench Pro
• Terminal-Bench 2.0

These results cover six frontier models: Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2, and GPT-5.4. The release also includes results from two related cyber evaluations—Cyber CTFs and The Last Ones—which use a different, partially overlapping set of models. The data accompany AISI’s paper, How Inference Compute Shapes Frontier LLM Evaluation, which studies how benchmark performance depends on inference-time compute and evaluation protocol.

When results are openly released with setup information, researchers and practitioners can examine individual studies more closely and compare findings across the wider ecosystem. Where other reports lack these details, releases like AISI’s provide verified reference points for interpreting evaluations in context—for example, by helping researchers understand how setup choices may influence reported performance. As more evaluators adopt EEE, open comparisons like these can support broader and more reliable meta-research.

We are excited about this adoption and look forward to further standardising and sharing evaluations with AISI and other AI evaluation organisations.

Contribute to the shared mission

• Model developers: Report verified evaluation results.
• Evaluation developers: Report benchmarks and run data using the Every Eval Ever schema.
• Evaluation, governance, and policy researchers: Explore Evaluation Cards by benchmark or model, or use it to examine the state of evaluation reporting as a whole.

About the EvalEval Coalition

The EvalEval Coalition is a research community developing scientifically grounded research and robust deployment infrastructure for the evaluation ecosystem. Its goal is to improve evaluation science, address the lack of consensus around documenting evaluation applicability and utility, and broaden coverage of the impacts that matter for scientific research and policy analysis.

The coalition’s flagship projects include Every Eval Ever, a shared schema and repository for evaluation results, and Evaluation Cards, which combines benchmark metadata, evaluation-run data, and model metadata into interpretable records. Together, they make it easier to understand when apparently similar scores were produced under meaningfully different conditions.

About the UK AI Security Institute

The UK AI Security Institute is a research organisation within the UK government’s Department for Science, Innovation and Technology. Its mission is to equip governments with a scientific understanding of the risks posed by advanced AI. AISI conducts research and builds infrastructure to understand advanced AI capabilities and impacts, develop and test mitigations, and inform policy.

Further reading

• How Inference Compute Shapes Frontier LLM Evaluation
• HiBayES: Improving LLM evaluation with hierarchical Bayesian modelling
• HiBayES paper
• OptStop paper
• Every Eval Ever
• Evaluation Cards

@misc{ghosh2026aisireproducible,
  author       = {Ghosh, Avijit and Chim, Jenny and Joshi, Deep and Yadav, Srishti and Kennedy, Wm. Matthew and Solaiman, Irene and McFadyen, Jessica and Tan, Lynn and Ududec, Cozmin},
  title        = {How {UK} {AISI} and {EvalEval} Are Making Benchmark Results Reproducible},
  year         = {2026},
  month        = sep,
  howpublished = {EvalEval Coalition Blog},
  url          = {https://evalevalai.com/infrastructure/2026/09/22/uk-aisi-evaleval-reproducible-benchmark-results/}
}

Founders Pledge (Global Catastrophic Risks Fund) has awarded the coalition funding to support development of Eval Cards. We are grateful to Sammy Martin and the Founders Pledge GCR team for their support of this work.

The Survival and Flourishing Fund (SFF) has also awarded the coalition support for our ongoing infrastructure and research work, and we are grateful to SFF for backing shared, open evaluation resources for the field.

Additionally, the Weizenbaum Project “Evaluating GenAI Evaluations,” funded by the German Federal Ministry of Research, Technology, and Space (BMFTR) and the State of Berlin, supports our Shared Task on Evaluating GenAI Evaluations, and we are grateful for this support of the coalition’s meta-evaluation research.

These contributions directly enable the coalition’s ongoing work on Every Eval Ever, Eval Cards, the Shared Task, and our collaborations with stakeholders including CAISI, UK AISI, and others working to bring greater coherence to AI evaluation.

For more about the EvalEval Coalition and how to join our efforts and Slack community, visit evalevalai.com.

@misc{lin2026grants,
  author       = {Lin, Michelle},
  title        = {Announcing Our New {EvalEval} Grant Support!},
  year         = {2026},
  month        = jul,
  howpublished = {EvalEval Coalition Blog},
  url          = {https://evalevalai.com/organization/2026/07/01/grant-announcement/}
}

AI progress is often measured with benchmarks like MMLU, HumanEval, and ARC-AGI. But what happens when the best models score almost the same?

In our ICML paper “When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation”, we study this growing problem: many popular benchmarks are losing their ability to separate between leading models.

Saturation, When Benchmarks Stop Measuring Progress

Benchmarks are useful when they can tell strong models apart from weaker ones. However, over time, top models start clustering together on benchmarks’ leaderboards. A model might score 91%, another 92%, and another 92.3%. The differences are so small that they may just reflect evaluation noise rather than real capability gains. This is what we call benchmark saturation.

Saturation is not the same as getting a perfect score. If the benchmark can no longer reliably measure differences between leading models, it stops being a useful measurement - even if plenty of room for improvement remains.

Measuring Saturation

To understand how widespread this problem is, we analyzed 60 benchmarks covering reasoning, coding, knowledge, multilingual tasks, factuality, and agentic systems.

We also developed an index to measure saturation. Instead of looking only at leaderboard scores, we asked: Can this benchmark still reliably tell top models apart? If score differences are smaller than the benchmark’s uncertainty, then those differences may not be meaningful.

The result: Nearly half of the benchmarks we studied already show high levels of saturation, meaning they have limited ability to distinguish between today’s leading models.

Common Assumptions Don’t Hold

We also tested several popular assumptions about what makes benchmarks last longer:

• “Keep the test set private.”
• “Use open-ended generation instead of multiple-choice questions.”
• “Evaluate models in many languages.”

Surprisingly, none of these factors showed a strong relationship with benchmark saturation once benchmark age was taken into account.

What seemed to matter more? If benchmarks are measurement tools, they need maintenance. Based on our findings, benchmark creators should consider:

• Larger and more diverse evaluation sets
• Dynamic benchmark updates
• Adversarial data collection
• Uncertainty-aware reporting
• Explicit criteria for benchmark revision or retirement

Saturation Isn’t Always Bad

If models genuinely master a capability, a saturated benchmark may be evidence of real progress - BUT the challenge is figuring out whether a benchmark is saturated because:

• Models have solved the task, or
• The benchmark has stopped measuring meaningful differences.

As models continue to improve, telling these two apart will become increasingly important.

Explore our Work

• Paper: https://arxiv.org/pdf/2602.16763
• Code & dataset: https://github.com/evaleval/benchmark-saturation

@misc{akhtar2026saturationblog,
  author       = {Akhtar, Mubashara and Sania, Jeba and D'Arcy, Jocelyn and Choshen, Leshem},
  title        = {When {AI} Benchmarks Stop Measuring Progress},
  year         = {2026},
  month        = jun,
  howpublished = {EvalEval Coalition Blog},
  url          = {https://evalevalai.com/2026/06/30/saturation-blog/}
}

@misc{akhtar2026aibenchmarksplateausystematic,
  author        = {Mubashara Akhtar and Anka Reuel and Prajna Soni and Sanchit Ahuja and Pawan Sasanka Ammanamanchi and Ruchit Rawal and Vilém Zouhar and Srishti Yadav and Chenxi Whitehouse and Dayeon Ki and Jennifer Mickel and Leshem Choshen and Marek Šuppa and Jan Batzner and Jenny Chim and Jeba Sania and Yanan Long and Hossein A. Rahmani and Christina Knight and Yiyang Nan and Jyoutir Raj and Yu Fan and Shubham Singh and Subramanyam Sahoo and Eliya Habba and Usman Gohar and Siddhesh Pawar and Robert Scholz and Arjun Subramonian and Jingwei Ni and Mykel Kochenderfer and Sanmi Koyejo and Mrinmaya Sachan and Stella Biderman and Zeerak Talat and Avijit Ghosh and Irene Solaiman},
  title         = {When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation},
  year          = {2026},
  eprint        = {2602.16763},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2602.16763}
}

• An interface to explore the largest and most comprehensive corpus yet assembled, with 101,955 reported evaluation results for 638 benchmarks run by 31 organizations on 5,816 models (as of 9 June 2026) based on our EvalEval sister project, Every Eval Ever (EEE)and IBM AutoBenchmark Cards.
• A front end that shows four novel signals about the state of evaluations as a whole: reproducibility, completeness, provenance, and comparability.
• An open platform that developers, evaluators, researchers, and anyone else in the evaluations community can use to centralize evaluation reporting.

EvalCards is in beta. Fixing evaluation reporting is a community effort that will benefit everyone but, as large as it is, the EvalEval Coalition won’t be able to do it alone. Help us by contributing and spreading the word!

Addressing fragmentation in the evaluation documentation landscape

AI evaluations are crucial for assessing and improving AI model performance across an increasingly wide variety of domains. Consequently, evaluations communities produce results at tremendous scale. However, these results are reported in different places and in different ways. Some use leaderboards, others model cards, still others blog posts, conference papers, posters, or articles. Although this flexibility can benefit certain stakeholders in different situations, it contributes to an increasingly serious problem: because so few evaluations are reported with enough information to properly interpret their findings, there is no way to compare results across sources, identify what a report leaves out, or trace a major claim back to the evidence it rests on.

Until today.

The EvalEval Coalition is excited to introduce the beta version of Evaluation Cards, a live interpretive layer over the public reporting ecosystem. For the first time, we have a tool that can reveal the state of reporting across the whole evaluation ecosystem, surfacing glaring gaps that have, until recently, remained undetected.

For example, three different organizations report GPT-5’s scores in MATH-500, but those range from 84.7% to 98.9% depending on configuration (which is not fully reported).

Evaluation Cards not only surfaces these discrepancies (which are accessible through existing reporting artifacts, albeit inefficiently), but also provides more granular detail about why these discrepancies exist (which is rarely obvious without detailed examination of each evaluation’s configuration, including specifics such as temperature or max tokens). This is Evaluation Cards’ main purpose: to provide a tool to answer deeper questions about the meaning of evaluation results.

A card for every evaluation

Evaluation Cards builds on years of hard work by others concerned with reporting practices and improving the state of evaluations. As a reporting artifact, Evaluation Cards is inspired by Datasheets¹, Model Cards², Benchmark Cards³, Audit Cards⁴, System Cards⁵, and BetterBench⁶, each of which specify what should be documented.

However, prior efforts largely stopped at prescription, leaving implementation for another day and important gaps to be filled. Each of those efforts covers only a narrow slice of the evaluation lifecycle, most artifacts are produced for one specific kind of audience, and most reporting proposals remain just that–proposals.

The Every Eval Ever (EEE) Schema moved the field forward by actually implementing a schema for structured evaluation reporting.

Evaluation Cards goes the rest of the way: composing the raw data that the ecosystem produces at scale into one canonical evaluation record:

Run data: Who ran the eval? On what model? With what generation settings? The source of this data comes from EEE, the EvalEval Coalition’s results reporting scheme and community repository for standardised eval run data.

Benchmark metadata: the benchmark’s measurement target, scoring, provenance, and known risks. This comes from AutoBenchmarkCards, which automatically and reliably extract information about a measurement instrument from conference papers and system cards.

Model metadata: release date, parameter count, weight accessibility, all automatically pulled from community catalogs (hub-stats, models.dev) covering both open and API-hosted models.

Furthermore, Evaluation Cards is open, built by and for the wider evaluations community. Contributors span institutions, sectors, continents, and specializations.

Evaluation Cards needs you!

Evaluation Cards surfaces extremely valuable information that shouldn’t be abstracted away. We’re looking for model developers to help foster transparency and ecological validity by contributing their data. We’re also looking for evaluators to share their data, setup, and config information. This way we can ensure the community is comparing evaluation data apples to apples.

Organizations that contribute their own evaluation data earn a verification badge. Verified contributors have submitted over 40% of Evaluation Cards’ data, and, through upcoming partnerships and engagements from policy and research organizations in the evaluation ecosystem, our goal is to grow this number quickly. Learn how your organization can become verified.

Four new interpretive signals

Having good model and evaluation data allows Evaluation Cards to do something no other project has done. Across these records (making legible 100,000+ results at launch), Evaluation Cards computes four interpretive signals that reveal valuable insights into the state and quality of evaluations. Signals respond to desiderata shared by evaluations stakeholders in the course of our own user interview program:

Reproducibility: Can someone else run this evaluation and get the same results?

Completeness: Is benchmark documentation sufficient to interpret scores?

Provenance: Who reported the score? And who has reproduced it?

Comparability: What is the measurement target and how faithfully do different setups evaluate against it?

Flags divergences in setups or reporting parties that likely reflect scores resulting from similar configurations rather than capability measurement targets, ensuring comparability of reported results.

These signals come from unflattening conventional evaluation reporting, which usually treats results as a flat triple comprising a model, a benchmark name, and a score. But evals are more than numbers, they are interpretive claims. Flattening discards structure that exists in almost every benchmark. For example, MATH has subject splits that vary widely in difficulty level. SWE-bench reports across various languages and setups. Open LLM Leaderboard aggregates several sub-benchmarks into a single figure.

A first glimpse of the entire evaluations ecosystem

The Evaluation Cards project has allowed us to see the state of evaluations as a whole. Aggregating the four signals across the evaluations reporting ecosystem provides invaluable insight into key questions: are models systematically achieving certain results on evaluations? And are those results meaningful?

Early observations across the corpus (5 June 2026 snapshot) reveal startling initial findings:

Evaluation reproducibility appears shockingly poor

Across more than 50,000 evaluation records, 96.5% are missing at least one field from the minimal set needed to re-run them. Max-tokens is absent 95.6% of the time, temperature 93.9% of the time. The gap appears to be worst amongst first-party reporters. Frontier models are no exception: 95% of GPT-5’s 213 documented results lack the parameters to reproduce them. Although sometimes these shortcomings are produced by data extraction failures, it points to a serious and pervasive problem.

Benchmark documentation substantially hampers the maturity of evaluations

Median completeness of benchmarks is a paltry 10.7% across all 635 benchmarks captured here. Scores are present, but documentation of construct definitions, validation, intended use, limitations, and provenance are missing far more often than not.

Scores reported by two or more parties diverge substantially

Nearly all (98.2%) of model-benchmark pairs are reported only by one party. Of the remainder, different parties report scores that vary substantially more than half the time (51.9%). Among safety benchmarks, third-parties make up most reporters—this could be because there are more third-party evaluators. Evaluation Cards surfaces these kinds of questions for further inquiry.

The field is doing good work. It is just doing that work in isolation. As a result, the divination of scores has been allowed to supplant the systematic production of scientific knowledge. Evaluation Cards reveals the extent of this potential loss. It also serves as a call to improve our practices of evaluation reporting. Otherwise the burden of interpretation will continue to fall on those least well positioned to accurately interpret evaluation results but often most responsible for making important technical governance or policy decisions.

Evaluation Cards is for everyone!

The evaluations community includes a diversity of stakeholders who are interested in evaluation results for many reasons. That’s why we’ve launched supporting different “reader modes” (Research and Summary) to support different user needs. Both modes read identical data. They differ only in how they display it. Users can easily toggle between modes without losing their place.

We are actively exploring additional stakeholder reader modes. Our roadmap is public, and ideas are welcome: https://changemap.co/evaleval/evalcards/

Research mode

Research reader mode was designed with AI evals researchers, model developers, and technical AI governance personae in mind. This mode foregrounds methodology and places at top information about missing fields, divergence because of setup differences, and full metric configuration.

Summary mode

Summary mode was designed for personae like nontechnical AI policy researchers, qualitative researchers, journalists and reporters, and readers who prefer narrative reporting. This mode presents more information in plain language, with benchmark-specific elaborations about the measurement target, central caveats, and intended uses.

Try it out using the toggle at top left:

Adopt, use, and contribute to Evaluation Cards

Evaluation Cards is built as a participatory, openly-governed initiative. We mean for it to be shaped by the AI evaluation community over time, not shipped once and abandoned. All of the code is open.

Model developers: Report your evaluations on Evaluation Cards so others can better understand your model’s performance. Review third-party reported results for discrepancies. Earn verification badges for uploading your own evaluation results!

Evaluation developers: Upload your evaluations so others can find them, run them, and use them to improve model performance or evaluations design.Review other evaluators’ results. Earn verification badges for uploading your own evaluation results

Research and policy community: Consult Evaluation Cards as a resource in your model-, evaluation-, or field-level analysis and cite us in papers, reports, or policy briefs.

Check out our contributor guides with more details here.

All stakeholders:

• Spread the word! Evaluation Cards promotes ecological validity in direct proportion to the number of those who know about it, use it, and contribute to it.
• Explore the corpus: Browse by model or Browse by evaluation
• Help us build Evaluation Cards (also, we’re hiring)
• Flag missing data or incorrect scores
• Become a verified contributor
• Give us your feedback

What’s next:

• See the Evaluation Cards roadmap
• Tutorials for integrations and use cases
• Research and contributor hackathons
• An annual EvalEvalIndex

Catch up with the EvalEval Coalition:

• Come to our FAccT ‘26 Tutorial
• Join our ACL ‘26 workshop
• Join the EvalEval community
• See our recent activity

@misc{kennedy2026evalcardsblog,
  author       = {Kennedy, Wm. Matthew and Reuel, Anka and Chim, Jenny and Tran, Andrew and Long, Yanan and Yadav, Srishti and Manghnani, Kabir and Choshen, Leshem and Manheim, David and Mickel, Jennifer and Ji, Jessica and Solaiman, Irene and Ghosh, Avijit},
  title        = {Introducing {Evaluation Cards}: A Live Interpretive Layer for Understanding the {AI} Evaluations Ecosystem},
  year         = {2026},
  month        = jun,
  howpublished = {EvalEval Coalition Blog},
  url          = {https://evalevalai.com/infrastructure/2026/06/09/evaluation-cards-launch/}
}

@misc{ghosh2026evaluationcardsinterpretivelayer,
  author        = {Avijit Ghosh and Anka Reuel and Jenny Chim and Wm. Matthew Kennedy and Srishti Yadav and Jennifer Mickel and Yanan Long and Andrew Tran and Anastassia Kornilova and Damian Stachura and Kevin Klyman and Felix Friedrich and Jeba Sania and Max Lamparth and Jan Batzner and Anoop Mishra and Eliya Habba and Yixiong Hao and Nathan Heath and Shalaleh Rismani and Usman Gohar and Andrea Loehr and David Manheim and Ruchira Dhar and Sree Harsha Nelaturu and Aarush Sinha and Leshem Choshen and Drishti Sharma and Ishan Khire and Amit Saha and Subramanyam Sahoo and Michael Hardy and Michael Alexander Riegler and Kabir Manghnani and Michelle Lin and Yanan Jiang and Yilin Huang and Asaf Yehudai and Jessica Ji and Aris Hofmann and Mubashara Akhtar and Nuno Moniz and Yacine Jernite and Stella Biderman and Zeerak Talat and Sanmi Koyejo and Mykel Kochenderfer and Irene Solaiman},
  title         = {Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting},
  year          = {2026},
  eprint        = {2606.09809},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2606.09809}
}

Summary. AI evaluation has crossed a cost threshold that changes who can do it. The Holistic Agent Leaderboard (HAL) recently spent about $40,000 to run 21,730 agent rollouts across 9 models and 9 benchmarks. A single GAIA run on a frontier model can cost $2,829 before caching. Exgentic's $22,000 sweep across agent configurations found a 33× cost spread on identical tasks, isolating scaffold choice as a first-order cost driver, and UK-AISI recently scaled agentic steps into the millions to study inference-time compute. In scientific ML, The Well costs about 960 H100-hours to evaluate one new architecture and 3,840 H100-hours for a full four-baseline sweep. While compression techniques have been proposed for static benchmarks, new agent benchmarks are noisy, scaffold-sensitive, and only partly compressible. Training-in-the-loop benchmarks are expensive by construction, and when you try to add reliability to these evals, repeated runs further multiply the cost.

Making static LLM benchmarks cheaper

The cost problem started before agents. When Stanford's CRFM released HELM in 2022, the paper's own per-model accounting showed API costs ranging from $85 for OpenAI's code-cushman-001 to $10,926 for AI21's J1-Jumbo (178B), and 540 to 4,200 GPU-hours for the open models, with BLOOM (176B) and OPT (175B) at the top end. Perlitz et al. (2023) restate the larger HELM cost pattern, and IBM Research notes that putting Granite-13B through HELM "can consume as many as 1,000 GPU hours." Across HELM's 30 models and 42 scenarios, the aggregate of reported costs and GPU compute came to roughly $100,000.

Another shocking observation came from Perlitz et al.'s analysis of EleutherAI's Pythia checkpoints: developers pay for evaluation repeatedly during model development. Pythia released 154 checkpoints for each of 16 models spanning 8 sizes, or 2,464 checkpoints if each model checkpoint is counted separately, so the community could study training dynamics. Running the LM Evaluation Harness across all those checkpoints turns eval into a multiplier on training: Perlitz et al. (2024) noted that evaluation costs "may even surpass those of pretraining when evaluating checkpoints." For small models, evaluation becomes the dominant compute line item across the whole development cycle. When we scale inference-time compute, we scale evaluation costs.

Perlitz et al. then asked how much of HELM actually carried the rankings. The result was striking: a 100× to 200× reduction in compute preserved nearly the same ordering, with larger reductions still useful for coarse grouping under the paper's tiered analysis. Flash-HELM turned that finding into a coarse-to-fine procedure: run cheap evaluations first, then spend high-resolution compute only on the top candidates. Much of HELM's compute was confirming rankings that the field could have inferred much more cheaply.

Other work reached the same conclusion from different angles. tinyBenchmarks compressed MMLU from 14,000 items to 100 anchor items at about 2% error using Item Response Theory. The Open LLM Leaderboard collapsed from 29,000 examples to 180. Anchor Points showed that as few as 1 to 30 examples could rank-order 87 language-model/prompt pairs on GLUE, and others followed, reducing dataset sizes by 90\%. Static benchmarks had a weakness you could exploit: model differences often concentrate in a small subset of items, so ranking can survive aggressive subsampling.

That trick weakened sharply once benchmarks moved from static predictions to agents.

Agent evals are messier

A very nice public accounting of agent evaluation comes from the Holistic Agent Leaderboard (Kapoor et al., ICLR 2026). HAL runs standardized agent harnesses across nine benchmarks covering coding, web navigation, science tasks, and customer service, with shared scaffolds and centralized cost tracking. The headline cost: $40,000 for 21,730 rollouts across 9 models and 9 benchmarks. By April 2026, the leaderboard had grown to 26,597 rollouts. Ndzomga's independent reproduction arrives at almost the same number: $46,000 across 242 agent runs.

Behind that aggregate, the cost of a single benchmark run varies by four orders of magnitude across HAL tasks, and by three orders within some individual benchmarks.

Behind these numbers is a blunt pricing fact. Claude Opus 4.1 charges $15 per million input tokens and $75 per million output. Gemini 2.0 Flash charges $0.10 and $0.40, a two-order-of-magnitude spread on input alone. Agent benchmarks rarely benchmark "the model" in isolation. They benchmark a model × scaffold × token-budget product, and small scaffold choices can multiply costs 10×.

Worse, higher spend does not reliably buy better results. On Online Mind2Web, Browser-Use with Claude Sonnet 4 cost $1,577 for 40% accuracy. SeeAct with GPT-5 Medium hit 42% for $171. The HAL paper notes "a 9× difference in cost despite just a two-percentage-point difference in accuracy." On GAIA, an HAL Generalist with o3 Medium cost $2,828 for 28.5% accuracy, while a different agent hit 57.6% for $1,686. CLEAR finds across 6 SOTA agents on 300 enterprise tasks that "accuracy-optimal configurations cost 4.4 to 10.8× more than Pareto-efficient alternatives" with comparable real-world performance.

The static-era toolkit should have helped, but it has only gone so far. Ndzomga's mid-difficulty filter, which selects tasks with 30 to 70% historical pass rates, achieves a 2× to 3.5× reduction while preserving rank fidelity under scaffold and temporal shifts. That is useful, but it falls far short of the 100× to 200× gains available for static benchmarks. When each item is a multi-turn rollout with its own variance, the unavoidable long trajectory per single question becomes the expensive object.

Some evals are just training

Some benchmarks escape the API-cost framing altogether because their evaluation protocol trains models from scratch.

The Well gives a very interesting example of this. It bundles 16 scientific machine-learning datasets spanning biological systems, fluid dynamics, magnetohydrodynamics, supernova explosions, viscoelastic instability, and active matter, totaling 15 TB. Using the paper's headline 16-dataset grid, the protocol leaves little room to economize: train each baseline model for 12 hours on a single H100, try five learning rates per (model, dataset) pair, repeat across four architectures and 16 datasets. That headline-grid sweep consumes 3,840 H100-hours, or roughly $9,600 under the conversion assumptions below. A single new architecture still costs about 960 H100-hours, or about $2,400.

Training one neural operator can take a single 12-hour H100 run, while evaluating it across the benchmark requires 80 such trainings. That asymmetry is what makes The Well important. In this corner of ML, evaluation compute exceeds training compute by roughly two orders of magnitude, reversing the old deep-learning mental model.

The same pattern recurs across SciML. PDEBench covers 11 PDE families and reports per-epoch timing tables across datasets and model families, but a clean per-architecture dollar figure depends on the chosen training protocol and hardware. MLE-Bench (OpenAI) sits between agent and training regimes. Each agent attempt at one of 75 Kaggle competitions runs 24 hours on a single A10 GPU, training real ML pipelines. The paper is explicit: "A single run of our main experiment setup of 24 hours per competition attempt requires 24 hours × 75 competitions = 1,800 GPU hours of compute," plus o1-preview consuming 127.5M input and 15M output tokens per seed. At $1.50 per A10-hour, the GPU floor alone is $2,700; adding o1-preview API usage brings a one-seed run to roughly $5,500. Three seeds × six models would therefore land near $100,000 before any additional grading or retry overhead.

METR's RE-Bench caps each of seven research engineering environments at 8 hours on 1 to 6 H100s. A single pass across the suite is therefore 56 to 336 H100-hours before adding repeated attempts, multiple seeds, or multiple agents; the human baseline, with 71 expert attempts, raises the implicit budget much further. Because the benchmark gives agents and humans the same wall-clock compute, a real-time training process sets the cost floor. A token budget no longer bounds it from above.

ResearchGym (ICLR 2026) makes the agent run actual ML research. Five test tasks (39 sub-tasks) drawn from ACL, ICLR, and ICML papers, including ACL Highlights, ICML Spotlight, ICLR Spotlight, and ICLR Oral categories, with the proposed methods withheld. The agent has to propose hypotheses, train models, and beat the original authors' baselines. The budget is tight: $10 in API plus 12 to 24 hours on a single GPU under 24 GB per task. A full pass (5 tasks × 24h × 3 seeds) consumes about 360 GPU-hours per agent.

The cost picture turns brutal in PaperBench. Twenty ICML 2024 Spotlight or Oral papers must be replicated from scratch, graded against rubric trees with 8,316 leaf-node criteria. Each rollout uses an A10 GPU for 12 hours, and the per-paper math is straightforward:

• $400 in API per o1 IterativeAgent rollout, times 20 papers, comes to about $8,000 per evaluation.
• Grading runs $66 per paper with the o3-mini judge, or $1,320 for the full benchmark.
• Using o1 as judge would push grading to about $830 per paper.

PaperBench Code-Dev drops execution on purpose. That choice halves rollout cost to about $4,000 and cuts grading to $10 per paper (85% lower). OpenAI built the variant because many groups cannot afford the full benchmark.

The historical precedent is NAS-Bench-101, whose tabular construction required over 100 TPU-years of training. Without that one-time investment, every NAS algorithm comparison would have cost 1 to 100+ GPU-hours per run, which would have made comparison pricier than the algorithms themselves.

As benchmarks move closer to real work, compression gets harder: static prediction leaves room for large savings, agent rollouts leave less, and in-the-loop training leaves almost none.

Reliability is the expensive part

Most of the costs above buy only single-run measurements with limited statistical power. When you measure reliability across repeated runs, static benchmarks, agent benchmarks, and training-in-the-loop benchmarks all become more expensive.

Agent reliability can fall hard when you stop treating one run as evidence. The best-known example comes from Yao et al.'s τ-bench, later reframed in CLEAR (Mehta, 2025): performance can drop from 60% on a single run to 25% under 8-run consistency. Kapoor et al.'s "AI Agents That Matter" found that simple baseline agents Pareto-dominate complex SOTA agents (Reflexion, LDB, LATS) on HumanEval at 50× lower cost. Their holdout analysis found that 7 of 17 benchmarks had no holdout set; among the 10 that did, only 5 held out tasks at the appropriate level of generality, so 12 of 17 failed their holdout criterion overall. The HAL paper notes that a "do-nothing" agent passes 38% of τ-bench airline tasks under the original construction. HAL's own log analysis revealed data leakage in the TAU-bench Few Shot scaffold, forcing its removal in December 2025.

Another recent reliability accounting comes from Rabanser, Kapoor et al.'s "Towards a Science of AI Agent Reliability", which proposes twelve metrics across consistency, robustness, predictability, and safety. Their finding: "recent capability gains have only yielded small improvements in reliability." HAL's internal analysis shows how much fragility hides behind aggregate accuracy. On SciCode and CORE-Bench, agents almost never completed a run without a tool-calling failure. On AssistantBench and CORE-Bench, environmental errors occurred in roughly 40% of runs. Agents violated explicit benchmark instructions in their final answer over 60% of the time on failed tasks.

A statistically credible HAL-style evaluation with k = 8 reruns per cell takes the $40K aggregate to roughly $320K. The same multiplier on PaperBench's $9,500-per-run cost pushes a single agent's evaluation past $75K, and on The Well, a multi-seed protocol takes the per-architecture cost from ~960 H100-hours to several thousand. Reliability acts as a multiplier on every cost category above.

HAL has paused new model evaluations to focus on reliability: the field's headline numbers still carry too much noise, and reducing that noise costs real money. And the figures above are lower bounds; many evaluators are already priced out.

What this means for ML as a field

Eval cost is now an accountability barrier

Academic groups, AI Safety Institutes, and journalists now hit the budget constraint before the technical one when they try to evaluate frontier agents independently. A single GAIA run can exceed an annual graduate student travel budget. A single PaperBench evaluation, including the LLM judge, runs about $9,500. Three-seed comparisons of six models, the kind of study one might publish, push above $150,000. The established practice of "running a benchmark once and reporting the accuracy number" has roughly the rigor of crash-testing one car in perfect weather. Moving past it requires money the academic system does not currently allocate as research compute.

The compute divide now includes evaluation

Ahmed, Wahed and Thompson (Science 2023) documented that industry models in 2021 were 29× larger than academic ones by parameter count, and that about 70% of AI PhDs went to industry in 2020 versus 21% in 2004. The original "compute divide" story mostly ignored evaluation because evaluation used to look cheap next to training. Many benchmarks have reversed that relationship. A lab that can fine-tune a 7B model can no longer assume it can afford the benchmarks the field takes seriously.

Cost-blind leaderboards reward waste

When leaderboards report raw accuracy and omit cost, researchers can rationally pour tokens into a problem until the number ticks up. The HAL paper finds that higher reasoning effort actually reduces accuracy in the majority of runs: extra inference compute does not reliably improve even the metric it is supposed to optimize. Pareto frontiers fix the comparison by ranking accuracy against cost. HAL implements them, but most leaderboards still do not.

If only frontier-lab compute budgets can produce statistically reliable benchmark numbers on the highest-cost agentic and scientific benchmarks, the social process of evaluating AI systems becomes concentrated inside the same labs that build them, rendering external validation partial, and sometimes absent, unless someone subsidizes the cost directly.

Cost summary across benchmark types

All figures normalized to USD per single evaluation. GPU compute converted at $2.50/H100-hour, $1.50/A10-hour; API and grading costs included where applicable. Pythia ("eval can exceed pretraining"), PDEBench (per-architecture cost depends on the selected training protocol and hardware), and NAS-Bench-101's 100 TPU-year construction cost are excluded because they do not normalize cleanly to a per-evaluation USD figure.

Stop paying twice for the same eval

One reason these numbers stay high is that the field keeps re-running the same evaluations. A frontier lab pays for a HAL sweep, an academic group pays again for a partial reproduction, an audit organization pays a third time for the model versions it cares about, and a journalist pays a fourth to spot-check the leaderboard. Most of those runs cover overlapping models on overlapping benchmarks. Almost none of the underlying instance-level outputs end up in a place where the next team can build on them, because results get reported as a single accuracy number in a PDF, in a model card table, or in a leaderboard entry that hides scaffold, prompt, and seed. The cost figures above are large in part because the field is paying retail every time, on artifacts the rest of the community could not reuse if it wanted to.

Standardized documentation is the cheapest lever available here, and it is the one reliability work needs anyway. If a $9,500 PaperBench rollout exports its full grading trace in a shared schema, the next group studying the same papers can spend its budget on new perturbations instead of repeating the baseline. If a multi-seed HAL run publishes per-trajectory tool-call logs, agent reliability research can answer questions that a single accuracy number cannot. The saving compounds: even a 2× reuse rate on the high-cost benchmarks would put more money back in the ecosystem than every compression technique combined.

Where this leaves us

The economics have changed. Not long ago, training was expensive and evaluation was cheap. For frontier LLMs trained at $50 million to $100 million, evaluation still looks like a rounding error, but that rounding error now costs tens of thousands of dollars per benchmark run and often leaves noisy results behind. For neural operators, ML research agents, and replication benchmarks, the ratio has flipped: a credible evaluation can cost more than training the candidate model.

We already know how to make static evaluation cheaper. Flash-HELM, tinyBenchmarks, and Anchor Points work. Agent evaluation has only partial fixes: mid-difficulty filtering helps, and Pareto-front leaderboards help, but the toolkit remains thin. Training-in-the-loop evaluation has no general compression method; tabular precomputation and tight budget caps can reduce cost only by narrowing what the benchmark measures. Reliability adds another layer because repeated runs raise the price of every protocol.

The field still talks as if capability sets the main constraint, but evaluation points to reliability as the tighter one. Governance institutions should want to measure the gap between single-run accuracy and pass^k consistency, yet that gap costs the most to measure. Static-benchmark compression does not transfer to agent or training-in-the-loop benchmarks, and mid-difficulty filtering remains the only credible partial substitute. Cost-blind leaderboards now mislead by design, because they reward extra spending without reporting what that spending bought.

Evaluation now has its own compute budgets, statistical methods, and failure modes. Its price also shapes who gets to evaluate powerful systems in the first place. Whoever can pay for the evaluation gets to write the leaderboard.

@misc{ghosh2026evalbottleneck,
  author       = {Ghosh, Avijit and Mai, Yifan and Channing, Georgia and Choshen, Leshem},
  title        = {{AI} evals are becoming the new compute bottleneck},
  year         = {2026},
  month        = apr,
  howpublished = {EvalEval Coalition Blog},
  url          = {https://evalevalai.com/research/2026/04/29/eval-costs-bottleneck/}
}

Since the launch, we have conducted an extensive literature review and semi-structured interviews with experts across the field to gain insights into prevailing approaches, methodological assumptions, and persistent challenges in evaluation practices. In this post, we provide an update on where that work stands today, share early themes emerging from our analysis, and highlight where additional input would be most valuable as the workstream moves forward.

We use ‘evaluation’ broadly to refer to the methods used to gather and interpret evidence about a GenAI system¹ regarding its performance, capabilities, propensities, and direct impacts. These include methods such as benchmarking, targeted in-context evaluations, system monitoring, automated red‑teaming, and human uplift studies. While GenAI evaluation can span many modalities, this post focuses on text-centered systems.

To date, we have conducted 15 semi-structured interviews (approximately 30 minutes each) with practitioners across leading AI labs, academia, government-linked institutions (including AI safety institutes), and evaluation-focused startups and non-profits. Their work spans development of evaluations, creation of policy, application of evaluations to models and products, and interpretation of evaluations for design and deployment decision making. Interviewees were selected in roughly equal proportions across each type of organization based on recent publications and snowball sampling, focusing on those actively shaping evaluation practice in the large language model (LLM) ecosystem. The interviews probed into methodologies, conceptual and statistical challenges, operational constraints, and gaps in the field of AI evaluations.

Here we draw on the first eight fully annotated interviews. Our findings center on three key themes: validity, practicality, and interpretability. The open questions and challenges within these themes can serve as a useful guide for researchers, policymakers, auditors, and engineers in the AI evaluation community, shaping future research directions and informing how decision-makers interpret and act on evaluation results. We discuss each of the themes in turn below.

Validity

Validity emerged as a central challenge in interviews about the GenAI evaluation space and is receiving increasing attention across the community.² Broadly, validity refers to the extent to which a measurement meaningfully captures the construct/behavior it claims to measure.

Interviewees highlighted challenges across multiple aspects of validity, namely construct validity (whether the concept of interest is clearly specified and theoretically grounded), content validity (whether test items and scoring criteria appropriately reflect the construct), and ecological validity (whether test items capture real-world usage³).

Across interviews, a clear message emerged: evaluation results can only be trusted if they are grounded in a precise specification (systematization) of the concept of interest. Terms like “bias” and “quality” are often used loosely, and this ambiguity trickles down to the measurement instruments on which evaluations depend. As a result, it is difficult to determine what evaluation results actually mean with respect to the concept of interest.

As interviewees put it:

“We can have 50 different benchmarks that all talk about measuring bias, but none of them say exactly what bias is […] I have no way of knowing which one to trust because they haven’t specified precisely what concept they were trying to capture.” (P1)

“Words mean different things to different people. We have to make sure that when we say we’re measuring misinformation, we are tightly defined. […] We call that context specification.” (P2)

The consequences extend to the test items themselves, where misalignment between test items and the target construct can undermine an entire benchmark:

“A lot of benchmarks have stuff that is kind of nonsense, and oftentimes not even particularly related to what the benchmark is trying to measure. So, for example, in [a popular benchmark professedly testing hazardous biosecurity knowledge], there are some history questions about facts about American and Soviet bio weapons programs in the seventies” (P3)

Interviewees also expressed dataset-related concerns associated with ecological validity. Prior to deployment, developers may lack access to relevant real-world usage data. Even after deployment, this data may not be accessible due to privacy considerations or may be unusable due to a low signal-to-noise ratio for rare concepts of interest. This prompts developers to turn to synthetic data, which comes with its own set of challenges:

“We use a lot of synthetic [data.] Basically, the pipeline is a rejection sampling pipeline where you have one model generate and another model filter […] Often, what we found is that the first thing [language models] generate looks almost correct, and then if you look at it for longer […] it makes you realize, this is not optimal.” (P4)

“We still really struggle with ecological validity of synthetic data. […] But, using real data is also not ideal because, if you’re looking for some rare phenomenon, the signal-to-noise ratio is going to be really low in real sample data. So synthetic is important for us to get a stronger signal.” (P1)

Even when the concept of interest is well-defined and the dataset is carefully constructed, evaluations can quickly lose relevance as usage shifts. Users may interact with a model differently as it becomes more capable, and developers may change how models are integrated into products. Such distributional shifts can quietly violate the assumptions underlying evaluation datasets.

“The distribution is changing very quickly… how a consumer uses a certain product that has AI with an old AI model might be very different from how they use the same product.” (P5)

Practicality

Interviewees emphasized that evaluation design is also shaped by real operational constraints (e.g., budgets, timelines, privacy), citing engineering and compute costs as key blockers, which influence how concepts of interest are prioritized.

“There are, you know, a lot of risks […] we have [an] internal repository where we’re tracking risks, and it’s a lot. We don’t have evaluations for all those risks. Especially if we’re trying to make it easier for [our] teams, we have to build these evaluations in the central platform in a central way, and that is […] quite resource-intensive to do that.” (P6)

“It’s a balancing step […] is performing that evaluation actually worth it with respect to the burdens and the benefits?” (P7)

Even when compute resources are not a problem, many GenAI evaluation tasks examine complex sociotechnical concepts that require substantial expertise to annotate and validate accurately. Accessing such expertise at large scales requires infrastructure and costs that developers may be unwilling to pay, favoring instead LLM-based judges.

“We never use human judges. … It just doesn’t scale like you would have to set up a pipeline, and then the humans don’t have time. It’s a coordination issue.” (P4)

More capable systems introduce further challenges. As new models and agentic systems emerge, evaluations become harder and more costly. Agent environments increase complexity, variance, and expense:

“As the models get more capable, it is harder and harder work to get good evals […] They are deployed as agents and environments […] that have greater complexity: Firstly, the variance in their performance is much much greater. There’s a lot more randomness in their interaction with the environment. And secondly, the cost goes up.” (P5)

Interpretability and Reporting

Interviewees also highlighted challenges in interpreting the evidence underlying evaluative claims. As benchmarks grow larger and more complex, translating results into relevant analyses for a variety of decision makers is becoming harder and more tooling-dependent (e.g., AI agents take 100s of steps to solve a single long horizon task, intermingling tool calls with reasoning tokens, making it hard for a single person to process them without access to tooling), furthering practicality issues.

As evaluation benchmarks grow in size, individual human review becomes impractical:

“Increasingly, as the benchmarks get bigger and more complicated, it’s harder to have this intuitive feel for the data. You’re more reliant on external tooling and systems to sanity check things than you were in the past.” (P5)

“Humans are just not capable of inspecting 10,000 runs through an environment. And so we need a lot of AI assistance to understand what is going on.” (P5)

More broadly, interviewees noted that stronger grounding in basic statistical practice could improve rigor and the reliability of resulting claims:

“There are massive, very common, huge statistical issues in the way that people go about doing their evaluations, whether it’s not including error bars, whether it’s conflating statistical significance with effect size, whether it’s just interpreting statistics sometimes.” (P3)

Ultimately, evaluation results are intended to inform decisions about AI models and systems in the real world. It remains a major challenge to build evaluations that stakeholders can trust at decision time:

“We know that the stakeholders who are using our evaluations to make decisions really want to be able to interpret them objectively. If they see that we conducted an evaluation and the output was a 3% failure rate, they want to know how that translates to what we expect for real users in the real world, does that mean that 3% of users on a day-to-day basis will experience harmful outputs?” (P1)

By synthesizing these challenges and open questions alongside emerging best practices and recommendations, we aim to help address them within the Science of Evaluations and broader evaluation practice space.

What’s Next?

This work is ongoing; we continue to analyze additional interviews and advance our literature review, synthesizing insights across AI and adjacent fields to learn established methodologies and identify where important gaps remain. We are also expanding our stakeholder interviews to ensure that the priorities we identify reflect the perspectives of researchers, practitioners, policymakers, and auditors across industries and geographies.

As we enter the next phase of the Science of Evaluations workstream, we welcome input from those developing or applying evaluation methods for generative AI and large language models (LLMs). This includes teams at labs engaged with development and evaluation of models, alongside academics, civil society, and industry practitioners. Responses will be reported only in anonymized, aggregated form, as in the current post.

If you are interested in contributing, suggesting interviewees, or sharing relevant work, we invite you to reach out at core@evalevalai.com. We see this effort as collaborative and ecosystem-wide, and we welcome participation as we continue to build the scientific foundations for AI evaluation.

@misc{scholz2026fieldnotes,
  author       = {Scholz, Robert and Atalla, Chad and Soni, Prajna and Staufer, Leon and Balayn, Agathe and Zouhar, Vilém and Mukhija, Namrata and Mayeesha, Tahsin and Paskov, Patricia and Majumdar, Subho and {EvalEval Coalition}},
  title        = {Field Notes: Challenges in {GenAI} Evaluation Science},
  year         = {2026},
  month        = mar,
  howpublished = {EvalEval Coalition Blog},
  url          = {https://evalevalai.com/research/2026/03/25/interview-field-insights/}
}

This rapid progression and wide range of evals led to fragmentation. Lacking standards, each framework reports different attributes in different formats, preventing the community from reliably comparing, replicating, determining which component fails, separating signal from noise, reusing others’ (often costly) evaluations, and performing large-scale analysis. This lack of reusability makes it difficult to build on the momentum of previous efforts. As model training has moved past the point where we retrain models from scratch or rewrite their training code, we must ask: why are we still rerunning every evaluation from scratch?

As part of a cross-institutional, cross-sector initiative, the EvalEval Coalition, we are announcing Every Eval Ever to improve the state of evaluation building and comparison:

(1) Defining a shared schema so results from different frameworks can be compared, and

(2) Providing a crowdsourced eval database¹ so researchers don’t have to start from scratch every time.

Today, we’re launching Every Eval Ever,² built upon valuable feedback from AI Eval Ecosystem actors including researchers and practitioners at the U.S. Center for AI Standards and Innovation (CAISI), EleutherAI, Hugging Face, Noma Security, Trustible, Inspect AI, Meridian, AVERI, Collective Intelligence Project, Stanford HELM, Weizenbaum, Evidence Prime, MIT, TUM, Northeastern, and IBM Research.

The Hidden Problem

The infrastructure is fragmented, and the results scattered. How does Model A compare to Model B on a given benchmark? One lab reports a five-shot score from its own harness. Another uses zero-shot through a different framework. A third pulls numbers from a leaderboard that does not disclose generation parameters. This fragmentation has concrete costs. Large-scale analysis of evaluation trends or improving evaluation methodologies requires weeks of data wrangling before any research can begin. If they are possible at all, without (re)running full leaderboards at extreme costs. Comparison is unreliable when evaluations use different settings but carry the same benchmark name. Reproducibility is elusive when the details are lacking and inconsistent. It is time for a change. We have seen this before in other parts of the ML pipeline. The community stopped retraining models from scratch or rewriting training code for each project long ago. Evaluations are next.

Why Us, Why Now

We just know the pain. The EvalEval Coalition is a community of researchers working to fix how AI evaluations are built, run, documented, shared, and compared. We worked on a myriad of projects where collecting evaluations restricts what can be done or takes most of the project’s efforts. Need examples? See 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 among others.

The urgency of standardized AI evaluation has reached a critical tipping point, driven by the shift toward evaluations as a primary mechanism of governance. With the EU AI Act and the U.S. Executive Order now mandating rigorous risk assessments, standardized data is no longer a luxury but a prerequisite for meaningful sociotechnical safety standards. This need is further intensified by the exploding complexity of modern AI, where frameworks like Inspect AI and HELM must navigate multi-turn agentic behaviors and human preferences that defy simple scoring.

Ultimately, failing to adopt reusable formats imposes a technical debt on the community, forcing researchers to waste resources rerunning redundant evaluations rather than advancing the scientific frontier. Oh, and of course, even what is shared is often just a single score per dataset, obscuring many questions.

What We’re Building

Every Eval Ever is a schema to describe evaluation results and a community collection of those results. It is by the community and for the community, made simple to contribute to, evaluation or code. Enough high-level, let’s get into the details.

The repository is organized by benchmark, model, and evaluation run. Each result file captures not just scores but the context you need to interpret and reuse them:

• who ran the evaluation,
• what model,
• with what settings,
• what these scores actually mean,
• and instance-level scores, if you have them.

data/
└── {benchmark}/
    └── {developer_name}/
        └── {model_name}/
            ├── {uuid}.json
            └── {uuid}.jsonl

The three components that make Every Eval Ever work 💙

📋 A metadata schema that defines the information needed for meaningful comparison of evaluation results

🔧 Validation that checks data against the schema before it enters the repository

🔌 Converters for popular evaluation tools like Inspect AI, HELM, and lm-eval-harness, so you can transform your existing evaluation logs into the standard format.

📋 The Schema

One thing we realized early on: evaluation harnesses are amazing — we love them all — but they were each built for their own purposes, and we cannot simply aggregate their scores. Take MMLU: the lm-eval-harness, HELM, and the original Berkeley implementation all evaluate the same dataset, but with different prompt formatting, different answer extraction methods, and different ordering of few-shot examples. The result? LLaMA 65B scored 0.637 on HELM but 0.488 on the EleutherAI harness — both called “MMLU score,” same dataset, big gap.

And then, think about all the evaluation data that doesn’t come from a harness at all: hand-built benchmarks, leaderboard scrapes, results pulled from research papers and blog posts. We wanted one format that could capture whether harness-run or hand-built alike, truly every evaluation ever. Here’s how the schema looks in practice, using LiveCodeBench Pro as an example from our repo.

First, we capture where the evaluation came from and who ran it:

{
  "source_metadata": {
    "source_name": "LiveCodeBench Pro",
    "source_type": "evaluation_platform",
    "source_organization_name": "LiveCodeBench",
    "evaluator_relationship": "third_party"
  }
}

The schema also records the exact model details — name, developer, and how it was accessed. Was it called through the developer’s own API (like OpenAI or Anthropic), through a third-party provider (like OpenRouter or Together AI), or run locally with an inference engine like vLLM? This isn’t just because we love eval metadata. The same model, accessed through different providers or run with different engine configurations, can produce different outputs — and therefore different scores.

Next, generation settings. We all know how much they matter — changing temperature or the number of samples alone can shift scores by several points. Yet they’re routinely absent from leaderboards and incomplete even in papers. When a model’s score is reported without this context, we’re left guessing whether differences reflect actual model capability or just different settings. So we want to capture this information — and where it’s missing, record that gap explicitly, so anyone interpreting the results knows what context they do and don’t have:

{
  "generation_config": {
    "generation_args": {
      "temperature": 0.2,
      "top_p": 0.95,
      "max_tokens": 2048
    },
    "additional_details": {
      "n_samples": 10,
      "stop_sequences": ["\n```"]
    }
  }
}

And then there’s the score itself. Let’s take a model on the coding benchmark HumanEval: scoring 0.31 on the first try (called pass@1) represents a fraction of coding problems it solved — higher would be better. On the contrary, if the same model scores again 0.31 but on RealToxicityPrompts, lower scores would be better. Every Eval Ever standardizes to enable better eval result interpretation:

{
  "evaluation_results": [
    {
      "evaluation_name": "code_generation",
      "metric_config": {
        "evaluation_description": "pass@1 on code generation tasks",
        "lower_is_better": false,
        "score_type": "continuous",
        "min_score": 0,
        "max_score": 1
      },
      "score_details": {
        "score": 0.31
      }
    }
  ]
}

🔧 Validation & Converters

Better eval infrastructure should be easy and frictionless for practitioners. That’s why we’re proud that Every Eval Ever provides:

Converters: If you’re already running evaluations with existing eval tools, you shouldn’t have to manually parse your results. Our converters transform evaluation logs into the Every Eval Ever format automatically. Converters for HELM, lm-eval, and Inspect AI are already available, with more on the way. One step, and your eval data fits in.¹

Validation: Validation runs automatically on every submission via Hugging Face Jobs — before any result file is merged, it’s checked against the schema to catch missing fields and structural issues early, not months later when someone tries to use the data.

🧩 Design Decisions

What counts as a unique evaluation for you? This can get quite a thorny question! Imagine two GSM8K runs might differ in prompt template, chat template, vLLM version, GPU kernels, or dataset version — each affecting scores. Within our own coalition, members used GSM8K for purposes as different as measuring mathematical reasoning and benchmarking speculative decoding.

We considered defining a canonical parameter set to fingerprint unique runs. In practice, the space of score-affecting variables is too large for any fixed set. Our solution: each run gets its own file, identified by a UUID, with as much metadata as possible captured alongside it. Deduplication and grouping happen at the analysis layer, not the schema layer. This keeps data lossless while letting consumers apply their own equivalence criteria.

On purpose we allowed reporting whatever one has. For many, this is an aggregation score with some metadata; for others, it is per-example data and a lot of hyperparameters. Why should you care? Hardly any evaluation research is done on the aggregation scores. You want to check whether there are biases, errors, redundant questions, questions that measure the wrong thing, aggregate datasets differently, and analyse errors. All of those questions require at least the model outputs or scores per example, not per dataset.

What’s Next

Every Eval Ever grew out of a need we kept running into in our own research. When EvalEval researchers mapped how social impact evaluations are reported across the field, examining 186 first-party reports and 183 third-party sources, the lack of a common format turned what should have been a straightforward analysis into weeks of manual data wrangling. That work made the case for why something like Every Eval Ever needed to exist: even though a lot of evaluation data is available open source, it is in incompatible formats, with no shared infrastructure to aggregate or compare it.

This schema enables research. Beyond just good documentation hygiene, researchers already used the repository in a multi-author EvalEval effort to analyze benchmark saturation across 60 benchmarks, finding that nearly half had lost their ability to differentiate top-performing models. Centralized, standardized evaluation data opens up more: seeing where the ecosystem is thin, which capabilities are over-measured, and which risks are neglected. With instance-level data, researchers can move beyond leaderboard averages to study item difficulty, robustness, and temporal drift. Every Eval Ever enables meta-evaluation: testing evaluation methods themselves to distinguish real progress from artifacts of setup and reporting. We need your help. We’re launching a Shared Task for practitioners alongside this post — two tracks for contributing public and proprietary eval data to the repository, with co-authorship for qualifying contributors and a workshop at ACL 2026 in San Diego.

Submissions open now, deadline May 1, 2026.

Get involved

• Try the schema 📋 : Hugging Face Dataset², Interactive Visualization³, and GitHub

• Join the Shared Task 🏁 : Call for Participation

• Join the community 💬 : Reach out to be added

@misc{batzner2026eeeblog,
  author       = {Batzner, Jan and Choshen, Leshem and Ghosh, Avijit and Nelaturu, Sree Harsha and Kornilova, Anastassia and Stachura, Damian and Mai, Yifan and Yehudai, Asaf and Reuel, Anka and Solaiman, Irene and Biderman, Stella},
  title        = {Every Eval Ever: Toward a Common Language for {AI} Eval Reporting},
  year         = {2026},
  month        = feb,
  howpublished = {EvalEval Coalition Blog},
  url          = {https://evalevalai.com/infrastructure/2026/02/17/everyevalever-launch/}
}

@misc{batzner2026evaleverunifyingschema,
  author        = {Jan Batzner and Sree Harsha Nelaturu and Damian Stachura and Anastassia Kornilova and Jon Crall and Tommaso Cerruti and Yanan Long and Yifan Mai and Sanchit Ahuja and Asaf Yehudai and Marek Šuppa and John P. Lalor and Oluwagbemike Olowe and Jatin Ganhotra and Brian H. Hu and Eliya Habba and Andrew M. Bean and Chang Liu and Sander Land and Steven Dillmann and Aniketh Garikaparthi and Elron Bandel and Saki Imai and James Edgell and Wm. Matthew Kennedy and Jenny Chim and Patrick Meusling and Asteria Kaeberlein and Venkata Ramachandra Karthik Chundi and Manasi Patwardhan and Martin Ku and Austin Meek and Leon Knauer and Brian Wingenroth and Srishti Yadav and Usman Gohar and Felix Friedrich and Michelle Lin and Jennifer Mickel and Arman Cohan and Stella Biderman and Irene Solaiman and Zeerak Talat and Anka Reuel and Mubashara Akhtar and Gjergji Kasneci and Avijit Ghosh and Leshem Choshen},
  title         = {Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results},
  year          = {2026},
  eprint        = {2606.14516},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2606.14516}
}

Acknowledgement and Feedback

We acknowledge feedback by, but not limited to, JJ Allaire (Inspect, Meridian Labs), Ryan Steed (US CAISI), Zeerak Talat (University of Edinburgh), Gal Moyal (Noma Security), Sean McGregor (AVERI), Baber Abbasi (EleutherAI), Joal Stein (WeVal/CiP), Elizabeth M. Daly (IBM Research), Aris Hofmann (IBM Research), Inge Vejsbjerg (IBM Research), Dhaval Salwala (IBM Research), Srishti Yadav (University of Copenhagen), Andrew Tran (Independent), Sanchit Ahuja (Northeastern), Marek Šuppa (Slido), Stefan Schmid (Weizenbaum, TUB), Gjergji Kasneci (TUM, MCML).

While still far from reaching universal adoption, many organizations post ‘release reports’ alongside new AI models. However, many reports typically focus on detailing the technical capabilities of models and often overlook, or in some cases, completely omitting any social impact risks evaluation.

Our paper, Who Evaluates AI’s Social Impacts? Mapping Coverage And Gaps In First And Third Party Evaluations, argues for more transparent and consistent reporting standards that document and put greater emphasis on social impact issues.

In our study, we analyze 186 first-party reports released at the time of an AI model’s release, 183 post-release reports, and incorporate developer interviews to understand the incentives behind model reporting. Release reports were analyzed based on seven social impact dimensions: Bias and Harm, Sensitive Content, Performance Disparity, Environmental Costs and Emissions, Privacy and Data, Financial Costs, and Moderation Labor (Solaiman et. al. 2023).

Our findings showed that there was a significant gap between release-time and post-release reports of social impact in AI evaluations.

Using a scale ranging from 0–3, we quantify how well the reports provided details of each social impact dimension. First-party release-time reports averaged just 0.72 out of 3, compared to 2.62 for post-release reports. Additionally, we found that certain categories, such as environmental cost and emission, have decreased substantially since late 2023. Our developer interviews gave us further insight into why this was happening. As one developer put it:

“The environment has become a touchy subject, and companies don’t want to make themselves look bad, so they just don’t report anything.”

Differences in Industry and Academia

In general, our findings show that companies such as Meta, Google, Anthropic, and more provided strong evaluations for their earlier models (e.g., Llama 1 and Llama 2), but fell short on scoring for recent releases.

In recent years, both academic and independent organizations have produced more rigorous evaluations across the social impact categories than their industry counterparts. However, these groups face intrinsic limitations due to limited access to internal data. In general, academic and independent organizations lack access to “internal data” such as compute metrics or data center location. This gap in information leads to reports that lack thorough social impact information.

Efforts to Close the Gap

This project, led by the Evaluating Evaluations (EvalEval) Coalition, highlights a growing discrepancy: as AI’s popularity increases, the transparency needed for governance is shrinking. Fixing this requires standardized reporting frameworks, shared tools, and safety policies that make safe reporting both accessible and possible.

Explore Our Work

• Paper
• Dataset
• Code

@misc{hameed2025hiddencosts,
  author       = {Hameed, Abdul and Kashif, Afifah and Tran, Andrew and Gohar, Usman},
  title        = {The Hidden Social Costs of {AI}},
  year         = {2025},
  month        = nov,
  howpublished = {EvalEval Coalition Blog},
  url          = {https://evalevalai.com/research/2025/11/12/hidden-cost-of-evals/}
}

@misc{reuel2026evaluatesaissocialimpacts,
  author        = {Anka Reuel and Avijit Ghosh and Jenny Chim and Andrew Tran and Yanan Long and Jennifer Mickel and Usman Gohar and Srishti Yadav and Pawan Sasanka Ammanamanchi and Mowafak Allaham and Hossein A. Rahmani and Mubashara Akhtar and Felix Friedrich and Robert Scholz and Michael Alexander Riegler and Jan Batzner and Eliya Habba and Arushi Saxena and Anastassia Kornilova and Kevin Wei and Prajna Soni and Yohan Mathew and Kevin Klyman and Jeba Sania and Subramanyam Sahoo and Olivia Beyer Bruvik and Pouya Sadeghi and Sujata Goswami and Angelina Wang and Yacine Jernite and Zeerak Talat and Stella Biderman and Mykel Kochenderfer and Sanmi Koyejo and Irene Solaiman},
  title         = {Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations},
  year          = {2026},
  eprint        = {2511.05613},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CY},
  url           = {https://arxiv.org/abs/2511.05613}
}