We're a coalition on evaluating evaluations (EvalEval)!
The flawed state of evaluations, lack of consensus for documenting evaluation applicability and utility, and disparate coverage of broader impact categories hinder adoption, scientific research, and policy analysis of generative AI systems. Coalition members are working collectively to improve the state of evaluations.
The Coalition brings together researchers and practitioners working on the methodology, infrastructure, and public understanding of AI evaluation, including the assessment of social impact. We pursue this through collaborative projects, shared infrastructure, publications, events, and engagement with policy and standards processes.
Our NeurIPS 2024 "Evaluating Evaluations" workshop kicked off the latest workstreams, highlighting tiny paper submissions that provoked and advanced the state of evaluations. Our original framework for establishing categories of social impact is now published as a chapter in the Oxford Handbook on Generative AI and we hope it will guide standards for Broader Impact analyses. We've previously hosted a series of workshops to hone the framework.
Since then, the Coalition has grown its work across research, infrastructure, and community. We have co-hosted several workshops, including Evaluating AI in Practice (2025) and the ACL 2026 Workshop on Evaluating Evaluations, and published at leading AI conferences including NeurIPS and ICML.
This cross‑sector and interdisciplinary coalition is governed by its bylaws (effective October 2026) and led by a Steering Committee and Chairs, with individual projects run by Project Leads.
Members take part in an individual capacity, and participation does not imply endorsement by their institutions.
Researchers, practitioners, and students are welcome to contribute to our mission. Send us an email to learn more about getting involved.
[email protected]
