ACL 2026 Workshop on Evaluating Evaluations (EvalEval)

Update: We are excited to meet you in San Diego! Save the date: on 03/July/2026 around 8pm, we will co-host a Social together with the GEM workshop. Both of our workshops will then be on the day after.

🗓️ Schedule 04/July/2026 Afternoon

👋 2:00 PM – 2:05 PM | Welcome and Introduction (5 mins)

Jennifer Mickel, EvalEval Workshop Co-Chair

🎙️ 2:05 PM – 2:45 PM | Panel Presentation (30 mins panel + 10 mins Q&A)

Moderator: Leshem Choshen, MIT, IBM Research, MIT-IBM Watson AI Lab

This panel brings together model developers and evaluation researchers to examine how evaluations are designed, interpreted, and used in real-world model development. We will surface key tensions — speed vs. rigor, benchmarks vs. deployment needs, and generality vs. task-specificity — while highlighting points of productive collaboration. The discussion aims to bridge perspectives from both communities and identify concrete pathways for more aligned, impactful evaluation practices in NLP.

👥 Panelists:

Angelina Wang, Cornell Tech
Sebastian Gehrmann, Bloomberg
Wesley Deng, Microsoft Research

☕ 2:45 PM – 2:50 PM | BREAK (5 mins)

🗣️ 2:50 PM – 3:20 PM | Six Oral Presentations (30 mins)

Moderator: Usman Gohar, Iowa State University

📄 Rigorous Interpretation Is a Form of Evaluation Isabelle Lee, Emmy Liu, Cathy Jiao, Brihi Joshi, Dani Yogatama, Fazl Barez, Michael Saxon

📄 Graduating the Benchmark Scale: Lessons from Thermometry Sean Trott, Oisín Parkinson-Coombs

📄 One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization Franziska Weeber, Vera Neplenbroek, Jan Batzner, Sebastian Padó

📄 Becoming Experienced Judges: Selective Test-Time Learning for Evaluators Seungyeon Jwa, Daechul Ahn, Reokyoung Kim, Dongyeop Kang, Jonghyun Choi

📄 LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking Lukas Helff, Quentin Delfosse, David Steinmann, Ruben Härle, Hikaru Shindo, Patrick Schramowski, Wolfgang Stammer, Kristian Kersting, Felix Friedrich

📄 Community-Centered Measurement of Cultural Content in AI Images Nari Johnson, Deepthi Sudharsan, Hamna, Samantha Dalal, Theo Holroyd, Anja Thieme, Hoda Heidari, Daniela Massiceti, Jennifer Wortman Vaughan, Cecily Morrison

🪧 3:20 PM – 4:20 PM | Poster Presentations (60 mins)

Moderator: Usman Gohar, Iowa State University

☕ 4:20 PM – 4:25 PM | BREAK (5 mins)

🔬 4:25 PM – 5:15 PM | Shared Task (50 mins)

Moderator: Jan Batzner, Weizenbaum Institute, Technical University Munich

AI evaluation results are scattered across leaderboards, papers, blog posts, and harness logs in incompatible formats, with different frameworks producing divergent scores and inconsistent metadata that hinder comparison, reuse, and cost reduction. Every Eval Ever is the first unifying schema and community-crowdsourced repository for all AI evaluation results — source-agnostic by design and at unprecedented scale: over 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats — and growing! At ACL, we present the ACL Every Eval Ever Shared Task and the fantastic community case studies it has already enabled on this database.

🏆 5:15 PM – 5:25 PM | EvalEval Community Awards (10 mins)

Moderator: Leshem Choshen, MIT, IBM Research, MIT-IBM Watson AI Lab

🎤 5:25 PM – 5:30 PM | Closing Remarks (5 mins)

Ichhya Pant, EvalEval Workshop Co-Chair

📖 Background

As Generative AI systems are increasingly integrated into real-world products and decision-making pipelines, evaluation has become a central yet challenging component of responsible AI development.¹ While evaluation research has advanced rapidly, gaps persist between research and practice: model developers often prioritize scalability and integration into development workflows, while evaluation researchers emphasize rigor, validity, and sociotechnical considerations. The EvalEval coalition’s multi author effort mapping first - and third-party social impact evaluations shows a clear division of labor, where developers underreport or deprioritize key impacts such as environmental costs, data provenance, and labor practices, while third-party evaluators provide broader but necessarily incomplete coverage, leaving critical gaps in accountability and comparability.²

This workshop focuses on AI evaluation in practice, centering the tensions and collaborations between model developers and evaluation researchers. Through a call for papers on contemporary challenges spanning methodological rigor, sociotechnical perspectives, scalability, community-informed evaluation, and real-world use alongside invited panels, the workshop aims to surface practical insights from across the evaluation ecosystem. The workshop will also host a shared task for building a unifying, standardized database of LLM Evaluations, encouraging shared infrastructure and actionable evaluation practices.

📝 Topics

Themes for submission include, but are not limited to:

1. Evaluation Methodology and Measurement Theory

Conceptualization: Operationalization and construct definition issues in evaluations of generative AI;
Validity: Construct, convergent, and discriminant validity of existing evaluations;
Reliability: Robustness, consistency, and generalizability of evaluation methods;
Metrics: Design, selection, and limitations of evaluation metrics, benchmarks, and scoring methods;
Reproducibility: Cross-model and cross-context reproducibility, standardization, and aggregation of evaluation results.

2. Evaluation Infrastructure, Cost, and Stakeholders

Infrastructure: Evaluation harnesses, tooling, platforms, and scalability of evaluation setups;³
Financial costs: Monetary costs of evaluations and documentation frameworks for tracking them;
Documentation: Transparency and reporting standards for evaluation processes and their limitations;
Stakeholders: Who evaluates, the relationship between evaluators and system developers, and the role of independent and third-party audits.

3. Evaluating Sociotechnical Impacts

State of sociotechnical evaluations of generative AI systems, drawing on categories such as those in the EvalEval social impact taxonomy:¹

Bias, stereotypes, and representational harms: Evaluations of stereotypes, disparate performance, inequality, marginalization, and community erasure;
Cultural values and sensitive content: Evaluations of linguistic diversity, sensitive content, trustworthiness, overreliance on outputs, and imposing norms and values;
Privacy and data protection: Evaluations of memorization, data leakage, contextual integrity, and personal privacy and sense of self;
Labor and creativity: Evaluations of data and content moderation labor, intellectual property, ownership, and labor market impacts;
Ecosystem and environment: Evaluations of environmental costs, carbon emissions, and widening resource gaps.

📄 Submission Guidelines

We welcome the following types of submissions:

Paper lengths:

📑 Full papers: 6–8 pages (excluding references and supplementary material)
📝 Short papers: Up to 4 pages (excluding references and supplementary material)
📋 Tiny papers: Up to 2 pages (excluding references and supplementary material), e.g. extended abstracts.

Submission types:

🔬 Research papers presenting original empirical or theoretical results
💡 Positions & Provocations that introduce novel perspectives or challenge conventional wisdom around broader social impact evaluation for generative AI

All submission types are welcome at any length tier. Accepted papers will be presented as posters, with a subset selected for spotlight oral presentation. All papers will be assessed based on their relevance to the workshop themes.

Papers should be submitted in the conference-provided format. The review process will be two-way anonymized; therefore, all identifying information must be removed from submissions.

📚 Publication

Submissions may include unpublished work as well as previously published or accepted papers, provided they do not violate dual-submission policies. Therefore, e.g., ACL 2026 Findings and Main papers, as well as ARR papers can be submitted.

Submissions are non-archival by default, but upon acceptance authors may opt in to archival publication. All accepted paper titles will be made publicly available.

📅 Important Dates

Submission Deadline: March 19th, 2026 ~~March 12th, 2026~~
Notification of Acceptance: April 28, 2026
Camera-ready paper due: May 14, 2026
Workshop at ACL in San Diego: July 3-4, 2026

All deadlines are specified in Anywhere on Earth (AoE).

🚀 Submission Site

All submissions to may be made via OpenReview.

🔍 Reviewer Recruitment

To support a fair, high-quality, and sustainable review process, we adopt a reciprocal reviewer recruitment expectation. Authors submitting papers to the workshop will be expected to serve as reviewers, helping ensure sufficient reviewing capacity and timely feedback for all submissions. This expectation must be clearly indicated during paper submission through the OpenReview system.

🧑‍🔬 Program Chairs

Mubashara Akhtar, ETH Zurich & ETH AI Center
Jan Batzner, Weizenbaum Institute, Technical University Munich
Leshem Choshen, MIT, IBM Research, MIT-IBM Watson AI Lab
Avijit Ghosh, Hugging Face
Usman Gohar, Iowa State University
Jennifer Mickel, Eleuther AI
Ichhya Pant, Independent
Zeerak Talat, University of Edinburgh

📬 Workshop Contact

Ichhya Pant: [email protected]
Jennifer Mickel: [email protected]
Jan Batzner: [email protected]

❓ FAQ

Can I attend this workshop online? The workshop is in-person at ACL 2026 in San Diego. At least one author of each accepted paper must present and register on-site.

How do I change from non-archival to archival? Please indicate in the form sent to all accepted authors whether you want your submission to be considered archival or non-archival.

Do I need to upload a Camera Ready if I selected non-archival? We encourage all authors to deanonymize their papers and incorporate the reviewers feedback in their revision. Nevertheless, Camera Ready instructions are tailored to archival submissions.

I’m waiting for my ARR decision — can I still submit to EvalEval? Yes! If your paper is later accepted at ACL, you would simply choose our non-archival option.

I do present already my Eval work at ACL 2026 Main or Findings, can I still submit? Yes, we are happy to have you in the room with us!

My paper does not fit into the topics listed in the Call for Papers, is it enough to propse a new benchmark? Please refer to at least one our main topic areas outlined in the Call for Papers: (1) Evaluation Methodology and Measurement Theory, (2) Evaluation Infrastructure, Cost, and Stakeholders, and (3) Evaluating Sociotechnical Impacts.

Can I also submit in the ICML format? No, please use the ARR formatting.

My position paper is 6 pages. Does that work? Yes, all submission types (research and positions/provocations) are welcome at any of the three length tiers.

What makes a good position paper? We had great success with positions at our last NeurIPS workshop! For instance, Cintaqia et al. (2025): Stop the Nonconsensual Use of Nude Images in Research — first presented at EvalEval, then a full NeurIPS position! Congratulations!

Solaiman, Talat et al. (2025). “Evaluating the Social Impact of Generative AI Systems in Systems and Society.” In The Oxford Handbook of the Foundations and Regulation of Generative AI. ↩
Reuel, Ghosh, Chim et al. (2025). “Who Evaluates AI’s Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations.” Preprint. ↩
ACL Shared Task ↩