Every Eval Ever
EvalEval Coalition — “We are a researcher community developing scientifically grounded research outputs and robust deployment infrastructure for broader impact evaluations.”
Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused. The three components that make it work:
- 📋 A metadata schema (eval.schema.json) that defines the information needed for meaningful comparison of evaluation results, including instance-level data
- 🔧 Validation that checks data against the schema before it enters the repository
- 🔌 Converters for Inspect AI, HELM, and lm-eval-harness, so you can transform your existing evaluation logs into the standard format
Project Components
Every Eval Ever is maintained across three connected components:
- GitHub repository: the
every_eval_everPython package with schema definitions, converters/adapters, tests, and core tooling. - EEE Datastore: the Hugging Face datastore that stores normalized Every Eval Ever evaluation data.
- EEE Validator: validator and EvalEvalBot checks used on datastore pull requests, built from repository logic plus additional checks that are being upstreamed.
Install the package:
pip install every-eval-ever
Optional converter dependencies:
pip install 'every-eval-ever[inspect]'
pip install 'every-eval-ever[helm]'
pip install 'every-eval-ever[all]'