Apertus 1.5 sets the bar for open, reproducible evaluation reporting
Congratulations to the Swiss AI Initiative on Apertus 1.5! It is one of the most open model releases to date, with weights and training data both public. Their release notes and technical report document the work in full.
Apertus is the first model developer to submit all of its evaluation results to Evaluation Cards and Hugging Face Community Evals, with enough detail for anyone to reproduce them. We have been waiting for a model developer to do this, so we’re really happy to see it happen.
What we did together
Apertus reported every evaluation for both final models in the Every Eval Ever (EEE) schema: 48 benchmarks for Apertus 1.5 8B and 47 for Apertus 1.5 70B. Each record says who ran the evaluation and how, down to the generation settings. Because the text evals ran on their public fork of lm-evaluation-harness, which EEE already has a converter for, you can rerun them yourself and see if you get the same numbers.
Those records now feed the Evaluation Cards for the 8B and 70B models, where each benchmark gets its own plot comparing Apertus with every other model reported on it. Here are two, for multilingual knowledge and instruction following, since those are the two areas the Apertus team singles out in their release notes.
Why this matters
A benchmark score on its own tells you surprisingly little. Without the setup behind it, you can’t fairly compare it with another model’s score, and you certainly can’t check it. Regulators are starting to ask for this detail too. When we mapped Evaluation Cards against AI governance requirements, the EEE schema covered 56 of the 60 evaluation-reporting elements in Article 55 of the EU AI Act and the GPAI Code of Practice.
The UK AI Security Institute already publishes its evaluations this way. Until now, though, no model developer had done it for their own release.
An open invitation to model developers
We’d love every model release to look like this. If you build models and want to report your evals the same way, here’s how to start:
- Write your results in the EEE schema. If you already use lm-evaluation-harness, the converter handles most of it. Your Eval Card picks them up from there.
- Cross-post them through Community Evals, so they appear on your Hugging Face model cards.
If you want a hand with your first submission, reach out through evalevalai.com. As for Apertus, we’re working with the team to make this the default for every release. Thank you to everyone at the Swiss AI Initiative for going first.
@misc{evaleval2026apertus,
author = {{EvalEval Coalition}},
title = {{Apertus} 1.5 Sets the Bar for Open, Reproducible Evaluation Reporting},
year = {2026},
month = oct,
howpublished = {EvalEval Coalition Blog},
url = {https://evalevalai.com/infrastructure/2026/10/15/apertus-open-reproducible-evals/}
}