2026 Microconference on Embedding Evaluators in Practice

📌 Key Details

When: Saturday, October 24, 2026 · 9:00 AM – 6:00 PM (PT)
Where: San Francisco, California (Exact location to be shared with confirmed attendees)
Format: Hybrid (in-person and remote)
Hosted by: EvalEval

Attendance is by invitation only. Discussions follow the Chatham House Rule.

📖 Background

Embedded evaluators are commonplace in public health¹, education², banking, and nuclear energy, and interest in applying this approach to AI evaluation is growing. The AI Evaluator Forum’s open letter calling for the Minimum Conditions for Embedding Evaluators has drawn more than 100 signatories, while Anthropic³ and OpenAI⁴ have committed to giving external evaluators sustained, employee-level access to internal systems. These developments raise a natural question. What should embedded AI evaluation look like in practice?

This microconference brings together third-party evaluators, model developers, policymakers, and civil society organizations to examine the conditions that make embedded AI evaluation independent, rigorous, and consequential. We look at which entities should host embedded evaluators, what access they need to internal systems such as training data, model weights, and APIs, and how that access is sustained. We also examine governance, including how evaluators are funded, selected, and validated, who sets their scope, and what protects their ability to report unfavorable findings while safeguarding sensitive information. Finally, we ask how findings translate into action and who is responsible for follow-through.

🎯 Goals

Clarify scope and assumptions. Identify which domains, systems, risks, and decisions embedded evaluations address across model, application, and downstream dependency layers, and where their limitations lie.
Identify conditions for independence and rigor. Examine evaluator selection, funding, sustained access, methods, transparency, and protections for evaluators and employees.
Connect findings to action. Explore how findings inform deployment decisions, incident response, and public accountability, including responsibility for follow-through.
Map agreement and unresolved tensions. Examine differences across stakeholder groups, how embedded evaluation could favor larger, better-resourced organizations, and how non-embedded findings retain salience.
Define collaborative next steps. Identify research questions, practical experiments, and implementation priorities, including a possible pilot embedded evaluation effort.

🗓️ Schedule 24/October/2026

👋 9:00 AM – 9:45 AM | Opening Remarks: Why We’re Here (45 mins)

Introduces the central question of what conditions make embedded frontier-AI evaluation independent, rigorous, and consequential, and sets the goal of a shared agenda for action.

🎙️ 9:45 AM – 10:30 AM | Evaluators’ Perspectives Panel (45 mins)

Third-party evaluators and academic researchers discuss independence, methodological rigor, access, resources, and reporting. What should embedded evaluation examine, and what can its findings credibly establish?

💬 10:30 AM – 11:00 AM | Breakout: Conditions for Trustworthy Evaluation (30 mins)

Participants identify practical conditions for independence and rigor, including evaluator selection, funding, scope, and publication rights.

🎙️ 11:00 AM – 11:45 AM | Impact Perspectives Panel (45 mins)

Panelists examine how evaluation findings inform governance and protect the public, what enables escalation, remediation, and accountability, and how embedded evaluation connects to broader oversight and downstream defenses.

💬 11:45 AM – 12:30 PM | Breakout: From Findings to Action (45 mins)

Participants map the path from a significant finding to a decision, response, and verification of follow-through, and identify gaps in authority, responsibility, and communication.

🍽️ 12:30 PM – 1:30 PM | LUNCH (60 mins)

🎙️ 1:30 PM – 2:15 PM | Developers’ Perspectives Panel (45 mins)

Panelists discuss how embedded evaluation works within development and deployment processes across a range of developers, including experiment review, monitoring, incident response, transparency, and protections for employees and evaluators.

💬 2:15 PM – 2:45 PM | Breakout: Making Embedded Evaluations Work in Practice (30 mins)

Participants identify implementation challenges and options for addressing them, including access, confidentiality, timelines, and coordination across research, security, legal, and governance teams.

⚖️ 2:45 PM – 3:45 PM | Tensions Breakout I: Clarifying Tradeoffs (60 mins)

Groups examine tensions such as access and independence, transparency and confidentiality, or evaluation scope and available resources, and how non-embedded evaluators retain credibility when embedded evaluators have privileged access.

⚖️ 3:45 PM – 4:45 PM | Tensions Breakout II: Developing Options (60 mins)

Groups compare possible approaches and identify where common ground is possible, where different arrangements fit different contexts, and where disagreements remain.

🧭 4:45 PM – 6:00 PM | Group Synthesis and Report-Out (75 mins)

Groups consolidate shared principles, disagreements, open questions, and possible recommendations (45 mins), then report back and identify priorities (30 mins).

📦 Intended Outputs

A set of proposed conditions for independent, rigorous, and consequential embedded evaluation
A record of substantive agreements, disagreements, and unresolved questions
A shared agenda for research, practical experimentation, and implementation
Concrete opportunities for continued cross-sector collaboration, including a pilot effort

🧑‍🔬 Organizers

Ichhya Pant, EvalEval Strategy and Community Engagement Co-Chair, Independent Researcher
Jennifer Mickel, EvalEval Strategy and Community Engagement Co-Chair, UC Berkeley
Sachin Pendse, UCSF (Host)
Avijit Ghosh, EvalEval Co-Founder, Hugging Face
Usman Gohar, EvalEval Public Relations Chair, Iowa State University
Jan Batzner, EvalEval Research Chair, TU Munich
Michelle Lin, EvalEval Funding Co-Chair, Mila
Wm. Matthew Kennedy, EvalEval Funding Co-Chair, AI+ Senior Research Fellow, King’s College London
Aspen Hopkins, MIT CSAIL
Kevin Wei, EvalEval Partnerships Chair, GovAI

📬 Contact

This event is invite-only. For questions, please contact Ichhya Pant.