Description

As GPAI models continue to improve, systems are increasingly touted as matching or exceeding expert performance in “problem solving”, “scientific reasoning” or “real-world software engineering.” These claims are based on common tests for different capabilities, known as “benchmarks”.

Yet, there is often misconception of what benchmarks actually measure, what conclusions about a system we can actually make and how to evaluate what actually matters for a use case. Just as other technology-driven industries, like healthcare and aviation, AI benchmarking needs reliable standards. With best practices in their infancy, it can be daunting for professionals to navigate GPAI evaluation.

In this masterclass, we distill and make accessible the most valuable insights from the field of GPAI benchmarking. The course covers:

• GPAI under the AI Act and benchmarking for systems with systemic risk
• Prominent industry benchmarks and adaptations to GPT-NL
• Insights from Algorithm Audit’s own benchmarking work
• Current issues in the field from a scientific perspective
• The design aspects that determine the quality of a benchmark

After attending, participants will be able to:

• Understand relevant obligations for GPAI systems under the AI Act
• Grasp how benchmarks tend to work “under the hood”
• Navigate common evaluation repositories and documentation
• Assess the suitability of existing benchmarks for their own work
• Reason when a custom approach is needed and how it may look

Date

3 November 2026

Address

The Hague Conference Centre (New Babylon), Anna van Buerenplein 29, 2595 DA Den Haag

Programme

• 09:30-10:00 Doors open
• 10:00-11:15 Introduction General Purpose AI (GPAI) and testing GPAI applications
• 11:15-11:30 Break – coffee, tea and refreshments
• 11:30-12:30 State-of-the-art concepts and developments in GPAI benchmarking
• 12:30-13:30 Lunch – catered
• 13:30-14:45 Case study from Dutch public sector
• 14:45-15:15 Break – coffee, tea and refreshments
• 15:15-16:30 Hands-on exercises to build practical experience
• 16:30-17:30 Drinks – catered

Fee

• €300 in-person participation (including, lunch, drinks, refreshments)

Audience

Professionals from private and public sector who regularly work with GPAI applications, such as implementation of generative AI solutions in work processes, testing GPAI capabilities and/or working on AI policy.