What does the tool do?
The tool helps find groups where an AI system or algorithm performs differently, which could indicate unfair treatment. It does this using a technique called clustering, which groups similar data points together (in a cluster). The tool doesn’t need information like gender, nationality, or ethnicity to find these patterns. Instead, it uses a bias score to measure deviations in the performace of the system, which you can choose based on your data.
What results does it give?
The tool finds groups (clusters) where performance of the algorithmic system is significantly deviating. It highlights the group with the worst bias score and creates a report called a bias analysis report, which you can download as a PDF. You can also download all the identified groups (clusters) in a .json file. Additionally, the tool provides visual summaries of the results, helping experts dive deeper into the identified deviations. Example below ⬇️.
What kind of data does it work with?
The tool works with data in a table format, consisting solely of numbers or categories. You just need to pick one column in the data to use as the bias score. This column should have numbers only, and you’ll specify whether a higher or lower number is better. For example, if you’re looking at error rates, lower numbers are better. For accuracy, higher numbers are better. The tool also comes with a demo dataset you can use by clicking “Try it out.”
Example of numerical data set:
| Age | Income | ... | Number of cars | Selected for investigation |
|---|---|---|---|---|
| 35 | 55.000 | ... | 1 | 1 |
| 40 | 45.000 | ... | 0 | 0 |
| ... | ... | ... | ... | ... |
| 20 | 30.000 | ... | 0 | 0 |
Is my data safe?
Yes! Your data stays on your computer and never leaves your organization’s environment. The tool runs directly in your browser, using your computer’s power to analyze the data. This setup, called ’local-only’, ensures no data is sent to cloud providers or third parties. Instructions for hosting the tool securely within your organization are available on Github.
Try the tool below ⬇️
Which steps does the tool undertake?
The unsupervised bias detection tool operates a series of steps:
Prepared by the user:
1. Dataset: The data must be provided in a tabular format. All columns, except the bias score column, should have uniform data types, e.g., either all numerical or all categorical. The bias score column must be numerical. Any missing values should be removed or replaced. The dataset should then be divided into training and testing subset, following a 80-20 ratio.
2. bias score: The user selects one column from the dataset to serve as the bias score. In step 3, clustering will be performed based on this chosen bias score. The chosen bias score must be numerical. Examples include metrics such as “being classified as high risk”, “error rate” or “selected for an investigation”.
Performed by the tool:
3. Hierarchical Bias-Aware Clustering (HBAC): The HBAC algorithm (detailed below) is applied to the training dataset. The centroids of the resulting clusters are saved and later used to assign cluster labels to data points in the test dataset.
4. Testing differences in bias score: Statistical hypothesis testing is performed to evaluate whether the most deviating cluster contains significantly more bias compared to the rest of the dataset. A two-sample t-test is used to compare the bias scores between clusters. For multiple hypothesis testing, Bonferonni correction should be applied. Further details can are available in our scientific paper.
A schematic overview of the above steps is depicted below.
How does the clustering algorithm work?
The Hierarchical Bias-Aware Clustering (HBAC) algorithm identifies clusters in the provided dataset based on a user-defined bias score. The objective is to find clusters with low variation in the bias score within each cluster and significant variation between clusters. HBAC iteratively finds clusters in the data using k-means (for numerical data) or k-modes clustering (for categorical data). For the initial split, HBAC takes the full dataset and splits it in two clusters. Cluster C – with the highest standard deviation of the bias score – is selected. Then, cluster C is divided into two candidate clusters C' and C''’. If the average bias score in either candidate cluster exceed the the average bias score in C, the candidate cluster with highest bias score is selected as a new cluster. This process repeats until the maximum number of iterations (max_iterations) is reached or the resulting cluster fails to meet the minimum size requirement (n_min). The pseudo-code of the HBAC algorithm is provided below.
The HBAC-algorithm is introduced by Misztal-Radecka and Indurkya in a scientific article as published in Information Processing and Management in 2021. Our implementation of the HBAC-algorithm advances this implementation by proposing additional methodological checks to distinguish real bias from noise, such as sample splitting, statistical hypothesis testing and measuring cluster stability. Algorithm Audit’s implementation of the algorithm can be found in the unsupervised-bias-detection pip package.
How should the results of the tool be interpreted?
The HBAC algorithm maximizes the difference in the bias score between clusters. To prevent incorrect conclusions that there is bias in the decision-making process under review when there truly is none, we split the dataset in training and test data, and hypothesis testing prevents us from (wrongly) concluding that there is a difference in the bias score while there is none. If statistically significant bias is detected, the outcome of the tool serves as a starting point for human experts to assess potential discrimination in the decision-making processes.
Web app – Unsupervised bias detection tool
The source code of the anolamy detection-algorithm is available on Github and as a pip package:
pip install unsupervised-bias-detection.The architecture to run web apps local-only is also available on Github.
The unsupervised bias detection tool has been applied in practice to audit a Dutch public sector risk profiling algorithm. Our team documented this use case in a scientific paper. The tool identified proxies for students with a non-European migration background in the risk profiling algorithm, specifically education level and distance between the student’s address and their parent(s)’ address. The results are also described in Appendix A of the below report. This report was sent to Dutch parliament on 22-05-2024.
What is local-only?
local-only computing is the opposite of cloud computing: the data is not uploaded to third-parties, such as a cloud providers, but is processed by your own computer. The data attached to the tool therefore doesn’t leave your computer or the environment of your organization. The tool is privacy-friendly because the data can be processed within the mandate of your organisation and doesn’t need to be shared with new parties. The unsupervised bias detection tool can also be hosted locally within your organization. Instructions, including the source code or the web app, can be found on Github.
Overview of local-only architecture
This tool is developed with support of public and philanthropic organisations.
Innovation grant Dutch Ministry of the Interior
Description
In partnership with the Dutch Executive Agency for Education and the Dutch Ministry of the Interior, Algorithm Audit has been developing and testing this tool from July 2024 to July 2025, supported by an Innovation grant from the annual competition hosted by the Dutch Ministry of the Interior. Project progress was shared at a community gathering on 13-02-2025.
SIDN Fund
Description
In 2024, the SIDN Fund supported Algorithm Audit to develop a first demo of the unsupervised bias detection tool.
This tool has received awards and is acknowledged by various stakeholders, including civil society organisations, industry representatives and academics.
Finalist Stanford’s AI Audit Challenge 2023
Description
Under the name Joint Fairness Assessment Method (JFAM) the unsupervised bias detection tool has been selected as a finalist in Stanford’s AI Audit Competition 2023.
OECD Catalogue of Tools & Metrics for Trustworthy AI
Description
The unsupervised bias detection tool is part of OECD’s Catalogue of Tools & Metrics for Trustworthy AI.
Key take-aways about unsupervised bias detection tool:
- Quantitative-qualitative research method: Data-driven bias testing combined with the balanced and context-sensitive judgment of human experts;
- Unsupervised bias detection: No user data needed on protected attributes (unsupervised learning);
- Anolamy detection: Scalable method based on statistical analysis;
- Detects complex bias: Identifies unfairly treated groups characterized by mixture of features, detects intersectional bias;
- Model-agnostic: Works for all binary classification algorithms and AI systems;
- Open-source and not-for-profit: User friendly and free to use for the entire AI auditing community.
Team
Floris Holstege
PhD-candidate Machine Learning, University of Amsterdam
Joel Persson PhD
Research Scientist, Spotify
Jurriaan Parie
Director, Algorithm Audit
Kirtan Padh
PhD-candidate Causal Inference and Machine Learning, TU München
Krsto Proroković
Freelance software developer and AI researcher
Mackenzie Jorgensen PhD
Researcher Alan Turing Institute, London
