Bias detection tool
What is the tool about?
The tool identifies potentially unfairly treated groups of similar users by an AI system. The tool returns clusters of users for which the system is underperforming compared to the rest of the data set. The tool makes use of clustering – an unsupervised statistal learning method. This means that no data are required on protected attributes of users, e.g., gender, nationality or ethnicity, to detect indirect discrimination, also referred to as higher-dimensional proxy or intersectional discrimination. The metric by which bias is defined can be manually chosen and is referred to as the performance metric.
What data can be processed?
Numerical and categorical data can be analysed. The type of data is automatically detected by the tool. The performance metric column should always contain numerical values. The user should indicate in the app whether a higher of lower value of the performance metric is considered to be better.
The tool contains a demo data set and a ‘Try it out’ button. More information can be found in the app.
Example of numerical data set:
| feat_1 | feat_2 | ... | feat_n | perf_metr |
|---|---|---|---|---|
| 10 | 1 | ... | 0.1 | 1 |
| 20 | 2 | ... | 0.2 | 1 |
| 30 | 3 | ... | 0.3 | 0 |
How is my data processed?
The tool is privacy preserving. It uses computing power of your own computer to analyze the attached data set. In this architectural setup, data is processed entirely on your device and it not uploaded to any third-party, such as cloud providers. This computing approach is called local-first and allows organisations to securely use tools locally. Instructions how the tool can be hosted locally, incl. source code, can be found here.
unsupervised-bias-detection.
What does the tool return?
The tool returns a pdf report or .json file with identified clusters. It specifically focusses on the identified cluster with highest bias and describes this cluster by the features that characterizes it. These results serve as a starting point for a deliberative assessment by human experts to evaluate potential discrimination and unfairness in the AI system under review. The tool also visualizes the outcomes.
Try the tool below ⬇️
Bias detection tool
Example output bias detection tool
Higher-dimensional proxy bias
Application of the tool in a real-world audit. Comparing supervised and unsupervised bias testing methods for establishing indirect discrimination.
Review of disinformation classifier by human experts
An advice commission believes there is a low risk of (higher-dimensional) proxy discrimination by the BERT-based disinformation classifier
Higher-dimensional proxy bias
Application of the tool in a real-world audit. Comparing supervised and unsupervised bias testing methods for establishing indirect discrimination.
Review of disinformation classifier by human experts
An advice commission believes there is a low risk of (higher-dimensional) proxy discrimination by the BERT-based disinformation classifier
Finalist Stanford’s AI Audit Challenge 2023
Under the name Joint Fairness Assessment Method (JFAM) our bias scan tool has been selected as a finalist in Stanford’s AI Audit Competition 2023.
OECD Catalogue of Tools & Metrics for Trustworthy AI
Algorithm Audit’s bias detection tool is part of OECD’s Catalogue of Tools & Metrics for Trustworthy AI.
Hierarchical Bias-Aware Clustering (HBAC) algorithm
The bias detection tool utilizes the Hierarchical Bias-Aware Clustering (HBAC) algorithm. HBAC processes input data according to the k-means (for numerical data) or k-modes (for categorical data) clustering algorithm. The HBAC-algorithm is introduced by Misztal-Radecka and Indurkya in a scientific article as published in Information Processing and Management (2021). Our implementation of the HBAC-algorithm can be found on Github. The methodology has been reviewed by a team of machine learning engineers and statisticians, and is continuously undergoing evaluation.
FAQ
Why this bias detection tool?
- Quantitative-qualitative joint method: Data-driven bias testing combined with the balanced and context-sensitive judgment of human experts;
- Unsupervised bias detection: No user data needed on protected attributes;
- Bias scan tool: Scalable method based on statistical learning to detect algorithmic bias;
- Detects complex bias: Identifies unfairly treated groups characterized by mixture of features, detects intersectional bias;
- Model-agnostic: Works for all AI systems;
- Open-source and not-for-profit: Easy to use and available for the entire AI auditing community.
By whom can the bias detection tool be used?
The bias detection tool allows the entire ecosystem involved in auditing AI, e.g., data scientists, journalists, policy makers, public- and private auditors, to use quantitative methods to detect bias in AI systems.
What does the tool compute?
A statistical method is used to compute for which clusters an AI system underperforms. A cluster is a group of data points sharing similar features. On these features the AI system is initially trained. The tool identifies and visualizes the found clusters automatically. The tool also assesses how individuals in a deviating cluster differ (in terms of the provided features) from other data points outside the cluster. The differences between these clusters are tested on statistical significance. All results can directly be downloaded as a pdf file.
The tool detects prohibited discrimination in AI?
No. The bias detection tool serves as a starting point to assess potentially unfair AI classifiers with the help of subject-matter expertise. The features of identified clusters are examined on critical links with protected grounds, and whether the measured disparities are legitimate. This is a qualitative assessment for which the context-sensitive legal doctrine provides guidelines, i.e., to assess the legitimacy of the aim pursued and whether the means of achieving that aim are appropriate and necessary. In a case study of Algorithm Audit – in which the bias detection tool was tested on a BERT-based disinformation classifier – a normative advice commission argued that the measured quantitative deviations could be legitimised. Legitimisation of unequal treatment is a context-sensitive taks for which legal frameworks exist, such an assessment of proportionality, necessity and suitability. This qualitative judgement will always be a human task.
How is my data processed?
The tool is privacy preserving. It uses computing power of your own computer to analyze a dataset. In this architectural setup, data is processed entirely on your device and it not uploaded to any third party, such as cloud providers. This local-only feature allows organisations to securely use the tool with proprietary data. The used software is also available as pip package unsupervised-bias-detection.
In sum
Quantitative methods, such as unsupervised bias detection, are helpful to discover potentially unfair treated groups of similar users in AI systems in a scalable manner. Automated identification of cluster disparities in AI models allows human experts to assess observed disparities in a qualitative manner, subject to political, social and environmental traits. This two-pronged approach bridges the gap between the qualitative requirements of law and ethics, and the quantitative nature of AI (see figure). In making normative advice, on identified ethical issues publicly available, over time a repository of case reviews emerges. We call case-based normative advice for ethical algorithm algoprudence. Data scientists and public authorities can learn from our algoprudence and can criticise it, as ultimately normative decisions regarding fair AI should be made within democratic sight.
Read more about algoprudence and how Algorithm Audit’s builds it.
Bias detection tool team
Floris Holstege
PhD-candidate Machine Learning, University of Amsterdam
Joel Persson PhD
Research Scientist, Spotify
Kirtan Padh
PhD-candidate Causal Inference and Machine Learning, TU München
Krsto Proroković
PhD-candidate, Swiss AI Lab IDSIA
Mackenzie Jorgensen
PhD-candidate Computer Science, King’s College London
