Bias scan

Quantitative-qualitative bias scan: To identify potentially unfair treated groups of similar users in AI systems. Working in tandem with the qualitative doctrine of law and ethics to assess fair AI.

Bias scan tool – What is it?

This bias scan tool identifies potentially unfairly treated groups of similar users by a binary algorithmic classifier. The bias scan identifies clusters of users that face a higher misclassification rate compared to the rest of the data set. Clustering is an unsupervised ML method, so no data is needed is required on protected attributes of users. The metric by which bias is defined can be manually chosen in advance: False Negative Rate (FNR), False Positive Rate (FPR), or Accuracy (Acc).

The tool returns a report which presents the cluster with the highest bias and describes this cluster by the features that characterizes it. This is quantitatively expressed by the (statistically significant) differences in feature means between the identified cluster and the rest of the data. The report also visualizes the outcomes.

Try the tool below ⬇️

Read an example how AI experts interpret the output of the bias scan tool

Generate a report

Select a bias metric

False Positive Rate (FPR)

True labels predicted to be false (false positive), proportional to all true labels. Learn more

False Negative Rate (FNR)

False labels predicted to be true (false negatives), proportional to all false labels. Learn more

Accuracy (Acc)

Sum of true positives (TPs) and true negatives (TNs), proportional to all predictions. Learn more

Upload data

Drag & drop or browse

Release to drop files here.

Select .csv file

File loaded

Selected bias metric:

No bias scan metric selected

Scanning!

Report is downloaded :-)

The report will be downloaded when ready
(in 5-15 seconds)

If your report is not downloaded, please check the names of your 'pred_label' and 'truth_label' columns, and whether all column values are numeric values. Or contact us.

Please read below FAQ section to learn more about the inner workings of this bias scan tool.

Algorithm

The implemented bias scan tool is based on k-means Hierarchical Bias-Aware Clustering (HBAC), as described by Misztal-Radecka and Indurkya in Information Processing and Management (2021) [link]. An implementation of the HBAC algorithm can be found on Github.

Download an example data set to use the bias scan tool.

Finalist Stanford's AI Audit Competition 2023

Under the name Joint Fairness Assessment Method (JFAM) our bias scan tool has been selected as a finalist in Stanford's AI Audit Competition 2023.

Input data

What input does the bias scan tool need?
A .csv file of max. 1GB with feature columns, predicted labels by the classifier and ground truth labels. Only the name of 'pred_label' and 'truth_label' are of importance, not the naming or order of the feature columns. All column values should be numeric and unscaled.

- Features: unscaled numeric values, e.g., feat_1, feat_2, ..., feat_n;

- Predicted label: 0 or 1;

- Truth label: 0 or 1;

- Bias metric: False Positive Rate (FPR), False Negative Rate (FNR) or Accuracy.

See example...

feat_1	feat_2	...	feat_n	pred_label	truth_label
10	1	...	0.1	1	1
20	2	...	0.2	1	0
30	3	...	0.3	0	0

Metric	Description
False Positive Rate (FPR)	The bias scan tool finds the cluster for which most true labels are predicted to be false, proportional to all true labels (False Positive Rate).
False Negative Rate (FNR)	The bias scan tool finds the cluster for which most false labels are predicted to be true, proportional to all false labels (False Negative Rate).
Accuracy	Sum of True Positives (TPs) and True Negatives (TNs), proportional to all predictions.

Example reports

An example report for the BERT-based disinformation detection (FPR) case study

Download

An example report for the BERT-based disinformation detection (FNR) case study

Download

FAQ

Why this bias scan?
– No data needed on protected attributes of users (unsupervised bias detection);
– Model-agnostic (AI binary classifiers only);
– Connecting quantitative tools with qualitative methods to assess fair AI;
– Developed open-source and not-for-profit.

By whom can the bias scan be used?
The bias scan tool allows the entire ecosystem involved in auditing AI, e.g., data scientists, journalists, policy makers, public- and private auditors, to use quantitative methods to detect bias in AI systems.

What does the tool compute?
A statistical method is used to compute which clusters are relatively often misclassified by an AI system. A cluster is a group of data points sharing similar features. The tool returns a report in which identified differences (between feature means) are visualized and statistical significant feature differences are tested (Welch’s two-samples t-test for unequal variances).

The tool detects prohibited discrimination in AI?
No. The bias scan tool serves as a starting point to assess potentially unfair AI classifiers with the help of subject-matter expertise. The features of identified clusters are examined on critical links with protected grounds, and whether the measured disparities are legitimate. This is a qualitative assessment for which the context-sensitive legal doctrine provides guidelines, i.e., to assess the legitimacy of the aim pursued and whether the means of achieving that aim are appropriate and necessary.

For what type of AI does the tool work?
Currently, only binary classification algorithms can be scanned. For instance, prediction of loan approval (yes/no), disinformation detection (true/false) or disease detection (positive/negative).

What happens with my data?
Your .csv file is uploaded to a AWS bucket, where it is processed. Once the clustering algorithm is finised the data is immediately deleted.

In sum
Quantitative methods, such as unsupervised bias scans, are helpful to discover potentially unfair treated groups of similar users in AI systems in a scalable manner. Automated identification of cluster disparities in AI models allows human experts to assess observed disparities in a qualitative manner, subject to political, social and environmental traits. This two-pronged approach bridges the gap between the qualitative requirements of law and ethics, and the quantitative nature of AI (see figure). In making normative advice, on identified ethical issues publicly available, over time a repository of 'techno-ethical jurisprudence' emerges; from which data scientists and public authorities can distill best practices to build fairer AI (see our case reviews).