This paper explains the potential biases that may arise throughout the lifecycle of artificial intelligence systems, taking an iterative approach to the processes of data generation, model development, and deployment.
We provide a definition of the typical issues associated with each stage and a specific mitigation strategy.
“The data is biased” is a common saying used to justify undesirable behaviours that arise from artificial intelligence systems. While this statement is not false, taking it out of context can lead to the idea of treating data as a static artifact divorced from the process that produced it.
The purpose of artificial intelligence models is to be able to generalize based on what they have already “learned” from existing data. However, problems can arise not only from the collection of data, but also from every stage of these systems’ lifecycle.
Figure 1: The life cycle of AI systems. This encompasses the processes of data generation, model development, and model deployment (adapted image). It is not a linear process, but rather a cyclical and continuous one, because in each iteration the models end up affecting the state of the world and, consequently, the data used to train them
Bias types across the ML lifecycle
| Bias Type | Lifecycle Phase | Definition | Mitigation Strategy |
| Historical Bias | Data generation | Risk of harming a specific group when models are trained on data that accurately reflect an unequal world. | Adjust label and feature distributions through systematic over- or under-sampling, while also addressing root causes such as label suitability and prediction objectives. |
| Representation Bias | Data generation | Occurs when the target population is defined in a way that does not adequately represent the use population, when certain groups are underrepresented, or when the sampling method is limited or uneven. | Redefine the target population or adjust sampling to include missing or underrepresented groups, ensuring labels and characteristics are accurate. |
| Measurement Bias | Data definition | Occurs when reatures o labels used like proxies to approximate reality are oversimplifications or when measurement accuracy is inconsistent across groups. | Use thoughtful, context-aware annotation processes involving domain experts to select proxies that more accurately reflect reality. |
| Aggregation Bias | Model definition | A single model is applied to subgroups that should be treated differently, assuming the mapping between inputs and labels is the same for everyone. | Tune the model to capture data complexity, or transform training data so the relationship between variables and labels is consistent across groups. |
| Learning Bias | Model learning | Modeling choices amplify performance disparities across different data samples. | Review optimization tasks by incorporating fairness metrics and understanding which variables are prioritized during training. |
| Evaluation Bias | Model evaluation | Benchmarks used to evaluate models do not represent the target population, producing statistically invalid performance generalizations. | Calculate metrics by group on granular data subsets and improve or replace benchmarks to reflect real improvements for the target user population. |
| Deployment Bias | Model deployment | Discrepancy between the problem the model is designed to solve and the way it is actually used. | Design systems that help users gauge confidence in predictions; use interpretable models and user-oriented interfaces combining multiple information sources. |
Author of the paper: Harini Suresh, John Guttag
Publisher or journal of publication: arXiv, Association for Computing Machinery (ACM) Digital Libray, MIT Library
The paper is available at the following link.
