Automated systems trained with real-world data have a tendency to discriminate against disadvantaged groups. For instance, an algorithm adopted by the U.K. government downgraded hundreds of thousands of students’ grades, disproportionately impacting those from tuition-free schools. One approach to alleviating the bias issue is imposing “fairness constraints” such that certain statistical measures are roughly equalized across groups. But the long-term effect of these constraints is poorly studied.
In an effort to shed light on the subject, researchers affiliated with the University of Michigan; KTH Royal Institute of Technology; the University of California, Santa Cruz; Carnegie Mellon University; and Microsoft Research coauthored a paper analyzing the impact of fairness constraints on the “equality” and “improvement” of a group’s well-being. They say the work is applicable to a range of domains, including recruitment and bank lending, where features like a person’s credit score are considered against their ability to repay a loan.
The researchers looked at the equilibrium of qualification rates in different groups under a general class of fairness constraints. Using a framework called Partially Observed Markov Decision Process (POMDP), they modeled sequential decision-making — i.e., a bank approving a loan — in different scenarios. POMDP accounts for the fact that people who experience a positive outcome — receiving a high test score, for instance — might feel less motivated to maintain this score or strive for better (the “lack of motivation” effect). However, the framework also allows that a person might gain access to better resources as a result of the outcome or feel inspired to achieve greater (the “leg-up” effect).
According to the researchers, in rare situations where there’s “natural equality” between groups, applying a fairness constraint can severely disrupt the equality. Indeed, imposing fairness only helps when the “leg-up” effect is more prominent than the “lack of motivation” effect; when the “lack of motivation” effect is dominant, imposing fairness should ideally be accompanied by other measures to dampen this effect.
“Imposing static fairness constraints is not always a valid intervention in terms of its long-term impact. In some cases it reinforces existing disparity; even when it could work, doing it right can be very hard due to its sensitivity to problem parameters,” the researchers wrote in a paper accepted by the NeurIPS 2020 machine learning conference. “Our findings show that the same fairness constraint can have opposite impact depending on the underlying problem scenarios, which highlights the importance of understanding real-world dynamics in decision-making systems.”
While the study doesn’t touch on the topics of transparency or explainability, many experts believe they’re a key to less prejudicial decision-making systems. In an article published in the Harvard Journal of Law and Technology in 2017, the coauthors argued that algorithms should offer people “counterfactual explanations,” disclose how they came to their decision, and suggest changes “that can be made to obtain a desirable outcome.” Some sources of data used to train algorithms will always contain more imbalances than others, of course, like datasets containing faces with labels for gender and race. But then the question becomes one of ethics rather than mathematics: understanding whether certain data should be applied to decision-making, considering the problems that might arise.
