From Soft Classifiers to Hard Decisions: How Fair Can We Be?

As algorithms are increasingly used to make important decisions in the lives of individuals, computer scientists have begun to investigate how to ensure that these decisions are made in a "fair" way. The field of algorithmic fairness seeks to analyze the process or results of classification mechanisms by investigating the way those mechanisms make mistakes, specifically if those mistakes are concentrated in a specific demographic of the population. However, defining this "fair" error balance is challenging, and in many cases, different notions of fairness are inconsistent or even incompatible with each other.

In this Cyber Alliance talk, BU Computer Science PhD student Sarah Scheffler will discuss how her work with her CS colleagues (Ran Canetti, Aloni Cohen, Nishanth Dikkala, Govind Ramnarayan, and Adam Smith) investigates how the fairness of a system changes when a transition is made from a soft "score" to a hard yes/no decision. Oftentimes, soft scoring algorithms are used as step one out of two: first, assign an individual a score based on information about them. Then, use that score along with a threshold to make a yes/no decision. Even starting from a scoring method that scores individuals from different demographics in a consistent "fair" way, the resulting yes/no decision may still be unfair. Their work analyzes several different methods of making these hard decisions, both in settings where the protected class membership is known and when it is unknown.

Ms. Scheffler will detail how they also investigate the possibility of allowing the algorithm to "defer" on some decisions, refusing to answer and waiting for a separate process to judge some individuals. When deferring, they show several methods for achieving fairness under multiple definitions which are usually incompatible with each other. This "concentrates" the unfairness inherent to a decision into the decision of whether not to defer, and allows a statistically "fair" decision to be made on the resulting non-deferred individuals.

There will be time for casual conversation and light refreshments before and after the presentation. Please RSVP to tgabs@bu.edu.