A<\/span>lgorithms are used to determine credit scores, which can mean the difference between owning a home and renting one. They are used in predictive policing, which suggests a likelihood that a crime will be committed, and in scoring how likely a criminal will reoffend. That data influences the severity of sentencing. As more decisions of greater importance are made by computer programs, the potential for harm grows. And the design of many algorithms is far from transparent, says Adam Smith<\/a>, a professor of computer science.<\/p>\n \u201cA lot of these systems are designed by private companies and their details are proprietary,\u201d says Smith, who is also a data science faculty fellow at the Rafik B. Hariri Institute for Computing and Computational Science & Engineering<\/a>. \u201cIt\u2019s hard to know what they are doing and who is responsible for the decisions they make. They are increasingly complex, and they are often hard to understand for laypeople and for the people about whom decisions are being made.\u201d<\/p>\n Recently, Smith and a joint team of BU-MIT computer scientists examined this problem, hoping to learn what, if anything, can be done to understand and minimize bias from decision-making systems that depend on computer programs.<\/p>\n The BU researchers\u2014Smith, Ran Canetti<\/a>, a professor of computer science and director of the Hariri Institute\u2019s Center for Reliable Information Systems & Cyber Security<\/a>, and Sarah Scheffler (GRS\u201923,\u201923), a computer science doctoral candidate\u2014worked with MIT PhD students to design systems whose decisions about all subsets of the population are equally accurate. They presented their work at the 2019 Association for Computing Machinery Conference on Fairness, Accountability, and Transparency<\/a>.<\/p>\n The researchers believe that a system that discriminates against people who have had a hard time establishing a credit history will perpetuate that difficulty, limiting opportunity for a subset of the population and preserving existing inequalities. What that means, they say, is that automated ranking systems can easily become self-fulfilling prophecies, whether they are ranking the likelihood of default on a mortgage or the quality of a university education.<\/p>\n \u201cOnce you\u2019ve got the same computer program making lots of decisions, any biases that exist are reproduced many times over on a larger scale,\u201d Smith says. That problem, the researchers say, will get worse as future algorithms use more outputs from past algorithms as inputs.<\/p>\n \u201cThe interaction between the algorithm and human behavior is such that if you create an algorithm and let it run, it can create a different society because humans interact with it,\u201d says Canetti. \u201cSo you have to be very careful how you design the algorithm.\u201d<\/p>\n But how exactly can an algorithm, which is basically a mathematical function, be biased?<\/p>\n Scheffler suggests two ways: \u201cOne way is with biased data. If your algorithm is based on historical data, it will soon learn that a particular institution prefers to accept men over women. Another way is that there are different accuracies on different parts of the population, so maybe an algorithm is really good at figuring out if white people deserve a loan, but it could have a high error rate for people who are not white. It could have 90 percent accuracy on one set of the population and 50 percent on another set.\u201d<\/p>\n \u201cThat\u2019s what we are looking at,\u201d says Smith. \u201cWe\u2019re asking, \u2018How is the system making mistakes?\u2019 and \u2018How are these mistakes spread across different parts of the population?\u2019\u201d<\/p>\n In May 2016, reporters from ProPublica, a nonprofit investigative newsroom, examined the accuracy of Compas, one of several algorithmic tools used by court systems to predict recidivism, or the likelihood that a criminal defendant will commit another crime. Compas gives people a risk score from 1 to 10, and those scores can be translated into probabilities: a score of 8 is the equivalent of a 70 percent chance of recidivism. Those with a risk score of 8 to 10 were considered \u201chigh risk.\u201d<\/p>\n When ProPublica researchers compared the tool\u2019s predicted risk of recidivism with actual recidivism rates over the following two years, they found that, in general, Compas got things right 61 percent of the time. They also found that predictions of violent recidivism were correct only 20 percent of the time.<\/p>\n More troubling, they found that black defendants were far more likely than white defendants to be incorrectly deemed more likely to commit crime again, and white defendants were more likely than black defendants to be incorrectly deemed a low risk to recidivate. According to ProPublica\u2019s article, this was a clear demonstration of bias by the algorithm.<\/p>\n In response, Northpointe Inc., the creator of Compas, published another study that argued that the Compas algorithm is in fact fair when measured by statistical average. Northpointe\u2019s software is widely used, and like many algorithmic tools, its calculations are proprietary, but the company did tell ProPublica that its formula for predicting who will recidivate is derived from answers to 137 questions whose answers come either from defendants or from criminal records.<\/p>\n Northpointe\u2019s study found that for each risk score, the fraction of white defendants who received a high risk score and recidivated (out of all white defendants who received a high risk score) roughly equals the fraction of black defendants who were considered high risk and recidivated, out of all black defendants who were named high risk.<\/p>\n \u201cProPublica and Northpointe came to different conclusions in their analyses of Compas\u2019 fairness. However, both of their methods were mathematically sound\u2014the opposition lay in their different definitions of fairness,\u201d Scheffler says.<\/p>\n The bottom line is that any imperfect prediction mechanism (either algorithmic or human) will be biased according to at least one of the two approaches: the error-balancing approach used by ProPublica, and the calibration method favored by Northpointe.<\/p>\n When it came to solving the problem of algorithmic bias, the BU-MIT research team created a method of identifying the subset of the population that the risk assessment system fails to judge fairly, and sending their score to a different system that is less likely to be biased. The goal is for the method to have the same accuracy on different subgroups of the population\u2014but there is no guarantee that these two systems have to work together fairly.<\/p>\n \u201cThere are many different measures of fairness,\u201d says Scheffler, \u201cand there are trade-offs between them. So to what extent are the two systems compatible with the notion of fairness we want to achieve?\u201d<\/p>\n Still, says Canetti, their research points to a possible way out of the statistical bias conundrum, one that could enable the design of algorithms that minimize the bias.<\/p>\n This work was supported by multiple National Science Foundation<\/a> awards, a Sloan Foundation Research Award, and a Clare Boothe Luce Graduate Fellowship<\/a>.<\/i><\/p>\n","protected":false},"excerpt":{"rendered":" Removing bias from the computer algorithms shaping our major life choices<\/p>\n","protected":false},"featured_media":38680,"template":"","department":[],"_links":{"self":[{"href":"https:\/\/www.bu.edu\/cas\/wp-json\/wp\/v2\/magazine-articles\/37402"}],"collection":[{"href":"https:\/\/www.bu.edu\/cas\/wp-json\/wp\/v2\/magazine-articles"}],"about":[{"href":"https:\/\/www.bu.edu\/cas\/wp-json\/wp\/v2\/types\/r_cas_magazine"}],"version-history":[{"count":19,"href":"https:\/\/www.bu.edu\/cas\/wp-json\/wp\/v2\/magazine-articles\/37402\/revisions"}],"predecessor-version":[{"id":39134,"href":"https:\/\/www.bu.edu\/cas\/wp-json\/wp\/v2\/magazine-articles\/37402\/revisions\/39134"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/cas\/wp-json\/wp\/v2\/media\/38680"}],"wp:attachment":[{"href":"https:\/\/www.bu.edu\/cas\/wp-json\/wp\/v2\/media?parent=37402"}],"wp:term":[{"taxonomy":"r_cas_department","embeddable":true,"href":"https:\/\/www.bu.edu\/cas\/wp-json\/wp\/v2\/department?post=37402"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nAdam Smith<\/span><\/div>\nThe problem of self-fulfilling predictions<\/h3>\n
The real-world impact of algorithmic bias<\/h3>\n
Overcoming algorithmic bias<\/h3>\n