Please Stop Explaining Black Box Models for High Stakes Decisions, Hacker News

1Introduction

There has been an increasing trend in healthcare and criminal justice to leverage machine learning (ML) for high-stakes prediction applications that deeply impact human lives. Many of the ML models are black boxes that do not explain their predictions in a way that humans can understand. The lack of transparency and accountability of predictive models can have (and has had already) severe consequences; there have been cases of people incorrectly denied parole(Wexler,2017), poor bail decisions leading to the release of dangerous criminals, ML-based pollution models stating that dangerous situations are safe(McGough,2018)and generally poor use of limited valuable resources in criminal justice, medicine, energy reliability, finance, and in other domains(Varshney and Alemzadeh,2016).

Rather than trying to create models that are inherently interpretable, there has been a recent explosion of work on “Explainable ML,” where a second (posthoc) model is created to explain the first black Explanations are often not reliable, and can be misleading, as we discuss below.If we instead use models that are inherently interpretable, they provide their own explanations, which are faithful to what the model actually computes.

In what follows, we discuss the problems with Explainable ML, followed by the challenges in Interpretable ML. This document is mainly relevant to high-stakes decision making and troubleshooting models, which are the main two reasons one might require an interpretable or explainable model. Interpretability is a domain-specific notion(Freitas,2014;Kodratoff,1994;Huysmans et al.,2011;Rüping,2006), so there cannot be an all-purpose definition. Usually, however, an interpretable machine learning model isconstrained in model formso that it is either useful to someone, or obeys structural knowledge of the domain, such as monotonicity(eg,Gupta et al.,2016), causality, structural (generative) constraints, additivity(Lou et al.,2013), or physical constraints that come from domain knowledge. Interpretable models can perform case-based reasoning for complex domains. Often for structured data, sparsity is a useful measure of interpretability, since humans can handle at most 7±2 cognitive entities at once(Miller,1956;Cowan,2010). Sparse models allow a view of how variables interactjointlyrather than individually. We will discuss several forms of interpretable machine learning models for different applications below, but there can never be a single definition; e.g., in some domains, sparsity is useful, and in others is it not. A vast number of papers in the field of applied statistics are interpretable predictive models. There is a spectrum between fully transparent models (where we understand how all the variables arejointlyrelated to each other) and models that are lightly constrained in model form (such as models that are forced to increase as one of the variables increases, or models that, all else being equal, prefer variables that domain experts have identified as important, see(Wang et al.,2018 b)

2Key Issues with Explainable ML

A black box model is either a function that is too complicated for any human to comprehend, or a function that is proprietary; it is a model that is difficult to troubleshoot. Deep learning models, for instance, tend to be black boxes because they are highly recursive. An explanation is a separate model that is supposed to replicate most of the behavior of a black box (eg, “people who have been delinquent on current credit are more likely to default on a new loan”).

I am concerned that the field of comprehensibility in machine learning has strayed away from the needs of real problems. This field dates back to the early 90 ‘s at least(seeFreitas,2014;Holte,1993), and there are a huge number of papers on interpretable ML in various fields, which do not have the word “ interpretable “or“ explainable “in the title, as the recent papers do. Recent work on explainability (rather than interpretability) contains and perpetuates critical misconceptions that have generally gone unnoticed, but that can have a lasting negative impact on the widespread use of machine learning models in society. Let us spend some time discussing this before discussing possible solutions.

(i) It is a myth that there is necessarily a trade-off between accuracy and interpretability.

There is a widespread belief that more complex models are more accurate, meaning that a complicated black box is necessary for top predictive performance. However, this is often not true, particularly when the data naturally have a good representation. When considering problems that have structured data with meaningful features, there is often no significant difference in performance between more complex classifiers (deep neural networks, boosted decision trees, random forests) and much simpler classifiers (logistic regression, decision lists) after preprocessing. In data science problems, where structured data with meaningful features are constructed as part of the data science process, there tends to be little difference between algorithms, assuming that the data scientist follows a standard process for knowledge discovery(such as KDD, CRISP-DM, or BigData, seeFayyad et al.,1996;Chapman et al.,2000;Agrawal et al.,2012).

Even for applications such as computer vision, where deep learning has major performance gains, and where interpretability is much more difficult to define, some forms of interpretability can be imbued directly into the models without losing accuracy. This will be discussed more later in the Challenges section.

Figure1, taken from the DARPA XAI BAA, exemplifies a blind belief in the myth of the accuracy-interpretability trade-off. This not a “real” figure, in that it was not generated by any data. The axes have no quantification (there is no specific meaning to the horizontal or vertical axes). The image appears to illustrate an experiment with a static dataset, where several machine learning algorithms are applied to the same dataset. However, this kind of smooth accuracy / interpretability / explainability trade-off is atypical in data science applications with meaningful features. Even if one were to quantify the interpretability / explainability axis and aim to show that such a trade-off did exist, it is not clear what algorithms would be applied to produce this figure. (Would one actually claim it is fair to compare the 1984 decision tree algorithm CART to a 2018 deep learning model and conclude that interpretable models are not as accurate?) One can always create an artificial trade-off between accuracy and interpretability / explainability by removing parts of a more complex model to reduce accuracy, but this is not representative of the analysis one would perform on a real problem. It is also not clear why the comparison should be performed on a static dataset, because any formal process for defining knowledge from data(Fayyad et al.,1996;Chapman et al.,2000;Agrawal et al.,2012))would require an iterative process, where one refines the data processing after interpreting the results. Generally, in the practice of data science, the small difference between performance of machine learning algorithms can often be overwhelmed by the ability to interpret results and process the data better at the next iteration.

Efforts working within a knowledge discovery process led me to work in interpretable machine learning(Rudin et al.,2010). This was a large-scale effort to predict electrical grid failures across New York City. The data were messy, including free text documents (trouble tickets), accounting data about electrical cables from as far back as the 1890 ‘s, inspections data from a brand new manhole inspections program; even the structured data were not easily integrated into a database, and there were confounding issues and other problems. Algorithms on a static dataset were at most 1% different in performance, but the ability to interpret and reprocess the data led to significant improvements in performance, including correcting problems with the dataset, and revealing false assumptions about the data generation process. The most accurate predictors we found were sparse models with meaningful features that were constructed through the iterative process.

The belief that there is always a trade-off between accuracy and interpretability has led many researchers to forgo theattemptto produce an interpretable model. This problem is compounded by the fact that researchers are now trained in deep learning, but not in interpretable machine learning. Worse, toolkits of machine learning algorithms offer little in the way of useful interfaces for interpretable machine learning methods.

To our knowledge, all recent review and commentary articles on this topic imply (implicitly or explicitly) that the trade-off between interpretability and accuracy generally occurs. It could be possible that there are application domains where a complete black box is required for a high stakes decision. As of yet, I have not encountered such an application.

(ii) Explainable ML methods provide explanations that are not faithful to what the original model computes.

Explanations must be wrong. They cannot have perfect fidelity with respect to the original model. If the explanation was completely faithful to what the original model computes, the explanation would equal the original model, and one would not need the original model in the first place, only the explanation. (In other words, this is a case where the original model would be interpretable.) This leads to the danger that the explanation method can be an inaccurate representation of the original model in parts of the feature space.

An inaccurate (low-fidelity) explanation model limits trust in the explanation, and by extension, trust in the black box that it is trying to explain. An explainable model that has a 90% agreement with the original model indeed explains the original model most of the time. However, an explanation model that is correct 90% of the time is wrong 10% of the time. If a tenth of the explanations are incorrect, one cannot trust the explanations, and thus one cannot trust the original black box. If we cannot know for certain whether our explanation is correct, we cannot know whether to trust either the explanation or the original model.

A more important misconception about explanations stems from the terminology “explanation,” which is often used in a misleading way, because explanation models do not always attempt to mimic the calculations made by the original model Even an explanation model that performs almost identically to a black box model might use completely different features, and is thus not faithful to the computation of the black box. Consider a black box model for criminal recidivism prediction, where the goal is to predict whether someone will be arrested within a certain time after being released from jail / prison. Most recidivism prediction models depend explicitly on age and criminal history, but do not explicitly depend on race. Since criminal history and age are correlated with race in all of our datasets , a fairly accurate explanation model could construct a rule such as “This person is predicted to be arrested because they are black.” This might be an accurate explanation model since it correctly mimics the predictions of the original model, but it would not be faithful to what the original model computes. This is possibly the main flaw identified by criminologists(Flores et al.,2016)in the Propublica analysis(Angwin et al.,2016;Larson et al.,2016)that accused the proprietary COMPAS recidivism model of being racially biased. Recidivism prediction will be discussed more later, as it is a key application where interpretable machine learning is necessary.

An easy fix to this problem is to change terminology. Let us stop calling approximations to black box model predictionsexplanations. For a model that does not use race, an automated explanation “This model predicts you will be arrested because you are black” is not an explanation of what the model is actually doing, and would be confusing to a judge, for instance. the methods that claim to produceexplanationsinstead compute usefulsummaries of predictionsmade by the original model. Rather than producing explanations that are faithful to the original model, they show trends in how predictions are related to the features. of predictions or trends rather than explanations would not be misleading, however it is possible that this terminology also would be misconstrued.

(iii) Explanations often do not make sense, or are incomplete.

Even if both models are correct (the original black box is correct in its prediction and the explanation model is correct in its explanation), it is possible that the explanation leaves out so much information that it makes no sense. I will give an example from image processing, for a low-stakes decision (not a high-stakes decision where explanations are needed), but where explanation methods are often demonstrated. Saliency maps are often considered to be explanatory. Saliency maps can be useful to determine what part of the observation is irrelevant, but this leaves out all information about how relevant informationisbeing used. Knowing where the network is looking within the image does not tell the user what it is doing with that part of the image, as illustrated in Figure2. An unfortunate trend in recent work is to show only examples of explanations forcorrectlylabeled observations (eg, Figure2would not appear). This practice can instill a false sense of confidence in the explanation method and in the black box.

Cat

Saliency maps are only one example of where explanations are so incomplete that they might not make sense, but similar arguments can be made with other kinds of explanation methods. Sometimes it is very hard to troubleshoot a black box. For instance, only recently have researchers noticed systematic inconsistencies in random forests(Au,2018).

(iv) Black box models are often not compatible in situations where information outside the database needs to be combined with a risk assessment.

In high stakes decisions, there are often considerations outside the database that need to be combined with a risk calculation. E.g., what if the circumstances of the crime are much worse than a generic assigned charge? There are often circumstances whose knowledge could either increase or decrease someone’s risk. But if the model is a black box, it is very difficult to manually calibrate how much this additional information should raise or lower the estimated risk. This issue arises constantly; for instance, the proprietary COMPAS model used in the U.S. Justice System for recidivism risk prediction does not depend on the seriousness of the current crime(Brennan et al.,2009), the judge is supposed to somehow incorporate that manually.

This is a problem for any black box method, whether or not an explanation is present.

Next, we discuss reasons why black box models with separate explanation models would be preferred over inherently interpretable models, even for high-stakes decisions.

3Key Issues with Interpretable ML

There are many cases where explanations are preferred over interpretable models, even for high-stakes decisions. However, for most applications, I am hopeful that there are ways around some of these problems, whether they are computational problems, or problems with training of researchers and availability of code. The first problem, however, is currently a major obstacle that I see no way of avoiding other than through policy, as discussed in the next section.

(i) Corporations can make profits from the intellectual property afforded to a black box.

Companies that charge for individual predictions would find their profits obliterated if an interpretable model were used instead.

Consider the COMPAS proprietary recidivism risk prediction tool discussed above that is in widespread use in the U.S. Justice System for predicting the probability that someone will be arrested after their release(Brennan et al.,2009).

The COMPAS model is equally accurate for recidivism prediction as the very simple three rule interpretable machine learning model involving only age and number of past crimes shown in Figure3below. However, there is no business model that would suggest profiting from the simple transparent model. The simple model in Figure3is an interpretable machine learning model. Even though the model looks like a rule of thumb, it is a full-blown machine learning model. A comparison of the COMPAS and CORELS models is in Table1. Standard machine learning tools and interpretable machine learning tools seem to be approximately equally accurate for predicting recidivism, even if we define recidivism in many different waysZeng et al.,2017;Tollenaar and van der Heijden,2013), based on predictions of different types of crimes using many different machine learning methods. This evidence, however, has not changed the momentum of the justice system towards proprietary models. As of this writing, California has recently eliminated its cash bail system, instead enforcing that decisions be made by algorithms; it is unclear whether COMPAS will be the algorithm used for this, despite the fact that it is not known to be any more accurate than other methods, such as the simple CORELS model.

As a important side note, typographical errors seem to be common in computing COMPAS, which sometimes determine bail decision outcomes(Wexler,2017;Rudin et al.,2018). This, unfortunately, is a drawback of using black box models for recidivism prediction, and is a type ofprocedural unfairness, whereby two individuals who are identical might be randomly given different parole or bail decisions.

COMPAS is not a machine learning model – it was not created by any standard machine learning algorithm. It was designed by experts based on carefully designed surveys and expertise, and it does not seem to depend heavily on past criminal history(Rudin et al.,2018). Interestingly, if the model were not proprietary, its documentation(Brennan et al .,2009)indicates that it is actually an interpretable predictive model. Revealing this model, however, would be revealing a trade secret.

Let us switch examples to consider the proprietary machine learning model by BreezoMeter, used by Google during the California wildfires of 2018, which predicted air quality as “good – ideal air quality for outdoor activities,” when air quality was dangerously bad according to multiple other models(McGough,2018), and people reported their cars covered in ash. The Environmental Protection Agency’s free, vigorously-tested air quality index would have provided a reliable result(Mannshardt and Naess,2018)

. How could BreezoMeter’s machine learning method be so badly wrong and put so many in danger? We will never find out, but BreezoMeter, who has probably made a profit from making these predictions, may not have developed this new technology if its models were forced to be transparent.

In medicine, there is a trend towards blind acceptance of black box models, which will open the door for companies to sell more models to hospitals. An example of where this can go wrong is given byZech et al.(2018), who noticed that their neural network was picking up on the word “portable” within an x-ray image, rather than the medical content of the image.If they had used an interpretable model, or even an explainable model, this issue would never have gone unnoticed.Zech et al.(2018)pointed out the issue of confounding generally; in fact, the plague of confounding haunts a vast number of datasets, and particularly medical datasets. This means that proprietary models for medicine can have serious errors.

The examples of COMPAS, Breezometer, and black box medical diagnosis all illustrate a problem with the business model for machine learning. In particular, there is a conflict of responsibility in the use of black box models for high-stakes decisions:the companies that profit from these models are not necessarily responsible for the quality of individual predictions. A prisoner serving an extra sentence due to a mistake entered in an overly-complicated risk score could suffer for years, whereas the company that constructed this complicated model is unaffected. On the contrary, the fact that the model was complicated and proprietary allowed the company to profit from it. In that sense, the model’s designers are not incentivized to be careful in its design, performance, and ease of use. These are some of the same types of problems affecting the credit rating agencies who priced mortgages in 2008; that is, these are the same problems that contributed to the financial crisis in the U.S.

(ii) Interpretable models can entail significant effort to construct, in terms of both computation and domain expertise.

As discussed above, interpretability usually translates in practice to a set of application-specific constraints on the model. Solving constrained problems is generally harder than solving unconstrained problems. Domain expertise is needed to construct the definition of interpretability for the domain, and the features for machine learning. For data that are unconfounded, complete, and clean, it is much easier to use a black box machine learning method than to troubleshoot and solve computationally hard problems. However, for high-stakes decisions, analyst time and computational time should not be as expensive relative to the quality of predictions than for other applications – it is worthwhile to put the extra effort and cost into constructing a high-quality model. But even so, many organizations do not have analysts who have the training or expertise to construct interpretable models at all.

Some companies have started to provide interpretable ML solutions using proprietary software. While this is a step in the right direction, it is not clear that the proprietary software is better than publicly available software.

As discussed earlier, interpretability constraints (like sparsity) lead to optimization problems that have been proven to be computationally hard in the worst case. The theetic hardness of these problems does not mean we cannot solve them in real cases, but these optimization problems are often difficult to solve. Major improvements have been made in the last decade(eg, seeCarrizosa et al.,2010;(Su et al.) ,), and some are discussed later in the challenges section. Explanation methods, on the other hand, are usually based on derivatives, which lead to easier gradient-based optimization.

Other reasons for preferring black boxes include the belief that they “uncover” hidden patterns in the data. However, it is not clear that a transparent model would not also uncover these same patterns, but in a transparent way.

4A Concrete Proposal for Responsible ML Policy

Currently the Generalized Data Protection Regulation and other AI regulation plans govern “right to an explanation,” where only an explanation is required, not an interpretable model(Goodman and Flaxman,2016). clear whether the explanation is required to be accurate, complete, or faithful to the underlying model(Wachter et al.,2017).

Let us consider a concrete proposal moving forward. Consider a mandate that, for certain high-stakes decisions,no black box should be deployed when there exists an interpretable model with the same level of performance. Companies that produce and sell black box models could then be held accountable if an equally accurate transparent model exists. It would be considered a form of false advertising to sell a black box model if there is an equally-accurate interpretable model. The onus will then fall on companies to produce black box models only when no transparent model exists for the same task.

This suggests a change in the business model for machine learning. Opacity is viewed as essential in protecting intellectual property, but it is at odds with the requirements of many domains that involve public health or welfare. However, the combination of opacity and explainability is not the only way to incentivize machine learning experts to invest in creating such systems. Compensation for developing an interpretable model could be provided in a lump sum, and the model could be released to the public. The creator of the model would not be able to profit from licensing the model over a period of time, but the fact that the models are useful for public good applications would make these problems appeal to academics and charitable foundations.

This proposal will not solve all problems, but it could at least rule out companies selling recidivism prediction models, possibly credit scoring models, and other kinds of models where we can construct accurate interpretable models. If applied too broadly, it could reduce industrial participation in cases where it might benefit society.

As mentioned earlier, I have not yet found a high-stakes application where a fully black box model is necessary, despite having worked on many applications. As long as we continue to allow for a broad definition of interpretability that is adapted to the domain, we should be able to improve decision making for serious tasks of societal importance. However, in order for people to design interpretable models, the technology must exist to do so. As discussed earlier, there is a formidable computational hurdle in designing interpretable models, even for standard structured data with already-meanin gful features.

5Algorithmic Challenges in Interpretable ML

What if every black box machine learning model could be replaced with one that was equally accurate but also interpretable? If we could do this, we would identify flaws in our models and data that we could not see before. Perhaps we could prevent some of the poor decisions in criminal justice and medicine that are caused by problems with using black box models. We could also eliminate the need for explanations that are misleading and often wrong.

Since interpretability is domain-specific, a large toolbox of possible techniques can come in handy. Below we expand on three of the challenges for interpretable machine learning that appear often. All three cases have something in common: people have been providing interpretable predictive models for these problems for decades, and the human-designed models look just like the type of model we want to create with machine learning. I also discuss some of our current work on these well-known problems.

By no means is this set of challenges close to encompassing the large number of domain-specific challenges that exist in creating interpretable models.

Challenge # 2: Construct optimal sparse scoring systems

Scoring systems have been designed by hand since at least the Burgess criminological model of 1928(Burgess,1928). This model was designed to predict whether a criminal would violate bail, where individuals received points for being a “ne’er do well” or a “recent immigrant” that increased their predicted probability of parole violation. (Of course, this model was not created using machine learning.) A scoring system is a sparse linear model with integer coefficients – the coefficients are the point scores. An example of a scoring system for criminal recidivism is shown in Figure4, which predicts whether someone will be arrested within 3 years of release. Scoring systems are used pervasively throughout medicine; there are hundreds of scoring systems developed by physicians. Again, the challenge is whether scoring systems – which look like they could have been produced by a human – can be produced by a machine learning algorithm, and be as accurate as any other model from any other machine learning algorithm.

There are several ways to formulate the problem of producing a scoring system. For instance, we could use a special case of (1), where the model size is the number of terms in the model. (Figure4is a machine learning model with 5 terms.) Sometimes, one can round the coefficients of a logistic regression model to produce a scoring system, but that does not tend to give accurate models, and does not tend to produce models that have particularly nice coefficients (such as 1 and -1 used in Figure(4)). However, solving ((1) ) or its variants is computationally hard, because the domain over which we solve the optimization problem is the integer lattice.

The model in Figure4is from the solution to a very hard optimization problem. Let us discuss this optimization problem briefly. The goal is to find the coefficients

**b

J,
J=1 …pfor the linear predictive model
f()z)

=

∑**jBJz[computer vision and pattern recognition] JwherezJis thejth covariate of a test observationz. In Figure1, thebJ‘s are the point scores, which turned out to be 1, -1, and 0 as a result of optimization, where only the nonzero coefficients are displayed in the figure. In particular, we want to solve:

min * (b) 1, * (b) 2,..,B p ∈{10,–9,.. .,(9),10}(1)n[computer vision and pattern recognition]n∑i=1log(1 exp(– p ∑(j)=(1)bjx(i),j)) λ∑J1 ,

where the point scoresb

jare constrained to be integers between – 10 and 10, the training observations are indexed by

i=1,.

.,n, andpis the total number of covariates fo r our data. Here the model size is the number of non-zero coefficients, and againλis the trade-off parameter. The first term is the logistic loss used in logistic regression. The problem is a hard problem, specifically a mixed-integer-nonlinear program (MINLP) whose domain is also the integer lattice.

Despite the hardness of this problem, new cutting plane algorithms have been able to solve this problem to optimality (or near-optimality) for arbitrarily large sample sizes and a average number of variables within a few minutes. The latest attempt at solving this problem is the RiskSLIM (Risk-Supersparse-Linear-Integer-Models) algorithm, which is a specialized cutting plane method that adds cutting planes only whenever the solution to a linear program is integer-valued, and otherwise performs branching.(Ustun and Rudin,2017).

This optimization problem is similar to what physicians attempt to solve manually, but without writing the optimization problem out like we did above. Because physicians do not use optimization tools to do this, accurate scoring systems tend to be difficult for physicians to create themselves. One of our collaborators spent months trying to construct a scoring system himself by adding and removing variables, rounding, and using other heuristics to decide which variables to add, remove, and round. RiskSLIM was useful for helping him with this task(Ustun et al.,2017). Formally,the second challenge is to create algorithms for scoring systems that are computationally efficient.Ideally we would increase the size of the optimal scoring system problems that current methods can practically handle by an order of magnitude.

Challenge # 3 Define interpretability for specific domains and create methods accordingly, including computer vision

Since interpretability needs to be defined in a domain-specific way, some of the most important technical challenges for the future are tied to specific important domains. Let us start with computer vision, for classification of natural images. There is a vast and growing body of research on posthoc explainability of deep neural networks, but not as much work in designinginterpretable neural networks. My goal in this section is to demonstrate that even for classic domains of machine learning, where latent representations of data need to be constructed, there could exist more interpretable models that are as accurate as black box models.

For computer vision in particular, there is not a clear definition of interpretability, and the sparsity-related models discussed above do not apply – sparsity in pixel space does not make sense. There can be many different ideas of what constitutes interpretability, even between different computer vision applications. However, if we can define interpretability somehow for our particular application, we can embed this definition into our algorithm.

Again, let us define what constitutes interpretability for any domain by consideringhow people explain to each otherthe reasoning processes behind complicated visual classification tasks. As it turns out, for classification of natural images, domain experts often direct our attention to different parts of the image and explain why these parts of the image were important in their reasoning process.

The question is whether we can construct network architectures for deep learning that can also do this. The network must then make decisions by reasoning about these parts of the image so that the explanations are real, and not posthoc.

In a recent attempt to do this,Li et al.(2018);Chen et al.((a))have been building architectures that append a special prototype layer to the end of the network. During training, the prototype layer finds parts of training images that act as prototypical parts of images. For instance, for bird classification, it might pick out a prototypical head of a blue jay, prototypical feathers of a blue jay, etc. The network also learns a similarity metric between parts of images. Thus, during testing, when a new image needs to be evaluated, the network finds parts of the test image that are similar to the prototypical parts it learned during training, as shown in Figure5 *************. The final class prediction of the network is based on the weighted sum of similarities to the prototypes; this is the sum of evidence throughout the image for a particular class. The explanations given by the network are the prototypes (and the weighted similarities to them).

Training this prototype network is not as easy as training an ordinary neural network; the tricks that have been developed for regular deep learning have not yet been developed for the prototype network. However, so far these prototype networks have been trained to be as accurate as the original black box deep neural networks before the prototype layer was added.

6Conclusion

If this whitepaper can shift the focus even slightly from the basic assumption underlying most work in Explainable ML – which is that a black box is necessary for accurate predictions – we will have considered this document a success.

If this document can encourage policy makers not to accept black box models without significant attempts at interpretable (rather than explainable) models, that would be even better.

If we can make people aware of the current challenges right now in interpretable machine learning, it will allow policy-makers the mechanism to demand that more effort should be made in ensuring safety and trust in our machine learning models for high-stakes decisions.

If we do not succeed at these efforts, it is possible that black box models will continue to be permitted when it is not safe to use them. Since the definition of what constitutes a viable explanation is unclear, even strong regulations such as “right to explanation” can be undermined with less-than-satisfactory explanations. Further, there will continue to be problems calibrating black box model predictions with outside information, and continued miscalculations of black box model inputs. This may continue to lead to poor decisions throughout our criminal justice system, incorrect safety guidance for air quality disasters, inexplicable loan decisions, and other widespread societal problems.

Acknowledgments

I would like to thank Fulton Wang, Chaofan Chen, Oscar Li, Tom Dietterich, Margo Seltzer, Elaine Angelino, Nicholas Larus-Stone, Elizabeth Mannshart, Maya Gupta, and several others who helped my thought processes in various ways, and particularly Berk Ustun, Ron Parr, and my father, Stephen Rudin, who went to considerable efforts to provide thoughtful comments and discussion.

References

• Agrawal et al. (2012)D. Agrawal, P. Bernstein, E. Bertino, S. Davidson, U. Dayal, M. Franklin,…, and J. Widom.Challenges and opportunities with big data: A white paper prepared for the computing community consortium committee of the computing research association.Technical report, 2012.URLhttp://cra.org/ccc/resources/ccc-led-whitepapers/.
• Angelino et al. (2018)Elaine Angelino, Nicholas Larus-Stone, Daniel Alabi, Margo Seltzer, and Cynthia Rudin.Certifiably optimal rule lists for categorical data.Journal of Machine Learning Research, 19: 1 – 79, 2018.
• Angwin et al. (2016)Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner.Machine bias.https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing, 2016.
• Au (2018)Timothy C. Au.Random forests, decision trees, and categorical predictors: The “Absent levels” problem.Journal of Machine Learning Research, 19: 1 – 30, 2018.
• Auer et al. (1995)Peter Auer, Robert C. Holte, and Wolfgang Maass.Theory and applications of agnostic pac-learning with small decision trees.In

  Machine Learning Proceedings 1995, pages 21 – 29. Morgan Kaufmann, San Francisco (CA), 1995.

• Breiman et al. (1984)Leo Breiman, Jerome Friedman, Charles J Stone, and Richard A Olshen.Classification and regression trees.CRC press, 1984.
• Brennan et al. (2009)Tim Brennan, William Dieterich, and Beate Ehret.Evaluating the predictive validity of the COMPAS risk and needs assessment system.(Criminal Justice and Behavior), 36 (1): 21 – 40, January 2009.
• Burgess (1928)Ernest W Burgess.Factors determining success or failure on parole.Illinois Committee on Indeterminate-Sentence Law and Parole Springfield, IL, 1928.
• Carrizosa et al. (2010)Emilio Carrizosa, Belen Martín-Barragán, and Dolores Romero Morales.Binarized support vector machines.(INFORMS Journal on Computing), 22 (1): 154 – 167, 2010.
• Chapman et al. (2000)Pete Chapman, Julian Clinton, Randy Kerber, Thomas Khabaza, Thomas Reinartz, Colin Shearer, and Rüdiger Wirth.CRISP-DM 1.0 – step-by-step data mining guide.Technical report, SPSS, 2000.
• Chen and Rudin (2018)Chaofan Chen and Cynthia Rudin.An optimization approach to learning falling rule lists.In

  Artificial Intelligence and Statistics (AISTATS), 2018.

• Chen et al. (2018 a)Chaofan Chen, Oscar Li, Chaofan Tao, Alina Barnett, Jonathan Su, and Cynthia Rudin.This

  looks like

  that: Deep learning for interpretable image recognition.ArXiv e-prints, June 2018 a.

• Chen et al. (2018 b)Chaofan Chen, Kancheng Lin, Cynthia Rudin, Yaron Shaposhnik, Sijia Wang, and Tong Wang.An interpretable model with globally consistent explanations for credit risk.In

  NIPS 2018 Workshop on Challenges and Opportunities for AI in Financial Services: the Impact of Fairness, Explainability, Accuracy, and Privacy, 2018 b.

• Cowan (2010)Nelson Cowan.The magical mystery four how is working memory capacity limited, and why?Current trends in psychological science, 19 (1): 51 – 57, 2010.
• Fayyad et al. (1996)Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth.From data mining to knowledge discovery in databases.AI Magazine, 17: 37 – 54, 1996.
• Flores et al. (2016)Anthony W. Flores, Christopher T. Lowenkamp, ​​and Kristin Bechtel.False positives, false negatives, and false analyzes: A rejoinder to “Machine bias: There’s software used across the country to predict future criminals. “Federal probation