One common way of assessing feature importance is based on simulating lack of knowledge about features (Lemaire, Féraud and Voisine, 2008; Robnik-Šikonja and Kononenko, 2008). For instance, one could compare the original model’s output with the output obtained when removing a specific feature. If the output changes, it means that the feature was important for the model prediction. Methods that use this approach often decompose each prediction into the individual contributions of each feature and use the decompositions as explanations, thus enabling the visualization of each instance-decision explanation separately. A notable challenge, however, is that interactions between features may lead to ambiguous explanations because the order in which features are removed may affect the importance attributed to each feature. Therefore, subsequent work proposed to assess feature importance by removing all possible subsets of features (rather than only one feature at a time), retraining models without the removed features, and comparing how predictions change (Štrumbelj, Kononenko and Robnik-Šikonja, 2009). However, such approaches may take hours of computation time and have been reported to handle only up to about 200 features. Alternative formulations have attempted to reduce computation time by sampling the space of feature combinations and by using imputation to deal with removed features, resulting in sampling-based approximations of the influence of each feature on the prediction (Strumbelj and Kononenko, 2010; Ribeiro, Singh and Guestrin, 2016; Datta, Sen and Zick, 2017; Lundberg and Lee, 2017).