This paper tackles the problem of explaining data-driven system decisions from a causal perspective: if the question we seek to answer is “why did the system make a specific decision”, we can instead ask “which inputs caused the system to make its decision?” This approach is advantageous because (a) it standardizes the form that an explanation can take; (b) it does not require all features to be part of the explanation, and (c) the explanations can be separated from the specifics of the model. Thus, we define an explanation as a set of features that is weakly causal (meaning that removing the set from the instance changes the decision) and irreducible (meaning that removing any subset of the features in the explanation would not change the decision). Our work generalizes previous work in counterfactual explanations in at least four important ways: (i) we explain system decisions (which may incorporate predictions from several predictive models) rather than model predictions, (ii) we do not enforce any specific method to remove features, (iii) our explanations can deal with feature sets with arbitrary dimensionality and data types, and (iv) we allow the tailoring of explanations to domain needs by introducing costs—for example, the costs of changing the features responsible for the decision. Finally, we showcase the approach using a real data set to illustrate its advantages over feature importance weights when the goal is to understand system decisions better. Future versions of this work will incorporate an extended section of the empirical analysis in: (1) high-dimensional settings, (2) settings in which explanations are expected to be used as recommendations with associated costs, and (3) settings where the decision-making procedure draws on more than one model.