G. Explanation Phase

Some problems are just too complex to explain, e.g, 20-layer neural network with 4,000 features.

It's exactly their intractability to our brains that makes them ideal for equation-generating algorithms to solve.

It is for that reason that some firms don’t care too much about explainability, anecdotally, firms like RenTech sometimes have no idea why their models are doing what they are doing.

Interpretable models (white-box) are inherently explainable, we don’t need to use methods like Permutation or Shapley value calculation to identify the feature effects.

Uninterpretable (black-box) models are not interpretable by nature, as such we need to use explainability methods like calculating Permutation importance or Shapley values.

Explainability methods seeks to close the gap in understanding between white-box and black-box models.

No matter how many explainability methods you use, they will always be estimates, and will never give you the intrinsic explanations of say linear regression models, i.e., parameter coefficients.

Importance measures are quantitative values that reflect the importance of each feature to the final predicted outcome.

Most models give you two of three values: direction, size, and significance.

Importance measures do not reflect the intrinsic predictive value of a feature and instead how important this feature is for a particular model.

If your model is more accurate, then not only are the predictions more accurate, but also the explainability outputs.

If a logistic regression has really good out-of-sample performance then the coefficients of the regression can be used to identify the most important features.

If the performance of the logistic regression model is good as measured by scores like Accuracy, F-1, or ROC (AUC) score, then we can trust the $\beta_{k}$ values more.

The coefficient values should also only be trusted when the data has been effectively normalized.

Through standardizing (subtract mean, divide by standard deviation of each feature) you effectively change the units to standard deviations above/below the mean.

So, a one standard deviation increase in $X_{1}$ corresponds to a $\beta_{1}$ increase in the log odds.

Previously we have looked at model-based importance methods like finding the coefficients of linear regression models.

Previously we have looked at model-based importance methods like finding the coefficients of linear regression models.

All model-based methods (regression parameters, split counting, and Mean Decrease Impurity) uses in-sample performance to estimate feature importance.

Model agnostic methods can both be performed in-sample and out of sample, we will start by looking at Mean Decrease Accuracy (MDA) and Single Feature Importance (SFI).

These methods can be applied to any classifier or regression model.

Model Based: we have see that there are some feature importance measure that only relates to specific models of interest, linear regression, decision trees, ensemble tree models etc.

Model Agnostic: there are also a range of model that can be applied to any model, as such they become good methods for black-box and obscure models.

These methods can either be global and only apply at the feature level (column) or local and apply to every prediction sample (row).

As a result local methods can be turned into global methods by simply averaging over all the prediction samples in the dataset.

Whereas global methods can never be used to obtain local explanations due to their design.