Local Interpretability of Models

Once trained an machine learning model, it can be deployed for prediction on new data. Sometimes, we care about not only the predictive values, but also how & why the prediction results are made.

SHAP

Based on the concept of Shapley Values in game theory, Shapley Additive Explanations(SHAP) has been proposed to explain each individual prediction. SHAP is adopted in SAP HANA PAL(so, also in hana_ml.algorithms.pal package) for the local interpretability of models on prediction data.

Generally speaking, the exact computation of SHAP values has exponential time complexity, it is prohibited in most cases. When it comes to specific machine learning models, variant versions of algorithms that can compute SHAP values approximately are implemented.

• For tree-based machine learning models, a tree SHAP explainer(tree SHAP) is provided. This method can approximately compute the Shapley values in polynomial time and satisfy the interpretation consistency. Additionally, an approximating method, Saabas is also implemented to calculate the attribution values, which takes only the single decision path into account and could run in a logarithmic time(over the number of nodes).

• For linear based models, for example, Linear Regression, a linear SHAP expainer(linear SHAP) could be provided. Linear SHAP uses so called background data, which are sampled from the sampled from the training dataset in the training phase and stored in the trained model, to estimate the SHAP values.

  Note

  Local interpretability for models like Exponential Regression and Generalized Linear Models(GLM) are computed w.r.t. linear response instead of the original one, also using linear SHAP in SAP HANA PAL.

• For nonlinear and non-tree based models(e.g. Support Vector Machine(SVM)), a kernel SHAP explainer(kernel SHAP) is provided. Kernel SHAP also uses background data that are sampled from the training dataset in the training phase(and stored in the trained model) to estimate the SHAP values. The only difference from linear models is that those background data are sampled from the training dataset with a kernel that emphasis samples with few or nearly full features value combinations.

Surrogate

When a model becomes over-complex, obtaining its SHAP explanations directly will be very time/resource-consuming no matter what kind of method is adopted. In this case, a simpler & local interpretable surrogate model that approximates the predictions of the underlying complex model is trained. Then, local explanations are obtained from the surrogate model instead of the original complex one. Surrogate model based interpretability is adopted in algorithms like LSTM and Attention(for feature-wise attributions).

Direct Explanation

For some algorithms with specific model structures, local interpretability sometimes can be computed directly. For example, for GRU model with Attension mechanism(i.e. class hana_ml.algorithms.pal.tsa.rnn.GRUAttention), time-dimensional-wise contributions can be directly inferred from the attention weights assigned.

Models/Algorithms in hana_ml.algorithms.pal Packages that Support Local Interpretability

• hana_ml.algorithms.pal.unified_classification.UnifiedClassification

  Note

  Local interpretability available only when the func parameter is set to one of the following values during class initialization: 'DecisionTree', 'RandomDecisionTree', 'HybridGradientBoostingTree', 'LogisticRegression'(additionally when multi_class is set as True, i.e. for multiclass logistic-regression only), 'NaiveBayes', 'MLP', 'SVM'.

• hana_ml.algorithms.pal.unified_regression.UnifiedRegression

  Note

  Local interpretability available only when the func parameter is set to one of the following values during class initialization: 'DecisionTree', 'RandomDecisionTree', 'HybridGradientBoostingTree', 'ExponentialRegression', 'MLP', 'SVM', 'GLM', 'LinearRegression'.

• hana_ml.algorithms.pal.neighbors.KNNClassifier

• hana_ml.algorithms.pal.neighbors.KNNRegressor

• hana_ml.algorithms.pal.tsa.AdditiveModelForecast

• hana_ml.algorithms.pal.tsa.arima.ARIMA

• hana_ml.algorithms.pal.tsa.bsts.BSTS

• hana_ml.algorithms.pal.tsa.lstm.LSTM

• hana_ml.algorithms.pal.tsa.rnn.GRUAttention

Key Relevant Parameters in hana-ml.algorithms.pal Package

In __init__():

• background_size : for models that supports linear/kernel SHAP (in particular, not for tree-based models), specify a positive value to activate local interpretability.

• background_random_state : set the random seed for sampling background data from training data.

In predict() or score():

• top_k_attributions : controls the number of top feature attributions to output

• attribution_method : for tree-based models in UnifiedClassification and UnifiedRegression, specify either 'saabas' or 'tree-shap' as the feature attribution method to activate local interpretability.

• sample_size : number of sampled combination of features in the implementation of linear/kernel SHAP.

• ignore_correlation : whether or not to ignore correlations between features, valid only for algorithms that uses linear SHAP for local interpretability in UnifiedClassification/UnifiedRegression.

Note

The availability of a specific parameter should be algorithm dependent, i.e. given an algorithm that supports local interpretability of models, not all parameters listed above are necessarily available.