Overview

Use the Model Compare component to compare models and learn the best algorithm for your predictive problem. Use in all scenarios (HANA and non-HANA).

Why Compare Models?

Comparing models in Expert Analytics enables you to try different algorithms and discover the best one to solve your predictive problem. When comparing the performance of two or more algorithms, you first use the Model Statistics component to calculate performance statistics for either Classification or Rregression algorithms. After which, the Model Compare component compares the calculated performance statistics to pick the best algorithm of those run at execution. Finally, the Model Compare component merges the results to provide a detailed summary on the best performing component.

Configuring Partitions

You can configure partition types in the Model Compare component for more control over your analysis chain. In the Properties Panel of the component, you can select either a Validate or Test partition to compare the performance of the models. The component slices a dataset into three subsets called Train, Validate and Test.

The component calculates performance results on every partition, but only on the partition that you select does it identify a winner. The result is the best component of those compared only.

Configuring KPIs

You can choose the type and comparison order of the Key Performance Indicators (KPIs) in your analysis chain.

The following tables define the KPIs specific to the Classification and Regression algorithms.
Table 1: Classification KPIs

KPI

Definition

Ki

Predictive power. A quality indicator that corresponds to the proportion of information contained in the target variable that the explanatory variables are able to explain.

Kr

Model reliability, or the ability to produce similar on new data. A robustness indicator of the models generated. It indicates the capacity of the model to achieve the same performance when it is applied to a new data set exhibiting the same characteristics as the training data set.

Ki & Kr

Predictive power and model reliability. Gives equal importance to the robustness and generalizing capabilities of the model. For more information, see the definitions above.

AUC

Area Under The Curve. Rank-based measure of the model performance or the predictive power calculated as the area under the Receiver Operating Characteristic curve (ROC).

S(KS)

The distance between the distribution functions of the two classes in binary classification (for example, Class 1 and Class 0). The score that generates the greatest separability between the functions is considered the threshold value for accepting or rejecting the target. The measure of seperability defines how well the model is able to distinguish between the records of two classes. If there are minor deviations in the input data, the model should still be able to identify these patterns and diiferentiate between the two. In this way, seperability is a metric of how good the model is; the greater the seperability, the greater the model. Note that the predictive model producing the greatest amount of separability between the two distributions is considered the superior model.

Gain % (Profit %)

The gain or profit that is realized by the model based on a percentage of the target population selection.

Lift %

The amount of lift that the trained model gives in comparison to a random model. It enables you to examine of the difference between a perfect model, a random model and the model created.

Table 2: Regression KPIs

KPI

Definition

Ki

Predictive power. A quality indicator that corresponds to the proportion of information contained in the target variable that the explanatory variables are able to explain.

Kr

Model reliability, or the ability to produce similar on new data. A robustness indicator of the models generated. It indicates the capacity of the model to achieve the same performance when it is applied to a new data set exhibiting the same characteristics as the training data set.

Ki & Kr

Predictive power and model reliability. Gives equal importance to the robustness and generalizing capabilities of the model. For more information, see the definitions above.

R2

The determination coefficient R2 is the proportion of variability in a dataset that is accounted for by a statistical model; the ratio between the variability (sum of squares) of the prediction and the variability (sum of squares) of the data.

L1

The mean absolute error L1 is the mean of the absolute values of the differences between predictions and actual results (for example, city block distance or Manhattan distance)

L2

The mean square error L2 is the square root of the mean of the quadratics errors (that is, Euclidian Distance or root mean squared error – RMSE).

Linf

The maximum error Linf is the maximum absolute difference between the predicted and actual values (upper bound); also know as the Chebyshev Distance.

ErrorMean

The mean of the difference between predictions and actual values.

ErrorStdDev

The dispersion of errors around the actual result.

Control over the order is important because if the top KPI cannot identify a winning algorithm, the component can perform calculations with the second KPI in the list, and so on. In addition, a precise percentage can be configured for the Gain and Lift parameters. The result is an even more accurate calculation when comparing two or more components.

Column Mapping

Column mapping in the Model Compare component enables you to map the output from two compared algorithms. The Column mapping section lists side-by-side the matching column types from both algorithms. A third column is the output column for the Model Compare component. This offers a one-to-one mapping between columns and serves as the result data schema for the Model Compare component. This will feed winning outputs into any following algorithms or components that you can add to the chain, such as a report or a decision tree. The data in the mapped columns comes from the winning component.

Columns are mapped only if the column types match. At first a default mapping is completed that is based on exact names, data and statistical types. After which it checks if the columns are of the same type.

Optionally, you can add or remove columns to include in the Model Compare result set.

The below image shows the Column Mapping panel of the Model Compare component in which you can configure the Partition and the KPIs (using the English language version as an example):

Comparing Two Components

You can perform a model comparison on multiple algorithms in one analysis. However, the Model Compare is designed to behave differently depending on the number of algorithms that you add to the comparison chain. On a model comparison chain that has two parent components, you can create a child node. The child node receives the output of the model comparison and displays it in a configurable mapping screen. This means that you can map the columns from two parent components into one for consumption by a child node. This enables you to perform further analysis on your chain. The Model Compare component displays the following icon when in two-component compare mode:

Comparing Three or More Components

You can perform a model comparison on multiple algorithms in one analysis. When Model Compare has three or more parents, the component becomes a terminal (or leaf) component. Therefore you cannot add a child component to perform further analysis after the original comparison. If you try to compare a third component, you receive an error message. The Model Compare component displays the following icon when comparing three or more components:

Results and Summary

The Results tab shows the Summary of the comparison results, and highlights the best component.

The feedback includes a star icon that indicates the best performing component. This is based on the comparison of performance statistics for the algorithms, which can be either classification or regression types. The Summary sorts the model algorithms in order of performance. It compares the results based on the partition selected, which can be either Test or Validate.

Titles display in the order set in the Model Compare component, with the bolded titles indicating those chosen for comparison. In the case of a classification algorithm, the Gain or Lift settings will default to 10% if you have not specified a percentage.