Comparing Two Models

Use the Model Compare component to identify the best performer from two algorithms to solve a complex problem in all scenarios (SAP HANA and other). Add a child component to perform further analysis.

Prerequisites: It is mandatory to use the Model Statistics and Partition components with the Model Compare component to create your model comparison chain.

Take the following steps to perform a two-component compare:

  1. In Expert Analytics, connect to a Data Source and navigate to the Predict room.
  2. From the Component List choose the Data Preparation section.
  3. Drag-and-drop a Partition component to the analysis editor. Alternatively, double-click the Partition component. Click OK.
  4. In the Algorithms section, drag-and-drop selected algorithms to the analysis editor. For example, if solving a classification problem, you might choose three classification algorithms, Auto Classification, R-CNR Tree, and Naïve Bayes.
  5. From the Data Preparation section, add Model Statistics components for each chosen algorithm. This enables Expert Analytics to calculate performance statistics on the dataset that the algorithms generate.
  6. Double-click the Model Statistics components to display the configuration options. Alternatively, click the context menu icon on the component and select Configure Settings. The result is a configured chain that can perform the model comparison.
  7. Set the Target and Predicted columns to compute performance statistics in both Model Statistics components.
  8. From the Data Preparation section, add the Model Compare component to the analysis editor.
  9. Drag-and-drop the Model Compare component that you have added to the analysis editor over both of the Model Statistics components that you want to compare. After which, the Model Compare component is linked to all the components that you wish to compare.
    Note For a two-component compare, the Model Compare component enables you to add a child node, which the component indicates by displaying the following icon:
  10. To start configuring the comparison, double-click the Compare component to view its configuration settings. Alternatively, on the component click the Settings icon and from the context menu, select Configure Settings.
  11. In the Model Compare dialog box, select a Validate or Test partition to compare the performance of various components connected to it.
    Note The Model Compare component uses the Validate setting by default to compare models.
  12. In the Performance KPI (Key Performance Indicator) section, take any of the following actions:
    1. Choose the KPIs for use and sort the order in which they should be compared. Control over the order is important because if the top KPI cannot identify a winning algorithm, the component can perform calculations with the second KPI in the list, and so on.
    2. Click the arrows to move KPIs up or down in the comparison order. The input components must be of the same type in the Model Statistics component. If not, an error message displays.
    3. Specify the percentage for the Gain comparison. The percentage of the target population must be between 1% and 100%, to one decimal point (for example, 15.3%).
    Note Classification has 7 KPIs = KI, KR, KI + KR, AUC, S(KS), Gain % and Lift %, whereas Regression has 9 KPIs = KI, KR, KI + KR, R2, L1, L2, LInf, ErrorMean and ErrorStdDev.
  13. When you have completed the configuration, click Done.
  14. The analysis chain is now fully configured and ready to be executed. The summary of the Model Statistics component shows the KPIs calculated for all partitions. Titles are in the order set in the Model Compare component and the Test partitions shows only when the Model Compare component exists and Test was selected for comparison.
    Note

    If all partitions are not available in the algorithm or the Model Statistics component, the component considers it as a chain without partition.

  15. Click the Run Analysis icon.
    Note

    The component calculates performance results on every partition, but only on the partition that you select does it identify a winner. The result is the best component of those compared only. It is advised that you ensure that the KI values are greater than 95% before deploying the component in production.

  16. The Results tab shows the Summary of the comparison results, and highlights the best component. The feedback includes the following information:
    1. A star icon indicates the best performing component. This is based on the comparison of performance statistics for the algorithms, which can be either classification or regression types. The Summary sorts the model algorithms in order of performance. It compares the results based on the partition selected, which can be either Test or Validate.
    2. Titles are in the order set in the Model Compare component, with the bolded titles indicating those chosen for comparison.
    3. In the case of a classification algorithm, the Profit or the Lift settings will default to 10% if you have not specified a percentage.
  17. Optionally, when you are using two parent components, you can extend the analysis by adding a child node with a mapping screen to the Model Compare component. To do so, right-click Model Compare and select Configure Settings. Alternatively, double-click the Model Compare component or press F5. After which, a default mapping occurs that is based on column name and type.
  18. Optionally, name the columns that result from the mapping for the child component to use. You can add or remove other columns of the same type. To map all other columns, manually add the additional rows.
    Note

    The data in the mapped columns comes from the winning component. None of the columns in the configuration window can be empty.

  19. Optionally, you can export the best model as a stored procedure for consumption. To do so, in the Model Compare component click the Settings icon and from the resulting context menu, select Export as Stored Procedure.
  20. Optionally, you can save and export the best chain directly from the Model Compare component. To do so, in Model Compare click the Settings icon and from the resulting context menu, select Save as Model.
You can now use the Partition, Model Statistics and Model Comparison components in unison to compare multiple algorithms to find the best one to use in a complex analysis.