OutlierProfiling

class hana_ml.algorithms.pal.outlier_profiling.OutlierProfiling(profiling='default')

This class provides a way to detect outliers within dataset from multiple perspective, through an outlier profiling. Namely, an outlier profiling is a parametric specification of multiple hana-ml classes/functionsfor outlier detection. It is defined by the detection methods used, together with each method's parameter specification.

Parameters

profilingdict or str, optional

Specifies the outlier profile in a dictionary, and for each key-value pair in that dictionary:

• key is the hana-ml classes/functions for outlier detection in str format. Valid options include:

  • 'KMeansOutlier' : see KMeansOutlier

  • 'grubbs_test' : for grubbs_test(), see grubbs_test()

  • 'iqr' : see iqr()

  • 'DBSCAN' : see DBSCAN

  • 'OneClassSVM' : see OneClassSVM

  • 'IsolationForest' : see IsolationForest

• value is the parameter setting for the hana-ml classes/functions defined by the corresponding key, also in a dictionary format, with keys being parameter names and values being parameter values. For each hana-ml classes/functions, its valid parameters are listed as follows:

  • KMeansOutlier: n_clusters, distance_level, contamination, sum_distance, init, max_iter, normalization, tol, distance_threshold

  • grubbs_test(): method, alpha

  • iqr(): multiplier

  • DBSCAN: minpts, eps, metric, minkowski_power, algorithm

  • OneClassSVM: c, kernel, degree, gamma, coef_lin, coef_const, shrink, tol, nu, scale_info, handle_missing

  • IsolationForest: n_estimators, max_samples, max_features, bootstrap, random_state, contamination

  Note

  contamination is not a parameter in the initialization method of class IsolationForest, but one in the predict() method, so please see the predict() method of IsolationForest for its description.

One can also use 'default' to set profiling the default one.

Defaults to 'default'.

Attributes

profilingdict

Stores the profiling used for outlier detection.

Methods

fit_predict(data[, key, ...])

Detection of outliers in the input data using outlier profiling.

fit_predict(data, key=None, categorical_variable=None, string_variable=None, variable_weight=None, grubbs_cols=None, iqr_cols=None)

Detection of outliers in the input data using outlier profiling.

Parameters

dataDataFrame

DataFrame containing the data used for outlier detection via outlier profiling.

There must be an ID column in data.

keystr, optional

Specifies the name of ID column in data.

Mandatory if data is not indexed by a single column.

Defaults to the single index column of data if there is one.

categorical_variablestr or a list of str, optional

Specifies which INTEGER columns should be treated as categorical, with all other INTEGER columns treated as continuous.

No default value.

string_variablestr or a list of str, optional

Indicates a string(i.e. of type VARCHAR/NVARCHAR) column storing not categorical data. Levenshtein distance is used to calculate similarity between two strings. Ignored if it is not a string column.

Effective only when DBSCAN is included in the outlier profiling.

By default all columns of type VARCHAR/NVARCHAR are categorical.

grubbs_colsstr or a list of str, optional

Specifies the (numerical)columns used for Grubbs' test.

Categorical columns specifies in categorical_variable are ignored automatically.

Effective only when grubbs_test is included in the outlier profiling.

Defaults to all numerical columns in data.

iqr_colsstr or a list of str, optional

Specifies the (numerical)columns used for Inter-Quantile-Range(IQR) test.

Categorical columns specifies in categorical_variable are ignored automatically.

Effective only when iqr is included in the outlier profiling.

Defaults to all numerical columns in data.

.. note ::

IsolationForest, grubbs_test, iqr apply only when data contains numerical columns. In the absence of such columns, the aforementioned three methods shall be disabled.

Returns

Dict

The detected outliers by different algorithms within the profiling, structured as follows:

• key : the algorithm applied for outlier detection

• value : the DataFrame containing the detected outliers by the algorithm specified by the key

Inherited Methods from PALBase

Besides those methods mentioned above, the OutlierProfiling class also inherits methods from PALBase class, please refer to PAL Base for more details.