online_univariate_analysis

hana_ml.algorithms.pal.stats.online_univariate_analysis(data, key=None, cols=None, categorical_variable=None, significance_level=None, trimmed_percentage=None, pre_model=None, pre_continuous_result=None, pre_categorical_result=None)

Online version of univariate analysis, useful for processing large-scale data in multiple rounds. Large datasets can be split into multiple batches and fed to the function one by one. Starting from the second batch, PAL estimates valid observations, lower quartile, median, upper quartile, and trimmed mean incrementally using the model from the previous round.

Parameters
dataDataFrame

DataFrame containing the data for the current batch.

keystr, optional

Name of the ID column.

If not provided, it is assumed that the input data has no ID column.

colslist of str, optional

List of column names to analyze.

If not provided, it defaults to all non-ID columns.

categorical_variablelist of str, optional

INTEGER columns specified in this list will be treated as categorical data.

By default, INTEGER columns are treated as continuous.

No default value.

Note

This parameter takes effect only during the initial calculation. In subsequent rounds, PAL reads the setting from the model.

significance_levelfloat, optional

The significance level when the function calculates the confidence interval of the sample mean.

Values must be greater than 0 and less than 1.

Defaults to 0.05.

Note

This parameter takes effect only during the initial calculation. In subsequent rounds, PAL reads the setting from the model.

trimmed_percentagefloat, optional

The ratio of data at both head and tail that will be dropped in the process of calculating the trimmed mean.

Value range is from 0 to 0.5.

Defaults to 0.05.

Note

This parameter takes effect only during the initial calculation. In subsequent rounds, PAL reads the setting from the model.

pre_modelDataFrame, optional

Model table output from the previous round of online univariate analysis, structured as follows:

  • ROW_INDEX, type INTEGER, row number.

  • MODEL_CONTENT, type NVARCHAR, model content.

For the first batch (initial run), leave this as None.

pre_continuous_resultDataFrame, optional

Continuous result table output from the previous round, structured as follows:

  • VARIABLE_NAME, type NVARCHAR(256), variable names.

  • STAT_NAME, type NVARCHAR(100), names of statistical quantities.

  • STAT_VALUE, type DOUBLE, values for the corresponding statistical quantities.

For the first batch (initial run), leave this as None.

pre_categorical_resultDataFrame, optional

Categorical result table output from the previous round, structured as follows:

  • VARIABLE_NAME, type NVARCHAR(256), variable names.

  • CATEGORY, type NVARCHAR(256), category names.

  • STAT_NAME, type NVARCHAR(100), names of statistical quantities.

  • STAT_VALUE, type DOUBLE, values for the corresponding statistical quantities.

For the first batch (initial run), leave this as None.

Returns
DataFrames

Model table for use as pre_model in the next round, structured as follows:

  • ROW_INDEX, type INTEGER, row number.

  • MODEL_CONTENT, type NVARCHAR, model content.

Statistics for continuous variables, structured as follows:

  • VARIABLE_NAME, type NVARCHAR(256), variable names.

  • STAT_NAME, type NVARCHAR(100), names of statistical quantities, including number of valid observation, min, lower quartile, median, upper quartile, max, mean, confidence interval for the mean (lower and upper bound), trimmed mean, variance, standard deviation, skewness, and kurtosis (14 quantities in total).

  • STAT_VALUE, type DOUBLE, values for the corresponding statistical quantities.

Statistics for categorical variables, structured as follows:

  • VARIABLE_NAME, type NVARCHAR(256), variable names.

  • CATEGORY, type NVARCHAR(256), category names of the corresponding variables. Null is also treated as a category.

  • STAT_NAME, type NVARCHAR(100), names of statistical quantities: number of observations, percentage of the current category for a variable (including null).

  • STAT_VALUE, type DOUBLE, values for the corresponding statistical quantities.

Examples

First batch:

>>> df1.collect()
      X1    X2  X3 X4
0    1.2  None   1  A
1    2.5  None   2  C
2    5.2  None   3  A
3  -10.2  None   2  A
4    8.5  None   2  C
5  100.0  None   3  B

Initial run (no pre-inputs):

>>> model, continuous, categorical = online_univariate_analysis(
...     data=df1,
...     categorical_variable=['X3'],
...     significance_level=0.05,
...     trimmed_percentage=0.2)

Second batch, passing previous outputs as pre-inputs:

>>> df2.collect()
     X1   X2  X3 X4
0   1.9  NaN   1  A
1   2.3  NaN   2  C
2   0.2  NaN   3  A
3   1.2  NaN   2  A
4  -7.5  0.7   2  C
5  80.0  0.3   3  B

>>> model2, continuous2, categorical2 = online_univariate_analysis(
...     data=df2,
...     pre_model=model,
...     pre_continuous_result=continuous,
...     pre_categorical_result=categorical)