MassivePipeline

class hana_ml.algorithms.pal.massive_pipeline.MassivePipeline(pipeline, pipeline_type=None, special_group_id='PAL_MASSIVE_PROCESSING_SPECIAL_GROUP_ID2')

Run a PAL pipeline (given as JSON) per group in a massive setting.

MassivePipeline is the massive counterpart of Pipeline when constructed with pipeline=... - i.e. an explicit PAL pipeline JSON that is forwarded to the backend verbatim. Every group in the input data is fitted with the same pipeline by default, or with a per-group pipeline supplied through group_pipelines.

This class deliberately does not support the steps-style composition offered by Pipeline; if you need to build a pipeline from (name, estimator) tuples, use Pipeline, extract generate_json_pipeline() output, and hand it here.

Parameters
pipelinestr or dict

A PAL pipeline JSON (string or dict) applied to every group by default.

Typical sources:

  • The best_pipeline_ output from a MassiveAutomaticClassification / MassiveAutomaticRegression / MassiveAutomaticTimeSeries run.

  • The generate_json_pipeline() output of a fitted Pipeline.

  • A hand-written operator graph such as {"HGBT_Regressor": {"args": {}, "inputs": {"data": "ROWDATA"}}}.

pipeline_type{'classifier', 'regressor', 'timeseries'}, optional

Task type used to shape the input / output columns. If omitted, it is inferred from the operator names in pipeline.

special_group_id{'PAL_MASSIVE_PROCESSING_SPECIAL_GROUP_ID', 'PAL_MASSIVE_PROCESSING_SPECIAL_GROUP_ID2'}, optional

The special group ID used for defaults shared across groups.

Defaults to 'PAL_MASSIVE_PROCESSING_SPECIAL_GROUP_ID2'.

Attributes
model_DataFrame

Fitted model, structured as follows:

  • 1st column: GROUP_ID.

  • 2nd column: ROW_INDEX.

  • 3rd column: MODEL_CONTENT.

info_DataFrame

Fitting statistics per group.

error_DataFrame

Error information for the fitting/prediction/scoring process.

Methods

cleanup_progress_log(connection_context)

Clean up the progress log.

delete_config_dict([operator_name, ...])

Deletes the content of the config dict.

disable_auto_sql_content([disable])

Disable auto SQL content logging.

disable_log_cleanup([disable])

Disable the log clean up.

disable_mlflow_autologging()

Disable the mlflow autologging.

disable_workload_class_check()

Disable the workload class check.

display_config_dict([operator_name, category])

Displays the config dict.

display_progress_table(connection_context)

Return the progress table.

enable_mlflow_autologging([schema, meta, ...])

Enable the mlflow autologging.

evaluate(data[, pipeline, key, features, ...])

Evaluates a pipeline.

fit(data, group_key[, key, features, label, ...])

Fit the pipeline per group via PAL_MASSIVE_PIPELINE_FIT.

get_best_pipeline()

Return the best pipeline.

get_best_pipelines()

Return the best pipeline.

get_config_dict()

Return the config_dict.

get_optimal_config_dict()

Return the optimal config_dict.

get_optimal_connections()

Return the optimal connections.

get_workload_classes(connection_context)

Return the available workload classes information.

make_future_dataframe([data, key, ...])

Create a new dataframe for time series prediction.

persist_progress_log()

Persist the progress log.

pipeline_plot([name, iframe_height])

Pipeline plot.

predict(data, group_key[, key, features, ...])

Predict per group via PAL_MASSIVE_PIPELINE_PREDICT.

score(data, group_key[, key, features, ...])

Score per group via PAL_MASSIVE_PIPELINE_SCORE.

set_progress_log_level(log_level)

Set progress log level to output scorings.

update_category_map(connection_context)

Updates the list of operators.

update_config_dict(operator_name[, ...])

Updates the config dict.

Examples

Replay a per-group pipeline JSON that came out of a massive AutoML run:

>>> pipe = MassivePipeline(pipeline={"HGBT_Regressor": {"args": {},
...                                                     "inputs": {"data": "ROWDATA"}}})
>>> pipe.fit(df_train, group_key='GROUP_ID', key='ID', label='Y')
>>> pipe.predict(df_test, group_key='GROUP_ID', key='ID').collect()

Per-group pipelines - each group gets its own JSON:

>>> pipe = MassivePipeline(pipeline=default_pipeline_json)
>>> pipe.fit(df_train, group_key='GROUP_ID', key='ID', label='Y',
...          group_pipelines={'g1': pipe_json_1, 'g2': pipe_json_2})

Zero-shot per-group forecast with the TSFM operator - no fit step required; predict auto-detects skip-fit:

>>> pipe = MassivePipeline(pipeline={"TSFM": {"args": {},
...                                           "inputs": {"data": "ROWDATA"}}})
>>> pipe.predict(combined_df, group_key='GROUP_ID', key='ID').collect()
fit(data, group_key, key=None, features=None, label=None, group_pipelines=None, categorical_variable=None, background_size=None, background_sampling_seed=None, model_table_name=None, use_explain=None, explain_method=None, group_params=None)

Fit the pipeline per group via PAL_MASSIVE_PIPELINE_FIT.

Parameters
dataDataFrame

The input data, containing a group column plus the usual key / features / label columns.

group_keystr

Name of the group column.

keystr, optional

Name of the ID column. Defaults to the single-column index of data if present, otherwise no ID column.

featureslist of str, optional

Names of the feature columns. Defaults to all non-group_key, non-key, non-label columns.

labelstr, optional

Name of the dependent variable. Defaults to the last non-group_key, non-key column (or the first such column for time-series).

group_pipelinesdict, optional

Per-group pipeline overrides {group_id: pipeline_json}. Groups not present in this dict use the pipeline supplied at construction time.

Defaults to None (every group uses the constructor's pipeline).

categorical_variablestr or list of str, optional

INTEGER columns to be treated as categorical.

background_sizeint, optional

Enables SHAP reason codes if set. Requires explain_method='kernelshap'.

background_sampling_seedint, optional

Seed for the background sampling.

model_table_namestr, optional

Custom HANA model table name.

use_explainbool, optional

Whether to store information for pipeline explanation.

explain_method{'kernelshap', 'globalsurrogate'}, optional

Explanation method.

group_paramsdict, optional

Per-group parameter overrides passed through to PAL.

Returns
A fitted MassivePipeline instance.
predict(data, group_key, key=None, features=None, model=None, show_explainer=False, top_k_attributions=None, random_state=None, sample_size=None, verbose_output=None, predict_args=None, group_params=None, output_prediction_interval=None, confidence_level=None, skip_fit=None)

Predict per group via PAL_MASSIVE_PIPELINE_PREDICT.

Parameters mirror MassiveAutomaticClassification.predict() / MassiveAutomaticTimeSeries.predict(); see those for details.

show_explainer is silently ignored for time-series pipelines (PAL does not implement PAL_MASSIVE_PIPELINE_EXPLAIN for TS). The output_prediction_interval and confidence_level arguments are only forwarded for time-series pipelines.

skip_fitbool, optional

Run in skip-fit mode: the caller doesn't have to call fit first. data must carry the historical context together with the forecast horizon in one table (target column populated on history rows, NULL on horizon rows). Internally the class runs _fit() on the history rows and then predicts the horizon rows - this transparently works for zero-shot operators (currently TSFM) whose fit step is effectively a no-op.

The model argument is ignored in this mode.

If left as None, predict auto-detects skip-fit: when no fitted model is available (neither model_ nor the model argument) yet the pipeline JSON references a zero-shot operator (currently TSFM), predict runs in skip-fit mode.

Defaults to None.

Returns
DataFrame

Predicted result per group.

score(data, group_key, key=None, features=None, label=None, model=None, random_state=None, top_k_attributions=None, sample_size=None, verbose_output=None, predict_args=None, group_params=None)

Score per group via PAL_MASSIVE_PIPELINE_SCORE.

Parameters mirror MassiveAutomaticClassification.score(); see that for details.

Returns
tuple of DataFrame

(result, statistics) per group.

cleanup_progress_log(connection_context)

Clean up the progress log.

Parameters
connection_contextConnectionContext

The connection object to a SAP HANA database.

delete_config_dict(operator_name=None, category=None, param_name=None)

Deletes the content of the config dict.

Parameters
operator_namestr, optional

Deletes the operator based on the given name in the config dict.

Defaults to None.

categorystr, optional

Deletes the whole given category in the config dict.

Defaults to None.

param_namestr, optional

Deletes the parameter based on the given name once the operator name is provided.

Defaults to None.

disable_auto_sql_content(disable=True)

Disable auto SQL content logging. Use AFL's default progress logging.

disable_log_cleanup(disable=True)

Disable the log clean up.

disable_mlflow_autologging()

Disable the mlflow autologging.

disable_workload_class_check()

Disable the workload class check. Please note that the MassiveAutomaticClassification/MassiveAutomaticRegression/MassiveAutomaticTimeSeries may cause large resource. Without setting workload class, there's no resource restriction on the training process.

display_config_dict(operator_name=None, category=None)

Displays the config dict.

Parameters
operator_namestr, optional

Only displays the information on the given operator name.

Defaults to None.

categorystr, optional

Only displays the information on the given category.

Defaults to None.

display_progress_table(connection_context)

Return the progress table.

Parameters
connection_contextConnectionContext

The connection object to a SAP HANA database.

Returns
DataFrame

Progress table.

enable_mlflow_autologging(schema=None, meta=None, is_exported=False, registered_model_name=None, delete_after_mlflow=False)

Enable the mlflow autologging.

Parameters
schemastr, optional

Defines the model storage schema for mlflow autologging.

Defaults to the current schema.

metastr, optional

Defines the model storage meta table for mlflow autologging.

Defaults to 'HANAML_MLFLOW_MODEL_STORAGE'.

is_exportedbool, optional

Determines whether export a HANA model to mlflow.

Defaults to False.

registered_model_namestr, optional

MLFlow registered_model_name.

Defaults to None.

delete_after_mlflowbool, optional

If True, the model saved to HANA Cloud via ModelStorage is deleted after it has been successfully exported to MLflow. This keeps the HANA schema free of accumulated model tables while all run artifacts remain available in MLflow.

Defaults to False.

evaluate(data, pipeline=None, key=None, features=None, label=None, categorical_variable=None, text_variable=None, resampling_method=None, fold_num=None, random_state=None)

Evaluates a pipeline.

Parameters
dataDataFrame

Data for pipeline evaluation.

pipelinejson str or dict

Pipeline to be evaluated.

keystr, optional

Name of the ID column.

Mandatory if data is not indexed, or the index of data contains multiple columns.

Defaults to the single index column of data if not provided.

featuresa list of str, optional

Names of the feature columns. If features is not provided, it defaults to all non-ID, non-label columns.

labelstr, optional

Name of the dependent variable.

Defaults to the name of the last non-ID column.

categorical_variablestr or a list of str, optional

Specifies which INTEGER columns should be treated as categorical, with all other INTEGER columns treated as continuous.

No default value.

text_variablestr or a list of str, optional

It indicates the text column.

Defaults to None.

resampling_methodcharacter, optional

The resampling method for pipeline model evaluation. For different pipeline, the options are different.

  • regressor: {'cv', 'stratified_cv'}

  • classifier: {'cv'}

  • timeseries: {'rocv', 'block'}

Defaults to 'stratified_cv' if the estimator in pipeline is a classifier, and defaults to(and can only be) 'cv' if the estimator in pipeline is a regressor, and defaults to 'rocv' if if the estimator in pipeline is a timeseries.

fold_numint, optional

The fold number for cross validation. If the value is 0, the function will automatically determine the fold number.

Defaults to 5.

random_stateint, optional

Specifies the seed for random number generator.

  • 0: Uses the current time (in seconds) as the seed.

  • Others: Uses the specified value as the seed.

Returns
DataFrame

Scores.

get_best_pipeline()

Return the best pipeline.

get_best_pipelines()

Return the best pipeline.

get_config_dict()

Return the config_dict.

get_optimal_config_dict()

Return the optimal config_dict. Only available when connections is used.

get_optimal_connections()

Return the optimal connections. Only available when connections is used.

get_workload_classes(connection_context)

Return the available workload classes information.

Parameters
connection_contextstr, optional

The connection to a SAP HANA instance.

make_future_dataframe(data=None, key=None, group_key=None, periods=1, increment_type='seconds', same_cadence=False)

Create a new dataframe for time series prediction.

Parameters
dataDataFrame, optional

The training data contains the index.

Defaults to the data used in the fit().

keystr, optional

The index defined in the training data.

Defaults to the specified key in fit() or the value in data.index or the PAL's default key column position.

group_keystr, optional

Specify the group id column.

This parameter is only valid when massive is True.

Defaults to the specified group_key in fit() or the first column of the dataframe.

periodsint, optional

The number of rows created in the predict dataframe.

Defaults to 1.

increment_type{'seconds', 'days', 'months', 'years'}, optional

The increment type of the time series.

Defaults to 'seconds'.

same_cadencebool, optional

When True, assumes every group shares the same time cadence (same latest key and same interval between consecutive keys). The library then infers the cadence once globally instead of per group, which is orders of magnitude faster for large group counts (e.g. tens of thousands of stores).

When False (default), cadence is inferred per group using a single set-based SQL query -- still much faster than the previous per-group Python loop.

Defaults to False.

Returns
DataFrame
persist_progress_log()

Persist the progress log.

pipeline_plot(name='my_pipeline', iframe_height=450)

Pipeline plot.

Parameters
namestr, optional

The name of the pipeline plot.

Defaults to 'my_pipeline'.

iframe_heightint, optional

The display height.

Defaults to 450.

set_progress_log_level(log_level)

Set progress log level to output scorings.

Parameters
log_level: {'full', 'full_best', 'specified'}

'full' prints all scores. 'full_best' prints all scores only for the 'current_best' of each generation; other pipelines print only the scores specified by SCORINGS. 'Specified' means all pipelines print only the specified scores.

update_category_map(connection_context)

Updates the list of operators.

Parameters
connection_contextstr, optional

The connection to a SAP HANA instance.

update_config_dict(operator_name, param_name=None, param_config=None)

Updates the config dict.

Parameters
operator_namestr

The name of operator.

param_namestr, optional

The parameter name to be updated. If the parameter name doesn't exist in the config dict, it will create a new one.

Defaults to None.

param_configany, optional

The parameter config value.

When operator_name is OutlierVoting and param_name is VOTING_COMBINATION, dict input is normalized to the JSON-formatted string format required by PAL.

Defaults to None.

Inherited Methods from PALBase

Besides those methods mentioned above, the MassivePipeline class also inherits methods from PALBase class, please refer to PAL Base for more details.