MassivePipeline¶
- class hana_ml.algorithms.pal.massive_pipeline.MassivePipeline(pipeline, pipeline_type=None, special_group_id='PAL_MASSIVE_PROCESSING_SPECIAL_GROUP_ID2')¶
Run a PAL pipeline (given as JSON) per group in a massive setting.
MassivePipelineis the massive counterpart ofPipelinewhen constructed withpipeline=...- i.e. an explicit PAL pipeline JSON that is forwarded to the backend verbatim. Every group in the input data is fitted with the same pipeline by default, or with a per-group pipeline supplied throughgroup_pipelines.This class deliberately does not support the
steps-style composition offered byPipeline; if you need to build a pipeline from(name, estimator)tuples, usePipeline, extractgenerate_json_pipeline()output, and hand it here.- Parameters
- pipelinestr or dict
A PAL pipeline JSON (string or dict) applied to every group by default.
Typical sources:
The
best_pipeline_output from aMassiveAutomaticClassification/MassiveAutomaticRegression/MassiveAutomaticTimeSeriesrun.The
generate_json_pipeline()output of a fittedPipeline.A hand-written operator graph such as
{"HGBT_Regressor": {"args": {}, "inputs": {"data": "ROWDATA"}}}.
- pipeline_type{'classifier', 'regressor', 'timeseries'}, optional
Task type used to shape the input / output columns. If omitted, it is inferred from the operator names in
pipeline.- special_group_id{'PAL_MASSIVE_PROCESSING_SPECIAL_GROUP_ID', 'PAL_MASSIVE_PROCESSING_SPECIAL_GROUP_ID2'}, optional
The special group ID used for defaults shared across groups.
Defaults to
'PAL_MASSIVE_PROCESSING_SPECIAL_GROUP_ID2'.
- Attributes
- model_DataFrame
Fitted model, structured as follows:
1st column: GROUP_ID.
2nd column: ROW_INDEX.
3rd column: MODEL_CONTENT.
- info_DataFrame
Fitting statistics per group.
- error_DataFrame
Error information for the fitting/prediction/scoring process.
Methods
cleanup_progress_log(connection_context)Clean up the progress log.
delete_config_dict([operator_name, ...])Deletes the content of the config dict.
disable_auto_sql_content([disable])Disable auto SQL content logging.
disable_log_cleanup([disable])Disable the log clean up.
Disable the mlflow autologging.
Disable the workload class check.
display_config_dict([operator_name, category])Displays the config dict.
display_progress_table(connection_context)Return the progress table.
enable_mlflow_autologging([schema, meta, ...])Enable the mlflow autologging.
evaluate(data[, pipeline, key, features, ...])Evaluates a pipeline.
fit(data, group_key[, key, features, label, ...])Fit the pipeline per group via
PAL_MASSIVE_PIPELINE_FIT.Return the best pipeline.
Return the best pipeline.
Return the config_dict.
Return the optimal config_dict.
Return the optimal connections.
get_workload_classes(connection_context)Return the available workload classes information.
make_future_dataframe([data, key, ...])Create a new dataframe for time series prediction.
Persist the progress log.
pipeline_plot([name, iframe_height])Pipeline plot.
predict(data, group_key[, key, features, ...])Predict per group via
PAL_MASSIVE_PIPELINE_PREDICT.score(data, group_key[, key, features, ...])Score per group via
PAL_MASSIVE_PIPELINE_SCORE.set_progress_log_level(log_level)Set progress log level to output scorings.
update_category_map(connection_context)Updates the list of operators.
update_config_dict(operator_name[, ...])Updates the config dict.
Examples
Replay a per-group pipeline JSON that came out of a massive AutoML run:
>>> pipe = MassivePipeline(pipeline={"HGBT_Regressor": {"args": {}, ... "inputs": {"data": "ROWDATA"}}}) >>> pipe.fit(df_train, group_key='GROUP_ID', key='ID', label='Y') >>> pipe.predict(df_test, group_key='GROUP_ID', key='ID').collect()
Per-group pipelines - each group gets its own JSON:
>>> pipe = MassivePipeline(pipeline=default_pipeline_json) >>> pipe.fit(df_train, group_key='GROUP_ID', key='ID', label='Y', ... group_pipelines={'g1': pipe_json_1, 'g2': pipe_json_2})
Zero-shot per-group forecast with the
TSFMoperator - nofitstep required;predictauto-detects skip-fit:>>> pipe = MassivePipeline(pipeline={"TSFM": {"args": {}, ... "inputs": {"data": "ROWDATA"}}}) >>> pipe.predict(combined_df, group_key='GROUP_ID', key='ID').collect()
- fit(data, group_key, key=None, features=None, label=None, group_pipelines=None, categorical_variable=None, background_size=None, background_sampling_seed=None, model_table_name=None, use_explain=None, explain_method=None, group_params=None)¶
Fit the pipeline per group via
PAL_MASSIVE_PIPELINE_FIT.- Parameters
- dataDataFrame
The input data, containing a group column plus the usual key / features / label columns.
- group_keystr
Name of the group column.
- keystr, optional
Name of the ID column. Defaults to the single-column index of
dataif present, otherwise no ID column.- featureslist of str, optional
Names of the feature columns. Defaults to all non-
group_key, non-key, non-labelcolumns.- labelstr, optional
Name of the dependent variable. Defaults to the last non-
group_key, non-keycolumn (or the first such column for time-series).- group_pipelinesdict, optional
Per-group pipeline overrides
{group_id: pipeline_json}. Groups not present in this dict use the pipeline supplied at construction time.Defaults to None (every group uses the constructor's pipeline).
- categorical_variablestr or list of str, optional
INTEGER columns to be treated as categorical.
- background_sizeint, optional
Enables SHAP reason codes if set. Requires
explain_method='kernelshap'.- background_sampling_seedint, optional
Seed for the background sampling.
- model_table_namestr, optional
Custom HANA model table name.
- use_explainbool, optional
Whether to store information for pipeline explanation.
- explain_method{'kernelshap', 'globalsurrogate'}, optional
Explanation method.
- group_paramsdict, optional
Per-group parameter overrides passed through to PAL.
- Returns
- A fitted
MassivePipelineinstance.
- A fitted
- predict(data, group_key, key=None, features=None, model=None, show_explainer=False, top_k_attributions=None, random_state=None, sample_size=None, verbose_output=None, predict_args=None, group_params=None, output_prediction_interval=None, confidence_level=None, skip_fit=None)¶
Predict per group via
PAL_MASSIVE_PIPELINE_PREDICT.Parameters mirror
MassiveAutomaticClassification.predict()/MassiveAutomaticTimeSeries.predict(); see those for details.show_explaineris silently ignored for time-series pipelines (PAL does not implementPAL_MASSIVE_PIPELINE_EXPLAINfor TS). Theoutput_prediction_intervalandconfidence_levelarguments are only forwarded for time-series pipelines.- skip_fitbool, optional
Run in skip-fit mode: the caller doesn't have to call
fitfirst.datamust carry the historical context together with the forecast horizon in one table (target column populated on history rows,NULLon horizon rows). Internally the class runs_fit()on the history rows and then predicts the horizon rows - this transparently works for zero-shot operators (currentlyTSFM) whose fit step is effectively a no-op.The
modelargument is ignored in this mode.If left as
None,predictauto-detects skip-fit: when no fitted model is available (neithermodel_nor themodelargument) yet the pipeline JSON references a zero-shot operator (currentlyTSFM), predict runs in skip-fit mode.Defaults to None.
- Returns
- DataFrame
Predicted result per group.
- score(data, group_key, key=None, features=None, label=None, model=None, random_state=None, top_k_attributions=None, sample_size=None, verbose_output=None, predict_args=None, group_params=None)¶
Score per group via
PAL_MASSIVE_PIPELINE_SCORE.Parameters mirror
MassiveAutomaticClassification.score(); see that for details.- Returns
- tuple of DataFrame
(result, statistics)per group.
- cleanup_progress_log(connection_context)¶
Clean up the progress log.
- Parameters
- connection_contextConnectionContext
The connection object to a SAP HANA database.
- delete_config_dict(operator_name=None, category=None, param_name=None)¶
Deletes the content of the config dict.
- Parameters
- operator_namestr, optional
Deletes the operator based on the given name in the config dict.
Defaults to None.
- categorystr, optional
Deletes the whole given category in the config dict.
Defaults to None.
- param_namestr, optional
Deletes the parameter based on the given name once the operator name is provided.
Defaults to None.
- disable_auto_sql_content(disable=True)¶
Disable auto SQL content logging. Use AFL's default progress logging.
- disable_log_cleanup(disable=True)¶
Disable the log clean up.
- disable_mlflow_autologging()¶
Disable the mlflow autologging.
- disable_workload_class_check()¶
Disable the workload class check. Please note that the MassiveAutomaticClassification/MassiveAutomaticRegression/MassiveAutomaticTimeSeries may cause large resource. Without setting workload class, there's no resource restriction on the training process.
- display_config_dict(operator_name=None, category=None)¶
Displays the config dict.
- Parameters
- operator_namestr, optional
Only displays the information on the given operator name.
Defaults to None.
- categorystr, optional
Only displays the information on the given category.
Defaults to None.
- display_progress_table(connection_context)¶
Return the progress table.
- Parameters
- connection_contextConnectionContext
The connection object to a SAP HANA database.
- Returns
- DataFrame
Progress table.
- enable_mlflow_autologging(schema=None, meta=None, is_exported=False, registered_model_name=None, delete_after_mlflow=False)¶
Enable the mlflow autologging.
- Parameters
- schemastr, optional
Defines the model storage schema for mlflow autologging.
Defaults to the current schema.
- metastr, optional
Defines the model storage meta table for mlflow autologging.
Defaults to 'HANAML_MLFLOW_MODEL_STORAGE'.
- is_exportedbool, optional
Determines whether export a HANA model to mlflow.
Defaults to False.
- registered_model_namestr, optional
MLFlow registered_model_name.
Defaults to None.
- delete_after_mlflowbool, optional
If True, the model saved to HANA Cloud via ModelStorage is deleted after it has been successfully exported to MLflow. This keeps the HANA schema free of accumulated model tables while all run artifacts remain available in MLflow.
Defaults to False.
- evaluate(data, pipeline=None, key=None, features=None, label=None, categorical_variable=None, text_variable=None, resampling_method=None, fold_num=None, random_state=None)¶
Evaluates a pipeline.
- Parameters
- dataDataFrame
Data for pipeline evaluation.
- pipelinejson str or dict
Pipeline to be evaluated.
- keystr, optional
Name of the ID column.
Mandatory if
datais not indexed, or the index ofdatacontains multiple columns.Defaults to the single index column of
dataif not provided.- featuresa list of str, optional
Names of the feature columns. If
featuresis not provided, it defaults to all non-ID, non-label columns.- labelstr, optional
Name of the dependent variable.
Defaults to the name of the last non-ID column.
- categorical_variablestr or a list of str, optional
Specifies which INTEGER columns should be treated as categorical, with all other INTEGER columns treated as continuous.
No default value.
- text_variablestr or a list of str, optional
It indicates the text column.
Defaults to None.
- resampling_methodcharacter, optional
The resampling method for pipeline model evaluation. For different pipeline, the options are different.
regressor: {'cv', 'stratified_cv'}
classifier: {'cv'}
timeseries: {'rocv', 'block'}
Defaults to 'stratified_cv' if the estimator in
pipelineis a classifier, and defaults to(and can only be) 'cv' if the estimator inpipelineis a regressor, and defaults to 'rocv' if if the estimator inpipelineis a timeseries.- fold_numint, optional
The fold number for cross validation. If the value is 0, the function will automatically determine the fold number.
Defaults to 5.
- random_stateint, optional
Specifies the seed for random number generator.
0: Uses the current time (in seconds) as the seed.
Others: Uses the specified value as the seed.
- Returns
- DataFrame
Scores.
- get_best_pipeline()¶
Return the best pipeline.
- get_best_pipelines()¶
Return the best pipeline.
- get_config_dict()¶
Return the config_dict.
- get_optimal_config_dict()¶
Return the optimal config_dict. Only available when connections is used.
- get_optimal_connections()¶
Return the optimal connections. Only available when connections is used.
- get_workload_classes(connection_context)¶
Return the available workload classes information.
- Parameters
- connection_contextstr, optional
The connection to a SAP HANA instance.
- make_future_dataframe(data=None, key=None, group_key=None, periods=1, increment_type='seconds', same_cadence=False)¶
Create a new dataframe for time series prediction.
- Parameters
- dataDataFrame, optional
The training data contains the index.
Defaults to the data used in the fit().
- keystr, optional
The index defined in the training data.
Defaults to the specified key in fit() or the value in data.index or the PAL's default key column position.
- group_keystr, optional
Specify the group id column.
This parameter is only valid when
massiveis True.Defaults to the specified group_key in fit() or the first column of the dataframe.
- periodsint, optional
The number of rows created in the predict dataframe.
Defaults to 1.
- increment_type{'seconds', 'days', 'months', 'years'}, optional
The increment type of the time series.
Defaults to 'seconds'.
- same_cadencebool, optional
When True, assumes every group shares the same time cadence (same latest key and same interval between consecutive keys). The library then infers the cadence once globally instead of per group, which is orders of magnitude faster for large group counts (e.g. tens of thousands of stores).
When False (default), cadence is inferred per group using a single set-based SQL query -- still much faster than the previous per-group Python loop.
Defaults to False.
- Returns
- DataFrame
- persist_progress_log()¶
Persist the progress log.
- pipeline_plot(name='my_pipeline', iframe_height=450)¶
Pipeline plot.
- Parameters
- namestr, optional
The name of the pipeline plot.
Defaults to 'my_pipeline'.
- iframe_heightint, optional
The display height.
Defaults to 450.
- set_progress_log_level(log_level)¶
Set progress log level to output scorings.
- Parameters
- log_level: {'full', 'full_best', 'specified'}
'full' prints all scores. 'full_best' prints all scores only for the 'current_best' of each generation; other pipelines print only the scores specified by SCORINGS. 'Specified' means all pipelines print only the specified scores.
- update_category_map(connection_context)¶
Updates the list of operators.
- Parameters
- connection_contextstr, optional
The connection to a SAP HANA instance.
- update_config_dict(operator_name, param_name=None, param_config=None)¶
Updates the config dict.
- Parameters
- operator_namestr
The name of operator.
- param_namestr, optional
The parameter name to be updated. If the parameter name doesn't exist in the config dict, it will create a new one.
Defaults to None.
- param_configany, optional
The parameter config value.
When
operator_nameisOutlierVotingandparam_nameisVOTING_COMBINATION, dict input is normalized to the JSON-formatted string format required by PAL.Defaults to None.