hana_ml.text.tm package¶
The text.tm package consists of the following section:
hana_ml.text.tm¶
- This module provides various functions of text minig. The following functions are available:
- hana_ml.text.tm.tf_analysis(data)¶
Perform Term Frequency(TF) analysis on the given document. TF is the number of occurrences of term in document.
- Parameters
- dataDataFrame
1st column, ID.
2nd column, Document content.
3rd column, Document category.
- Returns
- A tuple of DataFrame
- TF-IDF result, structured as follows:
TM_TERM.
TM_TERM_FREQUENCY.
TM_IDF_FREQUENCY.
TF_VALUE.
IDF_VALUE.
TF_IDF_VALUE.
- Document term frequency table, structured as follows:
ID.
TM_TERM.
TM_TERM_FREQUENCY.
- Document category table, structured as follows:
ID.
Document category.
Examples
The input DataFrame df:
>>> df.collect() ID CONTENT CATEGORY 0 doc1 term1 term2 term2 term3 term3 term3 CATEGORY_1 1 doc2 term2 term3 term3 term4 term4 term4 CATEGORY_1 2 doc3 term3 term4 term4 term5 term5 term5 CATEGORY_2 3 doc4 term3 term4 term4 term5 term5 term5 term5 term5 term5 CATEGORY_2 4 doc5 term4 term6 CATEGORY_3 5 doc6 term4 term6 term6 term6 CATEGORY_3
Invoke tf_analysis function:
>>> tfidf= tf_analysis(df)
Output:
>>> tfidf[0].head(3).collect() TM_TERMS TM_TERM_TF_F TM_TERM_IDF_F TM_TERM_TF_V TM_TERM_IDF_V 0 term1 1 1 0.030303 1.791759 1 term2 3 2 0.090909 1.098612 2 term3 7 4 0.212121 0.405465
>>> tfidf[1].head(3).collect() ID TM_TERMS TM_TERM_FREQUENCY 0 doc1 term1 1 1 doc1 term2 2 2 doc1 term3 3
>>> tfidf[2].head(3).collect() ID CATEGORY 0 doc1 CATEGORY_1 1 doc2 CATEGORY_1 2 doc3 CATEGORY_2
- hana_ml.text.tm.text_classification(pred_data, ref_data=None, k_nearest_neighbours=None, thread_ratio=None, lang='EN', index_name=None, created_index=None)¶
This function classifies (categorizes) an input document with respect to sets of categories (taxonomies).
- Parameters
- pred_dataDataFrame
The prediction data for classification.
1st column, ID.
2nd column, Document content.
- ref_dataDataFrame or a tuple of DataFrame,
- DataFrame, reference data
1st column, ID.
2nd column, Document content.
3rd column, Document Category.
- The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
- 1st DataFrame
1st column, TM_TERM.
2nd column, TF_VALUE.
3rd column, IDF_VALUE.
- 2nd DataFrame
1st column, ID.
2nd column, TM_TERM.
3rd column, TM_TERM_FREQUENCY.
- 3rd DataFrame
1st column, ID.
2nd column, Document category.
- k_nearest_neighboursint, optional
Number of nearest neighbors (k).
Defaults to 1.
- thread_ratiofloat, optional
Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.
Defaults to 0.0.
- langstr, optional
Only for on-premise HANA instance, specify the language type.
Defaults to 'EN'.
- index_namestr, optional
Only for on-premise HANA instance, specify the index name. If None, it will be generated.
- created_index{"index": xxx, "schema": xxx, "table": xxx}, optional
Only for on-premise HANA instance. Used the created index on the given table.
- Returns
- DataFrame (Cloud)
- Text classification result, structured as follows:
Predict data ID.
TARGET.
- Statistics table, structured as follows:
Predict data ID.
Training data ID.
Distance.
- DataFrame (On-Premise)
- Text classification result, structured as follows:
Predict data ID.
RANK.
CATEGORY_SCHEMA.
CATEGORY_TABLE.
CATEGORY_COLUMN.
CATEGORY_VALUE.
NEIGHBOR_COUNT.
SCORE.
Examples
The input DataFrame df:
>>> df.collect() ID CONTENT CATEGORY 0 doc1 term1 term2 term2 term3 term3 term3 CATEGORY_1 1 doc2 term2 term3 term3 term4 term4 term4 CATEGORY_1 2 doc3 term3 term4 term4 term5 term5 term5 CATEGORY_2 3 doc4 term3 term4 term4 term5 term5 term5 term5 term5 term5 CATEGORY_2 4 doc5 term4 term6 CATEGORY_3 5 doc6 term4 term6 term6 term6 CATEGORY_3
Invoke text_classification:
>>> res = text_classification(df.select(df.columns[0], df.columns[1]), df)
Result on a SAP HANA Cloud instance:
>>> res[0].head(1).collect() ID TARGET 0 doc1 CATEGORY_1
Result on a SAP HANA On-Premise instance:
>>> res[0].head(1).collect() ID RANK CATEGORY_SCHEMA CATEGORY_TABLE CATEGORY_COLUMN CATEGORY_VALUE NEIGHBOR_COUNT 0 doc1 1 "PAL_USER" "TM_CATEGORIZE_KNN_DT_6_REF_TBL" "CATEGORY" CATEGORY_1 1 ... SCORE ...0.5807794005266924131092309835366905
This function returns the top-ranked related documents for a query document based on Term Frequency - Inverse Document Frequency(TF-IDF) result or reference data..
- Parameters
- pred_dataDataFrame
1st column, Document content.
- ref_dataDataFrame or a tuple of DataFrame,
- DataFrame, reference data
1st column, ID
2nd column, Document content
3rd column, Document Category
- The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
- 1st DataFrame, TF-IDF Result
1st column, TM_TERM
2nd column, TF_VALUE
3rd column, IDF_VALUE
- 2nd DataFrame, Doc Term Freq Table
1st column, ID
2nd column, TM_TERM
3rd column, TM_TERM_FREQUENCY
- 3rd DataFrame, Doc Category Table
1st column, ID
2nd column, Document category
- topint, optional
Only show top N results. If 0, it shows all.
Defaults to 0.
- thresholdfloat, optional
Only the results which score bigger than this value will be put into the result table.
Defaults to 0.0.
- langstr, optional
Only for on-premise SAP HANA instance, specify the language type.
Defaults to 'EN'.
- index_namestr, optional
Only for on-premise SAP HANA instance, specify the index name.
- thread_ratiofloat, optional
Only for cloud version. Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.
Defaults to 0.0.
- created_index{"index": xxx, "schema": xxx, "table": xxx}, optional
Only for on-premise HANA instance. Used the created index on the given table.
- Returns
- DataFrame
Examples
The input DataFrame ref_df:
>>> ref_df.collect() ID CONTENT CATEGORY 0 doc1 term1 term2 term2 term3 term3 term3 CATEGORY_1 1 doc2 term2 term3 term3 term4 term4 term4 CATEGORY_1 2 doc3 term3 term4 term4 term5 term5 term5 CATEGORY_2 3 doc4 term3 term4 term4 term5 term5 term5 term5 term5 term5 CATEGORY_2 4 doc5 term4 term6 CATEGORY_3 5 doc6 term4 term6 term6 term6 CATEGORY_3
The input DataFrame pred_df:
>>> pred_df.collect() CONTENT 0 term2 term2 term3 term3
Invoke the function on a SAP HANA Cloud instance: tfidf is a DataFrame returned by tf_analysis function, please refer to the examples section of tf_analysis for its content.
>>> get_related_doc(pred_df, tfidf).collect() ID SCORE 0 doc2 0.891550 1 doc1 0.804670 2 doc3 0.042024 3 doc4 0.021225
Invoke the function on a SAP HANA On-Premise instance:
>>> res = get_related_doc(df_test1_onpremise, df_onpremise) >>> res.collect() ID RANK TOTAL_TERM_COUNT TERM_COUNT CORRELATIONS FACTORS ROTATED_FACTORS CLUSTER_LEVEL CLUSTER_LEFT 0 doc2 1 6 3 None None None None None 1 doc1 2 6 3 None None None None None 2 doc3 3 6 3 None None None None None 3 doc4 4 9 3 None None None None None ... CLUSTER_RIGHT HIGHLIGHTED_DOCUMENT HIGHLIGHTED_TERMTYPES SCORE ... None None None 0.8915504731053067732915451415465213 ... None None None 0.8046698732333942283290184604993556 ... None None None 0.04202449735779462125506711345224176 ... None None None 0.02122540837399113089478674964993843
This function returns the top-ranked related terms for a query term based on Term Frequency - Inverse Document Frequency(TF-IDF) result or reference data.
- Parameters
- pred_dataDataFrame
1st column, Document content.
- ref_dataDataFrame or a tuple of DataFrame,
- DataFrame, reference data
1st column, ID.
2nd column, Document content.
3rd column, Document Category.
- The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
- 1st DataFrame
1st column, TM_TERM.
2nd column, TF_VALUE.
3rd column, IDF_VALUE.
- 2nd DataFrame
1st column, ID.
2nd column, TM_TERM.
3rd column, TM_TERM_FREQUENCY.
- 3rd DataFrame
1st column, ID.
2nd column, Document category.
- topint, optional
Show top N results. If 0, it shows all.
Defaults to 0.
- thresholdfloat, optional
Only the results which score bigger than this value will be put into a result table.
Defaults to 0.0.
- langstr, optional
Only for on-premise HANA instance, specify the language type.
Defaults to 'EN'.
- index_namestr, optional
Only for on-premise HANA isntance, specify the index name.
- thread_ratiofloat, optional
Only for cloud version. Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.
Defaults to 0.0.
- created_index{"index": xxx, "schema": xxx, "table": xxx}, optional
Only for on-premise HANA instance. Used the created index on the given table.
- Returns
- DataFrame
Examples
The input DataFrame ref_df:
>>> ref_df.collect() ID CONTENT CATEGORY 0 doc1 term1 term2 term2 term3 term3 term3 CATEGORY_1 1 doc2 term2 term3 term3 term4 term4 term4 CATEGORY_1 2 doc3 term3 term4 term4 term5 term5 term5 CATEGORY_2 3 doc4 term3 term4 term4 term5 term5 term5 term5 term5 term5 CATEGORY_2 4 doc5 term4 term6 CATEGORY_3 5 doc6 term4 term6 term6 term6 CATEGORY_3
The input DataFrame pred_df:
>>> pred_df.collect() CONTENT 0 term3
Invoke the function on a SAP HANA Cloud instance,
>>> get_related_term(pred_df, ref_df).collect() ID SCORE 0 term3 1.000000 1 term2 0.923760 2 term1 0.774597 3 term4 0.550179 4 term5 0.346410
Invoke the function on a SAP HANA On-Premise instance:
>>> res = get_related_term(pred_df, ref_df) >>> res.collect() RANK TERM NORMALIZED_TERM TERM_TYPE TERM_FREQUENCY DOCUMENT_FREQUENCY CORRELATIONS 0 1 term3 term3 noun 7 4 None 1 2 term2 term2 noun 3 2 None 2 3 term1 term1 noun 1 1 None 3 4 term4 term4 noun 9 5 None 4 5 term5 term5 noun 9 2 None ... FACTORS ROTATED_FACTORS CLUSTER_LEVEL CLUSTER_LEFT CLUSTER_RIGHT SCORE ... None None None None None 1.0000003613794823387195265240734440 ... None None None None None 0.9237607645314674931213971831311937 ... None None None None None 0.7745969491648266869177064108953346 ... None None None None None 0.5501794128048571597133786781341769 ... None None None None None 0.3464102866993003515538873671175679
- hana_ml.text.tm.get_relevant_doc(pred_data, ref_data=None, top=None, threshold=None, lang='EN', index_name=None, thread_ratio=None, created_index=None)¶
This function returns the top-ranked documents that are relevant to a term based on Term Frequency - Inverse Document Frequency(TF-IDF) result or reference data.
- Parameters
- pred_dataDataFrame
1st column, Document content.
- ref_dataDataFrame or a tuple of DataFrame,
- DataFrame, reference data
1st column, ID.
2nd column, Document content.
3rd column, Document Category.
- The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
- 1st DataFrame
1st column, TM_TERM.
2nd column, TF_VALUE.
3rd column, IDF_VALUE.
- 2nd DataFrame
1st column, ID.
2nd column, TM_TERM.
3rd column, TM_TERM_FREQUENCY.
- 3rd DataFrame
1st column, ID.
2nd column, Document category.
- topint, optional
Show top N results. If 0, it shows all.
Defaults to 0.
- thresholdfloat, optional
Only the results which score bigger than this value will be put into a result table.
Defaults to 0.0.
- langstr, optional
Only for on-premise HANA instance, specify the language type.
Defaults to 'EN'.
- index_namestr, optional
Only for on-premise HANA isntance, specify the index name.
- thread_ratiofloat, optional
Only for cloud version. Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.
Defaults to 0.0.
- created_index{"index": xxx, "schema": xxx, "table": xxx}, optional
Only for on-premise HANA instance. Used the created index on the given table.
- Returns
- DataFrame
Examples
The input DataFrame ref_df:
>>> ref_df.collect() ID CONTENT CATEGORY 0 doc1 term1 term2 term2 term3 term3 term3 CATEGORY_1 1 doc2 term2 term3 term3 term4 term4 term4 CATEGORY_1 2 doc3 term3 term4 term4 term5 term5 term5 CATEGORY_2 3 doc4 term3 term4 term4 term5 term5 term5 term5 term5 term5 CATEGORY_2 4 doc5 term4 term6 CATEGORY_3 5 doc6 term4 term6 term6 term6 CATEGORY_3
The input DataFrame pred_df:
>>> pred_df.collect() CONTENT 0 term3
Invoke the function on a SAP HANA Cloud instance:
>>> get_relevant_doc(pred_df, ref_df).collect() ID SCORE 0 doc1 0.774597 1 doc2 0.516398 2 doc3 0.258199 3 doc4 0.258199
Invoke the function on a SAP HANA On-Premise instance:
>>> res = get_relevant_doc(pred_data, ref_data, top=4) >>> res.collect() ID RANK TOTAL_TERM_COUNT TERM_COUNT CORRELATIONS FACTORS ROTATED_FACTORS CLUSTER_LEVEL CLUSTER_LEFT 0 doc1 1 6 3 None None None None None 1 doc2 2 6 3 None None None None None 2 doc3 3 6 3 None None None None None 3 doc4 4 9 3 None None None None None ... CLUSTER_RIGHT HIGHLIGHTED_DOCUMENT HIGHLIGHTED_TERMTYPES SCORE ... None None None 0.7745969491648266869177064108953346 ... None None None 0.5163979661098845319600059156073257 ... None None None 0.2581989830549422659800029578036629 ... None None None 0.2581989830549422659800029578036629
- hana_ml.text.tm.get_relevant_term(pred_data, ref_data=None, top=None, threshold=None, lang='EN', index_name=None, thread_ratio=None, created_index=None)¶
This function returns the top-ranked relevant terms that describe a document based on Term Frequency - Inverse Document Frequency(TF-IDF) result or reference data.
- Parameters
- pred_dataDataFrame
1st column, Document content.
- ref_dataDataFrame or a tuple of DataFrame,
- DataFrame, reference data
1st column, ID.
2nd column, Document content.
3rd column, Document Category.
- The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
- 1st DataFrame
1st column, TM_TERM.
2nd column, TF_VALUE.
3rd column, IDF_VALUE.
- 2nd DataFrame
1st column, ID.
2nd column, TM_TERM.
3rd column, TM_TERM_FREQUENCY.
- 3rd DataFrame
1st column, ID.
2nd column, Document category.
- topint, optional
Show top N results. If 0, it shows all.
Defaults to 0.
- thresholdfloat, optional
Only the results which score bigger than THRESHOLD will be put into a result table.
Defaults to 0.0.
- langstr, optional
Only for on-premise HANA instance, specify the language type.
Defaults to 'EN'.
- index_namestr, optional
Only for on-premise HANA isntance, specify the index name.
- thread_ratiofloat, optional
Only for cloud version. Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.
Defaults to 0.0.
- created_index{"index": xxx, "schema": xxx, "table": xxx}, optional
Only for on-premise HANA instance. Used the created index on the given table.
- Returns
- DataFrame
Examples
The input DataFrame ref_df:
>>> ref_df.collect() ID CONTENT CATEGORY 0 doc1 term1 term2 term2 term3 term3 term3 CATEGORY_1 1 doc2 term2 term3 term3 term4 term4 term4 CATEGORY_1 2 doc3 term3 term4 term4 term5 term5 term5 CATEGORY_2 3 doc4 term3 term4 term4 term5 term5 term5 term5 term5 term5 CATEGORY_2 4 doc5 term4 term6 CATEGORY_3 5 doc6 term4 term6 term6 term6 CATEGORY_3
The input DataFrame pred_df:
>>> pred_df.collect() CONTENT 0 term3
Invoke the function on a SAP HANA Cloud instance,
>>> get_relevant_term(pred_df, ref_df).collect() ID SCORE 0 term3 1.0
Invoke the function on a SAP HANA On-Premise instance:
>>> res = get_relevant_term(pred_df, ref_df) >>> res.collect() RANK TERM NORMALIZED_TERM TERM_TYPE TERM_FREQUENCY DOCUMENT_FREQUENCY CORRELATIONS 0 1 term3 term3 noun 7 4 None
... FACTORS ROTATED_FACTORS CLUSTER_LEVEL CLUSTER_LEFT CLUSTER_RIGHT SCORE ... None None None None None 1.000002901113076436701021521002986
- hana_ml.text.tm.get_suggested_term(pred_data, ref_data=None, top=None, threshold=None, lang='EN', index_name=None, thread_ratio=None, created_index=None)¶
This function returns the top-ranked terms that match an initial substring based on Term Frequency - Inverse Document Frequency(TF-IDF) result or reference data.
- Parameters
- pred_dataDataFrame
1st column, Document content.
- ref_dataDataFrame or a tuple of DataFrame,
- DataFrame, reference data
1st column, ID.
2nd column, Document content.
3rd column, Document Category.
- The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
- 1st DataFrame
1st column, TM_TERM.
2nd column, TF_VALUE.
3rd column, IDF_VALUE.
- 2nd DataFrame
1st column, ID.
2nd column, TM_TERM.
3rd column, TM_TERM_FREQUENCY.
- 3rd DataFrame
1st column, ID.
2nd column, Document category.
- topint, optional
Show top N results. If 0, it shows all.
Defaults to 0.
- thresholdfloat, optional
Only the results which score bigger than this value will be put into a result table.
Defaults to 0.0.
- langstr, optional
Only for on-premise HANA instance, specify the language type.
Defaults to 'EN'.
- index_namestr, optional
Only for on-premise HANA isntance, specify the index name.
- thread_ratiofloat, optional
Only for cloud version. Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.
Defaults to 0.0.
- created_index{"index": xxx, "schema": xxx, "table": xxx}, optional
Only for on-premise HANA instance. Used the created index on the given table.
- Returns
- DataFrame
Examples
The input DataFrame ref_df:
>>> ref_df.collect() ID CONTENT CATEGORY 0 doc1 term1 term2 term2 term3 term3 term3 CATEGORY_1 1 doc2 term2 term3 term3 term4 term4 term4 CATEGORY_1 2 doc3 term3 term4 term4 term5 term5 term5 CATEGORY_2 3 doc4 term3 term4 term4 term5 term5 term5 term5 term5 term5 CATEGORY_2 4 doc5 term4 term6 CATEGORY_3 5 doc6 term4 term6 term6 term6 CATEGORY_3
The input DataFrame pred_df:
>>> pred_df.collect() CONTENT 0 term3
Invoke the function on a SAP HANA Cloud instance,
>>> get_suggested_term(pred_df, ref_df).collect() ID SCORE 0 term3 1.0
Invoke the function on a SAP HANA On-Premise instance:
>>> res = get_suggested_term(pred_df, ref_df) >>> res.collect() RANK TERM NORMALIZED_TERM TERM_TYPE TERM_FREQUENCY DOCUMENT_FREQUENCY SCORE 0 1 term3 term3 noun 7 4 0.999999999999999888977697537484346
- class hana_ml.text.tm.TFIDF¶
Bases:
hana_ml.algorithms.pal.pal_base.PALBaseClass for term frequency–inverse document frequency. Examples -------- Input dataframe for analysis:
>>> df_train.collect() ID CONTENT 0 doc1 term1 term2 term2 term3 term3 term3 1 doc2 term2 term3 term3 term4 term4 term4 2 doc3 term3 term4 term4 term5 term5 term5 3 doc5 term3 term4 term4 term5 term5 term5 term5 term5 term5 4 doc4 term4 term6 5 doc6 term4 term6 term6 term6
Creating TFIDF instance:
>>> tfidf = TFIDF()
Performing text_collector() on given dataframe:
>>> idf, _ = tfidf.text_collector(data=self.df_train)
>>> idf.collect() TM_TERMS TM_TERM_IDF_VALUE 0 term1 1.791759 1 term2 1.098612 2 term3 0.405465 3 term4 0.182322 4 term5 1.098612 5 term6 1.098612
Performing text_tfidf() on given dataframe:
>>> result = tfidf.text_tfidf(data=self.df_train)
>>> result.collect() ID TERMS TF_VALUE TFIDF_VALUE 0 doc1 term1 1.0 1.791759 1 doc1 term2 2.0 2.197225 2 doc1 term3 3.0 1.216395 3 doc2 term2 1.0 1.098612 4 doc2 term3 2.0 0.810930 5 doc2 term4 3.0 0.546965 6 doc3 term3 1.0 0.405465 7 doc3 term4 2.0 0.364643 8 doc3 term5 3.0 3.295837 9 doc5 term3 1.0 0.405465 10 doc5 term4 2.0 0.364643 11 doc5 term5 6.0 6.591674 12 doc4 term4 1.0 0.182322 13 doc4 term6 1.0 1.098612 14 doc6 term4 1.0 0.182322 15 doc6 term6 3.0 3.295837
Methods
add_attribute(attr_key, attr_val)Function to add attribute.
Disable with hint.
Enable no inline.
Enable parallel by parameter partitions.
Return the execute_statement for training.
Get PAL fit parmeters.
Parse sql lines containing the parameter definitions.
Return the execute_statement for predicting.
Get PAL predict parmeters.
Return the execute_statement for scoring.
Get PAL score parmeters.
Extract the specific function call of the PAL function from the sql code.
Checks if the model can be saved.
load_model(model)Function to load fitted model.
set_scale_out([route_to, no_route_to, ...])HANA statement routing.
text_collector(data)Its use is primarily compute inverse document frequency of documents which provided by user.
text_tfidf(data[, idf])Its use is primarily compute term frequency - inverse document frequency by document.
- text_collector(data)¶
Its use is primarily compute inverse document frequency of documents which provided by user.
- Parameters
- dataDataFrame
Data to be analysis. The first column of the input data table is assumed to be an ID column.
- Returns
- DataFrame
Inverse document frequency of documents.
Extended table.
- text_tfidf(data, idf=None)¶
Its use is primarily compute term frequency - inverse document frequency by document.
- Parameters
- dataDataFrame
Data to be analysis.
The first column of the input data table is assumed to be an ID column.
- idfDataFrame, optional
Inverse document frequency of documents.
- Returns
- DataFrame
Term frequency - inverse document frequency by document.
- add_attribute(attr_key, attr_val)¶
Function to add attribute.
- disable_with_hint()¶
Disable with hint.
- enable_no_inline()¶
Enable no inline.
- enable_parallel_by_parameter_partitions()¶
Enable parallel by parameter partitions.
- get_fit_execute_statement()¶
Return the execute_statement for training.
- get_fit_parameters()¶
Get PAL fit parmeters.
- Returns
- Array of tuples, where each tuple describes a parameter like (name, value, type)
- get_parameters()¶
Parse sql lines containing the parameter definitions. In the sql code all the parameters are defined by four arrays, where the first one contains the parameter name, and one of the other three contains the value fitting to the parameter, while the other two are NULL. This format should be changed into a simple key-value based storage.
- Returns
- dict of array of tuples, where each tuple describes a parameter like (name, value, type)
- get_predict_execute_statement()¶
Return the execute_statement for predicting.
- get_predict_parameters()¶
Get PAL predict parmeters.
- Returns
- Array of tuples, where each tuple describes a parameter like (name, value, type)
- get_score_execute_statement()¶
Return the execute_statement for scoring.
- get_score_parameters()¶
Get PAL score parmeters.
- Returns
- Array of tuples, where each tuple describes a parameter like (name, value, type)
- get_store_procedure()¶
Extract the specific function call of the PAL function from the sql code. Nevertheless it only detects the synonyms that have to be resolved afterwards
- Returns
- The procedure name synonym
- CALL "SYS_AFL.PAL_RANDOM_FORREST" (...) -> SYS_AFL.PAL_RANDOM_FORREST"
- is_fitted()¶
Checks if the model can be saved. To be overridden if the model is not stored in model_ attribute.
- Returns
- bool
True if the model is ready to be saved.
- load_model(model)¶
Function to load fitted model.
- Parameters
- modelDataFrame
HANA DataFrame for fitted model.
- set_scale_out(route_to=None, no_route_to=None, route_by=None, route_by_cardinality=None, data_transfer_cost=None, route_optimization_level=None, workload_class=None)¶
HANA statement routing.
- Parameters
- route_tostr, optional
Routes the query to the specified volume ID or service type.
- no_route_tostr or list of str, optional
Avoids query routing to a specified volume ID or service type.
- route_bystr, optional
Routes the query to the hosts related to the base table(s) of the specified projection view(s).
- route_by_cardinalitystr or list of str, optional
Routes the query to the hosts related to the base table(s) of the specified projection view(s) with the highest cardinality from the input list.
- data_transfer_costint, optional
Guides the optimizer to use the weighting factor for the data transfer cost. The value 0 ignores the data transfer cost.
- route_optimization_level{'mininal', 'all'}, optional
Guides the optimizer to compile with ROUTE_OPTIMIZATION_LEVEL (MINIMAL) or to default to ROUTE_OPTIMIZATION_LEVEL. If the MINIMAL compiled plan is cached, then it compiles once more using the default optimization level during the first execution. This hint is primarily used to shorten statement routing decisions during the initial compilation.
- workload_class, str, optional
Routes the query via workload class. ROUTE_TO() statement hint has higher precedence than WORKLOAD_CLASS() statement hint.