hana_ml.text.tm package

The text.tm package consists of the following section:

hana_ml.text.tm

This module provides various functions of text minig. The following functions are available:
hana_ml.text.tm.tf_analysis(data)

Perform Term Frequency(TF) analysis on the given document. TF is the number of occurrences of term in document.

Parameters
dataDataFrame
  • 1st column, ID.

  • 2nd column, Document content.

  • 3rd column, Document category.

Returns
A tuple of DataFrame
TF-IDF result, structured as follows:
  • TM_TERM.

  • TM_TERM_FREQUENCY.

  • TM_IDF_FREQUENCY.

  • TF_VALUE.

  • IDF_VALUE.

  • TF_IDF_VALUE.

Document term frequency table, structured as follows:
  • ID.

  • TM_TERM.

  • TM_TERM_FREQUENCY.

Document category table, structured as follows:
  • ID.

  • Document category.

Examples

The input DataFrame df:

>>> df.collect()
      ID                                                  CONTENT       CATEGORY
0   doc1                      term1 term2 term2 term3 term3 term3     CATEGORY_1
1   doc2                      term2 term3 term3 term4 term4 term4     CATEGORY_1
2   doc3                      term3 term4 term4 term5 term5 term5     CATEGORY_2
3   doc4    term3 term4 term4 term5 term5 term5 term5 term5 term5     CATEGORY_2
4   doc5                                              term4 term6     CATEGORY_3
5   doc6                                  term4 term6 term6 term6     CATEGORY_3

Invoke tf_analysis function:

>>> tfidf= tf_analysis(df)

Output:

>>> tfidf[0].head(3).collect()
  TM_TERMS TM_TERM_TF_F  TM_TERM_IDF_F  TM_TERM_TF_V  TM_TERM_IDF_V
0    term1            1              1      0.030303       1.791759
1    term2            3              2      0.090909       1.098612
2    term3            7              4      0.212121       0.405465
>>> tfidf[1].head(3).collect()
     ID TM_TERMS  TM_TERM_FREQUENCY
0  doc1    term1                  1
1  doc1    term2                  2
2  doc1    term3                  3
>>> tfidf[2].head(3).collect()
      ID    CATEGORY
0   doc1  CATEGORY_1
1   doc2  CATEGORY_1
2   doc3  CATEGORY_2
hana_ml.text.tm.text_classification(pred_data, ref_data=None, k_nearest_neighbours=None, thread_ratio=None, lang='EN', index_name=None, created_index=None)

This function classifies (categorizes) an input document with respect to sets of categories (taxonomies).

Parameters
pred_dataDataFrame

The prediction data for classification.

  • 1st column, ID.

  • 2nd column, Document content.

ref_dataDataFrame or a tuple of DataFrame,
  • DataFrame, reference data
    • 1st column, ID.

    • 2nd column, Document content.

    • 3rd column, Document Category.

The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
  • 1st DataFrame
    • 1st column, TM_TERM.

    • 2nd column, TF_VALUE.

    • 3rd column, IDF_VALUE.

  • 2nd DataFrame
    • 1st column, ID.

    • 2nd column, TM_TERM.

    • 3rd column, TM_TERM_FREQUENCY.

  • 3rd DataFrame
    • 1st column, ID.

    • 2nd column, Document category.

k_nearest_neighboursint, optional

Number of nearest neighbors (k).

Defaults to 1.

thread_ratiofloat, optional

Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.

Defaults to 0.0.

langstr, optional

Only for on-premise HANA instance, specify the language type.

Defaults to 'EN'.

index_namestr, optional

Only for on-premise HANA instance, specify the index name. If None, it will be generated.

created_index{"index": xxx, "schema": xxx, "table": xxx}, optional

Only for on-premise HANA instance. Used the created index on the given table.

Returns
DataFrame (Cloud)
Text classification result, structured as follows:
  • Predict data ID.

  • TARGET.

Statistics table, structured as follows:
  • Predict data ID.

  • Training data ID.

  • Distance.

DataFrame (On-Premise)
Text classification result, structured as follows:
  • Predict data ID.

  • RANK.

  • CATEGORY_SCHEMA.

  • CATEGORY_TABLE.

  • CATEGORY_COLUMN.

  • CATEGORY_VALUE.

  • NEIGHBOR_COUNT.

  • SCORE.

Examples

The input DataFrame df:

>>> df.collect()
      ID                                                  CONTENT       CATEGORY
0   doc1                      term1 term2 term2 term3 term3 term3     CATEGORY_1
1   doc2                      term2 term3 term3 term4 term4 term4     CATEGORY_1
2   doc3                      term3 term4 term4 term5 term5 term5     CATEGORY_2
3   doc4    term3 term4 term4 term5 term5 term5 term5 term5 term5     CATEGORY_2
4   doc5                                              term4 term6     CATEGORY_3
5   doc6                                  term4 term6 term6 term6     CATEGORY_3

Invoke text_classification:

>>> res = text_classification(df.select(df.columns[0], df.columns[1]), df)

Result on a SAP HANA Cloud instance:

>>> res[0].head(1).collect()
       ID     TARGET
0    doc1 CATEGORY_1

Result on a SAP HANA On-Premise instance:

>>> res[0].head(1).collect()
     ID RANK  CATEGORY_SCHEMA                   CATEGORY_TABLE    CATEGORY_COLUMN  CATEGORY_VALUE  NEIGHBOR_COUNT
0  doc1    1       "PAL_USER" "TM_CATEGORIZE_KNN_DT_6_REF_TBL"         "CATEGORY"      CATEGORY_1               1
...                               SCORE
...0.5807794005266924131092309835366905

This function returns the top-ranked related documents for a query document based on Term Frequency - Inverse Document Frequency(TF-IDF) result or reference data..

Parameters
pred_dataDataFrame
  • 1st column, Document content.

ref_dataDataFrame or a tuple of DataFrame,
  • DataFrame, reference data
    • 1st column, ID

    • 2nd column, Document content

    • 3rd column, Document Category

The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
  • 1st DataFrame, TF-IDF Result
    • 1st column, TM_TERM

    • 2nd column, TF_VALUE

    • 3rd column, IDF_VALUE

  • 2nd DataFrame, Doc Term Freq Table
    • 1st column, ID

    • 2nd column, TM_TERM

    • 3rd column, TM_TERM_FREQUENCY

  • 3rd DataFrame, Doc Category Table
    • 1st column, ID

    • 2nd column, Document category

topint, optional

Only show top N results. If 0, it shows all.

Defaults to 0.

thresholdfloat, optional

Only the results which score bigger than this value will be put into the result table.

Defaults to 0.0.

langstr, optional

Only for on-premise SAP HANA instance, specify the language type.

Defaults to 'EN'.

index_namestr, optional

Only for on-premise SAP HANA instance, specify the index name.

thread_ratiofloat, optional

Only for cloud version. Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.

Defaults to 0.0.

created_index{"index": xxx, "schema": xxx, "table": xxx}, optional

Only for on-premise HANA instance. Used the created index on the given table.

Returns
DataFrame

Examples

The input DataFrame ref_df:

>>> ref_df.collect()
      ID                                                  CONTENT       CATEGORY
0   doc1                      term1 term2 term2 term3 term3 term3     CATEGORY_1
1   doc2                      term2 term3 term3 term4 term4 term4     CATEGORY_1
2   doc3                      term3 term4 term4 term5 term5 term5     CATEGORY_2
3   doc4    term3 term4 term4 term5 term5 term5 term5 term5 term5     CATEGORY_2
4   doc5                                              term4 term6     CATEGORY_3
5   doc6                                  term4 term6 term6 term6     CATEGORY_3

The input DataFrame pred_df:

>>> pred_df.collect()
                   CONTENT
0  term2 term2 term3 term3

Invoke the function on a SAP HANA Cloud instance: tfidf is a DataFrame returned by tf_analysis function, please refer to the examples section of tf_analysis for its content.

>>> get_related_doc(pred_df, tfidf).collect()
       ID       SCORE
0    doc2    0.891550
1    doc1    0.804670
2    doc3    0.042024
3    doc4    0.021225

Invoke the function on a SAP HANA On-Premise instance:

>>> res = get_related_doc(df_test1_onpremise, df_onpremise)
>>> res.collect()
   ID    RANK   TOTAL_TERM_COUNT  TERM_COUNT  CORRELATIONS  FACTORS  ROTATED_FACTORS  CLUSTER_LEVEL  CLUSTER_LEFT
0  doc2     1                  6           3          None     None             None           None          None
1  doc1     2                  6           3          None     None             None           None          None
2  doc3     3                  6           3          None     None             None           None          None
3  doc4     4                  9           3          None     None             None           None          None
... CLUSTER_RIGHT  HIGHLIGHTED_DOCUMENT  HIGHLIGHTED_TERMTYPES                                   SCORE
...          None                  None                   None    0.8915504731053067732915451415465213
...          None                  None                   None    0.8046698732333942283290184604993556
...          None                  None                   None   0.04202449735779462125506711345224176
...          None                  None                   None   0.02122540837399113089478674964993843

This function returns the top-ranked related terms for a query term based on Term Frequency - Inverse Document Frequency(TF-IDF) result or reference data.

Parameters
pred_dataDataFrame
  • 1st column, Document content.

ref_dataDataFrame or a tuple of DataFrame,
  • DataFrame, reference data
    • 1st column, ID.

    • 2nd column, Document content.

    • 3rd column, Document Category.

The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
  • 1st DataFrame
    • 1st column, TM_TERM.

    • 2nd column, TF_VALUE.

    • 3rd column, IDF_VALUE.

  • 2nd DataFrame
    • 1st column, ID.

    • 2nd column, TM_TERM.

    • 3rd column, TM_TERM_FREQUENCY.

  • 3rd DataFrame
    • 1st column, ID.

    • 2nd column, Document category.

topint, optional

Show top N results. If 0, it shows all.

Defaults to 0.

thresholdfloat, optional

Only the results which score bigger than this value will be put into a result table.

Defaults to 0.0.

langstr, optional

Only for on-premise HANA instance, specify the language type.

Defaults to 'EN'.

index_namestr, optional

Only for on-premise HANA isntance, specify the index name.

thread_ratiofloat, optional

Only for cloud version. Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.

Defaults to 0.0.

created_index{"index": xxx, "schema": xxx, "table": xxx}, optional

Only for on-premise HANA instance. Used the created index on the given table.

Returns
DataFrame

Examples

The input DataFrame ref_df:

>>> ref_df.collect()
      ID                                                  CONTENT       CATEGORY
0   doc1                      term1 term2 term2 term3 term3 term3     CATEGORY_1
1   doc2                      term2 term3 term3 term4 term4 term4     CATEGORY_1
2   doc3                      term3 term4 term4 term5 term5 term5     CATEGORY_2
3   doc4    term3 term4 term4 term5 term5 term5 term5 term5 term5     CATEGORY_2
4   doc5                                              term4 term6     CATEGORY_3
5   doc6                                  term4 term6 term6 term6     CATEGORY_3

The input DataFrame pred_df:

>>> pred_df.collect()
  CONTENT
0   term3

Invoke the function on a SAP HANA Cloud instance,

>>> get_related_term(pred_df, ref_df).collect()
        ID       SCORE
0    term3    1.000000
1    term2    0.923760
2    term1    0.774597
3    term4    0.550179
4    term5    0.346410

Invoke the function on a SAP HANA On-Premise instance:

>>> res = get_related_term(pred_df, ref_df)
>>> res.collect()
  RANK  TERM  NORMALIZED_TERM  TERM_TYPE  TERM_FREQUENCY  DOCUMENT_FREQUENCY  CORRELATIONS
0    1 term3            term3       noun               7                   4          None
1    2 term2            term2       noun               3                   2          None
2    3 term1            term1       noun               1                   1          None
3    4 term4            term4       noun               9                   5          None
4    5 term5            term5       noun               9                   2          None
... FACTORS  ROTATED_FACTORS  CLUSTER_LEVEL  CLUSTER_LEFT  CLUSTER_RIGHT                                 SCORE
...    None             None           None          None           None  1.0000003613794823387195265240734440
...    None             None           None          None           None  0.9237607645314674931213971831311937
...    None             None           None          None           None  0.7745969491648266869177064108953346
...    None             None           None          None           None  0.5501794128048571597133786781341769
...    None             None           None          None           None  0.3464102866993003515538873671175679
hana_ml.text.tm.get_relevant_doc(pred_data, ref_data=None, top=None, threshold=None, lang='EN', index_name=None, thread_ratio=None, created_index=None)

This function returns the top-ranked documents that are relevant to a term based on Term Frequency - Inverse Document Frequency(TF-IDF) result or reference data.

Parameters
pred_dataDataFrame
  • 1st column, Document content.

ref_dataDataFrame or a tuple of DataFrame,
  • DataFrame, reference data
    • 1st column, ID.

    • 2nd column, Document content.

    • 3rd column, Document Category.

The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
  • 1st DataFrame
    • 1st column, TM_TERM.

    • 2nd column, TF_VALUE.

    • 3rd column, IDF_VALUE.

  • 2nd DataFrame
    • 1st column, ID.

    • 2nd column, TM_TERM.

    • 3rd column, TM_TERM_FREQUENCY.

  • 3rd DataFrame
    • 1st column, ID.

    • 2nd column, Document category.

topint, optional

Show top N results. If 0, it shows all.

Defaults to 0.

thresholdfloat, optional

Only the results which score bigger than this value will be put into a result table.

Defaults to 0.0.

langstr, optional

Only for on-premise HANA instance, specify the language type.

Defaults to 'EN'.

index_namestr, optional

Only for on-premise HANA isntance, specify the index name.

thread_ratiofloat, optional

Only for cloud version. Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.

Defaults to 0.0.

created_index{"index": xxx, "schema": xxx, "table": xxx}, optional

Only for on-premise HANA instance. Used the created index on the given table.

Returns
DataFrame

Examples

The input DataFrame ref_df:

>>> ref_df.collect()
      ID                                                  CONTENT       CATEGORY
0   doc1                      term1 term2 term2 term3 term3 term3     CATEGORY_1
1   doc2                      term2 term3 term3 term4 term4 term4     CATEGORY_1
2   doc3                      term3 term4 term4 term5 term5 term5     CATEGORY_2
3   doc4    term3 term4 term4 term5 term5 term5 term5 term5 term5     CATEGORY_2
4   doc5                                              term4 term6     CATEGORY_3
5   doc6                                  term4 term6 term6 term6     CATEGORY_3

The input DataFrame pred_df:

>>> pred_df.collect()
                   CONTENT
0                    term3

Invoke the function on a SAP HANA Cloud instance:

>>> get_relevant_doc(pred_df, ref_df).collect()
       ID       SCORE
0    doc1    0.774597
1    doc2    0.516398
2    doc3    0.258199
3    doc4    0.258199

Invoke the function on a SAP HANA On-Premise instance:

>>> res = get_relevant_doc(pred_data, ref_data, top=4)
>>> res.collect()
     ID    RANK   TOTAL_TERM_COUNT  TERM_COUNT  CORRELATIONS  FACTORS  ROTATED_FACTORS  CLUSTER_LEVEL  CLUSTER_LEFT
0  doc1       1                  6           3          None     None             None           None          None
1  doc2       2                  6           3          None     None             None           None          None
2  doc3       3                  6           3          None     None             None           None          None
3  doc4       4                  9           3          None     None             None           None          None
... CLUSTER_RIGHT  HIGHLIGHTED_DOCUMENT  HIGHLIGHTED_TERMTYPES                                   SCORE
...          None                  None                   None    0.7745969491648266869177064108953346
...          None                  None                   None    0.5163979661098845319600059156073257
...          None                  None                   None    0.2581989830549422659800029578036629
...          None                  None                   None    0.2581989830549422659800029578036629
hana_ml.text.tm.get_relevant_term(pred_data, ref_data=None, top=None, threshold=None, lang='EN', index_name=None, thread_ratio=None, created_index=None)

This function returns the top-ranked relevant terms that describe a document based on Term Frequency - Inverse Document Frequency(TF-IDF) result or reference data.

Parameters
pred_dataDataFrame
  • 1st column, Document content.

ref_dataDataFrame or a tuple of DataFrame,
  • DataFrame, reference data
    • 1st column, ID.

    • 2nd column, Document content.

    • 3rd column, Document Category.

The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
  • 1st DataFrame
    • 1st column, TM_TERM.

    • 2nd column, TF_VALUE.

    • 3rd column, IDF_VALUE.

  • 2nd DataFrame
    • 1st column, ID.

    • 2nd column, TM_TERM.

    • 3rd column, TM_TERM_FREQUENCY.

  • 3rd DataFrame
    • 1st column, ID.

    • 2nd column, Document category.

topint, optional

Show top N results. If 0, it shows all.

Defaults to 0.

thresholdfloat, optional

Only the results which score bigger than THRESHOLD will be put into a result table.

Defaults to 0.0.

langstr, optional

Only for on-premise HANA instance, specify the language type.

Defaults to 'EN'.

index_namestr, optional

Only for on-premise HANA isntance, specify the index name.

thread_ratiofloat, optional

Only for cloud version. Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.

Defaults to 0.0.

created_index{"index": xxx, "schema": xxx, "table": xxx}, optional

Only for on-premise HANA instance. Used the created index on the given table.

Returns
DataFrame

Examples

The input DataFrame ref_df:

>>> ref_df.collect()
      ID                                                  CONTENT       CATEGORY
0   doc1                      term1 term2 term2 term3 term3 term3     CATEGORY_1
1   doc2                      term2 term3 term3 term4 term4 term4     CATEGORY_1
2   doc3                      term3 term4 term4 term5 term5 term5     CATEGORY_2
3   doc4    term3 term4 term4 term5 term5 term5 term5 term5 term5     CATEGORY_2
4   doc5                                              term4 term6     CATEGORY_3
5   doc6                                  term4 term6 term6 term6     CATEGORY_3

The input DataFrame pred_df:

>>> pred_df.collect()
  CONTENT
0   term3

Invoke the function on a SAP HANA Cloud instance,

>>> get_relevant_term(pred_df, ref_df).collect()
        ID   SCORE
0    term3     1.0

Invoke the function on a SAP HANA On-Premise instance:

>>> res = get_relevant_term(pred_df, ref_df)
>>> res.collect()
  RANK  TERM  NORMALIZED_TERM  TERM_TYPE  TERM_FREQUENCY  DOCUMENT_FREQUENCY  CORRELATIONS
0    1 term3            term3       noun               7                   4          None

... FACTORS ROTATED_FACTORS CLUSTER_LEVEL CLUSTER_LEFT CLUSTER_RIGHT SCORE ... None None None None None 1.000002901113076436701021521002986

hana_ml.text.tm.get_suggested_term(pred_data, ref_data=None, top=None, threshold=None, lang='EN', index_name=None, thread_ratio=None, created_index=None)

This function returns the top-ranked terms that match an initial substring based on Term Frequency - Inverse Document Frequency(TF-IDF) result or reference data.

Parameters
pred_dataDataFrame
  • 1st column, Document content.

ref_dataDataFrame or a tuple of DataFrame,
  • DataFrame, reference data
    • 1st column, ID.

    • 2nd column, Document content.

    • 3rd column, Document Category.

The ref_data could also be a tuple of DataFrame, reference TF-IDF data:
  • 1st DataFrame
    • 1st column, TM_TERM.

    • 2nd column, TF_VALUE.

    • 3rd column, IDF_VALUE.

  • 2nd DataFrame
    • 1st column, ID.

    • 2nd column, TM_TERM.

    • 3rd column, TM_TERM_FREQUENCY.

  • 3rd DataFrame
    • 1st column, ID.

    • 2nd column, Document category.

topint, optional

Show top N results. If 0, it shows all.

Defaults to 0.

thresholdfloat, optional

Only the results which score bigger than this value will be put into a result table.

Defaults to 0.0.

langstr, optional

Only for on-premise HANA instance, specify the language type.

Defaults to 'EN'.

index_namestr, optional

Only for on-premise HANA isntance, specify the index name.

thread_ratiofloat, optional

Only for cloud version. Specifies the ratio of total number of threads that can be used by this function. The range of this parameter is from 0 to 1, where 0 means only using 1 thread, and 1 means using at most all the currently available threads. Values outside this range are ignored and this function heuristically determines the number of threads to use.

Defaults to 0.0.

created_index{"index": xxx, "schema": xxx, "table": xxx}, optional

Only for on-premise HANA instance. Used the created index on the given table.

Returns
DataFrame

Examples

The input DataFrame ref_df:

>>> ref_df.collect()
      ID                                                  CONTENT       CATEGORY
0   doc1                      term1 term2 term2 term3 term3 term3     CATEGORY_1
1   doc2                      term2 term3 term3 term4 term4 term4     CATEGORY_1
2   doc3                      term3 term4 term4 term5 term5 term5     CATEGORY_2
3   doc4    term3 term4 term4 term5 term5 term5 term5 term5 term5     CATEGORY_2
4   doc5                                              term4 term6     CATEGORY_3
5   doc6                                  term4 term6 term6 term6     CATEGORY_3

The input DataFrame pred_df:

>>> pred_df.collect()
  CONTENT
0   term3

Invoke the function on a SAP HANA Cloud instance,

>>> get_suggested_term(pred_df, ref_df).collect()
        ID     SCORE
0    term3       1.0

Invoke the function on a SAP HANA On-Premise instance:

>>> res = get_suggested_term(pred_df, ref_df)
>>> res.collect()
  RANK   TERM  NORMALIZED_TERM  TERM_TYPE  TERM_FREQUENCY  DOCUMENT_FREQUENCY                                SCORE
0    1      term3            term3       noun                   7                   4  0.999999999999999888977697537484346
class hana_ml.text.tm.TFIDF

Bases: hana_ml.algorithms.pal.pal_base.PALBase

Class for term frequency–inverse document frequency. Examples -------- Input dataframe for analysis:

>>> df_train.collect()
        ID      CONTENT
    0       doc1    term1 term2 term2 term3 term3 term3
    1       doc2    term2 term3 term3 term4 term4 term4
    2       doc3    term3 term4 term4 term5 term5 term5
    3       doc5    term3 term4 term4 term5 term5 term5 term5 term5 term5
    4       doc4    term4 term6
    5       doc6    term4 term6 term6 term6

Creating TFIDF instance:

>>> tfidf = TFIDF()

Performing text_collector() on given dataframe:

>>> idf, _ = tfidf.text_collector(data=self.df_train)
>>> idf.collect()
        TM_TERMS    TM_TERM_IDF_VALUE
    0       term1       1.791759
    1       term2       1.098612
    2       term3       0.405465
    3       term4       0.182322
    4       term5       1.098612
    5       term6       1.098612

Performing text_tfidf() on given dataframe:

>>> result = tfidf.text_tfidf(data=self.df_train)
>>> result.collect()
        ID      TERMS       TF_VALUE        TFIDF_VALUE
    0       doc1    term1   1.0             1.791759
    1       doc1    term2   2.0             2.197225
    2       doc1    term3   3.0             1.216395
    3       doc2    term2   1.0             1.098612
    4       doc2    term3   2.0             0.810930
    5       doc2    term4   3.0             0.546965
    6       doc3    term3   1.0             0.405465
    7       doc3    term4   2.0             0.364643
    8       doc3    term5   3.0             3.295837
    9       doc5    term3   1.0             0.405465
    10      doc5    term4   2.0             0.364643
    11      doc5    term5   6.0             6.591674
    12      doc4    term4   1.0             0.182322
    13      doc4    term6   1.0             1.098612
    14      doc6    term4   1.0             0.182322
    15      doc6    term6   3.0             3.295837

Methods

add_attribute(attr_key, attr_val)

Function to add attribute.

disable_with_hint()

Disable with hint.

enable_no_inline()

Enable no inline.

enable_parallel_by_parameter_partitions()

Enable parallel by parameter partitions.

get_fit_execute_statement()

Return the execute_statement for train, predict and score.

get_fit_parameters()

Get PAL fit parmeters.

get_parameters()

Parse sql lines containing the parameter definitions.

get_predict_execute_statement()

Return the execute_statement for train, predict and score.

get_predict_parameters()

Get PAL predict parmeters.

get_score_execute_statement()

Return the execute_statement for train, predict and score.

get_score_parameters()

Get PAL score parmeters.

get_store_procedure()

Extract the specific function call of the PAL function from the sql code.

is_fitted()

Checks if the model can be saved.

load_model(model)

Function to load fitted model.

set_scale_out([route_to, no_route_to, ...])

ERN client settings.

text_collector(data)

Its use is primarily compute inverse document frequency of documents which provided by user.

text_tfidf(data[, idf])

Its use is primarily compute term frequency - inverse document frequency by document.

text_collector(data)

Its use is primarily compute inverse document frequency of documents which provided by user.

Parameters
dataDataFrame

Data to be analysis. The first column of the input data table is assumed to be an ID column.

Returns
DataFrame
  • Inverse document frequency of documents.

  • Extended table.

text_tfidf(data, idf=None)

Its use is primarily compute term frequency - inverse document frequency by document.

Parameters
dataDataFrame

Data to be analysis.

The first column of the input data table is assumed to be an ID column.

idfDataFrame, optional

Inverse document frequency of documents.

Returns
DataFrame
  • Term frequency - inverse document frequency by document.

add_attribute(attr_key, attr_val)

Function to add attribute.

disable_with_hint()

Disable with hint.

enable_no_inline()

Enable no inline.

enable_parallel_by_parameter_partitions()

Enable parallel by parameter partitions.

get_fit_execute_statement()

Return the execute_statement for train, predict and score.

get_fit_parameters()

Get PAL fit parmeters.

Returns
Array of tuples, where each tuple describes a parameter like (name, value, type)
get_parameters()

Parse sql lines containing the parameter definitions. In the sql code all the parameters are defined by four arrays, where the first one contains the parameter name, and one of the other three contains the value fitting to the parameter, while the other two are NULL. This format should be changed into a simple key-value based storage.

Returns
dict of array of tuples, where each tuple describes a parameter like (name, value, type)
get_predict_execute_statement()

Return the execute_statement for train, predict and score.

get_predict_parameters()

Get PAL predict parmeters.

Returns
Array of tuples, where each tuple describes a parameter like (name, value, type)
get_score_execute_statement()

Return the execute_statement for train, predict and score.

get_score_parameters()

Get PAL score parmeters.

Returns
Array of tuples, where each tuple describes a parameter like (name, value, type)
get_store_procedure()

Extract the specific function call of the PAL function from the sql code. Nevertheless it only detects the synonyms that have to be resolved afterwards

Returns
The procedure name synonym
CALL "SYS_AFL.PAL_RANDOM_FORREST" (...) -> SYS_AFL.PAL_RANDOM_FORREST"
is_fitted()

Checks if the model can be saved. To be overridden if the model is not stored in model_ attribute.

Returns
bool

True if the model is ready to be saved.

load_model(model)

Function to load fitted model.

Parameters
modelDataFrame

HANA DataFrame for fitted model.

set_scale_out(route_to=None, no_route_to=None, route_by=None, route_by_cardinality=None, data_transfer_cost=None, route_optimization_level=None, workload_class=None)

ERN client settings.

Parameters
route_tostr, optional

Routes the query to the specified volume ID or service type.

no_route_tostr or list of str, optional

Avoids query routing to a specified volume ID or service type.

route_bystr, optional

Routes the query to the hosts related to the base table(s) of the specified projection view(s).

route_by_cardinalitystr or list of str, optional

Routes the query to the hosts related to the base table(s) of the specified projection view(s) with the highest cardinality from the input list.

data_transfer_costint, optional

Guides the optimizer to use the weighting factor for the data transfer cost. The value 0 ignores the data transfer cost.

route_optimization_level{'mininal', 'all'}, optional

Guides the optimizer to compile with ROUTE_OPTIMIZATION_LEVEL (MINIMAL) or to default to ROUTE_OPTIMIZATION_LEVEL. If the MINIMAL compiled plan is cached, then it compiles once more using the default optimization level during the first execution. This hint is primarily used to shorten statement routing decisions during the initial compilation.

workload_class, str, optional

Routes the query via workload class. ROUTE_TO() statement hint has higher precedence than WORKLOAD_CLASS() statement hint.