Show TOC

Configuring Language Recognition ManuallyLocate this document in the navigation structure


Language recognition takes place using the lexicon software of third-party providers and the TREX preprocessor. You can configure language recognition.

Naming Convention

  • Central directory for executable files < CENTRAL_DIR>

    • UNIX: usr/SAP/<SAPSID>/SYS/exe/nuc/<OS>

    • Windows: <drive>:usr\SAP\<SAPSID>\SYS\exe\nuc\<OS>

      As part of the CPE (Central Patch Environment), the sapcpe program takes on the automatic synchronization of executable files and copies them from the central directory for executable files, <CENTRAL_DIR>, to the local directory for executable files, <TREX_DIR>\exe. When you restart TREX, the system automatically launches the sapcpe program. During all subsequent starts, sapcpe checks whether or not the local executable files are up-to-date and copies new or changed executable files from the central directory to the local directory, <TREX_DIR>\exe.

  • TREX installation directory <TREX_INSTALL>

    • UNIX: /usr/sap/<sapsid>/trx<instance_number>/<TREX_host_name>

    • Windows: <disk_drive>:\usr\sap\<SAPSID>\TRX<instance_number>\<TREX_host_name>

Modifying Language Recognition with Lexicon Software

Language recognition with lexicon software includes the following areas:

  • Configure additional languages

    You can retrospectively configure language recognition for additional languages. When TREX is installed, you select the languages to be identified by language recognition.


    Only activate the languages that appear in your documents and that you also want to index. Doing this optimizes the performance of the language recognition procedure and of indexing in general. Moreover, the less languages used, the better the results of language recognition.

  • Disregarding parts of HTML or XML documents

    You can configure the system so that certain parts of HTML or XML documents are ignored when the language recognition procedure takes place. Documents that are to be indexed and are in HTML or XML format often contain elements (such as JavaScript programs) that damage the performance of the language recognition procedure.

  • Changing the number of characters for language recognition

    Language recognition using lexicon software is set up so that only a certain amount of characters are taken into consideration. This is usually set to its optimum value at delivery. If it turns out that languages are not being recognized correctly, you can increase the quantity of text that is taken into consideration.


    In certain cases, language recognition might not deliver the correct language. In particular, problems can occur when processing documents that are very short or that contain a large number of abbreviations or words loaned from another language.

You modify the lexicon software language recognition by editing the std.langid-config configuration file on the TREX preprocessor. The settings are valid for all indexes on the preprocessor. If you are using more than one TREX preprocessor, you need to modify the configuration file of each preprocessor.

  1. Open the std.langid-config configuration file in the central directory for executable files, <CENTRAL_DIR>\lexicon, in a text editor.

  2. In the section <encodings-languages-covered>, check the list of languages to be taken into consideration for the language recognition procedure. The list is under <list key = "utf_8">.

    Delete languages that you do not need, or flag them using <!-- -->.

    You can add more languages to the list as needed as long as the languages in question are supported by the language recognition service. The following list shows which languages you can use, and gives the entry that you enter into the list for each language.



Chinese (simplified)

<item key = "simplified-chinese" />

Chinese (traditional)

<item key = "traditional-chinese" />


<item key = "danish" />


<item key = "german" />


<item key = "english" />


<item key = "finnish" />


<item key = "french" />


<item key = "dutch" />


<item key = "italian" />


<item key = "japanese" />


<item key = "korean" />

Norwegian (Bokmal)

<item key = "bokmal" />

Norwegian (Nynorsk)

<item key = "nynorsk" />


<item key = "portuguese" />


<item key = "swedish" />


<item key = "spanish" />




<item key = "arabic" />


<item key = "greek" />


<item key = "hebrew" />


<item key = "polish" />


<item key = "romanian" />


<item key = "russian" />


<item key = " thai" />


<item key = "czech" />


<item key = " turkish " />


<item key = "hungarian" />

Only limited text-mining functions are currently available for these additional languages. For more information about these languages, see Supported Languages with Restricted Functions.


In the following example, English, French, and Danish are taken into consideration. Italian is not.


<list key = "utf_8">

<item key = "english" />

<item key = "french" />

<item key = "german" />

<!-- item key = "italian" -->


All documents are converted to URF-8 Unicode format before language recognition takes place. Therefore only the section <list key = "utf_8"> is relevant in the language list. The other types of coding do not need to be modified.

  1. The section <remove-markup-content> contains a list of markings that are ignored when language recognition takes place. All texts with these markings are ignored.


    The following example shows a section of the list.


    <item key = "applet" />

    <item key = "code" />

    <item key = "script" />


    <item key = "title" />

    For example, if JavaScript programs (marked by <script>) occur in HTML documents, they are ignored when language recognition takes place.

    If necessary, you can add more elements to the list, or remove existing elements from the list.


    You have French documents that also contain a short summary in English. The summary is marked with the tag <English-Abstract>: <English-Abstract> This is the abstract in English. ... </English-Abstract>. Add the line <item key = "english-abstract" /> to the list mentioned above.

    On the other hand, if you want text marked with <title> to be taken into consideration for language recognition, you need to remove the line <item key = "title" /> from the list.

  2. The section <detection-buffer-size> determines the quantity of text that is taken into consideration when a document is subjected to the language recognition procedure. You can increase this value if you think that the quantity of text is too small.


    However, this should only be done in exceptional circumstances. The larger the quantity of text, the longer language recognition, and therefore indexing, takes.

    The value in the section <detection-buffer-size> cannot be greater than the value in the section <langid-buffer-size>.

  3. Save the file and close the text editor.

  4. Restart TREX.

    For the changes to the std.langid-config configuration file in the <CENTRAL_DIR>\lexicon directory to take effect, you must restart TREX. When you restart TREX, the sapcpe program copies the changed configuration filesfrom the central directory for executable files, <CENTRAL_DIR>\lexicon, to the local TREX directory, <TREX_DIR>\exe\lexicon, and overwrites the std.langid-config configuration file there.


    You can also use the TREX admin tool (standalone), area Start of the navigation path Landscape Next navigation step Ini End of the navigation path, to change the std.langid-config configuration file and then have the changes take effect by restarting the TREX preprocessor. Note that only the file in the <TREX_DIR>\exe\lexicon directory is changed if you use this method. If you have changed the std.langid-config configuration file in the central directory, <CENTRAL_DIR>\lexicon, as described above and restarted TREX, the system overwrites the changed file in the local directory, <TREX_DIR>\exe\lexicon, during the automatic synchronization by the CPE and the changes are lost.

Modifying Language Recognition Using the TREX Preprocessor

The language of documents with only a small amount of text cannot be reliably identified, therefore TREX preprocessor language recognition is only activated if at least seven terms (default value) can be recognized for each document. You can change this value if you are using TREX in a scenario with very short sentences.

You make modifications for TREX preprocessor language recognition in the TREXPreprocessor.ini configuration file.

  1. Open the <TREX_INSTALL>\TREXPreprocessor.ini configuration file with a text editor.

  2. In the [lexicon] section, change the min_valid_tokens parameter.

    The default value of this parameter is 7. Choose a lower value if you want the TREX preprocessor to try to identify the language of documents with fewer terms per document.

  3. Restart the TREX preprocessor.

    You need to stop and restart the preprocessor for the new settings to take effect. You do this using the TREX admin tool (standalone), using the function for starting and stopping the TREX servers. Note that the TREX daemon automatically restarts the server after it has been stopped. The settings are valid for all documents indexed after the TREX preprocessor is restarted.


    The new settings do not affect documents that have already been indexed. This means that if, for example, a document that has already been indexed has been assigned to the wrong language, it must be reindexed.