Language recognition takes place first using the lexicon software of third-party providers and then using the TREX preprocessor. You can configure both types of language recognition.
Naming Convention
As part of the CPE (Central Patch Environment), the sapcpe program takes on the automatic synchronization of executable files and copies them from the central directory for executable files, <CENTRAL_DIR>, into the local directory for executable files, <TREX_DIR>\exe. When you restart TREX, the system automatically launches the sapcpe program. During all subsequent starts, sapcpe checks whether or not the local executable files are up-to-date and copies new or changed executable files from the central directory to the local directory, <TREX_DIR>\exe.
Modifying Language Recognition with Lexicon Software
Language recognition with lexicon software includes the following areas:
You can retrospectively configure language recognition for additional languages. When TREX is installed, you select the languages to be identified by language recognition.
Only activate the languages that appear in your documents and that you also want to index. Doing this optimizes the performance of the language recognition procedure and of indexing in general. Moreover, the less languages used, the better the results of language recognition.
You can configure the system so that certain parts of HTML or XML documents are ignored when the language recognition procedure takes place. Documents that are to be indexed and are in HTML or XML format often contain elements (such as JavaScript programs) that damage the performance of the language recognition procedure.
Language recognition using lexicon software is set up so that only a certain number of characters are taken into consideration. This is usually set to its optimum value at delivery. If it turns out that languages are not being recognized correctly, you can increase the quantity of text that is taken into consideration.
In certain cases, language recognition might not deliver the correct language. In particular, problems can occur when processing documents that are very short or that contain a large number of abbreviations or words loaned from another language.
You modify the lexicon software language recognition by editing the std.langid-config configuration file on the TREX preprocessor. The settings are valid for all indexes on the preprocessor. If you are using more than one TREX preprocessor, you need to modify the configuration file of each preprocessor.
Delete languages that you do not need, or flag them using <!-- -->.
You can add more languages to the list as needed as long as the languages in question are supported by the language recognition service. The following list shows which languages you can use, and gives the entry that you enter into the list for each language.
Languages supported by TREX
Language | Entry |
---|---|
Chinese (simplified) |
<item key = "simplified-chinese" /> |
Chinese (traditional) |
<item key = "traditional-chinese" /> |
Danish |
<item key = "danish" /> |
German |
<item key = "german" /> |
English |
<item key = "english" /> |
Finnish |
<item key = "finnish" /> |
French |
<item key = "french" /> |
Dutch |
<item key = "dutch" /> |
Italian |
<item key = "italian" /> |
Japanese |
<item key = "japanese" /> |
Korean |
<item key = "korean" /> |
Norwegian (Bokmal) |
<item key = "bokmal" /> |
Norwegian (Nynorsk) |
<item key = "nynorsk" /> |
Portuguese |
<item key = "portuguese" /> |
Swedish |
<item key = "swedish" /> |
Spanish |
<item key = "spanish" /> |
Languages supported by TREX with limited functionality
Language | Entry |
---|---|
Arabic |
<item key = "arabic" /> |
Greek |
<item key = "greek" /> |
Hebrew |
<item key = "hebrew" /> |
Polish |
<item key = "polish" /> |
Romanian |
<item key = "romanian" /> |
Russian |
<item key = "russian" /> |
Thai |
<item key = "thai" /> |
Czech |
<item key = "czech" /> |
Turkish |
<item key = "turkish" /> |
Hungarian |
<item key = "hungarian" /> |
Only limited text-mining functions are currently available for these additional languages. For more information about these languages, seeSupported Languages with Restricted Functionality.
In the following example, English, French, and Danish are taken into consideration. Italian is not.
<encodings-languages-covered>
<list key = "utf_8">
<item key = "english" />
<item key = "french" />
<item key = "german" />
<!-- item key = "italian" -->
...
All documents are converted to URF-8 Unicode format before language recognition takes place. Therefore only the section <list key = "utf_8"> is relevant in the language list. The other types of coding do not need to be modified.
The following example shows a section of the list.
<remove-markup-content>
<item key = "applet" />
<item key = "code" />
<item key = "script" />
...
<item key = "title" />
For example, if JavaScript programs (marked by <script>) occur in HTML documents, they are ignored when language recognition takes place.
If necessary, you can add more elements to the list, or remove existing elements from the list.
You have French documents that also contain a short summary in English. The summary is marked with the tag <English-Abstract>: <English-Abstract> This is the abstract in English. ... </English-Abstract>. Add the line <item key = "english-abstract" />to the list mentioned above.
On the other hand, if you want text marked with <title> to be taken into consideration for language recognition, you need to remove the line <item key = "title" /> from the list.
However, this should only be done in exceptional circumstances. The larger the quantity of text, the longer language recognition, and therefore indexing, takes.
The value in the section <detection-buffer-size> cannot be greater than the value in the section <langid-buffer-size>.
For the changes to the std.langid-config configuration file in the <CENTRAL_DIR>\lexicon directory to take effect, you must restart TREX. When you restart TREX, the sapcpe program copies the changed configuration files from the central directory for executable files, <CENTRAL_DIR>\lexicon, to the local TREX directory, <TREX_DIR>\exe\lexicon, and overwrites the std.langid-config configuration file there.
You can also use the TREX admin tool (stand-alone), area Landscape → Ini to change the std.langid-configconfiguration file and then have the changes take effect by restarting the TREX preprocessor. Note that only the file in the <TREX_DIR>\exe\lexicon directory is changed if you use this method. If you have changed the std.langid-config configuration file in the central directory, <CENTRAL_DIR>\lexicon, as described above and restarted TREX, the system overwrites the changed file in the local directory, <TREX_DIR>\exe\lexicon, during the automatic synchronization by the CPE and the changes are lost.
Modifying Language Recognition with the TREX Preprocessor
The language of documents with only a small amount of text cannot be reliably identified, therefore TREX preprocessor language recognition is only activated if at least seven terms (default value) can be recognized for each document. You can change this value if you are using TREX in a scenario with very short sentences.
You make modifications for TREX preprocessor language recognition in the TREXPreprocessor.ini configuration file.
The default value of this parameter is 7. Choose a lower value if you want the TREX preprocessor to try to identify the language of documents with fewer terms per document.
You need to stop and restart the preprocessor for the new settings to take effect. You do this with theTREX admin tool (standalone), using the function forstarting and stopping the TREX servers. Note that the TREX daemon automatically restarts the server after it has been stopped. The settings are valid for all documents indexed after the TREX preprocessor is restarted.
The new settings do not affect documents that have already been indexed. This means that if, for example, a document that has already been indexed has been assigned to the wrong language, it must be reindexed.