Import Data to Your Model

The process to import data differs depending on your model configuration, the data source, and the platform you are working on.

This section only describes how to import data to a model. After you import data, it must be prepared so that it's ready to be consumed both in models and stories. For more information about data preparation, check out Import and Prepare Fact Data in an Account Model, or Import and Prepare Fact Data in a Standard Model.

In the image below, you can see all the data sources supported when importing data to a model. Click any of them to access a detailed procedure.

This image is interactive. Hover over each area for a description. Click highlighted areas for more information.

Building a Query

Build a query to import data from an app or a data source into a new or existing model.

To build a query, you move data elements from the Available Data area into the Selected Data and Filters areas to create new measures and dimensions to be imported to your model, and apply filters to them.

An entity () can be expanded to select one or more of its members. The following icons represent types of data you can select, and their properties:

Icon Type

Entities

Numerical properties

Generic properties (strings, Boolean values, UUIDs, and so on)

Complex properties

In SAP Universe queries – the set of data that can be selected from and/or filtered on (similar to the Entity in other queries)

Date type properties

In SAP Universe queries – data that can be filtered on

When building a query, you can select the icon to search for specific values in the Available Data area, select the icon to display the list of reports from your source data in real time, or select the icon to change the way values are displayed:

  • Description – The Description is displayed and used to search values.

  • Description and ID – The Description and ID are both displayed. The ID is used to search values.

  • ID – The ID is displayed and used to search values.

Filters

When you add a data element to the Filters area, you can use operators, such as equal to, except, greater than, and so on, and select the icon to open the Select Values From dialog. In this dialog, you can search and select the values you want to filter on. The list displays the unique values from the first 1000 records. It is possible to search for a value not displayed in the list using the search function.

Some entities, such as Boolean values, can have only a single filter value. Entities such as strings or integer values can have multiple values in a filter. To select multiple values, select the check box of each value you want to add to the filter or Select All to select all the values in the list. Selecting multiple values for a single filter treats the filter as a large OR function, filtering out values that do not match any of the selected values.

Incremental Load Filters

When you create a query, you can set an Incremental Load. An incremental load is based on time or numeric data fields and lets you bring the newest set of data since the last import when you refresh. For example, you can set an incremental load based on the "Release Date" field of a "Product" entity to bring in the newest data based on the release date.

You can set an incremental load while creating a new data source or on an existing data source.

To set an incremental load, in the Filters area, select Set Incremental Load and then drag and drop an incremental-load-enabled property to the incremental load box that appears.

After you create the model, the incremental load appears in the Import Settings section of the Data Management screen.

The next time the source is refreshed, the latest value will be recorded for the Incremental Load Filter. Once the Incremental Load Filter has a value, you should switch the import method to Append.

Deleting or Replacing an Existing Incremental Load Filter

If you change the Incremental Load Filter to a different field, the new filter will not take effect until after the next refresh.

Example:

You set an Incremental Load based on the CreateDate field, and the latest row contains the date "Jan 14, 2018", so the system saves that value. The next time a data refresh occurs, a filter will be applied to grab only data that is later than "Jan 14, 2018".

If you change the Incremental Load to a different field, for example TransactionID, the system does not overwrite the old value until the next refresh. If you change the Incremental Load Filter back to CreateDate before a refresh, the load will still have the old value and, if you refresh now, it will still grab anything after "Jan 14, 2018".

To ensure the data is correct when you change an Incremental Load filter, make sure you perform the following steps:

  1. Change the Incremental Load Filter to a new field you want.

  2. Change the import method accordingly because the Incremental Load Filter will not apply right after you’ve switched it.

  3. Perform a manual refresh - The first refresh right after replacing or deleting the old incremental load filter will bring in the full set of data. Confirm a valid value is present for the incremental load filter after the manual refresh.

    Example of an Incremental Load Filter with a valid value
  4. Change the import method. More information.

Allow Data Import and Model Export with a File Server

The SAP Analytics Cloud agent must be configured in order to allow importing data from a file server or exporting a model to a file server.

Prerequisites

The SAP Analytics Cloud agent must be installed. For more information, see Installing SAP Analytics Cloud Agent.
Data import from file server or model export to file server will assume the file permission level based on the user that started the Tomcat process. For import, read permission is required. For export, write permission is required. The user depends on your Tomcat setup. This may be the Windows system user, a specific user account that started Tomcat as a service in Windows, or the user that executes startup.bat for Windows or startup.sh for Linux. The system administrator must ensure this user has permissions to access the local share or the network share on a different machine.

Context

For importing data, a file server location allowlist must be configured. Only file server paths allowed by this allowlist can be accessed when creating a model from file, or importing data from files.

For exporting models, another file server location allowlist must be configured. Only file server paths allowed by this allowlist can be accessed when exporting a model.

Procedure

  1. Create an allowlist file as a text document: allowlist.txt.
    Example allowlist.txt:
    \\<YourHostName>\Import
    The file must follow the following format:
    • The file must be encoded in UTF-8 format.

    • Each line in the file should specify only one allowed path. All descendant paths will be allowed.

    • The folder path is system specific. If the agent is running on a Microsoft web application server, the path should follow Windows format. For example: C:\folder1\folder2 or \\servername\volume\path. If the agent is running on a Linux system, the path should follow Linux format. For example, /net/abc/

    • Multiple lines are allowed.

  2. Define the file server allowlist using one of the following methods:
    • Add the allowlist location in your Tomcat JVM properties.

      In the Apache Tomcat GUI tool, go to the Java tab. Under Java Options, add one of these lines:

      • For import data: -DSAP_CLOUD_AGENT_FILE_SERVER_WHITELIST_PATH=<AllowlistFilepath>

      • For export model: -DSAP_CLOUD_AGENT_FILE_SERVER_EXPORT_WHITELIST_PATH=<AllowlistFilepath>

      Alternatively, you can add the allowlist location to JAVA_OPTS. Please refer to the Apache Tomcat guide for detail.

    • Set the operating system environment variable SAP_CLOUD_AGENT_FILE_SERVER_WHITELIST_PATH or SAP_CLOUD_AGENT_FILE_SERVER_EXPORT_WHITELIST_PATH to the location of the allowlist file.

    You must specify the complete path up to and including the allowlist file. For example, C:\<full path to file>\allowlist.txt.

  3. Restart the SAP Analytics Cloud agent.
  4. In SAP Analytics Cloud, go to (Main Menu) Start of the navigation path Next navigation step End of the navigation path System Start of the navigation path Next navigation step End of the navigation path Administration Start of the navigation path Next navigation step End of the navigation path System Configuration.
  5. Select (Edit).
  6. For data import, turn on Allow model import from File Server. For model export, turn on Allow model export to File Server.
  7. Select (Save).
  8. Go to (Main Menu) Start of the navigation path Next navigation step End of the navigation path Connection Start of the navigation path Next navigation step End of the navigation path Connections Start of the navigation path Next navigation step End of the navigation path (Add Connection) Start of the navigation path Next navigation step End of the navigation path Import Data Connection Start of the navigation path Next navigation step End of the navigation path File Server.
  9. In the dialog, enter a Connection Name, Description, and the Folder Path.

    The Folder Path should match the path listed in your allowlist file.

  10. Select Create.

Results

The environment variables and allowlist files are polled once a minute.