File/Hadoop CSV Output Adapter Studio Properties
Adapter type: toolkit_file_csv_output. The File/Hadoop CSV Output adapter writes delimiter-separated data to a file or a Hadoop Distributed File System (HDFS). Set these properties for the File/Hadoop CSV Output adapter in the adapter properties dialog of SAP Event Stream Processor Studio. The File/Hadoop CSV Output adapter uses SimpleDateFormat formatting codes.
If you use the CCL ATTACH ADAPTER statement to attach an adapter, you must supply the adapter type.
| Property Label | Description |
|---|---|
| Directory |
Type: string (Required) Specify the absolute path to the data files that you want the adapter to write to. For example, <username>/<foldername>. The default value is ".", meaning the current directory in which the adapter is running. Alternatively, you can leave this value empty and specify the absolute path in the File property. To use Hadoop system files, use an HDFS folder instead of a local file system folder. For example, hdfs://<hdfsserver>:9000/<foldername>/<subfoldername>/<leaffoldername>. To use Hadoop, download the binaries for Hadoop version 1.2.1 or
2.2.0 from http://hadoop.apache.org For version 2.2.0, copy these files over to
%STREAMING_HOME%\adapters\framework\libj:
Use a forward slash for both UNIX and Windows paths. |
| File (in Directory) |
Property ID: file Type: string (Required) Specify the file to write to. |
| Append Mode |
Property ID: appendMode Type: boolean (Advanced) Specify whether data should be appended to an existing file. The default value is false. |
| File Size Limit |
Property ID: fileSizeLimit Type: integer (Advanced) Specify the maximum size, in bytes, of the output file. If this property is set, the adapter starts writing a new file every time the size of the current output file becomes greater than this value. The files are named <filename>, <filename>.001, <filename>.002, and so on where <filename> is the value of the File element. If you do not specify a value or you specify a value of 0 for this property, the output file does not have a size limit. No default value. |
| Time Based Rotate |
Property ID: timeBasedRotate Type: boolean (Advancedl) Specify whether to rotate files at predefined intervals. The default value is false. |
| Time Based Rotate Interval |
Property ID: timeBasedRotateInterval Type: interval (Advanced) Specify the amount of time, in seconds, to wait between file rotations. The default value is 24 hours. |
| Time Based Rotate Start At |
Property ID: timeBasedRotateStartAt Type: time (Advanced) Specify the time of the first file rotation. The
supported formats are:
If you do not specify a time zone, the adapter defaults to UTC. |
| Time Stamp in File Names |
Property ID: timestampInFilenames Type: boolean (Advanced) Specify whether to append the system time, in UTC, to the output file name when a new output file is created. The default value is false. |
| Time Stamp In File Names Format |
Property ID: timestampInFilenamesFormat Type: string (Advanced) Specify the timestamp format that gets appended to the output file name. Valid formats include any formats accepted by the Java SimpleDateFormat class. For example, yyyy-MM-dd_HH-mm-ss. On Windows, the following symbols are not permitted in the file name: \ / : * ? " < > |. |
| Column Delimiter |
Property ID: csvDelimiter Type: string (Advanced) Specify the symbol used to separate the columns. The default value is a comma, ",". |
| SecondDate Format |
Property ID: csvSecondDateFormat Type: string (Advanced) Specify the format for parsing SecondDate values. For example, yyyy-MM-dd'T'HH:mm:ss is the default value. |
| MsDate Format |
Property ID: csvMsDateFormat Type: string (Advanced) Specify the format for parsing MsDate values. For example, yyyy-MM-dd'T'HH:mm:ss.SSS is the default value. |
| Time Format |
Property ID: csvTimeFormat Type: string (Advanced) Specify the format for parsing time values. For example, HH:mm:ss is the default value. |
| BigDatetime Format |
Property ID: csvBigDatetimeFormat Type: string (Advanced) Specify the format for parsing bigdatetime values. For example, yyyy-MM-dd'T'HH:mm:ss.SSSSSS is the default value. Using less than six Ss gives precision to that exact number of Ss and ignores values past that specification. Using more than six Ss truncates any values beyond the sixth, and replaces them with zero. This may result in slower behavior. |
| Has Header |
Property ID: csvHasHeader Type: boolean (Advanced) Indicate whether the first line of the file contains the description of the CSV fields. The default value is false. |
| Prepend stream name, opcode |
Property ID: csvPrependStreamNameOpcode Type: boolean (Optional) If set to true, the adapter prepends the stream name and the opcode in each row of data that is generated. The default value is true. |
| Batch Size |
Property ID: BatchSize Type: int (Advanced) If the adapter is running in GD (guaranteed delivery) mode, specify the number of message rows after which the adapter issues a commit command to the external data source and a GD commit to the stream to which the adapter is attached. If the adapter is runing without GD mode, specify the number of message rows after which the adapter issues a commit command to the external data source to which it is attached. The default value is 1. Increasing this value improves performance at the expense of latency. It also increases memory consumption in the ESP Server because the uncommitted rows need to be preserved for redelivery in case of failure. If running this adapter in GD mode, set either this or the BatchPeriod property to greater than 0. If neither property is set to greater than 0, a warning is sent and this property is set to 1. |
| Batch Period |
Property ID: BatchPeriod Type: int (Advanced) Specify the number of seconds after which the adapter issues a commit command to the external data source and a GD commit to the stream to which the adapter is attached. If the adapter is running without GD mode, specify the number of seconds after which the adapter issues a commit command to the external data source. The default value is 0. |
|
PropertySet |
Property ID: propertyset Type: string (Advanced) Specifies the name of the property set. Property sets are reusable sets of properties that are stored in the project configuration file. Using these sets allows you to move adapter configuration properties out of the CCL file and into the CCR file. If you specify the same properties in the project configuration file and the ATTACH ADAPTER statement, the values in the property set override the values defined in the ATTACH ADAPTER statement. No default value. |
| Include Base Content |
Property ID:outputBase Type: boolean (Optional) If set to true, the adapter outputs the initial stream contents in addition to stream updates. If this option is enabled and the adapter is running in GD mode, once the adapter has done a GD commit on the entire base data, the ESP Server does not redeliver the base data on adapter restart and only sends deltas that are saved for delivery. The default value is false. |
| Only Base Content |
Property ID: onlyBase Type: boolean (Advanced) Sends a one-time snapshot of initial contents in a stream. Default value is false. If set to true, outputBase automatically becomes true. |
|
Enable GD Mode |
Property ID: EnableGdMode Type: boolean (Advanced) Specifies whether the adapter runs in guaranteed delivery (GD) mode. GD ensures that data continues to be processed in the case that the ESP Server fails, or the destination (third-party server) fails or does not respond for a long time. See Guaranteed Delivery in the SAP Event Stream Processor: Developer Guide for details on enabling GD for your project. The default value is false. |
|
Enable GD Cache |
Property ID: EnableGdCache Type: boolean (Advanced)If set to true, only rows that can be recovered (that is, checkpointed) by the ESP Server on restart are sent to the end source. Other rows are cached internally by the adapter. The default value is false. |
|
GD Subscriber Name |
Property ID: GdSubscriberName Type: string (Advanced) If the adapter is running in GD mode (the EnableGDMode property is set to true), specify a unique name to identify the GD subscription client. If this value is empty when running in GD mode, the adapter logs an error and shuts down. |
