File Reader
Reads files from disk using a compatible parser.
You can create FileReader sources in the web UI using Source Preview.
See Supported reader-parser combinations) for parsing options.
File Reader properties
property | type | default value | notes |
---|---|---|---|
Block Size | Integer | 64 | amount of data in KB for each read operation |
Compression Type | String | Set to | |
Directory | String | Specify the path to the directory containing the file(s). The path may be relative to the Striim installation directory (for example, Samples/PosApp/appdata) or from the root. | |
Include Subdirectories | Boolean | False | Set to True if the files are written to subdirectories of the |
Position By EOF | Boolean | True |
|
Rollover Style | String | Default | Set to |
Skip BOM | Boolean | True | If set to True, when the wildcard value specifies multiple files, Striim will read the Byte Order Mark (BOM) in the first file and skip the BOM in all other files. If set to False, it will read the BOM in every file. |
Wildcard | String | Specify the name of the file, or a wildcard pattern to match multiple files (for example, *.xml).
|
The output type is WAevent except when using Avro Parser or JSONParser.
File Reader sample code
When used with DSV Parser, the type for the output stream can be created automatically from the file header (see Creating the FileReader output stream type automatically).
Striim also provides templates for creating applications that read from files and write to various targets. See Creating an application using a template for details.
An example from the PosApp sample application:
CREATE SOURCE CsvDataSource USING FileReader ( directory:'Samples/PosApp/appData', wildcard:'posdata.csv', positionByEOF:false ) PARSE USING DSVParser ( header:Yes, trimquote:false ) OUTPUT TO CsvStream;
See PosApp for a detailed explanation and MultiLogApp for additional examples.
Creating the output stream type automatically
When FileReader is used with DSV Parser, the type for the output stream can be created automatically from the file header using OUTPUT TO <stream name> MAP(filename:'<source file name>')
. For example:
CREATE SOURCE CsvDataSource USING FileReader ( directory:'Samples/PosApp/appData', wildcard:'posdata*.csv', positionByEOF:false ) PARSE USING DSVParser ( header:Yes, trimquote:false ) OUTPUT TO CsvStream MAP(filename:’posdata*.csv’);
Notes:
The specified source file must exist when the source is created.
The header must be the first line of the file (the HeaderLineNo setting is ignored by MAP).
If multiple files are specified by the
wildcard
property, the header will be taken from the first one read.All files must be like the first one read, with headers in the first line and the same number of fields.
Creating the FileReader output stream type automatically
When FileReader is used with DSV Parser, the type for the output stream can be created automatically from the file header using OUTPUT TO <stream name> MAP(filename:'<source file name>')
. A regular, unmapped output stream must also be specified. For example:
CREATE SOURCE PosSource USING FileReader ( wildcard: 'PosDataPreview*.csv', directory: 'Samples/PosApp/appData', positionByEOF:false ) PARSE USING DSVParser ( header:Yes, trimquote:false ) OUTPUT TO PosSource_Stream, OUTPUT TO PosSource_Mapped_Stream MAP(filename:'PosDataPreview.csv');
Notes:
When you use a MAP clause, you may not specify the Column Delimit Till. Header Line No, Line Number, or No Column Delimiter properties.
The file specified in the MAP clause must be in the directory specified by FileWriter's directory property when the source is created.
The header must be the first line of that file.
The column names in the header can contain only alphanumeric, _ (underscore) and $ (dollar sign) characters, and may not begin with numbers.
All files to be read must be similar to the one specified in the MAP clause, with headers (which will be ignored) in the first line and the same number of fields.
All fields in the output stream type will be of type String.
In this release, this feature is available only in the console, the MAP clause cannot be edited in the web UI, and changing the Wildcard property value in the web UI will break the source.