10.6.2 Flow of data processing during file output

The following figure shows the flow and details of data processing during file output by an output adaptor.

Figure 10-20 Flow and details of data processing during file output

[Figure]

For details about the data formats handled by an output adaptor, see (1) Data formats handled by an output adaptor. For details about the processing, see the subsections beginning with (2) Tuple reception.

Organization of this subsection
(1) Data formats handled by an output adaptor
(2) Tuple reception
(3) Mapping
(4) Format conversion
(5) Data output by the file output connector

(1) Data formats handled by an output adaptor

The data formats handled by an output adaptor are the common record and the output record. The common record is the same as the format handled by input adaptors. For details about the common record, see the description of the common record in 10.2.2(1) Data formats handled by an input adaptor. This subsection discusses the output record.

Output record
An output record is a row of data that is output to an output file. An output adaptor treats one row of data as one output record.

(2) Tuple reception

You define information about tuple reception in the output stream definition in the adaptor configuration definition file.

Tuple reception involves receiving tuples sent from the output stream and converting them to common records.

The figure below shows an example of tuple reception from the output stream using an output adaptor. This example receives tuples from output stream s1.

Figure 10-21 Example of tuple reception by an output adaptor

[Figure]

(3) Mapping

You specify information about mapping in the mapping definition in the adaptor configuration definition file.

The two types of mapping are mapping between record and stream and mapping between records. The table below provides an overview of these types of mapping.

Table 10-3 Overview of mapping

No.Type of mappingDescription
1Mapping between record and streamA common record (mapping source) that was obtained during tuple reception is converted to a common record (mapping target) according to the output format.
Mapping between record and stream is always performed after tuple reception.
2Mapping between recordsA common record obtained from mapping between record and stream is edited to obtain a target common record.
If necessary, mapping between records is performed after the mapping between record and stream but before format conversion.
You can use this type of mapping to change field names in the source common record or to delete fields that are not needed for the next callback processing.
You can also use built-in functions# to obtain character strings and time values from source common records and apply them to target common records.
You can specify multiple definitions for mapping between records.
#
You use the function attribute in the map tag in the adaptor configuration definition file to specify the built-in functions that can be used for mapping between records. For details about the built-in functions that can be specified in the function attribute, see 9.11.2 Mapping definition.

The figure below shows an example of mapping using an output adaptor. This example converts the fields t and url to the fields time and url according to the adaptor's output format and then converts them to the target common record.

Figure 10-22 Example of mapping between record and stream using an output adaptor

[Figure]

Reference note
After performing mapping between record and stream, you can perform mapping between records and record filtering, as necessary (in any order).
For details about record filtering, see 10.4 Record filtering.

(4) Format conversion

You define information about format conversion in the format conversion definition in the adaptor configuration definition file.

Format conversion involves conversion from a common record to an output record.

The figure below shows an example of format conversion from common record to output record. This example converts a common record that has field values of the TIME and character string types to an output format that consists of three fields delimited by the space.

Figure 10-23 Example of format conversion from common record to output record

[Figure]

The table below shows the structure of the common record in the above example and the tags that are specified in the format conversion definition.

Table 10-4 Definition files to be specified and record structure

Record structureTag
Record name: R1
Record structure: ($_time)[Figure]($_method)[Figure]($_url)
record tag (record definition)
Field name: time, Type: TIMEfield tag (field definition)
Field name: method, Type: STRING
Field name: url, Type: STRING
Legend:
[Figure]: Single-byte space

For details about the data types that can be converted and the settings for the structure of common records, see 9.11.1 Format conversion definition

(5) Data output by the file output connector

You use the file output connector definition in the adaptor configuration definition file to define information about file output.

A file output connector outputs obtained output records to the defined output directory.

The table below describes the structures for the files that can be output by a file output connector.

File structureDescription
WraparoundFiles are created according to the output file creation rules specified in the file output connector definition. When the number of files reaches the maximum value, the first file used is overwritten.
Non-wraparoundFiles are created according to the output file creation rules specified in the file output connector definition. When the number of files reaches the maximum value, the output adaptor stops record output and discards the remaining records.