3.2 Inputting files

To perform stream data processing on data files, such as log files, you use the file input connector as the input callback.

The file input connector extracts records to be processed from an input source file. Because these records are retrieved as input records, the format conversion callback must be used to convert them to common records so that the stream data processing engine can process them. The following figure shows the positioning and processing of the callbacks involved in file input.

Figure 3-5 Positioning and processing of the callbacks involved in file input

[Figure]

  1. The file input connector extracts the first line (record) from the input file. The record that it extracts is called an input record.
  2. The format conversion callback converts the input record to a common record.
    Reference note
    You can also extract and process multiple records at a time from the input source file.