3.4 Filtering records

To perform stream data processing only on specific records, you use a filter as the data editing callback.

Reference note
As explained in 2.3.2 Retrieving data that satisfies a condition, you can perform filter processing using CQL, but we recommend that you use the adaptor to improve performance.

For example, if you are monitoring temperatures from a number of observation sites and you want to summarize and analyze temperatures from only one particular observation site, you can filter on that observation site's ID.

Only common records can be filtered. If the input source is a file, after an input record is extracted by the file input connector, you must use the format conversion callback to convert it to a common record before filtering it.

When specifying the evaluation conditions you want to filter on, you can use any of the record formats and values that are defined in the records. The following figure shows the positioning and processing of the callback involved in record filtering.

Figure 3-7 Positioning and processing of the callback involved in record filtering

[Figure]

  1. The records passed to the filter are first filtered by record format. Only records of record format R1 meet the first condition, so only these records are selected for processing by the next condition. Records that do not satisfy this condition are passed to the next callback.
  2. After the records are filtered by record format, they are then filtered by record value. This condition specifies that only those records whose ID has a value of 1 are to be passed to the next callback. In this way, only those records that satisfy both conditions will be processed by the next callback. Records that do not satisfy these conditions are discarded.