1.1 A data processing system that analyzes the "right now"

Our societal infrastructure has been transformed by the massive amounts of data being packed into our mobile telephones, IC cards, home appliances, and other electronic devices. As a result, the amount of data handled by data processing systems continues to grow daily. The ability to quickly summarize and analyze this data can provide us with valuable new insights. To be useful, any real-time data processing system must have the ability to create new value from the massive amounts of data that is being created every second.

Stream Data Platform - AF responds to this challenge by giving you the ability to perform stream data processing. Stream data processing gives you real-time summary analysis of the large quantities of time-sequenced data that is always being generated, as soon as the data is generated.

For example, think how obtaining real-time summary information on what was searched for from peoples PCs and mobile phones could increase your product sales opportunities. If a particular product becomes a hot topic on product discussion sites, you expect the demand for it to increase, so more people would tend to search for that product on the various search sites. You can identify such products by using stream data processing to analyze the number of searches in real-time and provide summary results. This information allows retail outlets to increase their orders for the product before the demand hits, and for the manufacturer to quickly ramp up production of the product.

On the IT systems side, demand for higher operating efficiencies and lower costs continues to grow. At the same time, the increasing use of virtualization and cloud computing results in ever larger and more complex systems, making it even more difficult for IT to get a good overview of their system's state of operation. This means that it often takes too long to detect and resolve problems when they occur. Now, by using stream data processing to monitor the operating state of the system in real-time, a problem can be quickly dealt with as soon as it occurs. Moreover, by analyzing trends and correlations in the information about the system's operations, warning signs can be detected, which can be used to prevent errors from ever occurring.

Adding Stream Data Platform - AF to your data processing system gives you a tool that is designed for processing these large volumes of data.

The following figure provides an overview of a configuration that uses Stream Data Platform - AF to implement stream data processing.

Figure 1-1 Overview of a stream data processing configuration that uses Stream Data Platform - AF

[Figure]

Introducing Stream Data Platform - AF into your stream data processing system allows you to perform summary analysis of data as it is being created.

For example, by using a stream data processing system to monitor system operations, you can summarize and analyze log files output by a server and HTTP packets sent over a network. These results can then be displayed on the dashboard, allowing you to monitor your system's operations in real-time. In this way, you can quickly resolve system problems as they occur, improving operation and maintenance efficiencies. You can also store the processing results in a file, allowing you to use other applications to further review or process the results.

To give you a better idea of how stream data processing carries out real-time processing, stream data processing is compared to conventional stored data processing in the following example.

Figure 1-2 shows conventional stored data processing.

Figure 1-2 Stored data processing

[Figure]

Data processed using stored data begins by storing the data sequentially in a database as it occurs. Processing is not actually performed until a user issues a query for the data stored in the database, and summary analysis results are returned. Because data that is already stored in a database is searched when the query is received, there is a time lag between the time the data is collected and the time the data summary analysis results are produced. In the figure, processing of data that was collected at 09:00:00 is performed by a query issued at 09:05:00, obviously lagging behind the time the data was collected.

Figure 1-3 shows stream data processing.

Figure 1-3 Stream data processing

[Figure]

With stream data processing, you pre-load a query (summary analysis scenario) that will perform incremental data analysis, thus minimizing the amount of computing that is required. Moreover, because data analysis is triggered by the data being input, there is no time lag between it and the time the data is collected, providing you with real-time data summary analysis. This kind of stream data processing, in which processing is triggered by the input data itself, is a superior approach for data that is generated sequentially.

Therefore, the ability to perform stream data processing that you gain by integrating Stream Data Platform - AF into your system allows you to get a real-time summary and analysis of the data.