HITACHI Inspire The Next

uCosminexus Stream Data Platform - Application Framework

Setup and Operation Guide

3020-3-V02(E)


Contents

Notices
Preface
Part 1: Before You Start
1. Overview of the Configuration and Operation of Stream Data Platform - AF
1.1 Flow from introduction to operation
1.2 Organization of this manual
Part 2: Design and Configuration
2. System Design
2.1 Flow of design
2.2 Evaluating the system configuration
2.2.1 Program configuration in a stream data processing system
2.2.2 Components of a stream data processing system
2.2.3 Configuration when standard adaptors are used
2.2.4 Configuration when custom adaptors are used
2.3 Evaluating the stream data processing engine
2.3.1 Evaluating the number of SDP servers
2.3.2 Evaluating the SDP server's time control method
2.3.3 Evaluating adjustment of the timestamps in tuples
2.4 Evaluating the adaptor configuration
2.4.1 Evaluating the adaptors to be used
2.4.2 Evaluation items when the standard adaptors are used
2.4.3 Evaluation items when custom adaptors are used
2.5 Evaluating the processing to be performed by the standard adaptors
2.5.1 Evaluating the data input methods
2.5.2 Evaluating the data editing methods
2.5.3 Evaluating the data output methods
2.6 Evaluating the summary analysis scenarios
2.6.1 Evaluating the stream data summary analysis scenarios
2.6.2 Evaluating the configuration of the working directories for executing query groups
2.7 Estimating the memory requirements
2.7.1 Estimating the memory requirements for the stream data processing engine
2.7.2 Estimating the memory requirements for standard adaptors
3. System Configuration
3.1 Flow of configuring
3.2 Directory structure
3.2.1 Structure of the installation directory
3.2.2 Structure of the working directories
3.3 Setting up an operating environment
3.3.1 Registering an administrator user
3.3.2 Creating a working directory
3.4 Creating the SDP server definition files
3.4.1 Settings that can be specified in the SDP server definition files
3.4.2 How to create the SDP server definition files
3.5 Creating the query definition files
3.5.1 Settings that can be specified in the query definition files
3.5.2 How to create the query definition files
3.6 Creating the adaptor definition files
3.6.1 Settings that can be specified in the adaptor definition files
3.6.2 How to create the adaptor definition files
3.7 Setting up the dashboard
3.7.1 Output settings for dashboard-display data
3.7.2 Setting up Dashboard Server
3.7.3 Setting up Dashboard Viewer
Part 3: Operation
4. System Operation
4.1 Flow of system operation
4.2 Starting the system
4.2.1 Starting the SDP server
4.2.2 Registering query groups
4.2.3 Starting query groups
4.2.4 Starting adaptors (standard adaptors)
4.2.5 Starting adaptors (custom adaptors)
4.2.6 Checking the status of adaptor groups
4.3 Using queries and query groups
4.3.1 Re-executing queries
4.3.2 Displaying the status of query groups
4.4 Shutting down the system
4.4.1 Terminating adaptors (standard adaptors)
4.4.2 Terminating adaptors (custom adaptors)
4.4.3 Terminating query groups
4.4.4 Shutting down the SDP server
4.5 Displaying analysis results on a dashboard
4.5.1 Starting Dashboard Server
4.5.2 Using Dashboard Viewer to display analysis results
4.5.3 Shutting down Dashboard Server
5. Modifying the System
5.1 Overview of system modification
5.2 Modifying the stream data processing engine
5.3 Modifying query groups
5.3.1 Changing settings in the property files
5.3.2 Changing the query definition file
5.4 Modifying adaptors
5.5 Modifying the dashboard
6. Troubleshooting
6.1 Error handling procedure
6.2 Data to be collected in the event of an error
6.3 Details of log files and trace files
6.3.1 Details of log files
6.3.2 Details of API trace information
6.3.3 Details of adaptor trace information
6.3.4 Details of tuple logs
6.3.5 Details of a thread dump
6.4 How to handle major problems
6.4.1 Problems during system operation
Part 4: Reference
7. Commands
Format of command explanations
List of commands
sdpcql (registers a query group)
sdpcqldel (deletes a query group)
sdpcqlstart (starts a query group)
sdpcqlstop (terminates a query group)
sdpls (displays the status of query groups)
sdpsetup (sets up an operating environment)
sdpstart (starts the SDP server)
sdpstartap (starts RMI-connection adaptors)
sdpstartinpro (starts in-process-connection adaptors)
sdpstop (stops the SDP server)
sdpstopap (terminates RMI-connection adaptors)
sdpstopinpro (terminates in-process-connection adaptors)
sdptplls (displays tuple information)
sdptplput (reloads tuples)
sdptrced (edits trace information)
8. SDP Server Definition Files
8.1 Format of SDP server definition file explanations
8.2 Notes about creating SDP server definition files
8.3 List of SDP server definition files
8.4 JavaVM options file for SDP servers (jvm_options.cfg)
8.5 JavaVM options file for RMI connections (jvm_client_options.cfg)
8.6 System configuration property file (system_config.properties)
8.6.1 Details of the system configuration property file (system_config.properties)
8.6.2 Details of the parameters in the system configuration property file (system_config.properties)
8.7 Query group property file
8.7.1 Details of the query group property file
8.7.2 Details of the parameters in the query group property file
8.7.3 Coding rules for conditional expressions
8.8 Stream property file
8.8.1 Details of the stream property file
8.8.2 Details of the parameters in the stream property file
8.9 In-process connection property file (user_app.adaptor-group-name-or-adaptor-name.properties)
8.10 Log file output property file (logger.properties)
8.11 List of JavaVM options
9. Adaptor Definition Files
9.1 Format of adaptor definition file explanations
9.2 Notes about creating adaptor definition files
9.3 List of adaptor definition files
9.4 Adaptor command definition file (AgentManagerDefinition.xml)
9.5 Adaptor configuration definition file (AdaptorCompositionDefinition.xml)
9.5.1 Overview of the adaptor configuration definition file (AdaptorCompositionDefinition.xml)
9.5.2 Adaptor configuration definition file namespaces
9.6 Common definition in the adaptor configuration definition file
9.6.1 Common definition
9.6.2 Adaptor trace definition
9.7 Adaptor group definition in the adaptor configuration definition file
9.7.1 In-process group definition
9.7.2 RMI group definition
9.8 Adaptor definition in the adaptor configuration definition file
9.8.1 Input adaptor definition
9.8.2 Output adaptor definition
9.9 CB definition in the adaptor configuration definition file
9.9.1 CB definition for input
9.9.2 CB definition for output
9.9.3 CB definition for editing
9.9.4 CB definition for sending
9.9.5 CB definition for receiving
9.10 CB definitions for input and output in the adaptor configuration definition file
9.10.1 File input connector definition
9.10.2 HTTP packet input connector definition
9.10.3 File output connector definition
9.10.4 Dashboard output connector definition
9.11 CB definitions for data editing in the adaptor configuration definition file
9.11.1 Format conversion definition
9.11.2 Mapping definition
9.11.3 Filter definition
9.11.4 Record extraction definition
9.12 CB definitions for sending and receiving in the adaptor configuration definition file
9.12.1 Input stream definition
9.12.2 Output stream definition
9.13 Coding examples for an adaptor configuration definition file
9.13.1 Example 1
9.13.2 Example 2
10. Details About Definitions in the Definition Files
10.1 Organization of this chapter
10.2 File input
10.2.1 Overview of file input
10.2.2 Flow of data processing during file input
10.2.3 File input settings
10.3 HTTP packet input
10.3.1 Overview of HTTP packet input
10.3.2 Flow of data processing during HTTP packet input
10.3.3 HTTP packet input settings
10.4 Record filtering
10.4.1 Overview of record filtering
10.4.2 Flow of data processing during record filtering
10.4.3 Record filtering settings
10.5 Record extraction
10.5.1 Overview of record extraction
10.5.2 Flow of data processing during record extraction
10.5.3 Record extraction settings
10.6 File output
10.6.1 Overview of file output
10.6.2 Flow of data processing during file output
10.6.3 File output settings
10.7 Dashboard output
10.7.1 Overview of dashboard output
10.7.2 Flow of data processing during dashboard output
10.7.3 Dashboard output settings
10.8 Timestamp adjustment for tuples
10.8.1 Scope of timestamp adjustment
10.8.2 Range of times to be adjusted
10.8.3 How to adjust the time
10.8.4 Processing after tuples have been input
10.8.5 Operation when the query group shuts down
10.8.6 Tuple retention period
10.8.7 Selecting tuples by filtering
10.8.8 Timestamp adjustment settings
11. Details of Flex Dashboard Settings
11.1 Organization of this chapter
11.2 Dashboard Server internal settings file (usrconf.properties)
11.3 Dashboard Viewer window layout file
11.3.1 Configuration of Dashboard Viewer window
11.3.2 Details of a Dashboard Viewer window layout file
11.4 Example definitions for dashboard output
11.4.1 Output adaptor definition (displaying the most recent data)
11.4.2 Output adaptor definition (displaying data including historical data)
Appendix
A. Reference Material for This Manual
A.1 Related publications
A.2 Conventions: Abbreviations for product names
A.3 Conventions: Acronyms
A.4 Conventions: KB, MB, GB, and TB
Index