uCosminexus Stream Data Platform - Application Framework

Setup and Operation Guide

3020-3-V02(E)

Contents

Notices

Preface

Part 1: Before You Start

1. Overview of the Configuration and Operation of Stream Data Platform - AF

1.1 Flow from introduction to operation

1.2 Organization of this manual

Part 2: Design and Configuration

2. System Design

2.1 Flow of design

2.2 Evaluating the system configuration

2.2.1 Program configuration in a stream data processing system

2.2.2 Components of a stream data processing system

2.2.3 Configuration when standard adaptors are used

2.2.4 Configuration when custom adaptors are used

2.3 Evaluating the stream data processing engine

2.3.1 Evaluating the number of SDP servers

2.3.2 Evaluating the SDP server's time control method

2.3.3 Evaluating adjustment of the timestamps in tuples

2.4 Evaluating the adaptor configuration

2.4.1 Evaluating the adaptors to be used

2.4.2 Evaluation items when the standard adaptors are used

2.4.3 Evaluation items when custom adaptors are used

2.5 Evaluating the processing to be performed by the standard adaptors

2.5.1 Evaluating the data input methods

2.5.2 Evaluating the data editing methods

2.5.3 Evaluating the data output methods

2.6 Evaluating the summary analysis scenarios

2.6.1 Evaluating the stream data summary analysis scenarios

2.6.2 Evaluating the configuration of the working directories for executing query groups

2.7 Estimating the memory requirements

2.7.1 Estimating the memory requirements for the stream data processing engine

2.7.2 Estimating the memory requirements for standard adaptors

3. System Configuration

3.1 Flow of configuring

3.2 Directory structure

3.2.1 Structure of the installation directory

3.2.2 Structure of the working directories

3.3 Setting up an operating environment

3.3.1 Registering an administrator user

3.3.2 Creating a working directory

3.4 Creating the SDP server definition files

3.4.1 Settings that can be specified in the SDP server definition files

3.4.2 How to create the SDP server definition files

3.5 Creating the query definition files

3.5.1 Settings that can be specified in the query definition files

3.5.2 How to create the query definition files

3.6 Creating the adaptor definition files

3.6.1 Settings that can be specified in the adaptor definition files

3.6.2 How to create the adaptor definition files

3.7 Setting up the dashboard

3.7.1 Output settings for dashboard-display data

3.7.2 Setting up Dashboard Server

3.7.3 Setting up Dashboard Viewer

Part 3: Operation

4. System Operation

4.1 Flow of system operation

4.2 Starting the system

4.2.1 Starting the SDP server

4.2.2 Registering query groups

4.2.3 Starting query groups

4.2.4 Starting adaptors (standard adaptors)

4.2.5 Starting adaptors (custom adaptors)

4.2.6 Checking the status of adaptor groups

4.3 Using queries and query groups

4.3.1 Re-executing queries

4.3.2 Displaying the status of query groups

4.4 Shutting down the system

4.4.1 Terminating adaptors (standard adaptors)

4.4.2 Terminating adaptors (custom adaptors)

4.4.3 Terminating query groups

4.4.4 Shutting down the SDP server

4.5 Displaying analysis results on a dashboard

4.5.1 Starting Dashboard Server

4.5.2 Using Dashboard Viewer to display analysis results

4.5.3 Shutting down Dashboard Server

5. Modifying the System

5.1 Overview of system modification

5.2 Modifying the stream data processing engine

5.3 Modifying query groups

5.3.1 Changing settings in the property files

5.3.2 Changing the query definition file

5.4 Modifying adaptors

5.5 Modifying the dashboard

6. Troubleshooting

6.1 Error handling procedure

6.2 Data to be collected in the event of an error

6.3 Details of log files and trace files

6.3.1 Details of log files

6.3.2 Details of API trace information

6.3.3 Details of adaptor trace information

6.3.4 Details of tuple logs

6.3.5 Details of a thread dump

6.4 How to handle major problems

6.4.1 Problems during system operation

Part 4: Reference

7. Commands

Format of command explanations

List of commands

sdpcql (registers a query group)

sdpcqldel (deletes a query group)

sdpcqlstart (starts a query group)

sdpcqlstop (terminates a query group)

sdpls (displays the status of query groups)

sdpsetup (sets up an operating environment)

sdpstart (starts the SDP server)

sdpstartap (starts RMI-connection adaptors)

sdpstartinpro (starts in-process-connection adaptors)

sdpstop (stops the SDP server)

sdpstopap (terminates RMI-connection adaptors)

sdpstopinpro (terminates in-process-connection adaptors)

sdptplls (displays tuple information)

sdptplput (reloads tuples)

sdptrced (edits trace information)

8. SDP Server Definition Files

8.1 Format of SDP server definition file explanations

8.2 Notes about creating SDP server definition files

8.3 List of SDP server definition files

8.4 JavaVM options file for SDP servers (jvm_options.cfg)

8.5 JavaVM options file for RMI connections (jvm_client_options.cfg)

8.6 System configuration property file (system_config.properties)

8.6.1 Details of the system configuration property file (system_config.properties)

8.6.2 Details of the parameters in the system configuration property file (system_config.properties)

8.7 Query group property file

8.7.1 Details of the query group property file

8.7.2 Details of the parameters in the query group property file

8.7.3 Coding rules for conditional expressions

8.8 Stream property file

8.8.1 Details of the stream property file

8.8.2 Details of the parameters in the stream property file

8.9 In-process connection property file (user_app.adaptor-group-name-or-adaptor-name.properties)

8.10 Log file output property file (logger.properties)

8.11 List of JavaVM options

9. Adaptor Definition Files

9.1 Format of adaptor definition file explanations

9.2 Notes about creating adaptor definition files

9.3 List of adaptor definition files

9.4 Adaptor command definition file (AgentManagerDefinition.xml)

9.5 Adaptor configuration definition file (AdaptorCompositionDefinition.xml)

9.5.1 Overview of the adaptor configuration definition file (AdaptorCompositionDefinition.xml)

9.5.2 Adaptor configuration definition file namespaces

9.6 Common definition in the adaptor configuration definition file

9.6.1 Common definition

9.6.2 Adaptor trace definition

9.7 Adaptor group definition in the adaptor configuration definition file

9.7.1 In-process group definition

9.7.2 RMI group definition

9.8 Adaptor definition in the adaptor configuration definition file

9.8.1 Input adaptor definition

9.8.2 Output adaptor definition

9.9 CB definition in the adaptor configuration definition file

9.9.1 CB definition for input

9.9.2 CB definition for output

9.9.3 CB definition for editing

9.9.4 CB definition for sending

9.9.5 CB definition for receiving

9.10 CB definitions for input and output in the adaptor configuration definition file

9.10.1 File input connector definition

9.10.2 HTTP packet input connector definition

9.10.3 File output connector definition

9.10.4 Dashboard output connector definition

9.11 CB definitions for data editing in the adaptor configuration definition file

9.11.1 Format conversion definition

9.11.2 Mapping definition

9.11.3 Filter definition

9.11.4 Record extraction definition

9.12 CB definitions for sending and receiving in the adaptor configuration definition file

9.12.1 Input stream definition

9.12.2 Output stream definition

9.13 Coding examples for an adaptor configuration definition file

9.13.1 Example 1

9.13.2 Example 2

10. Details About Definitions in the Definition Files

10.1 Organization of this chapter

10.2 File input

10.2.1 Overview of file input

10.2.2 Flow of data processing during file input

10.2.3 File input settings

10.3 HTTP packet input

10.3.1 Overview of HTTP packet input

10.3.2 Flow of data processing during HTTP packet input

10.3.3 HTTP packet input settings

10.4 Record filtering

10.4.1 Overview of record filtering

10.4.2 Flow of data processing during record filtering

10.4.3 Record filtering settings

10.5 Record extraction

10.5.1 Overview of record extraction

10.5.2 Flow of data processing during record extraction

10.5.3 Record extraction settings

10.6 File output

10.6.1 Overview of file output

10.6.2 Flow of data processing during file output

10.6.3 File output settings

10.7 Dashboard output

10.7.1 Overview of dashboard output

10.7.2 Flow of data processing during dashboard output

10.7.3 Dashboard output settings

10.8 Timestamp adjustment for tuples

10.8.1 Scope of timestamp adjustment

10.8.2 Range of times to be adjusted

10.8.3 How to adjust the time

10.8.4 Processing after tuples have been input

10.8.5 Operation when the query group shuts down

10.8.6 Tuple retention period

10.8.7 Selecting tuples by filtering

10.8.8 Timestamp adjustment settings

11. Details of Flex Dashboard Settings

11.1 Organization of this chapter

11.2 Dashboard Server internal settings file (usrconf.properties)

11.3 Dashboard Viewer window layout file

11.3.1 Configuration of Dashboard Viewer window

11.3.2 Details of a Dashboard Viewer window layout file

11.4 Example definitions for dashboard output

11.4.1 Output adaptor definition (displaying the most recent data)

11.4.2 Output adaptor definition (displaying data including historical data)

Appendix

A. Reference Material for This Manual

A.1 Related publications

A.2 Conventions: Abbreviations for product names

A.3 Conventions: Acronyms

A.4 Conventions: KB, MB, GB, and TB

Index