This subsection explains how to estimate the memory requirements for the stream data processing engine.
To specify the memory requirements for the stream data processing engine, you use parameters in the JavaVM options file for SDP servers (jvm_options.cfg), one of the SDP server definition files, as shown in the table below.
Table 2-8 Parameters used to specify the memory requirements for the stream data processing engine
No. | Parameter | Description |
---|---|---|
1 | SDP_INITIAL_MEM_SIZE | Specifies the initial size for the Java heap. The default value is 512 megabytes. |
2 | SDP_MAX_MEM_SIZE | Specifies the maximum size of the Java heap. The default value is 1,024 megabytes. |
If the default memory size is not sufficient, change the parameter values as appropriate. For stream data processing, a fixed amount of memory is required for the stream data processing engine, as described below. To reduce the frequency of garbage collection, specify a sufficient value for each parameter based on the values obtained below.
For details about the parameters in the JavaVM options file for SDP servers, see 8.4 JavaVM options file for SDP servers (jvm_options.cfg).
The following subsections explain each of the stream data processing engine memory requirement items.
The memory required for running an SDP server is 145 megabytes.
The memory required for one tuple is determined by the size of one tuple. You can use the following formula to determine the size of one tuple:
Size of one tuple (bytes) = 500# + size of user-defined data area |
In the formula above, size of user-defined data area is determined by the data types and number of data items defined by the user in the CQL schema. You can use the following formula to determine the size of the user-defined data area:
Size of user-defined data area (bytes) = (256# x number of user-defined data items) + (length of the character string defined as string data x 2) |
As the sizes of the windows used in window operations increase, the number of tuples held by the SDP server increases, and the amount of memory required for retaining the tuples also increases. The size of a window is specified by the number of tuples (for ROWS window) or time (for RANGE window).
You can use the formulas below to determine the amount of memory required when window operations are used. In the case of a RANGE window, the formula depends on whether or not the time division function is specified.
For a ROWS window: Size of memory required (bytes) = A x number of tuples specified in the ROWS window + 50,000# For a RANGE window (when time division function is not specified): Size of memory required (bytes) = A x time specified in the RANGE window x number of tuples input per unit of time + 50,000# For a RANGE window (when time division function is specified): Size of memory required (bytes) = A x time specified in the RANGE window / mesh interval specified in the time division function x number of tuples input per mesh interval + 50,000# |
Specification of the time division function is useful when it is difficult to predict the frequency of data input or when you want to limit the size of the available memory. Note that there are prerequisites and limitations to the specification of the time division function. For details about specification of the time division function, see the uCosminexus Stream Data Platform - Application Framework Application Development Guide.
You can change the maximum number of elements in a stream queue by using the engine.maxQueueSize parameter in the system configuration property file (system_config.properties), one of the SDP server definition files. Changing the maximum number of elements affects the size of memory used by the stream queue. If you change the maximum number of elements, you can use the following formula to determine the amount of memory required per stream queue:
Memory required per stream queue (bytes) =A x B |
For details about the parameters in the system configuration property file, see 8.6 System configuration property file (system_config.properties).
You can change the API trace buffers by using the following parameters in the system configuration property file (system_config.properties), one of the SDP server definition files:
Changing any of these parameters affects the size of the memory used for API trace information. You can use the following formula to determine the memory used for API trace information when the API trace buffer are changed:
Memory required for API trace information (kilobytes) = A x B + C |
For details about the parameters in the system configuration property file, see 8.6 System configuration property file (system_config.properties).
You can change the timing of outputting tuple logs to files by using the tpl.outputTrigger parameter in the system configuration property file (system_config.properties), one of the SDP server definition files.
Specifying BUFFER (default value) as the timing of outputting tuple logs to files in the tpl.outputTrigger parameter affects the size of the memory required for acquiring tuple logs per stream.
You can use the formula shown below to determine the size of the memory required for acquiring tuple logs per stream when you specify BUFFER as the timing of outputting tuple logs to files:
Memory required for acquiring tuple logs per stream (kilobytes) = A x B |
For details about the parameters in the system configuration property file, see 8.6 System configuration property file (system_config.properties).
You can use the following formula to determine the size of the memory required per query group:
Memory required per query group (megabytes) = 2#1 x number of queries contained in the registered query group x 128#2 |
You can use the following formula to determine the size of the memory required for using the timestamp adjustment function:
Memory required for timestamp adjustment (bytes) = number of input streams x number of tuples per unit of time x time adjustment range x A |
The values of number of tuples per unit of time and time adjustment range in the formula depend on the value of the stream.timestampAccuracy parameter in the system configuration property file (system_config.properties), query group property file, or stream property file, all of which are SDP server definition files. The table below shows the relationship between the stream.timestampAccuracy parameter value and the value in the formula.
stream.timestampAccuracy parameter value | Number of tuples per unit of time | Time adjustment range |
---|---|---|
unuse | 0 | 0 |
Other than unuse | Number of tuples per unit time | Time adjustment range# + 1 |
For details about the parameters in each file, see 8.6 System configuration property file (system_config.properties), 8.7 Query group property file, and 8.8 Stream property file.