Estimating the memory requirements for standard adaptors

uCosminexus Stream Data Platform - Application Framework Setup and Operation Guide

2.7.2 Estimating the memory requirements for standard adaptors

This subsection explains how to estimate the memory requirements for the standard adaptors.

To specify the memory requirements for the standard adaptors, you use parameters in the JavaVM options file for SDP servers (jvm_options.cfg), one of the SDP server definition files, or the JavaVM options file for RMI connections (jvm_client_options.cfg), as shown in the table below.

Table 2-9 Parameters used to specify the memory requirements for the standard adaptors (JavaVM options file for SDP servers (jvm_options.cfg))

No.	Parameter	Description
1	SDP_INITIAL_MEM_SIZE	Specifies the initial size for the Java heap. The default value is 512 megabytes.
2	SDP_MAX_MEM_SIZE	Specifies the maximum size of the Java heap. The default value is 1,024 megabytes.

Table 2-10 Parameters used to specify the memory requirement for the standard adaptors (JavaVM options file for RMI connections (jvm_client_options.cfg))

No.	Parameter	Description
1	SDP_INITIAL_MEM_SIZE	Specifies the initial size for the Java heap. The default value is 256 megabytes.
2	SDP_MAX_MEM_SIZE	Specifies the maximum size of the Java heap. The default value is 512 megabytes.

If the default memory size is not sufficient, change the parameter values as appropriate. To run the standard adaptors, you need the amount of memory estimated for the stream data processing engine in 2.7.1 Estimating the memory requirements for the stream data processing engine plus a fixed amount of memory for the standard adaptors, as described below. To reduce the frequency of garbage collection, specify a sufficient value for each parameter based on the values obtained below.

For details about the parameters in each file, see 8.4 JavaVM options file for SDP servers (jvm_options.cfg) or 8.5 JavaVM options file for RMI connections (jvm_client_options.cfg).

The following subsections explain each of the standard adaptor memory requirement items.

Organization of this subsection: (1) Memory required for running adaptors; (2) Memory required for one adaptor configured as an in-process group; (3) Memory required for one adaptor configured as an RMI group; (4) Memory required for record processing (when a file input connector is used); (5) Memory required for record processing (when an HTTP packet input connector is used); (6) Memory required for record processing (when a file output connector or dashboard output connector is used); (7) Memory required for record accumulation (when record extraction or dashboard output connector is used)

(1) Memory required for running adaptors

(2) Memory required for one adaptor configured as an in-process group

(3) Memory required for one adaptor configured as an RMI group

(4) Memory required for record processing (when a file input connector is used)

When a file input connector is used, you can use the following formula to determine the size of the memory required for record processing:

Memory required for record processing (bytes) =
(memory required for processing one record x number of records processed at one time)
+ (A x number of records processed at one time)

Legend:: A: Size per tuple shown in 2.7.1(2) Memory required for one tuple

Memory required for processing one record

In the formula above, the memory required for processing one record can be obtained from the following formula:

Memory required for processing one record (bytes) =
(500 x number of field definitions in the format conversion definition^#)
+ (maximum length of common record)

#: This is the number of field definitions (field tags) in the format conversion definition (FormatDefinition tags) in the adaptor configuration definition file, one of the adaptor definition files.

Number of records processed at one time

In the above format, the number of records processed at one time can be obtained from the following formula:

Number of records processed at one time (bytes) =
number of records read by a file input connector at one time^#

#: This is the readUnit attribute value in the input definition (input tag) in the file input connector definition (FileInputConnectorDefinition) in the adaptor configuration definition file, one of the adaptor definition files.

For details about the definitions in the adaptor configuration definition file, see 9.5 Adaptor configuration definition file (AdaptorCompositionDefinition.xml).

(5) Memory required for record processing (when an HTTP packet input connector is used)

If you use an HTTP packet input connector, you can use the following formula to determine the size of memory required for record processing:

Memory required for record processing (bytes) =
(memory required for processing one record x number of records processed at one time)
+ (A x number of records processed at one time)

Legend:: A: Size per tuple shown in 2.7.1(2) Memory required for one tuple

Memory required for processing one record

In the formula above, memory required for processing one record can be obtained from the following formula:

Memory required for processing one record (bytes) =
(500 x number of field definitions in the HTTP packet input connector definition^#)
+ (maximum length of common record)

#: This is the number of field definitions (field tags) in the HTTP packet input connector definition (HttpPacketInputConnectorDefinition tag) in the adaptor configuration definition file, one of the adaptor definition files.

Number of records processed at one time:

In the formula above, number of records processed at one time can be obtained from the following formula:

Number of records processed at one time (bytes) =
number of records that are read by the HTTP packet input connector at one time^#

#: This is the unit attribute value in the output definition (output tag) in the HTTP packet input connector definition (HttpPacketInputConnectorDefinition tag) in the adaptor configuration definition file, one of the adaptor definition files.

For details about the definitions in the adaptor configuration definition file, see 9.5 Adaptor configuration definition file (AdaptorCompositionDefinition.xml).

(6) Memory required for record processing (when a file output connector or dashboard output connector is used)

If you use a file output connector or a dashboard output connector, you can use the following formula to determine the amount of memory required for record processing:

Memory required for record processing (bytes) =
(memory required for processing one record x number of records processed at one time)
+ (A x number of records processed at one time)

Legend:: A: Size per tuple shown in 2.7.1(2) Memory required for one tuple

Memory required for processing one record

In the formula above, memory required for processing one record can be obtained from the following formula:

Memory required for processing one record (bytes) =
(500 x number of mapping information definitions in the mapping definition^#)
+ (maximum length of common record)

#: This is the number of mapping information definitions (map tags) of mappings between records and streams in the mapping definition (MappingDefinition tag) of the adaptor configuration definition file, one of the adaptor definition files.; For details about the definitions in the adaptor configuration definition file, see 9.5 Adaptor configuration definition file (AdaptorCompositionDefinition.xml).

Number of records processed at one time: In the formula above, number of records processed at one time is 1,024 (maximum number of tuples that can be received).

(7) Memory required for record accumulation (when record extraction or dashboard output connector is used)

If you use record extraction or a dashboard output connector, you can use the following formula to determine the memory required for record accumulation:

Memory required for record accumulation (bytes) =
(memory required for processing one record x number of records to be accumulated)

Memory required for processing one record

In formula above, memory required for processing one record depends on the connector being used, as described below.

When a file input connector is used to perform record extraction
See memory required for processing one record in (4) Memory required for record processing (when a file input connector is used).
When an HTTP packet input connector is used to perform record extraction
See memory required for processing one record in (5) Memory required for record processing (when an HTTP packet input connector is used).
When a dashboard output connector is used
See memory required for processing one record in (6) Memory required for record processing (when a file output connector or dashboard output connector is used).

Number of records to be accumulated

In the formula above, number of records to be accumulated depends on whether record extraction or a dashboard output connector is used, as described below.

When record extraction is used
Use the size attribute value in the extraction condition group definition (extractions tag) in the record extraction definition (RecordExtractionDefinition tag) in the adaptor configuration definition file, one of the adaptor definition files.
When a dashboard output connector is used
Use the MaxNum attribute value in the dashboard output connector definition (DashboardOutputConnectorDefinition tag) in the adaptor configuration definition file, one of the adaptor definition files.

For details about the definitions in the adaptor configuration definition file, see 9.5 Adaptor configuration definition file (AdaptorCompositionDefinition.xml).