2.7.4 Managing Trend Data by IM management node

Organization of this subsection

(1) Data to be stored in the trend data management DB

Trend data (time series data) stored in the trend data management DB is uniquely identified by the metric name and label. Labels are also used to link IM management nodes to trend data. When browsing trend data, you can specify a label to get trend data associated with the IM management node.

The format and examples of metric names and labels are shown below.

Metric name and label format

metric-name{label-name="label-value" label-name="label-value"...}

Examples of metric names and labels

windows_logical_disk_free_bytes{instance="host1:9001" volume="C:"}
windows_logical_disk_free_bytes{instance="host1:9001" volume="D:"}
windows_logical_disk_free_bytes{instance="host2:9001" volume="C:"}

Trend data (time series data) is formed by multiple samples. Each sample consists of the following items:

Millisecond-precision timestamps (UNIX time)
Float64 (double-precision floating-point type) value (performance data)

The following is an example of time series data.

Table 2‒14: Sample time-series data
Metric name			windows_logical_disk_free_bytes
label and value			instance="host1:9001" volume="C:"	instance="host1:9001" volume="D:"	instance="host2:9001" volume="C:"
Time series data	Sample	Timestamp	`1624260855`	`1624260855`	`1624260855`
	Sample	Performance data	`55153717248.0`	`923198898.0`	`194208931840.0`
	Sample	Timestamp	`1624260870`	`1624260870`	`1624260870`
	Sample	Performance data	`55151808512.0`	`923162116.0`	`194208688168.0`
	Sample	Timestamp	`1624260885`	`1624260885`	`1624260885`
	Sample	Performance data	`55153377280.0`	`923146812.0`	`194208931976.0`
	Sample	Timestamp	`1624260900`	`1624260900`	`1624260900`
	Sample	Performance data	`55157182464.0`	`923138684.0`	`194208921684.0`

To Page Top

(2) Retrieving data stored in the trend data management DB

Prometheus query (PromQL) is used to retrieve data stored in the trend data management DB. For supported PromQL language specifications, see 2.7.4(7)About PromQL.

PromQL can retrieve trend data in two types:

Getting with the instant Vector type

Represents a collection of performance data at a point in time, usually at the time of data acquisition.

For example, in Table 2-14 Sample time-series data, it refers to the collection of performance-data 55153717248.0, 923198898.0, and 194208931840.0 for timestamp 1624260855.
Getting with the range Vector type

Represents a collection of performance data from multiple points in time from a point in time (usually the time the data was retrieved).

For example, in Table 2-14 Sample time-series data, it refers to the collection of performance data 55153377280.0, 923146812.0, 194208931976.0 for timestamp 1624260885, and the collection of performance data 55157182464.0, 923138684.0, and 194208921684.0 for timestamp 1624260900.

To get trend data, you can filter and retrieve only those with a specific label by specifying the metric name and label.

If there is no trend data matching the specified label, trend data is not returned (performance data will not be returned with 0 if no trend data exists).

For example, if you filter time-series data shown in Table 2-14 Sample time-series data by specifying windows_logical_disk_free_bytes{volume="C:"} in PromQL expression, you can get the set of performance data corresponding to the label that contains volume="C:" as shown below.

Performance data 55153717248.0, 55151808512.0, 55153377280.0, 55157182464.0 corresponding to the labels instance="host1:9001" volume="C:"
Performance data 194208931840.0, 194208688168.0, 194208931976.0, 194208921684.0 corresponding to the label instance="host2:9001" volume="C:"

In addition, various aggregation results can be obtained by aggregating or calculating the retrieved trend data using the aggregation operator, operator, or function of the PromQL statement.

Notes:

PromQL statements specified in the API provided by JP1/IM - Manager must be written so that the data of the final summary result is an instant Vector type.

Also, if the retrieved data is of type range Vector, you must use an aggregate operator, operator, or function to create a PromQL statement so that the final aggregate data is of type instant Vector.

To Page Top

(3) Ability to write trend data

This function writes trend data to the trend data management DB.

Use JP1/IM - Manager's Write Trend Data REST API to write trend data to the Trend Data Management DB via the Trend Data Management Service (Promscale).

For details on the write trend data API, see 5.11.3 Write Trend Data in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

The data to be written to the trend data management DB is shown below.

Types of data	Description	Specify multiple	Can be omitted
Metric name	Information that represents the type of trend data.	not possible	not possible
Label	Additional information about trend data, such as the host from which trend data is issued. Data in the format "`label-name:label-value`".	possible	possible
Trend data (time series data)	Data in the format of "`time,value`".	possible	not possible

Types of data

Description

Specify multiple

Can be omitted

Metric name

Information that represents the type of trend data.

not possible

Label

Additional information about trend data, such as the host from which trend data is issued.

Data in the format "label-name:label-value".

possible

Trend data (time series data)

Data in the format of "time,value".

possible

not possible

Multiple trend data writing operations can be executed at the same time. Exclusive control of data writing is performed in the trend data management DB.

To Page Top

(4) Ability to browse trend data

This function refers to trend data from the trend data management DB.

Use the jp1TrendDataService.getTrendData method that can be used in the product plug-in to reference trend data from the trend data management DB via the trend data management service (Promscale). JP1/IM - Manager (Intelligent Integrated Management Base) issues a query (PromQL) to the trend data management service using REST API to retrieve trend data.

For details on the jp1TrendDataService.getTrendData method, see 4.5.17 jp1TrendDataService.getTrendData in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

The trend data reference process can be performed at the same time as the write operation. The data that can be referenced is the data that has been written to when the trend data management DB receives the reference request.

■Referring to trend data linked to the IM management node

In the jp1TrendDataService.getTrendData method that retrieves trend data from the trend data management DB, you can refer to the trend data associated with the IM management node by setting the replacement string $jp1im_TrendData_labels in the PromQL statement specified in the promQLQuery argument.

To Page Top

(5) Trend data deletion function

See 2.7.2(3) Deleting trend data.

To Page Top

(6) About Trend Data Objects

See 7.2.3(1) Trend Data Object in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

To Page Top

(7) About PromQL

(a) Get trend data by specifying metric names and labels

Prometheus queries (PromQL) can retrieve trend data in the following two types.

Getting with the instant Vector type

Point-in-time performance data (instant vector) can be obtained in one of the following formats:

metric-name{label^#}

metric-name

#

Do not specify an empty character for the label value. If specified, it may take time to acquire performance data.

The Time Series Data Acquisition API provided by JP1/IM - Manager acquires performance data from the start date and time of the time series data specified in member startTime to the end date and time of the time series data specified in member endTime at the trend data acquisition interval (in seconds) specified in member step.

The retrieved data may have more than 10 decimal places.

If you want the retrieved data to be rounded up, down, or rounded, use the functions of the PromQL statement (ceil, flor, round, etc.).

For details of the functions of the PromQL statement, see 2.7.4(7)(f) Available functions.
Getting with the range Vector type

Performance data (range vectors) from multiple points in time can be obtained in one of the following formats:

metric-name{label^#1}[Time-back^#²]

metric-name[Time-back^#2]

#1

Do not specify an empty character for the label value. If specified, it may take time to acquire performance data.

#2

The time to go back is specified as a number and in units (s, m, h, d, w, or y).

(b) Replacing SID-associated labels

In the API provided by JP1/IM - Manager, the replacement string $jp1im_TrendData_labels can be used for the labels specified in the PromQL statement. If you use $jp1im_TrendData_labels, you must specify it in and $jp1im_TrendData_labels. For details about PromQL to be specified in the JP1/AJS3 metric definition file (metrics_ajs_rootjobnet.conf) of the job monitoring function, see Setup for linking JP1/IM3 in the JP1/Automatic Job Management System 3 Linkage Guide. In addition, only PromQL described in that explanation are supported.

Note that the string after replacing $jp 1im_TrendData_labels must be specified so that it is correct as a PromQL statement.

If the replacement string $jp1im_TrendData_labels is specified in the PromQL statement, it will be replaced with the following string indicated by BNF notation.

<$jp1im_TrendData_labels>     ::= {__name__=~".+"} | (<condition>)
<condition>      ::= { <key_value_list> } | <condition> or { <key_value_list> }
<key_value_list> ::= <key_value> | <key_value_list>, <key_value >
<key_value>      ::= KEY="VALUE#"

#: The backslash (\) in the value of VALUE is specified in PromQL with \\ and double quotation marks (") replaced with \".

Depending on the number of labels specified for the SIDs and the number of SIDs, the strings that are replaced are shown below.

■When the number of labels specified for SID is 1

When the number of SIDs is 1

{KEY1=VALUE1}
When the number of SIDs is 2 or more

{SID1.KEY1=SID1.VALUE1} or

{SID2.KEY1=SID2.KEY2} ...
When the number of SIDs is 0

Do not issue a query (PromQL) to the Trend Data Management Service (Promscale) and return 0 data.

■When the number of labels specified for SID is 2 or more

When the number of SIDs is 1

{KEY1=VALUE1,KEY2=VALUE2 ...}
When the number of SIDs is 2 or more

{SID1.KEY1=SID1.VALUE1,SID1.KEY2=SID1.VALUE2...} or

{SID2.KEY1=SID2.VALUE1,SID2.KEY2=SID2.VALUE2...} ...
When the number of SIDs is 0

Do not issue a query (PromQL) to the Trend Data Management Service (Promscale) and return 0 data.

■When the number of labels specified for SID is 0

When the number of SIDs is 1

{__name__=~".+"}
When the number of SIDs is 0

Do not issue a query (PromQL) to the Trend Data Management Service (Promscale) and return 0 data.

■When the number of labels specified for SIDs is 2 or more, and some are mixed and some are not.

When the number of SIDs is 2 or more

{__name__=~".+"}
Other than the above

Not applicable.

Notes:

The PromQL statement specified in the API provided by JP1/IM - Manager must be written so that the data of the final summary result is an instant Vector type.

Also, if the retrieved data is of type range Vector, you must use an aggregate operator, operator, or function to create a PromQL statement so that the final aggregate data is of type instant Vector.

(c) Replacing the backdated timestamp of range Vector

API provided by JP1/IM - Manager can use "$stepTime{minSeconds=\"minimum-seconds\"}" as the backdated timestamp to range Vector type specified by PromQL expression.

If "$stepTime{minSeconds=\"minimum-seconds\"}" is specified in PromQL expression, it will be replaced by the time obtained by the following formula. Specify a positive integer for minimum-seconds.

Formulas for dashboards
```
Trend data acquisition range (seconds) / (number of plots set in the panel - 1)
```
If the value obtained by the calculation formula is a decimal, the fractional part is truncated to an integer.
Formulas for trend data display
```
Trend data acquisition range (seconds) / countPerInstance^#
```
If the value obtained by the calculation formula is a decimal, the fractional part is rounded up to an integer.

#: When displayed in the Trends tab of the integrated operation viewer, 60 is assumed. For details about countPerInstance, see Parameters in 5.11.2 Time-series data acquisition in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

However, if the value obtained by the calculation expression is smaller than the value specified by minimum-seconds, "$stepTime{minSeconds="minimum-seconds"}" is replaced with the time specified by minimum-seconds.

(d) Available aggregate operators

The following are the aggregate operators that can be used with PromQL: You can use aggregate operators to aggregate retrieved data.

Operator	Description
sum	Return the total.
count	Returns the number of performance values.
avg	Returns the average value.
min	Returns the minimum value.
max	Returns the maximum value.

The above aggregate operators can be labeled as follows.

by Label-name

Summarizes each specified label.
without Label-name

Ignores the specified label and aggregates.

(e) Available operators

The following operators can be used with PromQL: You can use the operator to calculate performance values.

■Calculation target

Depending on the combination of calculation targets, the operation of the operator differs. The operation of the operator for each combination to be calculated is shown below.

Combinations of calculation targets	instant Vector	range Vector	scalar
instant Vector	Add per label	Error	Add scalar values for each label
range Vector	Error	Labels, add per timestamp	Labels, add per timestamp
scalar	Add scalar values for each label	Labels, add per timestamp	Add scalar-to-scal

■Operations on labels

When calculating between instant vectors and range vectors, you can specify the following operations for labels.

ignoring

Perform calculations ignoring only the specified labels.
on

The calculation takes into account only the specified labels.

■Operator

The following operators are available:

Operator	Description
`+`	Between instant vectors and range vectors Adds each value of the same label. For instant Vector and scalar, instant Vector and scalar Add the scalar value to each performance value of the Vector. For scalar-to-scal Add the values as is. Other than the above Returns a calculation error.
`-`	Between instant vectors and range vectors Subtracts the value of the same label. For instant Vector and scalar, instant Vector and scalar Subtract the scalar value from each performance value in the Vector. For scalar-to-scal Subtract the value as is. Other than the above Returns a calculation error.
`*`	Between instant vectors and range vectors Multiply the values of the same label. For instant Vector and scalar, instant Vector and scalar Multiply each performance value of the Vector by the value of scalar. For scalar-to-scal Multiply the value as is. Other than the above Returns a calculation error.
`/`	Between instant vectors and range vectors Divides the value of the same label. For instant Vector and scalar, instant Vector and scalar Divide each performance value in the Vector by the scalar value. For scalar-to-scal ivide the value as is. Other than the above Returns a calculation error.
`%`	Between instant vectors and range vectors Find the remainder of the value of the same label. For instant Vector and scalar, instant Vector and scalar Find the remainder of each performance value in the Vector divided by the scalar value. For scalar-to-scal Find the remainder of the value. Other than the above Returns a calculation error.
`^`	Between instant vectors and range vectors Returns the "right-hand value" power of the left-hand value between the same labels. For instant Vector and scalar, instant Vector and scalar Returns the "scalar" power of each performance value in the Vector. For scalar-to-scal Returns the "right-hand value" power of the left-hand value. Other than the above Returns a calculation error.

■Comparator

The following are the available comparators:

Comparator	Description
`==`	Between instant vectors and range vectors Returns only data with the same value between the same labels. For instant Vector and scalar, instant Vector and scalar Returns each performance value of Vector whose value is "scalar value". For scalar-to-scal Returns a calculation error. You must specify the bool modifier. Other than the above Returns a calculation error.
`!=`	Between instant vectors and range vectors Returns only data with different values between the same labels. For instant Vector and scalar, instant Vector and scalar Of the performance values of Vector, those whose value is not "scalar value" are returned. For scalar-to-scal Returns a calculation error. You must specify the bool modifier. Other than the above Returns a calculation error.
`>`	Between instant vectors and range vectors The performance values of the Vector are compared, and only data that is larger on the left side than on the right side is returned. For instant Vector and scalar, instant Vector and scalar Of each performance value of Vector, only data greater than the scalar value is returned. For scalar-to-scal Returns a calculation error. You must specify the bool modifier. Other than the above Returns a calculation error.
`<`	Between instant vectors and range vectors The performance value of the vector is compared, and only data that is smaller on the left side than on the right side is returned. For instant Vector and scalar, instant Vector and scalar Of each performance value of Vector, only data less than the scalar value is returned. For scalar-to-scal Returns a calculation error. You must specify the bool modifier. Other than the above Returns a calculation error.
`>=`	Between instant vectors and range vectors Compares the performance values of vectors and returns only data that is larger or the same value on the left side than the right side. For instant Vector and scalar, instant Vector and scalar Of each performance value of Vector, only data with a value greater than or equal to the value of scalar is returned. For scalar-to-scal Returns a calculation error. You must specify the bool modifier. Other than the above Returns a calculation error.
`<=`	Between instant vectors and range vectors Compares the performance values of vectors and returns only data that is smaller or the same value on the left side than on the right side For instant Vector and scalar, instant Vector and scalar Of each performance value of Vector, only data with a value less than or equal to the value of scalar is returned. For scalar-to-scal Returns a calculation error. You must specify the bool modifier. Other than the above Returns a calculation error.

■Logical operators

The following logical operators are available:

Logical operators	Description
`and`	Returns data on the left side only if there is data on the corresponding right side.
`or`	If the left side has a sample, the data on the left side is returned, and if the left side does not have a sample, the data on the right side is returned.
`unless`	If there is data on the right side, the data is not returned, and if there is no data on the right side, the data on the left side is returned.

(f) Available functions

The following functions can be used with PromQL:

Function name	Description	Argument type^#	Return Type^#
abs	Make the performance value absolute.	instant Vector	instant Vector
absent	An empty element is returned if there is a performance value, and the element with a value of 1 is returned if there is no performance value.	instant Vector	instant Vector
absent_over_time	For all time series data with a specified period, An empty element is returned if there is a performance value, and the element with the value 1 is returned if there is no performance value.	range Vector	instant Vector
avg_over_time	Returns the average of the values in the time period.	range Vector	instant Vector
ceil	The performance value is rounded up to the nearest decimal point and returned.	instant Vector	instant Vector
changes	Returns the number of times the value has changed.	range Vector	instant Vector
clamp	If the performance value in the Vector is less than or equal to the minimum value, it rounds to the minimum value, and if it is greater than or equal to the maximum value, it rounds to the maximum value.	First argument instant Vector Second argument scalar Third argument scalar	instant Vector
clamp_max	If the performance value in the Vector is greater than or equal to the maximum value, it is rounded to the maximum value.	First argument instant Vector Second argument scalar	instant Vector
clamp_min	If the performance value in the Vector is less than or equal to the minimum value, it is rounded to the minimum value.	First argument scalar Second argument scalar	instant Vector
count_over_time	Returns the number of pieces of data for the specified time period.	range Vector	instant Vector
delta	Returns the difference between the first and last values of the specified time period.	range Vector	instant Vector
floor	The performance value is rounded down to the nearest decimal point and returned.	instant Vector	instant Vector
increase	Returns the amount of value growth for the specified time period.	range Vector	instant Vector
irate	Returns the percentage increase per second for the specified period (compared to the previous performance value only).	range Vector	instant Vector
label_replace	For each time series of data, replace the label with a regular expression. Argument 1: Target data Second argument: Replacement label name Third argument: Replaced string Fourth argument: Label name to replace Fifth argument: Source string (regular expression)	First argument instant Vector From the second argument onwards string	instant Vector
label_join	For each time series of data, combine the labels. Argument 1: Target data Second argument: Destination label Third argument: Separator string From the fourth argument onwards: Binding source label	First argument instant Vector From the second argument onwards string	instant Vector
last_over_time	Returns the latest data for each range of time series data.	range Vector	instant Vector
max_over_time	Returns the largest amount of data for each range of time series data.	range Vector	instant Vector
min_over_time	Returns the smallest data for each range of time series data.	instant Vector	instant Vector
predict_linear	From the performance value of the metric specified in the first argument, the performance value after the number of seconds specified in the second argument is predicted and returned. For the first argument, specify a metric whose metric type is `Gauge`.	First argument range Vector Second argument scalar	instant Vector
rate	Returns the percentage increase per second for the specified period.	range Vector	instant Vector
resets	Returns the number of times the performance value has been reset (decreased).	range Vector	instant Vector
round	Returns a value rounded to the nearest decimal point.	First argument instant Vector From the second argument onwards scalar	instant Vector
stddev_over_time	Returns the standard deviation.	range Vector	instant Vector
sum_over_time	Return the total.	range Vector	instant Vector

#

instant Vector

Data at a specific time.
range Vector

Data at multiple times. It is a collection of instant Vector data for multiple times.
scalar

Numeric.
string

String.

To Page Top