2.7.4 Managing Trend Data by IM management node
- Organization of this subsection
(1) Data to be stored in the trend data management DB
Trend data (time series data) stored in the trend data management DB is uniquely identified by the metric name and label. Labels are also used to link IM management nodes to trend data. When browsing trend data, you can specify a label to get trend data associated with the IM management node.
The format and examples of metric names and labels are shown below.
-
Metric name and label format
metric-name{label-name="label-value" label-name="label-value"...}
-
Examples of metric names and labels
windows_logical_disk_free_bytes{instance="host1:9001" volume="C:"}
windows_logical_disk_free_bytes{instance="host1:9001" volume="D:"}
windows_logical_disk_free_bytes{instance="host2:9001" volume="C:"}
Trend data (time series data) is formed by multiple samples. Each sample consists of the following items:
-
Millisecond-precision timestamps (UNIX time)
-
Float64 (double-precision floating-point type) value (performance data)
The following is an example of time series data.
Metric name |
windows_logical_disk_free_bytes |
||||
---|---|---|---|---|---|
label and value |
instance="host1:9001" volume="C:" |
instance="host1:9001" volume="D:" |
instance="host2:9001" volume="C:" |
||
Time series data |
Sample |
Timestamp |
1624260855 |
1624260855 |
1624260855 |
Performance data |
55153717248.0 |
923198898.0 |
194208931840.0 |
||
Sample |
Timestamp |
1624260870 |
1624260870 |
1624260870 |
|
Performance data |
55151808512.0 |
923162116.0 |
194208688168.0 |
||
Sample |
Timestamp |
1624260885 |
1624260885 |
1624260885 |
|
Performance data |
55153377280.0 |
923146812.0 |
194208931976.0 |
||
Sample |
Timestamp |
1624260900 |
1624260900 |
1624260900 |
|
Performance data |
55157182464.0 |
923138684.0 |
194208921684.0 |
(2) Retrieving data stored in the trend data management DB
Prometheus query (PromQL) is used to retrieve data stored in the trend data management DB. For supported PromQL language specifications, see 2.7.4(4)About PromQL.
PromQL can retrieve trend data in two types:
-
Getting with the instant Vector type
Represents a collection of performance data at a point in time, usually at the time of data acquisition.
For example, in Table 2-14 Sample time-series data, it refers to the collection of performance-data 55153717248.0, 923198898.0, and 194208931840.0 for timestamp 1624260855.
-
Getting with the range Vector type
Represents a collection of performance data from multiple points in time from a point in time (usually the time the data was retrieved).
For example, in Table 2-14 Sample time-series data, it refers to the collection of performance data 55153377280.0, 923146812.0, 194208931976.0 for timestamp 1624260885, and the collection of performance data 55157182464.0, 923138684.0, and 194208921684.0 for timestamp 1624260900.
To get trend data, you can filter and retrieve only those with a specific label by specifying the metric name and label.
If there is no trend data matching the specified label, trend data is not returned (performance data will not be returned with 0 if no trend data exists).
For example, if you filter time-series data shown in Table 2-14 Sample time-series data by specifying windows_logical_disk_free_bytes{volume="C:"} in PromQL expression, you can get the set of performance data corresponding to the label that contains volume="C:" as shown below.
-
Performance data 55153717248.0, 55151808512.0, 55153377280.0, 55157182464.0 corresponding to the labels instance="host1:9001" volume="C:"
-
Performance data 194208931840.0, 194208688168.0, 194208931976.0, 194208921684.0 corresponding to the label instance="host2:9001" volume="C:"
In addition, various aggregation results can be obtained by aggregating or calculating the retrieved trend data using the aggregation operator, operator, or function of the PromQL statement.
- Notes:
-
PromQL statements specified in the API provided by JP1/IM - Manager must be written so that the data of the final summary result is an instant Vector type.
Also, if the retrieved data is of type range Vector, you must use an aggregate operator, operator, or function to create a PromQL statement so that the final aggregate data is of type instant Vector.
(a) Ability to write trend data
This function writes trend data to the trend data management DB.
Use JP1/IM - Manager's Write Trend Data REST API to write trend data to the Trend Data Management DB via the Trend Data Management Service (Promscale).
For details on the write trend data API, see 5.11.3 Write Trend Data in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
The data to be written to the trend data management DB is shown below.
Types of data |
Description |
Specify multiple |
Can be omitted |
---|---|---|---|
Metric name |
Information that represents the type of trend data. |
not possible |
not possible |
Label |
Additional information about trend data, such as the host from which trend data is issued. Data in the format "label-name:label-value". |
possible |
possible |
Trend data (time series data) |
Data in the format of "time,value". |
possible |
not possible |
Multiple trend data writing operations can be executed at the same time. Exclusive control of data writing is performed in the trend data management DB.
(b) Ability to browse trend data
This function refers to trend data from the trend data management DB.
Use the jp1TrendDataService.getTrendData method that can be used in the product plug-in to reference trend data from the trend data management DB via the trend data management service (Promscale). JP1/IM - Manager (Intelligent Integrated Management Base) issues a query (PromQL) to the trend data management service using REST API to retrieve trend data.
For details on the jp1TrendDataService.getTrendData method, see 4.5.17 jp1TrendDataService.getTrendData in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
The trend data reference process can be performed at the same time as the write operation. The data that can be referenced is the data that has been written to when the trend data management DB receives the reference request.
■Referring to trend data linked to the IM management node
In the jp1TrendDataService.getTrendData method that retrieves trend data from the trend data management DB, you can refer to the trend data associated with the IM management node by setting the replacement string $jp1im_TrendData_labels in the PromQL statement specified in the promQLQuery argument.
(c) Trend data deletion function
Refer to 2.7.2(3) Deleting trend data.
(3) About Trend Data Objects
See 7.2.3(1) Trend Data Object in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
(4) About PromQL
(a) Get trend data by specifying metric names and labels
Prometheus queries (PromQL) can retrieve trend data in the following two types.
-
Getting with the instant Vector type
Point-in-time performance data (instant vector) can be obtained in one of the following formats:
metric-name{label}
metric-name
The Time Series Data Acquisition API provided by JP1/IM - Manager acquires performance data from the start date and time of the time series data specified in member startTime to the end date and time of the time series data specified in member endTime at the trend data acquisition interval (in seconds) specified in member step.
The retrieved data may have more than 10 decimal places.
If you want the retrieved data to be rounded up, down, or rounded, use the functions of the PromQL statement (ceil, flor, round, etc.).
For details of the functions of the PromQL statement, see "2.7.4(4)(e) Available functions".
-
Getting with the range Vector type
Performance data (range vectors) from multiple points in time can be obtained in one of the following formats:
metric-name{label}[Time-back#]
metric-name[Time-back #]
- #
-
The time to go back is specified as a number and in units (s, m, h, d, w, or y).
(b) Replacing SID-associated labels
In the API provided by JP1/IM - Manager, the replacement string $jp1im_TrendData_labels can be used for the labels specified in the PromQL statement.
If the replacement string $jp1im_TrendData_labels is specified in the PromQL statement, it will be replaced with the following string indicated by BNF notation.
<$jp1im_TrendData_labels> ::= {__name__=~".+"} | (<condition>) <condition> ::= { <key_value_list> } | <condition> or { <key_value_list> } <key_value_list> ::= <key_value> | <key_value_list>, <key_value > <key_value> ::= KEY="VALUE#" |
- #
-
The backslash (\) in the value of VALUE is specified in PromQL with \\ and double quotation marks (") replaced with \".
Depending on the number of labels specified for the SIDs and the number of SIDs, the strings that are replaced are shown below.
- ■When the number of labels specified for SID is 1
-
-
When the number of SIDs is 1
{KEY1=VALUE1}
-
When the number of SIDs is 2 or more
{SID1.KEY1=SID1.VALUE1} or
{SID2.KEY1=SID2.KEY2} ...
-
When the number of SIDs is 0
Do not issue a query (PromQL) to the Trend Data Management Service (Promscale) and return 0 data.
-
- ■When the number of labels specified for SID is 2 or more
-
-
When the number of SIDs is 1
{KEY1=VALUE1,KEY2=VALUE2 ...}
-
When the number of SIDs is 2 or more
{SID1.KEY1=SID1.VALUE1,SID1.KEY2=SID1.VALUE2...} or
{SID2.KEY1=SID2.VALUE1,SID2.KEY2=SID2.VALUE2...} ...
-
When the number of SIDs is 0
Do not issue a query (PromQL) to the Trend Data Management Service (Promscale) and return 0 data.
-
- ■When the number of labels specified for SID is 0
-
-
When the number of SIDs is 1
{__name__=~".+"}
-
When the number of SIDs is 0
Do not issue a query (PromQL) to the Trend Data Management Service (Promscale) and return 0 data.
-
- ■When the number of labels specified for SIDs is 2 or more, and some are mixed and some are not.
-
-
When the number of SIDs is 2 or more
{__name__=~".+"}
-
Other than the above
Not applicable.
-
- Notes:
-
The PromQL statement specified in the API provided by JP1/IM - Manager must be written so that the data of the final summary result is an instant Vector type.
Also, if the retrieved data is of type range Vector, you must use an aggregate operator, operator, or function to create a PromQL statement so that the final aggregate data is of type instant Vector.
(c) Available aggregate operators
The following are the aggregate operators that can be used with PromQL: You can use aggregate operators to aggregate retrieved data.
Operator |
Description |
---|---|
sum |
Return the total. |
count |
Returns the number of performance values. |
avg |
Returns the average value. |
min |
Returns the minimum value. |
max |
Returns the maximum value. |
The above aggregate operators can be labeled as follows.
-
by Label-name
Summarizes each specified label.
-
without Label-name
Ignores the specified label and aggregates.
(d) Available operators
The following operators can be used with PromQL: You can use the operator to calculate performance values.
- ■Calculation target
-
Depending on the combination of calculation targets, the operation of the operator differs. The operation of the operator for each combination to be calculated is shown below.
Combinations of calculation targets
instant Vector
range Vector
scalar
instant Vector
Add per label
Error
Add scalar values for each label
range Vector
Error
Labels, add per timestamp
Labels, add per timestamp
scalar
Add scalar values for each label
Labels, add per timestamp
Add scalar-to-scal
- ■Operations on labels
-
When calculating between instant vectors and range vectors, you can specify the following operations for labels.
-
ignoring
Perform calculations ignoring only the specified labels.
-
on
The calculation takes into account only the specified labels.
-
■Operator
The following operators are available:
Operator |
Description |
---|---|
+ |
|
- |
|
* |
|
/ |
|
% |
|
^ |
|
■Comparator
The following are the available comparators:
Comparator |
Description |
---|---|
== |
|
!= |
|
> |
|
< |
|
>= |
|
<= |
|
■Logical operators
The following logical operators are available:
Logical operators |
Description |
---|---|
and |
Returns data on the left side only if there is data on the corresponding right side. |
or |
If the left side has a sample, the data on the left side is returned, and if the left side does not have a sample, the data on the right side is returned. |
unless |
If there is data on the right side, the data is not returned, and if there is no data on the right side, the data on the left side is returned. |
(e) Available functions
The following functions can be used with PromQL:
Function name |
Description |
Argument type# |
Return Type# |
---|---|---|---|
abs |
Make the performance value absolute. |
instant Vector |
instant Vector |
absent_over_time |
For all time series data with a specified period, the performance value is returned if there is a performance value, and the element with the value 1 is returned if there is no performance value. |
range Vector |
instant Vector |
avg_over_time |
Returns the average of the values in the time period. |
range Vector |
instant Vector |
ceil |
The performance value is rounded up to the nearest decimal point and returned. |
instant Vector |
instant Vector |
changes |
Returns the number of times the value has changed. |
range Vector |
instant Vector |
clamp |
If the performance value in the Vector is less than or equal to the minimum value, it rounds to the minimum value, and if it is greater than or equal to the maximum value, it rounds to the maximum value. |
First argument instant Vector Second argument scalar Third argument scalar |
instant Vector |
clamp_max |
If the performance value in the Vector is greater than or equal to the maximum value, it is rounded to the maximum value. |
First argument instant Vector Second argument scalar |
instant Vector |
clamp_min |
If the performance value in the Vector is less than or equal to the minimum value, it is rounded to the minimum value. |
First argument scalar Second argument scalar |
instant Vector |
count_over_time |
Returns the number of pieces of data for the specified time period. |
range Vector |
instant Vector |
delta |
Returns the difference between the first and last values of the specified time period. |
range Vector |
instant Vector |
floor |
The performance value is rounded down to the nearest decimal point and returned. |
instant Vector |
instant Vector |
increase |
Returns the amount of value growth for the specified time period. |
range Vector |
instant Vector |
irate |
Returns the percentage increase per second for the specified period (compared to the previous performance value only). |
range Vector |
instant Vector |
label_replace |
For each time series of data, replace the label with a regular expression. Argument 1: Target data Second argument: Replacement label name Third argument: Replaced string Fourth argument: Label name to replace Fifth argument: Source string (regular expression) |
First argument instant Vector From the second argument onwards string |
instant Vector |
label_join |
For each time series of data, combine the labels. Argument 1: Target data Second argument: Destination label Third argument: Separator string From the fourth argument onwards: Binding source label |
First argument instant Vector From the second argument onwards string |
instant Vector |
max_over_time |
Returns the largest amount of data for each range of time series data. |
range Vector |
instant Vector |
min_over_time |
Returns the smallest data for each range of time series data. |
instant Vector |
instant Vector |
rate |
Returns the percentage increase per second for the specified period. |
range Vector |
instant Vector |
resets |
Returns the number of times the performance value has been reset (decreased). |
range Vector |
instant Vector |
round |
Returns a value rounded to the nearest decimal point. |
First argument instant Vector From the second argument onwards scalar |
instant Vector |
sum_over_time |
Return the total. |
range Vector |
instant Vector |
- #
-
-
instant Vector
Data at a specific time.
-
range Vector
Data at multiple times. It is a collection of instant Vector data for multiple times.
-
scalar
Numeric.
-
string
String.
-