2.7.2 Trend data Management Database
By building a trend data management DB in the intelligent integrated management database, trend data collected from the managed database can be centrally managed by the trend data management DB.
- Organization of this subsection
(1) Database configuration
See 2.7.1(1)(a) Database configuration in Intelligent Integrated Management Database.
(2) Installation and Setup
Build a trend data management DB in the intelligent integrated management database to manage trend data.
For information about building an Intelligent Integrated Management Database (Trend Data Management DB), see the description about construction of Intelligent Integrated Management Database in the JP1/Integrated Management 3 - Manager Configuration Guide.
(3) Deleting trend data
(a) Functional Overview
This function automatically deletes trend data that has passed the retention period from the trend data management DB starting from the time of trend data (in the format of "time, value").
The default retention period is 32 days (46080 minutes).
The retention period can be changed on the setup command for the Intelligent Integrated Administration Database (jimgndbsetup). The upper limit is 1096 days (3 years).
However, changing the settings requires a restart of the Trend Data Management Service.
- ■Operation image
-
For example, if the retention period is "32 days" and the execution time of the delete function is "2022/01/21 09:05:42.055", trend data for the past 32 days from "2022/01/21 09:05:42.055", that is, "2021/12/20 09:05:42.055" to "2022/01/21 09:05:42.055" will be saved and trend data outside this period will be deleted.
The following is an example of whether each trend data stored in the database is deleted under the above conditions:
Trend data stored in a database
Whether it will be deleted
:
:
Deleted
Data on 2021/12/20 09:05:42.017
Data on 2021/12/20 09:06:42.017
Not deleted
Data on 2021/12/20 09:07:42.017
:
:
Data on 2022/01/21 09:05:42.017
(b) Setting and changing the retention period
You can set and change the retention period of trend data with the jimgndbsetup command.
- ■Setting the retention period
-
-
For physical host environments
Specify the retention period# in item "RETENTION" of the Intelligent Integrated Management Database Setup Information File (jimgndbsetupinfo.conf).
-
For clustered environment
Specify the retention period# in the item "RETENTION" in the cluster environment intelligent integrated management database setup information file (jimgndbclustersetupinfo.conf).
- #
-
Trend data that has passed the set retention period is deleted periodically (every 30 minutes).
The following is an example of the entry "RETENTION".
# IM GNDATABASE SERVICE - Retention
RETENTION=32
For details about Intelligent Integrated Management Database setup information file, and cluster environment Intelligent Integrated Management Database setup information file, see Intelligent Integrated Management Database setup information file (jimgndbsetupinfo.conf) and Cluster environment Intelligent Integrated Management Database setup information file (jimgndbclustersetupinfo.conf) in Chapter 2 Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
-
- ■Running the jimgndbsetup command
-
An example of running the jimgndbsetup command is shown below.
jimgndbsetup -f Setup-information-file-name#
- #
-
-
For physical host environments
Intelligent Integrated Management Database Setup Information File Name
-
For clustered environment
Cluster Environment Intelligent Integrated Management Database Setup Information File Name
-
For details about the jimgndbsetup command, see jimgndbsetup in Chapter 1. Commands in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
(c) When to automatically delete trend data
Trend data is automatically deleted every 30 minutes starting when the retention period is set by the jimgndbsetup command. If the retention period of trend data is the default (32 days (46080 minutes)), trend data whose retention period has passed 46080 minutes is deleted every 30 minutes.
If the database is stopped at the time of automatic deletion of trend data, automatic deletion of trend data is executed when the database is started. In this case, the next automatic deletion operation is scheduled 30 minutes after the automatic deletion process completes.
Also, the timing and interval for deleting trend data cannot be changed by the user.
You can check the scheduled date and time of deletion of trend data with the jimgndbstatus command.
For details about the jimgndbstatus command, see jimgndbstatus in Chapter 1. Commands in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
(d) About saving trend data for deletion
Trend data that has passed the retention period is deleted from the trend data management DB. If there is trend data (time series data) that you want to use even after the retention period has elapsed, use the time-series data acquisition API# of JP1/IM - Manager (Intelligent Integrated Management Platform) to output and save the corresponding data.
For details on the time-series data acquisition API, see 5.11.2 Time-series data acquisition in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
- #
-
JP1/IM - Agent product plug-in is required. JP1/IM - Use the time-series data acquisition method (__timeSeriesDataGet method) for acquiring trend information implemented by the agent's product plug-in.
■ Use case "Output and save trend data (time series data) to be deleted monthly"
Do the following:
-
List the SID and metric of the IM management node specified in the time series data acquisition API in advance.
- •Checking the SID of the output target JP1/IM - Agent
-
Check the SID of JP1/IM - Agent that outputs time series data using the API for acquiring IM management node information.
SID of JP1/IM - Agent is identified from the node of Prometheus SID (_JP1PC-IMB_ JP1/IM agent management base hostname/_JP1PC-M_P hostname/_HOST_P hostname/_PROMETHEUS_) or agent SID (_JP1PC-IMB_JP1/IM agent management base hostname/_JP1PC-M_P hostname/_JP1PC-AHOST_I hostname/_HOST_I hostname/_JP1PC-A_jp1_pc_nodelabel).
P Hostname: Host name of the host on which the Prometheus server is set up
I Hostname: Host name of the monitored host
jp1pc_nodelabel: setting value in jp1_pc_nodelabel of scrape_configs in Prometheus configuration file (jpc_prometheus_server.yml)
For details on the IM management node information acquisition API, see 5.8.3 IM management node information acquisition in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
The following is an image of executing an API for retrieving IM management node information using the OSS curl command on a Windows command prompt.
- Run Login API
curl -XPOST -H "Content-Type: application/json" -c path-of-the-output-destination-text-file-A -d "{\"user\":\"JP1-user-name\",\"password\":\"password\"}" http://hostname:20703/im/api/v1/login
- Execute API for retrieving IM management node information
curl -GET -H "Content-Type: application/json" -b path-of-text-file-A http://hostname:20703/im/api/v1/nodes/configInfo
- Execution result of API for IM management node information acquisition (JSON format)
{"simtData":
[{
"sid":"XXXX/XXXX",
"value":{"component":"/HITACHI/JP1/XXXX","visible":"true","label":"XXXX"}
},{
.......
},{
"sid":"YYYY/YYYY/YYYY",
"value":{ ......
"methods":[.......,"__metricListGet","__timeSeriesDataGet",......],
......}
},{
......
}]
}
- •Checking which metrics can be output
-
Use the metric list acquisition API to check the metric name of the time series data.
For details on the metric list acquisition API, see 5.11.1 Metric list acquisition in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
The following is an image of executing the metric list acquisition API using the OSS curl command on a Windows command prompt.
- Example of executing the Login API
curl -XPOST -H "Content-Type: application/json" -c path-of-the-output-destination-text-file-A -d "{\"user\":\"JP1-user-name\",\"password\":\"password\"}" http://hostname:20703/im/api/v1/login
- Example of executing the metric list acquisition API
curl -XPOST -H "Content-Type: application/json" -b path-of-text-file-A -d "{\"sid\":\"JP1/IM-Agent-SID\"}" http://hostname:20703/im/api/v1/nodes/metrics
- Example of metric list acquisition API execution result (JSON format)
{"metrics":[
{
"name":Metric-name,
"default":Default-settings,
"label":Metric-display-name,
"description":"Metric-description"
} {
............
}
]}
-
Specify the SID of the JP1/IM - Agent node, the metric, and the output period (start date and time and end date and time) and execute the time-series data acquisition API.
- •Get trend data
-
Use the time series data acquisition API to get the metric name of trend data (time series data). Since the API for acquiring time series data is executed by combining SID and metric, if there are multiple SIDs or metrics of JP1/IM - Agent nodes for which you want to output trend data, you need to execute the number of SIDs × number of metrics.
For details on the time-series data acquisition API, see 5.11.2 Time-series data acquisition in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
The maximum number of data that can be acquired by the Time Series Data Acquisition API is 30,000. For example, if the number of data per instance is one, the number of data will be 44,640 if data for × month (31 days) per minute is output, so it is necessary to divide the output target period into two times. The necessity of splitting and the number of splits depend on the data interval, output target period, and metric content (number of data per instance).
In addition, since the time series data acquisition API can fetch data in JSON format, it is necessary to extract trend data from JSON format data.
Specifically, since the "data" of "timeSeriesData" is stored in the format "{"time":date-and-time-data, "value":(metric)-value-at-that-date-and-time}," it is necessary to extract the data of "date-and-time-data" and "(metric)-value-at-that-date-and-time".
The following is an example of a request and response for the Time Series Data Acquisition API.
- Example Time Series data retrieval API Request
POST http://hostname:20703/im/api/v1/nodes/timeSeries
{
"sid":"_JP1PC-IMB-IMMHOST/_JP1PC-M_IMAHOST/_HOST_IMAHOST/_PROMETHEUS_",
"metric": "cpu_used_rate",
"startTime": "2021-06-01T00:00:00Z",
"endTime": "2021-06-15T23:59:59Z",
"countPerInstance":1,
"instanceCount":30000
}
- Time Series data acquisition API Response Example
< HTTP/1.1 200 OK
< Content-Type: application/json
{
"metric":"cpu_used_rate",
"timeSeriesData":[
{
"unit":"%",
"data":[
{"time":"2021-06-01T00:00:00Z","value":14.04},
......
]
],
"exceedCountDetected": {
"countPerInstance": false,
"instanceCount": false
}
}
The following is an image of executing the time series data acquisition API using the OSS curl command on a Windows command prompt and outputting one month's worth of data in two parts.
- Example of executing the Login API
curl -XPOST -H "Content-Type: application/json" -c path-of-the-output-destination-text-file-A -d "{\"user\":\"JP1-user-name\",\"password\":\"password\"}" http://hostname:20703/im/api/v1/login
- Example of executing Time Series data acquisition API <1st time>
curl -XPOST -H "Content-Type: application/json" -b path-of-text-file-A -d "{\"sid\":\"JP1/IM-Agent-SID\",
\"metric\":\"metric-name\",
\"startTime\":\"2021-06-01T00:00:00Z\",
\"endTime\":\"2021-06-15T23:59:59Z\",
\"countPerInstance\":30000,
\"instanceCount\":1}"
http://hostname:20703/im/api/v1/nodes/timeSeries
- Example of execution result of Time Series data acquisition API (JSON format) <1st time>
{"metric":
"cpu_used_rate",
"timeSeriesData":[
{"unit":"%",
"data":[
{"time":"2021-06-01T00:00:00Z","value":10.480632},
......
]}
],
"exceedCountDetected":{"countPerInstance":false,"instanceCount":false}
}
- Example of executing Time Series data acquisition API <2nd time>
curl -XPOST -H "Content-Type: application/json" -b path-of-text-file-A -d "{\"sid\":\"JP1/IM-Agent-SID\",
\"metric\":\"metric-name\",
\"startTime\":\"2021-06-16T00:00:00Z\"
\"endTime\":\"2021-06-30T23:59:59Z\",
\"countPerInstance\":30000,
\"instanceCount\":1}"
http://hostname:20703/im/api/v1/nodes/timeSeries
- Example of execution result of Time Series data acquisition API (JSON format) <2nd time>
{"metric":
"cpu_used_rate",
"timeSeriesData":[
{"unit":"%",
"data":[
{"time":"2021-06-16T00:00:00Z","value":9.560327},
......
]}
],
"exceedCountDetected":{"countPerInstance":false,"instanceCount":false}
}
(e) Manual deletion of trend data
If you want to delete trend data in the Intelligent Integrated Management Database without stopping operation, delete trend data by directly manipulating the table with the superuser role (Postgres).
To remove trend data for Linux 7, Linux 8, or Linux 9, follow these steps:
-
Run the following command to connect to the Intelligent Integrated Administration database with the superuser role:
psql -d imgndb -U postgres -W -p port-number-of-the-Trend-Data-Management-database
When prompted for a password, enter the password shown below.
2BXY7aVkbicj
-
Run the following command to get a list of metric names:
select * from metric;
When you execute the above command, you can get the values of metric_name (metric name obtained as trend data) and table_name (name of the table that stores trend data for time series).
-
Check the table corresponding to the metric name confirmed in step 2 (repeat the number of metrics from step 3 onwards).
The name of the table corresponding to the metric name is "prom_data. Metric Name".
You can run the following command to check the number of records in the table corresponding to each metric name.
select count(*) from prom_data.metric-name;
-
Check the number and content of records to be deleted.
As an example, if you want to delete records one month before the current time (keep records from the last one month), you can check the number of records in the table corresponding to each metric name by executing the following command.
select count(*) from prom_data.metric-name where time <= cast(now() - interval# '1 months' as timestamp);
You can also run the following command to check the contents of the record to be deleted.
select * from prom_data.metric-name where time <= cast(now() - interval# '1 months' as timestamp);
- #
-
The unit of the value specified by interval can also be minutes, hour, etc.
You can also specify a specific date, as shown in the following example.
select * from prom_data.metric-name where time <= '2021-09-30';
-
Delete trend data.
Since trend data is managed in units called chunks (child tables by date and time interval), trend data is deleted in chunks.
As an example, if you want to delete data older than one month, run the following command:
select drop_chunks('prom_data.metric-name', older_than => interval# '1 months');
- #
-
The unit of the value specified by interval can also be minutes, hour, etc.
You can also specify a specific date, as shown in the following example.
select drop_chunks('prom_data.metric-name', older_than => '2021-09-30');
If you delete trend data in chunks, chunks containing only the data for the period specified at the time of deletion will be deleted. Therefore, data from the specified period is not deleted if it is stored in a chunk that contains data outside the specified period.
(f) Automatic deletion of trend data (size cap)
Monitor the amount of disk space used by Trend data Management Database. If the amount of disk space used exceeds the standard value (90% of the value specified in TDDBDISKMAX of Intelligent Integrated Management Database setup information file, or cluster environment Intelligent Integrated Management Database setup information file), remove the old trend data to reduce the amount of disk space used.
You can use Intelligent Integrated Management Database setup command to set the following items:
-
Max Disk Requirements for Trend data Management Database
Trend data is automatically deleted when it exceeds 90% of the specified value.
-
Duration of trend data to be deleted
When automatic deletion of trend data is executed, the data of the specified period from the oldest trend data is deleted.
- ■Monitoring Disk Requirements and Trend Data Deletion Timing
-
Monitors disk requirements and deletes trend data (when the threshold is exceeded) as one task and performs this task at 1-minute intervals. (The next task is executed one minute after the completion of one task.)
- ■About the duration of trend data to be deleted
-
Specify in TDDBCUTOFFTERM of Intelligent Integrated Management Database setup information file, or cluster environment Intelligent Integrated Management Database setup information file.