3.14.8 Cooperation with user's own Prometheus
The following describes how to work with user-defined Prometheus.
-
Verifying the prerequisites
The prerequisite is that JP1/IM - Manager,Prometheus, Alertmanager, Exporter is installed and set up.
-
Publish client secret
Execute JP1/IM - Manager's client secret publishing API(/im/api/v1/secret/client/generateInfo) to obtain client secret #.
- #
-
Client secret can be retrieved only once, so be sure to keep a copy of it.
Here is an example of the message body for a request and response:
-
Message body of the request
{ "clientId": ">prometheus" }
-
Message body of the response
{ "secret": "kABNY0UFwa4K+W5KhLbkva945Whx+sguH65QinFggaW3CZaB6bA0MkgNgKN2FSSs1kORnQHpWUCHitkeBUzEmYHY09d9BR+tg73vSfbh05xStE7qB1idgfzisqhlqu7R+Q+XNA", "lastUpdateTime": "2023-05-18T07:59:23Z" }
For details about client secret issue API, see 5.4.5 IM Client secret issue in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
-
Editing prometheus.yml
Edits prometheus.yml directly under Prometheus destination (the path where prometheus.exe resides).
An example is shown below. Bold text indicates points added or changed from the default. Specify values in italics to suit your environment.
(Example of description)
# my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). external_labels: jp1_pc_prome_hostname: "Prometheus-hostneme" # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: - Prometheus-hostneme:Prometheus-Port-number # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: - "alerting_rules.yml#1" # - "first_rules.yml" # - "second_rules.yml" # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'OPTIONAL NAME #2' file_sd_configs: - files: - 'file_sd_config_any name.yml#3' relabel_configs: - target_label: jp1_pc_nodelabel replacement: Windows exporter metric_relabel_configs: - source_labels: ['__name__'] regex: 'windows_cs_physical_memory_bytes|windows_cache_copy_read_hits_total|windows_cache_copy_reads_total|windows_cpu_time_total|windows_logical_disk_free_bytes|windows_logical_disk_idle_seconds_total|windows_logical_disk_read_bytes_total|windows_logical_disk_read_latency_seconds_total|windows_logical_disk_read_seconds_total|windows_logical_disk_reads_total|windows_logical_disk_requests_queued|windows_logical_disk_size_bytes|windows_logical_disk_write_bytes_total|windows_logical_disk_write_latency_seconds_total|windows_logical_disk_write_seconds_total|windows_logical_disk_writes_total|windows_memory_available_bytes|windows_memory_cache_bytes|windows_memory_cache_faults_total|windows_memory_page_faults_total|windows_memory_pool_nonpaged_allocs_total|windows_memory_pool_paged_allocs_total|windows_memory_swap_page_operations_total|windows_memory_swap_pages_read_total|windows_memory_swap_pages_written_total|windows_memory_system_cache_resident_bytes|windows_memory_transition_faults_total|windows_net_bytes_received_total|windows_net_bytes_sent_total|windows_net_bytes_total|windows_net_packets_sent_total|windows_net_packets_received_total|windows_system_context_switches_total|windows_system_processor_queue_length|windows_system_system_calls_total' action: 'keep' remote_write: - url: http://JP1/IM - Manager hostname: Intelligent Integrated Management Base portnumber/im/api_system remote_timeout: 30s send_exemplars: false queue_config: capacity: 10000 max_shards: 200 min_shards: 4 max_samples_per_send: 3000 batch_send_deadline: 10s min_backoff: 100ms max_backoff: 10s basic_auth: username: 'Client ID specified in the client secret publishing API in JP1/IM - Manager' password: 'Client secret issued by the client secret publishing API in JP1/IM - Manager'
- #1
-
Specify the filename of alert configuration file described in step 6.
- #2
-
Write a scrape definition for each Exporter.
- #3
-
Specify the filename of the discovery configuration file as described in step 4.
-
Creating a discovery configuration file
Create a discovery configuration file (file_sd_config_any name.yml) and place it directly under Prometheus destination.
An example is shown below. In italics, specify a value appropriate for your environment.
(Example of description)
- targets: - Hostname of Exporter: Port number of Exporter labels: jp1_pc_exporter: any-name #1 jp1_pc_category: platform jp1_pc_trendname: any-name #2
- #1
-
Specify any name that does not start with "JPC_".
- #2
-
You must specify a name that matches the "Any Prometheus trend name" part of the filename of metric definition file (metrics_any Prometheus trend name.conf).
-
Editing alertmanager.yml
Edits alertmanager.yml directly under Alertmanager destination (the path where alertmanager.exe resides).
An example is shown below. Bold text indicates points added from the default. Specify values in italics to suit your environment.
(Example of description)
route: receiver: 'JP1IMDD' group_by: ['...'] continue: false group_wait: 5s group_interval: 5s repeat_interval: 7d receivers: - name: 'JP1IMDD' webhook_configs: - send_resolved: true url: 'http:// JP1/IM - Manager Host Name: Intelligent Integrated Management Platform port number/im/api_system/v1/events/transform' http_config: basic_auth: username: 'Client ID specified in the client secret publishing API in MJP1/IM - Manager' password: 'Client ID specified in the client secret publishing API in JP1/IM - Manager'
-
Creating a alert configuration file
Create an alert-definition file (with any name) and place it directly under Prometheus destination.
An example is shown below. If you want to create an alert definition file other than the example description, see Definition example in alert configuration file (jpc_alerting_rules.yml) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
(Example of definition)
groups: - name: windows_exporter rules: - alert: cpu_used_rate(Windows exporter) expr: 0 < 100 - (avg by (instance,job,jp1_pc_nodelabel,jp1_pc_exporter) for: 3m labels: jp1_pc_product_name: "/HITACHI/JP1/JPCCS2" jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO" jp1_pc_severity: "Error" jp1_pc_eventid: "0401" jp1_pc_metricname: "windows_cpu_time_total" annotations: jp1_pc_firing_description: "CPU utilization has exceeded the threshold value (0%). value={{$value}}%" jp1_pc_resolved_description: "CPU utilization has fallen below the threshold (0%). "
-
Creating a metric definition file
Create a metric definition file and place it in the following path:
JP1/IM - Manager install directory
\conf\imdd\plugin\jp1pccs\user\
If the targets you define are JP1/IM - Agent monitorable Exporter and metric, you can use the files provided by JP1/IM - Agent (JP1/IM - Manager install directory \conf\imdd\plugin\jp1pccs\*.conf).
-
Starting Exporter ,Alertmanager,Prometheus
Start Exporter ,Alertmanager,Prometheus.
-
Generating and reflecting trees
Start the command prompt in administrator mode and run the following command:
-
jddsetacccessuser (not required if already executed)
-
jddcreatetree
-
jddupdatetree
-
-
Checking the Tree
Log in to integrated operation viewer and verify that nodes of Prometheus, Alertmanager Exporter are created.