2.19.2 Settings of JP1/IM - Agent

(a) Edit the configuration files (for Linux)

See 1.21.2(1)(a) Edit the configuration files (for Windows).

(b) Changing unit definition file (for Linux only)

The following lists unit definition file storage locations and filenames:

• Storage destination: /usr/lib/systemd/system/

• File name: jpc_ Service name.service

To change unit definition file, follow these steps:

1. Login to integrated agent host.

2. Stop JP1/IM - Agent service.

3. Edit unit definition file.

4. Execute the following command to reflect the unit definition.

   # systemctl daemon-reload

5. Start JP1/IM - Agent service.

(c) Change command-line options (for Linux)

On ExecStart line of unit definition file, change the command-line options.

For editing methods, see 2.19.2(1)(b) Changing unit definition file (for Linux only).

(a) Change Integrated manager to connect to (for Linux) (optional)

See 1.21.2(2)(a) Change Integrated manager to connect to (for Windows) (optional).

(b) Changing Ports (for Linux) (Optional)

See 1.21.2(2)(b) Change the port (for Windows) (optional).

(c) Deploy a CA certificate (for Linux) (optional)

See 1.21.2(2)(c) Place CA certificate (for Windows) (optional).

(d) Modify settings related to Action Execution (on Linux) (optional)

See 1.21.2(2)(d) Modify settings related to Action Execution (for Windows) (optional).

(e) Setup the proxy authentication's authentication ID and Password (optional)

See 1.21.2(2)(e) Setup the proxy authentication's authentication ID and Password (for Windows) (optional).

(f) Change the user of Action Execution (mandatory)

Change action.username in imagent configuration file (jpc_imagent.json).

For setup procedure, see 2.19.2(1)(a) Edit the configuration files (for Linux).

(a) Changing Ports (For Linux) (Optional)

The listen port used by Prometheus server is specified in --web.listen-address option of prometheus command.

For details about how to change prometheus command options, see 2.19.2(1)(c) Change command-line options (for Linux). For details of --web.listen-address option, see If you want to change command line options in Unit definition file (jpc_program-name.service) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

The default port is "20713". If port number is changed, review setup of the firewall and prohibit accessing from outside. However, if you want to monitor Prometheus server with external shape monitoring by Blackbox exporter in other host, allow it to be accessed. In such cases, consider security measures such as limiting the source IP address as required.

(b) Add the alert definition (for Linux) (optional)

See 1.21.2(3)(b) To Add the alert definition (for Windows) (optional).

(c) Add a Blackbox exporter scrape job (for Linux) (optional)

See 1.21.2(3)(c) Add Blackbox exporter scrape job (for Windows) (optional).

(d) Add a user-defined Exporter scrape job (for Linux) (optional)

See 1.21.2(3)(d) Add user-defined Exporter scrape job (for Windows) (optional).

(e) Change RemoteWrite destination (for Linux) (optional)

See 1.21.2(3)(e) Changing Remote Write destination (for Windows) (optional).

(f) Configuring service monitor settings (for Linux) (Optional)

To use the service monitoring function in an environment where the version is upgraded and installed from JP1/IM - Agent 13-00 to 13-01 or later, configure the following settings. This setting is not required if JP1/IM - Agent 13-01 or later is newly installed.

(Omitted)
scrape_configs:
(Omitted)
  - job_name: 'jpc_node'
(Omitted)
    metric_relabel_configs:
      - source_labels: ['__name__']
        regex: 'node_network_receive_bytes_total|node_network_transmit_bytes_total|-- Omitted --|node_vmstat_pswpin|node_vmstat_pswpout|node_systemd_unit_state'
        action: 'keep'
      - source_labels: ['__name__']
        regex: 'node_systemd_unit_.*'
        target_label: 'jp1_pc_trendname'
        replacement: 'node_exporter_service'
      - source_labels: ['__name__']
        regex: 'node_systemd_unit_.*'
        target_label: 'jp1_pc_category'
        replacement: 'service'
      - source_labels: ['__name__','name']
        regex: 'node_systemd_unit_.*;(.*)'
        target_label: 'jp1_pc_nodelabel'
        replacement: ${1}
      - regex: jp1_pc_multiple_node
        action: labeldrop

- targets:
  - hostname:20716
  labels:
    jp1_pc_exporter: JPC Node exporter
    jp1_pc_category: platform
    jp1_pc_trendname: node_exporter
    jp1_pc_multiple_node: "{__name__=~'node_systemd_unit_.*'}"

(g) Set when the IM management node label name (jp1_pc_nodelabel value) exceeds the upper limit (for Linux) (Optional)

See 1.21.2(3)(g) Configure the settings when the label name (jp1_pc_nodelabel value) of the IM management node exceeds the upper limit (for Windows) (Optional).

(h) Setting for executing the SAP system log extract command using Script exporter (for Linux) (Optional)

See 1.21.2(3)(h) Setting for executing the SAP system log extract command using Script exporter (for Windows) (Optional).

(a) Changing Ports (For Linux) (Optional)

The listen port used by Alertmanager is specified in --web.listen-address option of alertmanager command.

For details about how to change alertmanager command options, see 2.19.2(1)(c) Change command-line options (for Linux). For details of --web.listen-address option, see If you want to change command line options in Unit definition file (jpc_program-name.service) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

The default port is "20714". If port number is changed, review setup of the firewall and prohibit accessing from outside. However, if you want to monitor Alertmanager with external shape monitoring by Blackbox exporter in other host, allow it to be accessed. In such cases, consider security measures such as limiting the source IP address as required.

(b) Changing the alert notification destination (for Linux) (optional)

See 1.21.2(4)(b) Changing the alert notification destination (for Windows) (optional).

(c) Setup silence (on Linux) (optional)

See 1.21.2(4)(c) Setup silence (for Windows) (optional).

(a) Changing Ports (Optional)

The listen port used by Node exporter is specified in --web.listen-address option of node_exporter command.

For details about how to change the options of the node_exporter command, see 2.19.2(1)(c) Change command-line options (for Linux). For details of --web.listen-address option, see If you want to change command line options in Unit definition file (jpc_program-name.service) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

The default port is "20716". If port number is changed, review setup of the firewall and prohibit accessing from outside.

(b) Change metric to collect (optional)

1. In the metric_relabel_configs of Prometheus configuration file (jpc_prometheus_server.yml), metric to be collected are defined separated by "|". Delete metric that you do not need to collect and add metric that you want to collect.

   For instructions on updating configuration file, see 1.21.2(1)(a) Edit the configuration files (for Windows).

   <Sample Setup>

     - job_name: 'jpc_node'
       :
       metric_relabel_configs:
         - source_labels: ['__name__']
           regex: 'node_boot_time_seconds|node_context_switches_total|node_cpu_seconds_total|node_disk_io_now|node_disk_io_time_seconds_total|node_disk_read_bytes_total|node_disk_reads_completed_total|.......|node_time_seconds|node_uname_info|node_vmstat_pswpin|node_vmstat_pswpout [Add metric here]'
           action: 'keep'

2. If required, define a trend view in metric definetion file.

   In Node exporter metric definition file, you define a trend view.

For descriptions, see Node exporter metric definition file (metrics_node_exporter.conf) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

1. Configure the service monitor settings.

   - Editing Node exporter's unit definition file (jpc_node_exporter.service).

   When monitoring services, edit the unit definition file (jpc_node_exporter.service) of Node exporter as shown in the underlined part below.

   [Unit]
   Description = JP1/IM3-Agent Linux metric collector
   After=local-fs.target remote-fs.target rsyslog.service network.target
   [Service]
   WorkingDirectory = @@installdir2@@/jp1ima/bin
   ExecStart = /bin/sh -c '"@@installdir1@@/jp1ima/bin/node_exporter" \
     --collector.cpu.info \
   (Omitted)
     --no-collector.supervisord \
     --collector.systemd \
     --collector.systemd.unit-include="Regular expressions to match unit file names" \
     --no-collector.tcpstat \
     --no-collector.textfile \
     --no-collector.thermal_zone \
   (Omitted)

   If "--no-collector.systemd" is set as the value of the argument to be set on the ExecStart line, change it to "--collector.systemd". If "--collector.systemd.unit-include" is not set, add a line and set the value to a regular expression that matches the unit file name of the unit you want to monitor.

   Depending on how regular expressions are specified, it may take some time to collect performance information. For more information, see G.4 Tips on using regular expressions section in the manual JP1/Integrated Management 3 - Manager Overview and System Design Guide.

   - Node exporter unit definition file (jpc_node_exporter.service) definition example

   The following shows a sample configuration for monitoring jpc_imagent.service and jpc_imagentproxy.service units.

   [Unit]
   Description = JP1/IM3-Agent Linux metric collector
   After=local-fs.target remote-fs.target rsyslog.service network.target
   [Service]
   WorkingDirectory = @@installdir2@@/jp1ima/bin
   ExecStart = /bin/sh -c '"@@installdir1@@/jp1ima/bin/node_exporter" \
     --collector.cpu.info \
   (Omitted)
     --no-collector.supervisord \
     --collector.systemd \
     --collector.systemd.unit-include="^(jpc_imagent|jpc_imagentproxy)\.service$" \
     --no-collector.tcpstat \
     --no-collector.textfile \
     --no-collector.thermal_zone \
   (Omitted)

Among the units that correspond to the above specification, the units for which automatic startup is enabled or the status is being executed are monitored. For more information, see the Specifying monitored services in 3.15.1(1)(d) Node exporter (Linux performance data collection capability) in JP1/Integrated Management 3 - Manager Overview and System Design Guide.

(a) Specifying monitored processes (required)

- Edit the Process exporter configuration file (jpc_process_exporter.yml)

Edit the Process exporter configuration file (jpc_process_exporter.yml) to define which processes are to be monitored.

By default, no process is to be monitored, and therefore you will uncomment the initial settings and then specify the processes you want to monitor in the Process exporter configuration file.

For details on the Process exporter configuration file, see Process exporter configuration file (jpc_process_exporter.yml) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

(b) Modifying monitoring metrics (optional)

- Edit the Prometheus configuration file (jpc_prometheus_server.yml)

If you want to change metrics to be collected, modify the metric_relabel_configs setting in the Prometheus configuration file (jpc_prometheus_server.yml).

For details on the Prometheus configuration file, see Prometheus configuration file (jpc_prometheus_server.yml) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

- Edit the Process exporter metric definition file (metrics_process_exporter.conf)

If you want to change metrics displayed in the Trends tab of the integrated operation viewer, edit the settings in the Process exporter metric definition file (metrics_process_exporter.conf).

For details on the Process exporter metric definition file, see Process exporter metric definition file (metrics_process_exporter.conf) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

(c) Changing Ports (Optional)

The listen port used by Process exporter is specified in --web.listen-address option of the process_exporter command.

For details about how to change the options of process_exporter command, see 2.19.2(1)(c) Change command-line options (for Linux). For details about --web.listen-address option, see If you want to change command line options in Unit definition file (jpc_program-name.service) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

The default port is "20721". If port number is changed, review setup of the firewall and prohibit accessing from outside.

Notes:

• When specifying the host name for this option, the same host name must be set for targets in the Process exporter discovery configuration file (jpc_file_sd_config_process.yml) on the same host.

• When specifying an IP address for this option, the host name that is resolved to the IP address specified for this option must be set for targets in the Process exporter discovery configuration file (jpc_file_sd_config_process.yml) on the same host.

(a) Changing Ports (For Linux) (Optional)

The listen port used by Blackbox exporter is specified in --web.listen-address option of the blackbox_exporter command.

For details about how to change the options of blackbox_exporter command, see 2.19.2(1)(c) Change command-line options (for Linux). For details about --web.listen-address option, see If you want to change command line options in Unit definition file (jpc_program-name.service) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

The default port is "20715". If port number is changed, review setup of the firewall and prohibit accessing from outside.

(b) Add, Modify, or Delete a Module. (Optional)

See 1.21.2(6)(b) Add, change, and delete modules (for Windows) (optional).

(c) Add, Modify, or Delete a monitoring target (for Linux) (mandatory)

See 1.21.2(6)(c) Add, change, or Delete the monitoring target (for Windows) (mandatory).

(d) Monitor HTTP through proxy (on Linux) (optional)

See 1.21.2(6)(d) Monitoring HTTP through proxy (for Windows) (optional).

(e) Setup the proxy authentication ID and Password (optional)

See 1.21.2(6)(e) Setup the proxy authentication ID and Password (for Windows) (Optional).

(f) Setup authentication ID, Password, and Bearer tokens for accessing the monitored Web Server (optional)

See 1.21.2(6)(f) Setup authentication ID, Password, and Bearer tokens for accessing the monitored Web Server (for Windows) (optional).

(a) Changing Ports (For Linux) (Optional)

See 1.21.2(7)(a) Changing Ports (For Windows) (Optional).

(b) Modify Setup to connect to CloudWatch (for Linux) (optional)

See 1.21.2(7)(b) Modify Setup to connect to CloudWatch (for Windows) (optional).

(c) Connect to CloudWatch through a proxy (for Linux) (optional)

See 1.21.2(7)(c) Connect to CloudWatch through a proxy (for Windows) (optional).

(d) Add AWS Services to be Monitored (Optional)

See 1.21.2(7)(d) Add AWS Services to be Monitored (Optional).

(e) Monitoring AWS Resources (Optional)

See 1.21.2(7)(e) Monitoring AWS Resources (Optional).

(f) Modify metric to Collect (Optional)

See 1.21.2(7)(f) Modify metric to Collect (Optional).

(a) Changing Setup of Common Definition file for Log Monitor (Optional)

See 1.21.2(9)(a) Changing Setup of Common Definition file for Log Monitor (for Windows) (Optional).

(b) Monitor Text-Format Logging File (Required)

1.21.2(9)(b) Monitoring the text-format logging File (for Windows) (Required)".

(c) Modifying the Monitoring Setup for Text-Format Logging File (Optional)

See 1.21.2(9)(c) Modifying the Monitoring Setup of the Text-Format Logging File (for Windows) (Optional).

(d) Delete Monitoring File in text-format setting

See 1.21.2(9)(d) Deleting the monitoring settings of the text log file (for Windows) (optional).

(e) Setup of the log metrics definition (Required)

See 1.21.2(9)(h) Setup of the log metrics definition (Required).

(f) Changing Ports (Optional)

See 1.21.2(9)(i) Changing Ports (Optional).

(g) Monitoring SAP system log information (Optional)

See 1.21.2(9)(j) Monitoring SAP system logging (Optional).

(h) Change monitoring settings for SAP system log information (Optional)

See 1.21.2(9)(k) Modify SAP system logging monitoring configuration (Optional)

(i) Delete the monitoring settings for log information in SAP systems (Optional)

See 1.21.2(9)(l) Remove SAP system logging monitoring configuration (Optional).

(j) Monitoring CCMS alert information for SAP systems (Optional)

See 1.21.2(9)(m) Monitoring CCMS alerting for SAP system (Optional).

(k) Change monitoring settings for CCMS alert information in SAP systems (Optional)

See 1.21.2(9)(n) Modify SAP system CCMS alert information monitoring settings (Optional).

(l) Delete the monitoring settings for CCMS alert information in SAP systems (Optional)

See 1.21.2(9)(o) Remove SAP system CCMS alert information monitoring settings (Optional).

(a) External shape monitoring by other-host Blackbox exporter

Prometheus server and Alertmanager services monitors from Blackbox exporter of integrated agent running on other hosts. The following tables show URL to be monitored.

For details about how to Add HTTP monitor of Blackbox exporter, see 1.21.2(6)(c) Add, change, or Delete the monitoring target (for Windows) (mandatory). For details about how to set the alert definition, see 1.21.2(3)(b) To Add the alert definition (for Windows) (optional)".

For an example of the alert definitions to be monitored by HTTP Monitor of Blackbox exporter, see 1.21.2(15) Setup of integrated agent process alive monitoring (for Windows) (optional).

(b) Alive Monitoring Processes by Process exporter

Imagentproxy service, imagentaction service, and Fluentd service are monitored by the operation status of the process monitor of Process exporter. The processes to be monitored are described in the following table.

For details about how to set up, see Process exporter configuration file (jpc_process_exporter.yml) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

For details about setting method of the alert definition, see 1.21.2(3)(b) To Add the alert definition (for Windows) (optional).

The following is a sample Process exporter configuration file that is monitored by Process exporter.

process_names:
  - name: "{{.ExeBase}};{{.Username}};{{.Matches.cmdline}}"
    exe:
     - /opt/jp1ima/bin/imagent
     - /opt/jp1ima/bin/imagentproxy
     - /opt/jp1ima/bin/imagentaction
 
  - name: "{{.ExeBase}};{{.Username}};{{.Matches.cmdline}}"
    exe:
     - /opt/jp1ima/bin/rotatelogs
    cmdline:
     - (?P<cmdline>.*/opt/jp1ima/logs/fluentd\.*)
 
  - name: "{{.ExeBase}};{{.Username}};{{.Matches.cmdline}}"
    exe:
     - /opt/jp1ima/lib/ruby/bin/ruby
    cmdline:
     - (?P<cmdline>.*jpc_fluentd_common\.conf.*)

Here is a sample alertdefinition that Process exporter monitors:

groups:
  - name: process_exporter
    rules:
    - alert: jp1_pc_procmon_imagent
      expr: 1 >  sum by (program, instance, job, jp1_pc_nodelabel, jp1_pc_exporter) (namedprocess_namegroup_num_procs{program="imagent"})
      for: 3m
      labels:
        jp1_pc_product_name: "/HITACHI/JP1/JPCCS2"
        jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO"
        jp1_pc_severity: "Error"
        jp1_pc_eventid: "1303"
        jp1_pc_metricname: "namedprocess_namegroup_num_procs"
      annotations:
        jp1_pc_firing_description: "The number of processes was less than the threshold Value (1). value={{$value}}"
        jp1_pc_resolved_description: "The number of processes exceeded the threshold Value (1)."
 
    - alert: jp1_pc_procmon_imagentproxy
      expr: 1 >  sum by (program, instance, job, jp1_pc_nodelabel, jp1_pc_exporter) (namedprocess_namegroup_num_procs{program="imagentproxy"})
      for: 3m
      labels:
        jp1_pc_product_name: "/HITACHI/JP1/JPCCS2"
        jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO"
        jp1_pc_severity: "Error"
        jp1_pc_eventid: "1303"
        jp1_pc_metricname: "namedprocess_namegroup_num_procs"
      annotations:
        jp1_pc_firing_description: "The number of processes was less than the threshold Value (1). value={{$value}}"
        jp1_pc_resolved_description: "The number of processes exceeded the threshold Value (1)."
 
    - alert: jp1_pc_procmon_imagentactoin
      expr: 1 >  sum by (program, instance, job, jp1_pc_nodelabel, jp1_pc_exporter) (namedprocess_namegroup_num_procs{program="imagentaction"})
      for: 3m
      labels:
        jp1_pc_product_name: "/HITACHI/JP1/JPCCS2"
        jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO"
        jp1_pc_severity: "Error"
        jp1_pc_eventid: "1303"
        jp1_pc_metricname: "namedprocess_namegroup_num_procs"
      annotations:
        jp1_pc_firing_description: "The number of processes was less than the threshold Value (1). value={{$value}}"
        jp1_pc_resolved_description: "The number of processes exceeded the threshold Value (1)."
 
    - alert: jp1_pc_procmon_fluentd_rotatelogs Log trapper(Fluentd) #1
      expr: 1 >  sum by (program, instance, job, jp1_pc_nodelabel, jp1_pc_exporter) (namedprocess_namegroup_num_procs{program="rotatelogs"})
      for: 3m
      labels:
        jp1_pc_product_name: "/HITACHI/JP1/JPCCS2"
        jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO"
        jp1_pc_severity: "Error"
        jp1_pc_eventid: "1303"
        jp1_pc_metricname: "namedprocess_namegroup_num_procs"
      annotations:
        jp1_pc_firing_description: "The number of processes was less than the threshold Value (1). value={{$value}}"
        jp1_pc_resolved_description: "The number of processes exceeded the threshold Value (1). "
 
    - alert: jp1_pc_procmon_fluentd_ruby Log trapper(Fluentd) #2
      expr: 2 >  sum by (program, instance, job, jp1_pc_nodelabel, jp1_pc_exporter) (namedprocess_namegroup_num_procs{program="ruby"}) #3
      for: 3m
      labels:
        jp1_pc_product_name: "/HITACHI/JP1/JPCCS2"
        jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO"
        jp1_pc_severity: "Error"
        jp1_pc_eventid: "1303"
        jp1_pc_metricname: "namedprocess_namegroup_num_procs"
      annotations:
        jp1_pc_firing_description: "The number of processes was less than the threshold Value (2). value={{$value}}"
        jp1_pc_resolved_description: "The number of processes exceeded the threshold Value (2). "

#1

If only log metrics feature is used, specify "jp1_pc_procmon_fluentd_prome_rotatelogs Log trapper(Fluentd)".

#2

If only log metrics feature is used, specify "jp1_pc_procmon_fluentd_prome_ruby Log trapper(Fluentd)".

#3

The Ruby process starts the number of workers + 1. For the threshold, specify the number of workers + 1. For details on the number of workers, see Log monitoring common definition file.

(c) Monitoring by Prometheus server up metric

Node exporter service, Process exporter service, Blackbox exporter service, and Yet another cloudwatch exporter service are monitored through Prometheus server alert-monitoring. For setting method of the alert definition, see 1.21.2(3)(b) To Add the alert definition (for Windows) (optional).

For an example of an alert definition that monitors up metric, see 1.21.2(15) Setup of integrated agent process alive monitoring (for Windows) (optional).