1.23.2 Setting up Node exporter for AIX

(a) Obtaining the setup archive file

Retrieve the setup archive file to use Node exporter for AIX. The setup archive file is located in JP1/IM - Agent installation folder.

The file names for the setup archive files are as follows.

JP1/IM-Agent-installation-folder/jp1ima/options/

• node_exporter_aix_VVRRSS.tar.Z (AIX version)

(b) Setting up JP1/IM - Manager

If JP1/IM - Manager 13-01 or later is newly installed, it does not need to be executed.

Perform the following steps when upgrading from a version earlier than JP1/IM - Manager 13-01.

(c) Setting up JP1/IM - Agent

If JP1/IM - Manager 13-01 or later is newly installed, it does not need to be executed.

Perform the following steps when upgrading from a version earlier than JP1/IM - Manager 13-01.

(Omitted)
   :
scrape_configs:
   :
  - job_name: 'jpc_node_aix'
    
    file_sd_configs:
      - files:
        - 'jpc_file_sd_config_node_aix.yml'
    
    relabel_configs:
      - target_label: jp1_pc_nodelabel
        replacement: AIX metric collector(Node exporter for AIX)
    
    metric_relabel_configs:
      - source_labels: ['__name__']
        regex: 'node_context_switches|node_cpu|aix_memory_iomu|aix_cpu_wait|aix_diskpath_wblks|aix_disk_rserv|aix_disk_rblks|aix_disk_wserv|aix_disk_wblks|aix_diskpath_rblks|node_filesystem_avail_bytes|node_filesystem_files_avail|node_filesystem_free_bytes|node_filesystem_free_bytes|node_filesystem_size_bytes|node_intr|node_load1|node_load15|node_load15|aix_netbuffer_inuse|aix_memory_real_avail|aix_netinterface_mtu|aix_netinterface_obytes|aix_netinterface_ierrors|aix_netinterface_ipackets|aix_netadapter_tx_bytes|aix_netinterface_collisions|aix_netadapter_tx_errors|aix_netinterface_opackets|aix_memory_pgspins|aix_memory_pgspouts'
        action: 'keep'

(a) Deploying the Node exporter for AIX

Extract node_exporter_aix_VVRRSS.tar.Z obtained in 1.23.2(1)(a) Obtaining the setup archive file to any directory on the monitored host (AIX). Do not specify the same directory configuration as JP1/IM - Manager or JP1/IM - Agent installation location in the destination directory.

When operating Node exporter for AIX on a logical host, place it in a shared directory.

(b) Output setting of system log

Set this item if the system log is not output.

Node exporter for AIX log is written to OS system log. Therefore, Node exporter for AIX logging settings are based on OS system logging settings. If you have not configured OS system logs to be printed, add the settings to /etc/syslog.conf.

The following shows an example configuration to add.

User.info /var/log/syslog.log rotate size 10m files 8

After adding the above settings, restart syslogd for the settings to take effect.

If you are using user for another application to output to the system log, the log is also output.

(c) Enabling service-registration for Node exporter for AIX

For information about enabling service registration for Node exporter for AIX, see 10.4.2(1) Enabling service registration in the JP1/Integrated Management 3 - Manager Administration Guide.

(d) Node exporter for AIX service registration activation confirmation

For details on checking the activation of the service registration of Node exporter for AIX, see 10.4.2(4) Checking the service status in the JP1/Integrated Management 3 - Manager Administration Guide.

(e) Starting Node exporter for AIX

For details on starting Node exporter for AIX service, see 10.4.2(6) Starting the service in the JP1/Integrated Management 3 - Manager Administration Guide.

(f) Adding a monitoring target

For details on adding a monitoring target, see 1.23.2(4)(a) Adding a monitoring target (Required).

(a) Deleteing a monitoring target host (AIX)

1. Deletes the monitoring target host (AIX)

   Remove the monitored hosts (AIX) listed in the (jpc_file_sd_config_node_aix.yml) targets.

   For details about Node exporter for AIX Discovery configuration file, see Node exporter for AIX discovery configuration file (jpc_file_sd_config_node_aix.yml) (2. Definition File) in the JP1/Integrated Management 3 - Manager Command Definition File, and API Reference.

   When deleting multiple monitored target hosts (AIX), delete multiple monitored target hosts (AIX) listed in targets, restart Prometheus, and then execute 1.23.2(3)(b) Stopping Node exporter for AIX and later for each of the deleted monitored hosts (AIX).

2. Restart Prometheus.

   Restart Prometheus for the changes to take effect.

(b) Stopping Node exporter for AIX

For details on stopping the service of Node exporter for AIX, see 10.4.2(7) Stopping the service in the JP1/Integrated Management 3 - Manager Administration Guide.

(c) Disabling Auto-Start for a Node exporter for AIX

If you have enabled auto-start of Node exporter for AIX, you must disable it.

For details on disabling auto-start of Node exporter for AIX, see 10.4.2(10) Disabling automatic startup in the JP1/Integrated Management 3 - Manager Administration Guide.

(d) Disabling Auto-Stop for a Node exporter for AIX

If you have enabled Node exporter for AIX auto-stop, you must disable it.

For details on disabling auto-stop of Node exporter for AIX, see 10.4.2(12) Disabling automatic stop in the JP1/Integrated Management 3 - Manager Administration Guide.

(e) Disabling service registration for Node exporter for AIX

If you have enabled service registration for the Node exporter for AIX, you must disable it.

For details on disabling the registration of Node exporter for AIX services, see 10.4.2(2) Disabling the registration of services in the JP1/Integrated Management 3 - Manager Administration Guide.

Make sure that Node exporter for AIX service registration is disabled after execution.

(f) Deleting Node exporter for AIX

To delete Node exporter for AIX, delete the directory that were placed in "1.23.2(2)(a) Deploying the Node exporter for AIX".

(a) Adding a monitoring target (Required)

The procedure for adding a monitoring target is as follows.

(b) Changing the port on Node exporter for AIX (Optional)

You must change the listen port on Node exporter for AIX and the port number that Prometheus will scrape.

The procedure for changing is as follows.

1. Stopping Prometheus service.

2. Stopping Node exporter for AIX service.

3. Change the listen port for Node exporter for AIX registered to the service listed in the following tables.

   OS	Host type	File to be changed
   AIX	For physical host operation	Services with subsystem named "jpc_node_exporter_aix"
   	For logical host operation	Services with subsystem named "jpc_node_exporter_aix_logical-host-name"

   For details on changing the registration of a service, see 10.4.2(3) Registering the service changed in the JP1/Integrated Management 3 - Manager Administration Guide.

4. Changes Node exporter for AIX port number that scrape Prometheus in the definition-file shown in the following tables.

   OS	Host type	File to be changed
   Windows	For physical host operation	JP1/IM-Agent-installation-folder\jp1ima\conf\jpc_file_sd_config_node_aix.yml
   	For logical host operation	JP1/IM-Agent-shared-folder\jp1ima\conf\jpc_file_sd_config_node_aix.yml
   Linux	For physical host operation	/opt/jp1ima/conf/jpc_file_sd_config_node_aix.yml
   	For logical host operation	JP1/IM-Agent-shared-directory/jp1ima/conf/jpc_file_sd_config_node_aix.yml

5. Start Node exporter for AIX.

6. Start Prometheus service.

7. Perform the settings in steps 1 to 6 and wait a few minutes after all services have started.

8. Refresh IM management node tree.

   Follow steps 2 to 5 in 1.19.3(1)(c) Creation and import of IM management node tree data (for Windows) (mandatory).

(c) Modify metric to collect (Optional)

1. Add metric to Prometheus configuration file (jpc_prometheus_server.yml).

   Modify the metrics listed in the metric_relabel_configs of the Prometheus configuration file (jpc_prometheus_server.yml).

   For Prometheus configuration file editing procedure, see 1.21.2(1)(a) Edit the configuration files (for Windows).

   Note that the supported metric is set to be collected by default.

     - job_name: 'jpc_node_aix'
         :
       metric_relabel_configs:
         - source_labels: ['__name__']
           regex: 'node_context_switches|node_cpu|aix_memory_iomu|aix_cpu_wait|aix_diskpath_wblks|aix_disk_rserv|aix_disk_rblks|aix_disk_wserv|.......|aix_netinterface_opackets|aix_memory_pgspins|aix_memory_pgspouts  (Add here)'
           action: 'keep'

2. Define trend display in metric definition file

   Optionally, define a metric in metric definition-file that you want to view in integrated operation viewer trend view. The primary metric is initially configured.

(d) Configuring Node exporter for AIX life-and-death monitoring (Optional)

Node exporter for AIX processes are monitored by up metric of Prometheus server.

For details on how to configure monitoring with Prometheus server's up metric, see 1.21.2(15)(c) Monitoring with Prometheus server up metrics.

(e) Monitoring processes on monitoring hosts (AIX) (Optional)

Monitor the processes on the monitoring hosts (AIX) using the following script-results in Script exporter:

• Script that connects SSH to the monitored host (AIX) and executes ps command on the monitored host (AIX)

The following are configuration steps and examples of specific scripts and definition files.

In the following example, integrated operation viewer's tree chart format creates a node with the process name under Platform category node of AIX's host node and displays a graphical representation of only metric with the Process Status. Script exporter metrics other than Process Status, are graphically displayed on the "monitoring-target-host-name(AIX):aix_process" node under Other Applications category node of the agent host node on which Script exporter is installed, and Process Status metric is not graphically displayed.

1. Create a script that connects SSH to the monitored host (AIX) and executes ps command on the monitored host (AIX).

   Create a script that connects to the monitoring host (AIX) with ssh command and executes ps command on the monitoring host (AIX). If you want to monitor multiple processes, create one script for each process you want to monitor.

   #

   It is necessary to enable SSH connection from the integrated agent host to the monitored host (AIX) in advance.

   You must also be able to connect SSH to anyone who runs the following scripts:

   Environments under which Script exporter runs	User running the script
   Windows	Users who run Script exporter services. (Local system account)
   Linux	root

   • When Script exporter runs in Windows

   Create the following batch and text files:

   - Batch file

   @echo off
   setlocal
    
   rem Name of the process to be monitored on rem monitored host (AIX) (specify with a space separator if you specify more than one)
   set PROCESS=node_exporter_aix logger
    
   rem Users who ssh connect to the monitoring target host (AIX)
   set USER=user1
    
   rem Private key file path in PuTTY format used to connect ssh to rem monitored host (AIX)
   Set PUTTY_ID_RSA="C:\tmp\ScriptExporter\id_rsa.ppk"
    
   rem Host name of the monitoring target host (AIX)
   set AIX_HOST=aixhost1
    
   rem ps command result output destination file (specify any location)
   set FILE1="C:\tmp\ScriptExporter\ps.txt"
    
   rem A file that outputs the contents to be read by the Script exporter (specify any location)
   set FILE2="C:\tmp\ScriptExporter\ps_for_exporter.txt"
    
   rem Specify rem plink.exe filepath
   Set PLINK_EXE="C:\Program Files\PuTTY\plink.exe"
    
   rem Specifies the text file path for rem (b)
   Set CMD_LIST="C:\tmp\ScriptExporter\command.txt"
    
   rem Destination file for plink.exe stderr (any location)
   set LOG_FILE="C:\tmp\ScriptExporter\ssh_log.txt"
    
   echo enter | %PLINK_EXE% %USER%@%AIX_HOST% -i %PUTTY_ID_RSA% -m %CMD_LIST% > %FILE1% 2>%LOG_FILE%
    
   rem Output metrics
   (echo # HELP script_aix_process AIX process status.&echo # TYPE script_aix_process gauge)> %FILE2%
    
   for %%i in (%PROCESS%) do call:findpsinfo %%i
    
   type %FILE2%
   endlocal
   exit /b
    
   :findpsinfo
   for /f "tokens=1-3* usebackq" %%i in (`findstr %1 %FILE1%`) do (
     set PNAME=%1
     set STATE=%%j
     call:putinfo
     exit /b
   )
   set PNAME=%1
   set STATE=
   call:putinfo
   exit /b
    
   :putinfo
   if "%STATE%"=="A" (
     echo script_aix_process{host="%AIX_HOST%",process="%PNAME%",state="A"} 1 >> %FILE2%
   ) else (
     echo script_aix_process{host="%AIX_HOST%",process="%PNAME%",state="A"} 0 >> %FILE2%
   )
   exit /b

   -Text file

   /usr/bin/ps -A -X -o comm,st,etime

   • When Script exporter runs in Linux

   Create the following shell script.

   #!/bin/sh
    
   # Process-name to be monitored on the monitored host (AIX) (specified by a space separator if more than one is specified)
   PROCESS=("node_exporter_aix" "logger")
    
   # Private key file used to connect ssh to the monitoring target host (AIX)
   ID_RSA=/temp/script_exporter/id_rsa
    
   # Users ssh connecting to monitoring target host (AIX)
   USER=guest
    
   # Hostname of the monitoring target host (AIX)
   AIX_HOST=aixhsot1
    
   # ps command result output destination file (specify any location)
   FILE1=/temp/script_exporter/ps.txt
    
   #A file that outputs the contents to be read by the Script exporter (specify any location)
   FILE2=/temp/script_exporter/ps_for_exporter.txt
    
   # Destination file for ssh stderr (any location)
   LOG_FILE=/temp/script_exporter/ssh_log.txt
    
   /usr/bin/ssh -i $ID_RSA $USER@$AIX_HOST /usr/bin/ps -A -X -o comm,st,etime >$FILE1 2>$LOG_FILE
    
   echo '# HELP script_aix_process AIX process status.'>$FILE2
   echo '# TYPE script_aix_process gauge'>>$FILE2
    
   for target in "${PROCESS[@]}" ; do
     psinfo=`grep -m1 "$target" $FILE1`
     find_flag=$?
     if [ $find_flag = 0 ]; then
       psinfo=($psinfo)
       if [ "${psinfo[1]}" = "A" ]; then
         echo "script_aix_process{host=\"$AIX_HOST\",process=\"$target\",state=\"A\"} 1">>$FILE2
       else
         echo "script_aix_process{host=\"$AIX_HOST\",process=\"$target\",state=\"A\"} 0">>$FILE2
       fi
     else
       echo "script_aix_process{host=\"$AIX_HOST\",process=\"$target\",state=\"A\"} 0">>$FILE2
     fi
   done
    
   cat $FILE2

2. Editing Script exporter configuration file and Prometheus configuration file.

   Edit the Script exporter configuration file and the Prometheus configuration file as follows.

   For configuration file editing procedure, see 1.21.2(1)(a) Edit the configuration files (for Windows).

   • Editing Script exporter configuration file

   Change the following:

   scripts:
     - name: monitoring-target-host-name(AIX) :aix_process
       command: file-path-of-the-script-created-in-step-1
       timeout:
         max_timeout: 30
         enforced: true

   • Editing Prometheus configuration file

   Change the metric_relabel_configs of Script exporter as follows:

     - job_name: 'jpc_script_exporter'
         :
       metric_relabel_configs:
         - source_labels: ['__name__']
           regex: 'script_success|script_duration_seconds|script_exit_code|script_aix_process'
           action: 'keep'
         - source_labels: [jp1_pc_script]
           target_label: jp1_pc_nodelabel
         - source_labels: ['__name__']
           target_label: jp1_pc_category
           regex: 'script_aix_process'
           replacement: platform
         - source_labels: ['__name__','process']
           regex: 'script_aix_process.*;(.*)'
           target_label: jp1_pc_nodelabel
           replacement: ${1}
         - source_labels: ['__name__','host']
           regex: 'script_aix_process.*;(.*)'
           target_label: instance
           replacement: ${1}
         - regex: (jp1_pc_script|jp1_pc_multiple_node|jp1_pc_agent_create_flag)
           action: labeldrop

3. Editing alert configuration file.

   For an alert definition that alerts you when a monitoring target process stops, edit alert configuration file as follows.

   For details on how to edit alert configuration file, see 1.21.2(1)(a) Edit the configuration files (for Windows).

   groups:
     - name: script_exporter
       rules:
       - alert: process_state(Script exporter)
         expr: script_aix_process{state="A"} == 0
         for: 3m
         labels:
           jp1_pc_product_name: "/HITACHI/JP1/JPCCS2"
           jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO"
           jp1_pc_severity: "Error"
           jp1_pc_eventid: "1404"
           jp1_pc_metricname: "script_aix_process"
         annotations:
           jp1_pc_firing_description: "The status of the process is not running."
           jp1_pc_resolved_description: "The status of the process is now running."

4. Create and import tree information for IM management nodes.

   Perform steps 1 to 3 and perform steps 2 to 5 in 1.19.3(1)(c) Creation and import of IM management node tree data (for Windows) (mandatory).

5. Editing Script exporter metric definition file.

   Editing Script exporter metric definition file as follows.

   For details on how to edit Script exporter metric definition file, see To edit the configuration files (for Windows) in 1.19.3(1)(a) Common way to setup.

   [
   :
     {
       "name": "script_exit_code",
       :
     },
     {
       "name": "script_aix_process",
       "default": true,
       "promql": "script_aix_process{state=\"A\"} and $jp1im_TrendData_labels",
       "resource_en": {
         "category": "script_aix_process",
         "label": "Process status",
         "description": "Status of processes on AIX hosts.",
         "unit": "-"
       },
       "resource_ja": {
         "category": "script_aix_process",
         "label": "Processing status",
         "description": "The state of the process on the AIX host.",
         "unit": "-"
       }
     }
   ]

   The monitored hosts (AIX) listed above have been tested with the following OS and packaging:

   • OS

     AIX7.2 TL3 SP2

   • Package

     Openssh.base.client 7.5.102.1500

     Openssh.base.server 7.5.102.1500

     Openssh.man.en_US 7.5.102.1500

     Openssh.msg.en_US 7.5.102.1500

(f) Changing the log destination of Node exporter for AIX (Optional)

Node exporter for AIX log is written to OS system log. To change Node exporter for AIX logging destination, use syslogd Settings and logger command I will.

The following shows an example in which local1.info is specified in-p of logger command, the log output destination of the system log is set to /var/log/jpc_node_exporter_aix/jpc_node_exporter_aix.log, and the log size is changed to 8 10MB, log sectors. ^#

#

If you are using local1 for other products to output to the system log, the output destination of the log is also changed. Note that the jpc_stop_node_exporter_aix command-log destination is not changed.

1. Add the following to /etc/syslog.conf:

   local1.info /var/log/jpc_node_exporter_aix/jpc_node_exporter_aix.log rotate size 10m files 8

2. Create a directory and file for the log output destination.

3. Restart syslogd.

4. Stop Node exporter for AIX.

5. Change the registration of a service.

   Run the following command.

   • For physical host operation

     chssys -p /bin/sh -s jpc_node_exporter_aix -u root -S -f 9 -n 15 -a "-c \"'Node-exporter-for-AIX-installation-directory/jp1ima/bin/node_exporter_aix' -CcADmdiabf -p 20730 2>&1 | logger -p local1.info\""

   • For logical host operation

     chssys -p /bin/sh -s jpc_node_exporter_aix_logical host name -u root -S -f 9 -n 15 -a "-c \"'Node-exporter-for-AIX-installation-directory/jp1ima/bin/node_exporter_aix' -CcADmdiabf -p 20730 2>&1 | logger -p local1.info\""

6. Start Node exporter for AIX.