2.19.2 Change Setup in JP1/IM - Agent

(a) Edit the configuration files (for Linux)

See "1.21.2(1)(a) Edit the configuration files (for Windows)".

(b) Changing unit definition file (for Linux only)

The following lists unit definition file storage locations and filenames:

• Storage destination: /usr/lib/systemd/system/

• File name: jpc_ Service name.service

To change unit definition file, follow these steps:

1. Login to integrated agent host.

2. Stop JP1/IM - Agent service.

3. Edit unit definition file.

4. Execute the following command to reflect the unit definition.

   # systemctl daemon-reload

5. Start JP1/IM - Agent service.

(c) Change command-line options (for Linux)

On ExecStart line of unit definition file, change the command-line options.

For editing methods, see " (b) Changing unit definition file (for Linux only)".

(a) Change Integrated manager to connect to (for Linux) (optional)

See " 1.21.2(2)(a) Change Integrated manager to connect to (for Windows) (optional)".

(b) Changing Ports (for Linux) (Optional)

See " 1.21.2(2)(b) Change the port (for Windows) (optional)".

(c) Deploy a CA certificate (for Linux) (optional)

See "1.21.2(2)(c) Place CA certificate (for Windows) (optional)".

(d) Modify settings related to Action Execution (on Linux) (optional)

See "1.21.2(2)(d) Modify settings related to Action Execution (for Windows) (optional)".

(e) Setup the proxy authentication's authentication ID and Password (optional)

See "1.21.2(2)(e) Setup the proxy authentication's authentication ID and Password (for Windows) (optional)".

(a) Changing Ports (For Linux) (Optional)

The listen port used by Prometheus server is specified in --web.listen-address option of prometheus command.

For details about how to change prometheus command options, see "2.19.2(1)(c) Change command-line options (for Linux)". For Details of --web.listen-address option, refer to "change command options" in "unit definition file (jpc_program-name_service.xml)"(2. definition File) in "JP1/Integrated Management 3 - Manager Command, Definition File, and API Reference" manual.

The default port is "20713". If Port number is changed, review Setup of the firewall and prohibit accessing from outside. However, if you want to monitor Prometheus server with external shape monitoring by Blackbox exporter in other host, allow it to be accessed. In such cases, consider security measures such as limiting the source IP address as required.

(b) Add the alert definition (for Linux) (optional)

See "1.21.2(3)(b) To Add the alert definition (for Windows) (optional)".

(c) Add a Blackbox exporter scrape job (for Linux) (optional)

See " 1.21.2(3)(c) Add Blackbox exporter scrape job (for Windows) (optional)".

(d) Add a user-defined Exporter scrape job (for Linux) (optional)

See "1.21.2(3)(d) Add user-defined Exporter scrape job (for Windows) (optional)".

(e) Change RemoteWrite destination (for Linux) (optional)

See " 1.21.2(3)(e) Changing Remote Write destination (for Windows) (optional)".

(a) Changing Ports (For Linux) (Optional)

The listen port used by Alertmanager is specified in --web.listen-address option of alertmanager command.

For details about how to change alertmanager command options, see "2.19.2(1)(c) Change command-line options (for Linux)". For Details of --web.listen-address option, refer to "change command options" in "unit definition file (jpc_program-name_service.xml)"(2. definition File) in "JP1/Integrated Management 3 - Manager Command, Definition File and API Reference" manual.

The default port is "20714". If Port number is changed, review Setup of the firewall and prohibit accessing from outside. However, if you want to monitor Alertmanager with external shape monitoring by Blackbox exporter in other host, allow it to be accessed. In such cases, consider security measures such as limiting the source IP address as required.

(b) Changing the alert notification destination (for Linux) (optional)

See " 1.21.2(4)(b) Changing the alert notification destination (for Windows) (optional)".

(c) Setup silence (on Linux) (optional)

See " 1.21.2(4)(c) Setup silence (for Windows) (optional)".

(a) Changing Ports (For Linux) (Optional)

The listen port used by Node exporter is specified in --web.listen-address option of node_exporter command.

For details on how to change the options of the node_exporter command, see "2.19.2(1)(c) Change command-line options (for Linux)". For information about --web.listen-address option, see the manual "JP1/Integrated Management 3 - Manager Command, Definition File and API Reference", section "unit definition file (jpc_program-name.service)" (2. definition File), "if you want to change command-line options".

The default port is "20716". If Port number is changed, review Setup of the firewall and prohibit accessing from outside.

(b) Change metric to collect (on Linux) (optional)

1. In the metric_relabel_configs of Prometheus configuration file (jpc_prometheus_server.yml), metric to be collected are defined separated by "|". Delete metric that you do not need to collect and add metric that you want to collect.

   For instructions on updating configuration file, see "1.21.2(1)(a) Edit the configuration files (for Windows)".

   <Sample Setup>

     - job_name: 'jpc_node'
       : 
       metric_relabel_configs:
         - source_labels: ['__name__']
           regex: 'node_boot_time_seconds| Node_context_switches_total| Node_cpu_seconds_total| Node_disk_io_now| Node_disk_io_time_seconds_total| Node_disk_read_bytes_total| Node_disk_reads_completed_total| ....... |node_time_seconds| Node_uname_info| Node_vmstat_pswpin| Node_vmstat_pswpout [Add metric here]'
           action: 'keep'

2. If required, define a trend view in metric definetion File.

   In Node exporter metric definition file, you define a trend view.

For descriptions, see "JP1/Integrated Management 3 - Manager Command, Definition File and API Reference", Node exporter metric definition file (metrics_node_exporter.conf)" (2. Definition File), in the manual.

(a) Changing Ports (For Linux) (Optional)

The listen port used by Blackbox exporter is specified in --web.listen-address option of the blackbox_exporter command.

For details on how to change the options of blackbox_exporter command, see "2.19.2(1)(c) Change command-line options (for Linux)". For information about --web.listen-address option, see the manual "JP1/Integrated Management 3 - Manager Command, Definition File and API Reference", section "unit definition file (jpc_program-name.service)" (2. definition File), "change command-line options".

The default port is "20715". If Port number is changed, review Setup of the firewall and prohibit accessing from outside.

(b) Add, Modify, or Delete a Module. (Optional)

See " 1.21.2(6)(b) Add, change, and delete modules (for Windows) (optional)".

(c) Add, Modify, or Delete a monitoring target (for Linux) (mandatory)

See " 1.21.2(6)(c) Add, change, or Delete the monitoring target (for Windows) (mandatory)".

(d) Monitor HTTP through proxy (on Linux) (optional)

See " 1.21.2(6)(d) Monitoring HTTP through proxy (for Windows) (optional)".

(e) Setup the proxy authentication ID and Password (optional)

See "1.21.2(6)(e) Setup the proxy authentication ID and Password (for Windows) (Optional)".

(f) Setup authentication ID, Password, and Bearer tokens for accessing the monitored Web Server (optional)

See "1.21.2(6)(f) Setup authentication ID, Password, and Bearer tokens for accessing the monitored Web Server (for Windows) (optional)".

(a) Changing Ports (For Linux) (Optional)

The listen port used by Yet another cloudwatch exporter is specified in -listen-address option of yet-another-cloudwatch-exporter command.

For details on how to change the options of yet-another-cloudwatch-exporter command, see "2.19.2(1)(c) Change command-line options (for Linux)". For Detail of -listen-address option, refer to "yet-another-cloudwatch-exporter command options" in "unit definition file (jpc_program-name.service)" in the manual "JP1/Integrated Management 3-Manager, Command Definition File and API Reference".

The default port is "20718". If Port number is changed, review Setup of the firewall and prohibit accessing from outside.

(b) Modify Setup to connect to CloudWatch (for Linux) (optional)

There are two ways to connect to CloudWatch from Yet another cloudwatch exporter: using an access key (hereinafter referred to as the access key method) and using an IAM role (hereinafter referred to as the IAM role method). If you install Yet another cloudwatch exporter on a host other than AWS/EC2, you can only use the access key method. If you are installing Yet another cloudwatch exporter on AWS/EC2, you can use the access key method or the IAM role method.

The procedure for connecting to CloudWatch is described in the following four patterns.

• Access Key Method (Part 1)

  Connect to CloudWatch as an IAM user in your AWS account

• Access Key Method (Part 2)

  Create multiple IAM users in your AWS account with the same role, and connect to CloudWatch with IAM users in this role

• IAM Role Method (Part 1)

  Connect to CloudWatch with an AWS account for which you have configured an IAM role

• IAM Role Method (Part 2)

  Connect to CloudWatch with multiple AWS accounts with the same IAM role

- When connecting to CloudWatch with access method (part 1)

1. Create an IAM policy "yace_policy" in your AWS account (1) and set the following JSON format information.

   { "Version": "2012-10-17", "Statement": [ { "Sid": "CloudWatchExporterPolicy", "Effect": "Allow", "Action": [ "tag:GetResources", "cloudwatch:ListTagsForResource", "cloudwatch:GetMetricData", "cloudwatch:ListMetrics" ], "Resource": "*" } ] }

2. Create an IAM group "yace_group" in the AWS account (1) and assign the IAM policy "yace_policy" created in step 1.

3. Create IAM user "yace_user" in AWS account (1) and belong to the IAM group "yace_group" created in step 2.

4. On the host of the monitoring module, create a credentials file in the "/root/.aws/" directory, and set the access key and secret access key of the IAM user "yace_user" created in step 3 in the [default] section of the credentials file.

- When connecting to CloudWatch with access method (part 2)

1. Create IAM policy "yace_policy" in AWS account (2) and set the same JSON format information as in step 1 of "Access method (Part 1)".

2. Create the IAM role "cross_access_role" in AWS account (2), select "Another AWS account" for [Select trusted entity type], and specify the account ID of AWS account (1) as the account ID.

3. Assign the IAM policy "yace_policy" created in step 1 to the IAM role "cross_access_role" created in step 2.

4. Create IAM policy "yace_policy" in AWS account (1) and set the same JSON format information as in step 1 of "Access method (Part 1)".

5. Create IAM policy "account2_yace_policy" in AWS account (1) and set the following JSON format information.

   { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "arn:aws:iam::AWS Account(2):role/cross_access_role" } ] }

   The underlined "cross_access_role" is the name of IAM role created in step 2.

6. Create an IAM group "yace_group" in your AWS account (1), and assign the IAM policy "yace_policy" created in step 1 and the IAM policy "account2_yace_policy" created in step 5.

7. Create IAM user "yace_user" in AWS account (1) and belong to the IAM group "yace_group" created in step 6.

8. On the host of the monitoring module, create a credentials file in the "/root/.aws/" directory, and set the access key and secret access key of the IAM user "yace_user" created in step 7 in the [default] section of the credentials file.

9. Add the following definition^# of AWS account (2) to the Yet another cloudwatch exporter configuration file (ya_cloudwatch_exporter.yml).

   discovery: exportedTagsOnMetrics: AWS/S3: - jp1_pc_nodelabel jobs: - type: AWS/S3 regions: - us-east-2 metrics: - name: BucketSizeBytes statistics: - Sum period: 300000 length: 400000 nilToZero: true - type: AWS/S3 regions: - us-east-2 roles: - roleArn: "arn:aws:iam::AWS Account(2):role/cross_access_role" metrics: - name: BucketSizeBytes statistics: - Sum period: 300000 length: 400000 nilToZero: true

   #

   Lines 1 to15 show the collection settings of AWS account (1), and lines 17 and later show the collection settings of AWS account (2).

   In the collection settings of AWS account (2), "roles.roleArn" must be specified. You can specify up to two AWS accounts for "roles.roleArn", but if you want to specify two or more accounts, please contact Hitachi Sales.

- When connecting to CloudWatch using the IAM role method (Part 1)

1. Create IAM policy "yace_policy" in AWS account (1) and set the same JSON format information as in step 1 of "Access method (Part 1)".

2. Create an IAM role "yace_role" in your AWS account (1), and select AWS service for [Select trusted entity type] and EC2 for [Select use case].

3. Assign the IAM policy "yace_policy" created in step 1 to the IAM role "yace_role" created in step 2.

4. Assign the IAM role "yace_role" created in steps 2~3 to the EC2 instance where the monitoring module of AWS account (1) is installed^#.

   #

   Open the EC screen of the AWS console and execute it in the menu of [Action] - [Security] - [Change IAM Role].

- When connecting to CloudWatch using the IAM role method (part 2)

1. Create IAM policy "yace_policy" in AWS account (2) and set the same JSON format information as in step 1 of "Access method (Part 1)".

2. Create the IAM role "cross_access_role" in AWS account (2), select "Another AWS account" for [Select trusted entity type], and specify the account ID of AWS account (1) as the account ID. Also, specify an external ID if necessary.

3. Create IAM policy "account2_yace_policy" in AWS account (1) and set the following JSON format information.

   { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "arn:aws:iam::AWS Account(2):role/cross_access_role" } ] }

   The underlined "cross_access_role" is the name of IAM role created in step 2.

4. Create an IAM role "yace_role" in your AWS account (1), and select AWS service for [Select trusted entity type] and EC2 for [Select use case].

5. Assign the IAM policy "account2_yace_policy" created in step 3 to the IAM role "yace_role" created in step 4.

   Assign the IAM role "yace_role" created in step 4 to the EC2 instance where the monitoring module of the AWS account (1) is installed.^#

   #

   Open the EC screen of the AWS console and execute it in the menu of [Action] - [Security] - [Change IAM Role].

6. Add the following definition^# of AWS account (2) to the Yet another cloudwatch exporter configuration file (ya_cloudwatch_exporter.yml).

   discovery: exportedTagsOnMetrics: AWS/S3: - jp1_pc_nodelabel jobs: - type: AWS/S3 regions: - us-east-2 roles: - roleArn: "arn:aws:iam:: AWS Account(2):role/cross_access_role" externalId: " External ID" metrics: - name: BucketSizeBytes statistics: - Sum period: 300000 length: 400000 nilToZero: true

   #

   Lines 9~11 show the collection settings for AWS account (2).

   In the collection settings of AWS account (2), "roles.roleArn" must be specified. You can specify up to two AWS accounts for "roles.roleArn", but if you want to specify two or more accounts, please contact Hitachi Sales.

   Specify "externalId" in the collection settings of your AWS account (2) only if you specified an external ID in step 2.

(c) Connect to CloudWatch through a proxy (for Linux) (optional)

If you need to connect to CloudWatch through a proxy, use the environment variable HTTPS_PROXY (the environment variable HTTP_PROXY is not available).

The format of Value specified in the environment-variable HTTPS_PROXY is shown below.

http:// Proxy User name : Password @Proxy server Host name : Port number

Important

Note that Value begins with "http://" in HTTPS_PROXY of the environment variable-name.

(d) Add AWS Services to be Monitored (Optional)

The following six AWS services are monitored by default: If you want to monitor other AWS services, follow the steps here.

• AWS/EC2

• AWS/Lambda

• AWS/S3

• AWS/DynamoDB

• AWS/States

• AWS/SQS

1. Add AWS service definition in Yet another cloudwatch exporter configuration file.

   For details about editing, see "1.21.2(1)(a) Edit the configuration files (for Windows)".

   Add AWS service definition to the underlined sections below.

   • discovery.exportedTagsOnMetrics

   - Description

   discovery:
     exportedTagsOnMetrics:
       AWS Service name:
         - jp1_pc_nodelabel

   - Sample Setup

   discovery:
     exportedTagsOnMetrics:
       AWS/EC2:
         - jp1_pc_nodelabel

   • discovery.jobs

   - Description

   discovery:
     : 
     jobs:
     - type: AWS Service name
       regions:
         - AWS region
       period: 0
       length: 600
       delay: 120
       metrics:

   - Sample Setup

   discovery:
     : 
     jobs:
     - type: AWS/EC2
       regions:
         - ap-northeast-1
       period: 0
       length: 600
       delay: 120
       metrics:

2. Add metric you want to collect.

   See "2.19.2(7)(f) Modify metric to Collect (Optional)".

(e) Monitoring AWS Resources (Optional)

For AWS resource to be monitored by Yet another cloudwatch exporter, the jp1_pc_nodelabel tag must be Setup to AWS resource that you want to monitor. See AWS documentation for how to set the tags for AWS resource.

For jp1_pc_nodelabel tag, Setup the following Value: Specify an alphanumeric character or hyphen within the range of 1 to 255 characters.

• For EC2

  Specify Host name.

• Other than EC2

  Specifies the text that is labeled in IM management node.

  Important

  • Setup a string that is unique within AWS services. You can Setup the same string for different services - for example, EC2 and Lambda.

  • Accounts with different YACE monitoring destinations must be different string. Even in different regions, for the same service, use different strings.

  • If a string is duplicated, only one IM management node is created.

The value set in jp1_pc_nodelabel tags is added as the value of jp1_pc_nodelabel label of samples collected by Yet another cloudwatch exporter.

(f) Modify metric to Collect (Optional)

1. Verify metric collected on CloudWatch.

   Verify that metric that you want to collect is collected on CloudWatch.

   In addition, you must have verified Setup for CloudWatch metric name and CloudWatch statistic types in preparation for Setup in the following steps.

   For information about CloudWatch metric name and CloudWatch statistical types, see "Amazon CloudWatch User Guide" in AWS documentation.

2. Add definition of CloudWatch metric to Yet another cloudwatch exporter configuration file.

   The underlined sections of discovery.jobs.metrics below describe CloudWatch metric definitions.

   Description	Sample Value
   discovery: : jobs: - type: AWS Service name regions: - AWS region period: 0 length: 600 delay: 120 metrics: - name: CloudWatch metric name 1 statistics: - CloudWatch statistic types - name: CloudWatch metric name 2 statistics: - CloudWatch statistic types - name: CloudWatch metric name 3 statistics: - CloudWatch statistic types :	CPUUtilization Average DiskReadBytes Sum DiskWriteBytes Sum

3. Add metric to Prometheus configuration file.

   Value of metric_relabel_configs lists metric to collect, separated by |. Add metric that you want to collect. Also, Delete metric that does not need to be collected. For the naming conventions for metric names, see the manual "exporter metric Naming Conventions" in "9.5.3(2)(f) Yet another cloudwatch exporter" in the manual "JP1/Integrated Management 3 - Manager Overview and System Design Guide"

   For information on updating Prometheus configuration file, see "1.21.2(1)(a) Edit the configuration files (for Windows)".

   <Sample Setup>

     - job_name: 'jpc_cloudwatch'
       : 
       metric_relabel_configs:
         - regex: 'tag_(jp1_pc_.*)'
           replacement: ${1}
           action: labelmap
         - regex: 'tag_(jp1_pc_.*)'
           action: 'labeldrop'
         - source_labels: ['__name__','jp1_pc_nodelabel']
           regex: '(aws_ec2_cpuutilization_average|aws_ec2_disk_read_bytes_sum|aws_ec2_disk_write_bytes_sum|aws_lambda_errors_sum|aws_lambda_duration_average|aws_s3_bucket_size_bytes_sum|aws_s3_5xx_errors_sum|aws_dynamodb_consumed_read_capacity_units_sum|aws_dynamodb_consumed_write_capacity_units_sum|aws_states_execution_time_average|aws_states_executions_failed_sum|aws_sqs_approximate_number_of_messages_delayed_su m| aws_sqs_number_of_messages_deleted_sum [Add metrics here as "|" separated by]);. +$'
           action: 'keep'

4. If required, define a trend view in metric definition File.

   For descriptions, see "JP1/Integrated Management 3 - Manager Command, Definition File and API Reference", "Yet another cloudwatch exporter metric definition file (metrics_ya_cloudwatch_exporter.conf" (2. Definition File).

(a) Changing Setup of Common Definition file for Log Monitor (Optional)

See "1.21.2(7)(a) Changing Setup of Common Definition file for Log Monitor (For Windows) (Optional)".

(b) Monitor Text-Format Logging File (Optional)

1.21.2(7)(b) Monitoring the text-format logging File (Windows) (optional)".

(c) Modifying the Monitoring Setup for Text-Format Logging File (Optional)

See " 1.21.2(7)(c) Modifying the Monitoring Setup of the Text-Format Logging File (Windows) (Optional)".

(d) Delete Monitoring File in text-format setting

See "1.21.2(7)(d) Delete Monitoring File in text-format settings (for Windows) (optional)".

(a) External shape monitoring by other-host Blackbox exporter

Prometheus server and Alertmanager services monitors from Blackbox exporter of integrated agent running on other hosts. The following tables show URL to be monitored.

For details about how to Add HTTP monitor of Blackbox exporter, see "1.21.2(6)(c) Add, change, or Delete the monitoring target (for Windows) (mandatory)". For details about how to set the alert definition, see "1.21.2(3)(b) To Add the alert definition (for Windows) (optional)".

For an example of the alert definitions to be monitored by HTTP Monitor of Blackbox exporter, see "1.21.2(9) Setup of integrated agent process alive monitoring (for Windows) (optional)".

Table 2‒12: URL monitored by HTTP monitoring of Blackbox exporter

Service	URL to monitor
Prometheus server	http:// Host name of integrated agent: Port number of Prometheus server /-/healthy
Alertmanager	http:// Host name of integrated agent: Port number of Alertmanager /-/healthy

(b) Alive Monitoring Processes by Process exporter

Imagentproxy service, imagentaction service, and Fluentd service are monitored by the operation status of the process monitor of Process exporter. The processes to be monitored are described in the following table.

For more information on how to set up, see Process exporter configuration file (Definition File for jpc_process_exporter.yml) (10. IM Exporter Definition Files) in the manual JP1/Integrated Management 3-Manager Command-Definition File API Reference.

For details about Setting method of the alert definition, see "1.21.2(3)(b) To Add the alert definition (for Windows) (optional)".

Table 2‒13: Processes monitored by the Process exporter

Service	Processes to monitor	Remarks
imagent	Agent pass /bin/imagent	Set this when you want to detect a case that starts up quickly after imagent described in "9.5.5(2) polling monitoring of JP1/IM agent control base" in JP1/Integrated Management 3-Manager Overview and System Design Guide manuals has stopped abnormally.
imagentproxy	Agent pass /bin/imagentproxy	Not applicable.
imagentaction	Agent pass /bin/imagentaction	Not applicable.
Fluentd	Agent pass /lib/ruby/bin/ruby	The "jpc_fluentd_common.conf" text on the command-line distinguishes it from ruby other than Fluentd.
Rotatelogs (only for Fluentd)	Agent pass /bin/ rotatelogs	The "Agent pass /logs/fluentd" text on the command line distinguishs it from rotatelogs other than Fluentd.

The following is a sample Process exporter configuration file that is monitored by Process exporter.

process_names:
  - name: ";{{.ExeBase}};;"
    exe:
     - /opt/jp1ima/bin/imagent
     - /opt/jp1ima/bin/imagentproxy
     - /opt/jp1ima/bin/imagentaction
 
  - name: ";{{.ExeBase}};;{{.Matches.logpath}}"
    exe:
     - /opt/jp1ima/bin/rotatelogs
    cmdline:
     - /opt/jp1ima/logs/(?P<logpath>fluentd)
 
  - name: ";{{.ExeBase}};;{{.Matches.defpath}}"
    exe:
     - /opt/jp1ima/lib/ruby/bin/ruby
    cmdline:
     - jpc_(?P<defpath>fluentd)_common.conf

Here is a sample alertdefinition that Process exporter monitors:

groups:
  - name: process_exporter
    rules:
    - alert: jp1_pc_procmon_imagent
      expr: 1 >  sum by (program, instance, job, jp1_pc_nodelabel, jp1_pc_exporter) (namedprocess_namegroup_num_procs{program="imagent"})
      for: 3m
      labels:
        jp1_pc_product_name: "/HITACHI/JP1/JPCCS2"
        jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO"
        jp1_pc_severity: "Error"
        jp1_pc_eventid: "1303"
        jp1_pc_metricname: "namedprocess_namegroup_num_procs"
      annotations:
        jp1_pc_firing_description: "The number of processes was less than the threshold Value (1). value={{$value}}"
        jp1_pc_resolved_description: "The number of processes exceeded the threshold Value (1).
 
    - alert: jp1_pc_procmon_imagentproxy
      expr: 1 >  sum by (program, instance, job, jp1_pc_nodelabel, jp1_pc_exporter) (namedprocess_namegroup_num_procs{program="imagentproxy"})
      for: 3m
      labels:
        jp1_pc_product_name: "/HITACHI/JP1/JPCCS2"
        jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO"
        jp1_pc_severity: "Error"
        jp1_pc_eventid: "1303"
        jp1_pc_metricname: "namedprocess_namegroup_num_procs"
      annotations:
        jp1_pc_firing_description: "The number of processes was less than the threshold Value (1). value={{$value}}"
        jp1_pc_resolved_description: "The number of processes exceeded the threshold Value (1).
 
    - alert: jp1_pc_procmon_imagentactoin
      expr: 1 >  sum by (program, instance, job, jp1_pc_nodelabel, jp1_pc_exporter) (namedprocess_namegroup_num_procs{program="imagentaction"})
      for: 3m
      labels:
        jp1_pc_product_name: "/HITACHI/JP1/JPCCS2"
        jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO"
        jp1_pc_severity: "Error"
        jp1_pc_eventid: "1303"
        jp1_pc_metricname: "namedprocess_namegroup_num_procs"
      annotations:
        jp1_pc_firing_description: "The number of processes was less than the threshold Value (1). value={{$value}}"
        jp1_pc_resolved_description: "The number of processes exceeded the threshold Value (1).
 
    - alert: jp1_pc_procmon_fluentd_rotatelogs Log trapper(Fluentd) #1
      expr: 1 >  sum by (program, instance, job, jp1_pc_nodelabel, jp1_pc_exporter) (namedprocess_namegroup_num_procs{program="rotatelogs"})
      for: 3m
      labels:
        jp1_pc_product_name: "/HITACHI/JP1/JPCCS2"
        jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO"
        jp1_pc_severity: "Error"
        jp1_pc_eventid: "1303"
        jp1_pc_metricname: "namedprocess_namegroup_num_procs"
      annotations:
        jp1_pc_firing_description: "The number of processes was less than the threshold Value (1). value={{$value}}"
        jp1_pc_resolved_description: "The number of processes exceeded the threshold Value (1). "
 
    - alert: jp1_pc_procmon_fluentd_ruby Log trapper(Fluentd) #2
      expr: 2 >  sum by (program, instance, job, jp1_pc_nodelabel, jp1_pc_exporter) (namedprocess_namegroup_num_procs{program="ruby"}) #3
      for: 3m
      labels:
        jp1_pc_product_name: "/HITACHI/JP1/JPCCS2"
        jp1_pc_component: "/HITACHI/JP1/JPCCS/CONFINFO"
        jp1_pc_severity: "Error"
        jp1_pc_eventid: "1303"
        jp1_pc_metricname: "namedprocess_namegroup_num_procs"
      annotations:
        jp1_pc_firing_description: "The number of processes was less than the threshold Value (2). value={{$value}}"
        jp1_pc_resolved_description: "The number of processes exceeded the threshold Value (2). "

#1

If only log metrics feature is used, specify "jp1_pc_procmon_fluentd_prome_rotatelogs Log trapper(Fluentd)".

#2

If only log metrics feature is used, specify "jp1_pc_procmon_fluentd_prome_ruby Log trapper(Fluentd)".

#3

The Ruby process starts the number of workers + 1. For the threshold, specify the number of workers + 1. For details on the number of workers, see "Log monitoring common definition file".

(c) Monitoring by Prometheus server up metric

Node exporter service, Process exporter service, Blackbox exporter service, and Yet another cloudwatch exporter service are monitored through Prometheus server alert-monitoring. For Setting method of the alert definition, see "1.21.2(3)(b) To Add the alert definition (for Windows) (optional)".

For an example of an alert definition that monitors up metric, see "1.21.2(9) Setup of integrated agent process alive monitoring (for Windows) (optional)".