Log metrics definition file (fluentd_any-name_logmetrics.conf)

Syntax

## Input
<worker worker-ids-used-for-the-log-metrics-feature>
  <source>
    @type prometheus
    bind bind-number
    port listening-port-number
    metrics_path metrics-path
  </source>
</worker>
 
<worker worker-id>
  <source>
    @type tail
    path absolute-path-of-the-log-file-monitored-by-the-log-metrics-feature
    tag user-specified-tag-value
    pos_file ../data/fluentd/tail/user-specified-tag-value.pos
[
    <parse>
      @type regular-expression-type
      expression regular-expressions-for-log-messages
      time_key time-variable-specified-in-expression
      time_format time_key-format
      types variable-name-specified-in-expression:type
    </parse>
]
  </source>
 
## Output
  <match tag-to-which-this-match-applies>
    @type prometheus
    <metric>
      name user-specified-log-metric-name
      type log-metric-type
      desc log-metric-description
      key key-for-determining-which-logs-are-converted-to-log-metrics
      buckets histogram-bucket
      <labels>
      label-key label-value
      </labels>
    </metric>
  </match>
</worker>

To Page Top

File

fluentd_any-name_logmetrics.conf

An any-name can contain half-width alphanumeric characters, hyphens, and underscores. The number of characters that can be specified is from 1 to 30.

To Page Top

Storage directory

For Windows

When using a physical host

Agent-path\jp1ima\conf\user\

When using a logical host

shared-folder\jp1ima\conf\user\

For Linux

When using a physical host

/opt/jp1ima/conf/user/

When using a logical host

shared-directory/jp1ima/conf/user/

To Page Top

Description

The Fluentd metric definition file used to read logs output by the application being monitored and convert them into log metrics. Create and place this file to use the log metrics feature.

In addition, in order to read this file, add an include setting to the log monitoring target definition file (jpc_fluentd_common_list.conf), and specify the name of this file.

To Page Top

Character encoding

UTF-8 (without BOM)

To Page Top

Linefeed code

In Windows; CR+LF

In Linux: LF

To Page Top

Timing in which definitions are reflected

Definitions are reflected in Fluentd behavior when restarting the Fluentd service.

To Page Top

Content description

- Input plug-in feature definitions

For details on the input plug-in feature, see 3.15.2(1) Input plug-in capability in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.

- Setup used for Prometheus scraping

Configure settings using the first <source> of the ## Input shown in Syntax.

Specify prometheus in @type.

bind-number (optional)

This specifies the binding interface when Prometheus scrapes metrics for this plug-in. If this item does not exist, the value is set to 0.0.0.0 by default.

listening-port-number (required)

Specify the listening port number 24820 used when Prometheus scrapes metrics for this plug-in. To change the listening port number, specify a different value.

metrics-path (optional)

This specifies the HTTP endpoint used when Prometheus scrapes metrics for this plug-in. If this item does not exist, the value is set to /metrics by default.

The number of characters that can be specified is from 1 to 255 (not including the / at the beginning).

worker-ids-used-for-the-log-metrics-feature (required)

Specify the worker ID of the worker used in "Setup for a log file to be monitored by the log metrics feature". To specify a single worker, use <worker worker-id>; to specify a range of multiple workers, use <worker worker-id (minimum value) - worker-id (maximum value)>.

- Setup for a log file to be monitored by the log metrics feature

Configure settings using the second <source> of the ## Input shown in Syntax.

For the first @type, specify tail. For the @type under <parse>, specify the supported range of the log monitoring feature (either regexp, syslog, csv, tsv, ltsv, multiline, or none).

For details on the tail and <parse> setup, see 3.15.3(2) Input plug-in functions (Input Plugins) for the 3.15.3 Log monitoring fuction by JP1/IM - Agent in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.

user-specified-tag-value (required)

Specify character strings separated by a dot (.). (Example: myapp.access)

Character strings can contain lower-case alphabetical characters, numeric characters, and underscores. The number of characters that can be specified is from 1 to 30.

regular-expression-type (required)

This specifies the format of log files targeted by the log metrics feature when read into Fluentd. The following shows the types that can be specified:

• regexp

• syslog

• csv

• tsv

• ltsv

• multiline

• none

regular-expressions-for-log-messages (required or optional)

This setting is only written when the regular-expression-type is set to regexp. Specify the regular expression of the log message you want to read as the target of the log metrics feature here. Specifications of regular expressions follow Ruby regular expressions.

variable-name-specified-in-expression (optional)

This specifies the variable name used to specify the type of numerical value data extracted in regular-expressions-for-log-messages.

type (optional)

This specifies the type of the variable name specified in the variable-name-specified-in-expression. The following shows types that can be specified:

• integer

• float

worker-id (required)

Specify the worker ID to be used to monitor a log file in this <source> section from among worker-ids-used-for-the-log-metrics-feature specified in "Setup used for Prometheus scraping".

- Output plug-in feature definitions

For details on the output plug-in feature, see 3.15.2(2) Output plug-in capability in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.

Set these definitions using ## Output shown in Syntax.

<match tag-to-which-this-match-applies> to </match> (optional)

In the match section, specify a tag to only apply match to Fluentd events assigned with the specified tag. If a tag is not specified, match is applied to all Fluentd events.

Specified values can contain half-width alphanumeric characters, hyphens, and underscores. The number of characters that can be specified is from 1 to 30.

<metric> to </metric>

Configure detailed log metrics settings in the metric section. Multiple metric sections can be defined for a single match section.

user-specified-log-metric-name (required)

This specifies the Prometheus metrics data name.

This must match the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*. The number of characters that can be specified is from 1 to 255.

log-metric-type (required)

This specifies the metric type of the log metric in Prometheus (counter or gauge).

• counter

  This represents metrics that contain values that increase monotonically. This is suited to expressing the number of requests processed, completed tasks, and the number of errors, among other values.

  Cumulative values are only reset when restarting Fluentd.

  If the key in the definition is empty, the metrics value increases in increments of 1 for each record, regardless of record contents.

  For metrics that could have declining values, specify gauge.

• gauge

  This represents metrics that contain values that can increase or decrease. This is suited to expressing temperature values, current memory usage, and the number of simultaneous requests, among other values.

log-metric-description (required)

Enter the description of the log metric specified in the description of user-specific metric definitions.

Specified values can contain half-width alphanumeric characters, hyphens, and underscores. The number of characters that can be specified is from 1 to 255.

key-for-determining-which-logs-are-converted-to-log-metrics (required or optional)

This specifies the key for determining targets for conversion to log metrics. If the log-metric-type is counter, this is optional. If the log-metric-type is gauge, this is required.

The number of characters that can be specified is from 1 to 255.

label-key (optional)

A key can be specified for labels assigned to Prometheus metrics data.

This must match the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*. This cannot begin with __. The number of characters that can be specified is from 1 to 255.

label-value (optional)

A value can be specified for the label key.

This can be specified using any Unicode characters other than control characters. The number of characters that can be specified is from 1 to 255.

When specifying a character string, the character string is set as the label value as is.

If either the label-key or the label-value is missing, the label will not be displayed.

The labels section, expressed using <labels> and </labels>, can be defined separately for both inside and outside the metric section. In addition, multiple labels (label sets including a label-key and a label-value) can be set. For details on labels, see Label and A higher-level label section and the label section under the metric section in 3.15.2(3)(b) prometheus output plug-in in the JP1/Integrated Management 3 - Manager Overview and System Design Guide 3.15.2 Log metrics by JP1/IM - Agent.

To Page Top

Example of definitions

## Input
<source>
  @type prometheus
  bind '0.0.0.0'
  port 24820
  metrics_path /metrics
</source>
 
## Extract target log message 1
<source>
  @type tail
  @id logmetrics_counter
  path /usr/lib/WebAppA/ControllerLog/ControllerLog.log
  tag WebAppA.Controller
  pos_file /posfile/WebAppA/ControllerLog.log.pos
  <parse>
    @type regexp
    expression /^(?<logtime>[^\[]*) \[(?<loglebel>[^\]]*)\] (?<class>[^\[]*) : endpoint "\/register" started. Target record: (?<record_num>\d[^\[]*).$/
    time_key logtime
    time_format %Y-%m-%d %H:%M:%S
    types record_num:integer
  </parse>
</source>
 
## Extract target log message 2
<source>
  @type tail
  @id logmeetrics_gauge
  path /usr/lib/WebAppA/GeneralLog/GeneralLog.log
  tag WebAppA.GeneralLog
  pos_file /posfile/WebAppA/GeneralLog.log.pos
  <parse>
    @type regexp
    expression /^(?<logtime>[^\[]*) \[(?<Status>[^\]]*)\] CPU Usage is (?<cpuvalue>\d[^%]*)%.$/
    time_key logtime
    time_format %Y-%m-%d %H:%M:%S
    types cpuvalue:float
  </parse>
</source>
 
## Output
## Define log metrics 1 and 2
<match WebAppA.ControllerLog>
  @type prometheus
  <metric>
    name logmetrics_request_endpoint_register
    type counter
    desc The request number of endpoint register
  </metric>
  <metric>
    name logmetrics_num_of_registeredrecord
    type counter
    desc The number of registered record
    key record_num
    <labels>
    loggroup ${tag_parts[0]}
    log ${tag_parts[1]}
    </labels>
  </metric>
</match>
 
## Define log metric 3
<match WebAppA.GeneralLog>
  @type prometheus
  <metric>
    name logmetrics_cpu_usage
    type gauge
    desc the CPU Usage of WebAppA.
    key cpuvalue
    <labels>
    loggroup ${tag_parts[0]}
    log ${tag_parts[1]}
    </labels>
  </metric>
</match>

To Page Top