12.5.3 Log metrics

(a) prometheus input plug-in

This plug-in is used to configure the settings for sending the metrics collected by the prometheus output plug-in to Prometheus. More specifically, it specifies an HTTP endpoint at which the Prometheus server scrapes collected metrics. Specify 20273 for the log metrics feature, and you can access the collected metrics at http://localhost:20273/metrics.

You configure the prometheus input plug-in in the log metrics definition file (fluentd_any-name_logmetrics.conf).

Setting example:

<source>
  @type prometheus
  bind bind-number
  port listening-port-number
  metrics_path metrics-path
</source>

For details on the above parameters, see Log metrics definition file (fluentd_any-name_logmetrics.conf) of 10. IM Exporter definition files in the manual JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

(b) prometheus output plug-in

The prometheus output plug-in uses the <match> section of Fluentd to generate and measure log metrics from fluentd records. It only reads fluentd records and therefore does not affect the records themselves.

You configure the prometheus output plug-in in the log metrics definition file (fluentd_any-name_logmetrics.conf).

Setting example:

<match>
  @type prometheus
  <metric>
    name user-specified-log-metric-name
    type log-metric-type
    desc log-metric-description
    key key-for-determining-which-logs-are-converted-to-log-metrics
    <labels>
      label-key-1 label-value-1
      label-key-2 label-value-2
    </labels>
  </metric>
</match>

For details on the above parameters, see Log metrics definition file (fluentd_any-name_logmetrics.conf) of 10. IM Exporter definition files in the manual JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

- Metric type

The log metrics feature can use the following types of Prometheus metric types.

• counter

  This indicates a metric with a monotonically increasing value. This is suitable to show numbers, such as the number of processed requests, of completed tasks, or of errors.

  The accumulated value can only be reset when you restart Fluentd.

  If key is blank in the definition, the metric value is incremented by one regardless of what is contained in the record.

  If your metric value may decrease, then use gauge.

  Setting example:

    <metric>
      name logmetrics1#1
      type counter#1
      desc The description of the logmetrics1.#1
      key hoge#2
      <labels>#2
        label-key-1 label-value-1
        label-key-2 label-value-2
      </labels>
    </metric>

  #1: Required.

  #2: Optional. For details on the labels section, see - Label below.

• gauge

  This indicates a metric with a value that can change in any increments or decrements. This is suitable to show numbers, such as temperatures, current memory usage, or the number of concurrent requests.

  Setting example:

    <metric>
      name logmetrics2#1
      type gauge#1
      desc The description of the logmetrics2.#1
      key hoge#1
      <labels>#2
        label-key-1 label-value-1
        label-key-2 label-value-2
      </labels>
    </metric>

  #1: Required.

  #2: Optional. For details on the labels section, see - Label below.

- Label

With the label section specified as <labels>, you can specify a static or dynamic value from a fluentd record as a Prometheus label.

Setting example:

<labels>
  label-key-1 label-value-1
  label-key-2 label-value-2
</labels>

Every label section uses the same format. Each single row in the label section enclosed by <labels> should have a key-value pair for one label. A string specified as a label value is used for the label as it is; however, you can specify the value by using a record accessor or placeholder as follows:

• Record accessor

  With a record accessor of Prometheus, you can specify the value of a label based on a nested fluentd field in a fluentd record. If you want to specify a label value with the record accessor, add a leading $. or $[ symbol.

  For example, if you want to access critical in the fluentd record below,

  use $.event.level or $['event']['level'].

  If you try to access a non-existent record with the record accessor, the label value will have an empty value.

  Example fluentd record:

  {
    "log": "some message",
    "event": {
       "level": "critical", 
       "detail": {
           "category": "application error"
        }
    }
  }

• Placeholder

  With a placeholder, you can specify the value of a label. Note that if you want to use a fluentd record, use the record accessor because the use of a fluentd record in a placeholder is deprecated.

  The following table lists the predefined placeholders.

  Placeholder	Value
  ${hostname}	Host name of the host where Fluentd is running
  ${worker_id}	fluentd worker ID
  ${tag}	Fluentd tag name
  ${tag_parts[N]}	N+1th element of the Fluentd tag#1
  ${tag_prefix[N]}	0 to N+1th element of the Fluentd tag#1
  ${tag_suffix[N]}	(tag-size#2 - N)th element of the Fluentd tag#1

  #1: The placeholder is available only in the prometheus output plug-in. If a value more than or equal to the tag-size^#2 is specified as N, the specified value is used as it is for the label value.

  #2: The tag-size refers to the number of segments separated by dots (.) in a tag (that is, the number of dots (.) + 1). For example, if your tag is 1.2.3, then its tag size is 3.

- A higher-level label section and the label section under the metric section

In the prometheus output plug-in, you can define multiple metric sections tagged as <metric>. A label for all subordinate metrics can be defined in a higher-level label section such as <match>, whereas a label for each metric can be defined in the label section of the metric section. When a label is defined at both levels, they are merged.

For example, in the following setting example, the LogMetrics1 metric has the tag, hostname, key, and data_type labels.

Setting example:

<match>
  @type Prometheus
  <metric>
    name LogMetrics1
    type counter
    desc The total number of foo in message.
    key foo
    <labels>#1
      key foo
      data_type ${type}
    </labels>
  </metric>
  <metric>
    name LogMetrics2
    type counter
    desc The total number of bar in message.
    key bar
    <labels>#2
      key bar
    </labels>
  </metric>
  <labels>#3
    tag ${tag}
    hostname ${hostname}
  </labels>
</match>

#1: The label section for metric LogMetrics1

#2: The label section for metric LogMetrics2

#3: A higher-level label section