9.5.4 Log monitoring function

(a) Log file path settings

Tail plug-in reads all files specified by an absolute path, regardless of the file content. To read log messages from a text-format log file, you must specify the absolute path of the log file in the definition file. For Windows, you cannot specify directories and files on network drives.

You can specify wildcards in the path string. The wildcard that can be used is "* (any 0 or more character string)". You can also specify multiple paths. To specify multiple paths, separate them with ",".

If the path of a file specified using a wildcard contains an unintended file, such as a binary format, the string in the file is read. This causes unintended Fluentd logging or JP1 of unintended messaging to occur. To avoid this problem, wildcards should be used in conjunction with strings to ensure that they correctly match the names of the monitored log files.

The following shows an example configuration.

• When monitoring all log files stored in /path/to/a directory and /path/to/a2 directory

  path /path/to/a/*, /path/to/a2/*

• When monitoring all log files in the third tier from the point of view of path, such as those under /path/to1/a directory and those under /path/to2/a2 directory (in this case, the log files stored under /path/to1 directory and /path/to2 directory are not monitored)

  path /path/*/*

• To monitor the wrapped log files (funcAlog1.log, funcAlog2.log, funcAlog3.log,...) stored in /path/to/b directory.

  path /path/to/b/funcAlog*.log

• When two or more log files are output to the same directory (/path/to/c)

  • fluentd_upB_tail.conf

  path /path/to/c/funcBlog*

  • fluentd_upC_tail.conf

  path /path/to/c/funcClog*

(b) Log files that can be monitored

You can monitor files that continue to be added to one log file, or files that are written by creating a new log file with a different file name when the log file reaches a certain capacity.

The following log file formats can be monitored:

• Sequential files (SEQ)

  A file that continues to be written to one log file, or a file that creates and writes a new log file with a different file name when the log file reaches a certain size.

• Sequential file (SEQ2)

  • For Windows

    After the file name is changed in the same volume, a file with the same name as the previous file name is created and a new log is written.

  • For Linux

    This file changes the file name or deletes the file, creates a file with the same name as the file name before the change or deletion, and writes a new log.

• Sequential file (SEQ3)

  • Windows only

    This file is used to delete a file, create a file with the same name as the file name before deletion, and write a new log.

• Wrap Around File (WRAP2)

  When a log file is wrapped around after reaching a certain size, it is a file in which data is deleted and data is written from the beginning again.

• UPD Types of Log Files (UPD)

  This function is used to monitor a log file in which an undefined character string, such as a date, is set in the file name. An undefined string part is specified with a wildcard.

(c) Log files that cannot be monitored

When the log file reaches a certain size, it cannot be monitored for a format that wraps around and overwrites the data again from the beginning.

(d) Character code of the log file that can be monitored

Character codes of log files that can be monitored are shown below.

• UTF-8 (default) (Japanese / English)

• UTF-16LE (Japanese / English)

• UTF-16BE (Japanese / English)

• Shift_JIS (Japanese)

• Windows-31J (Japanese-language)

• C (English)

• GB18030 (Chinese)

When monitoring UTF-8 and C log files, from_encoding and encoding are not specified. If you want to monitor logfiles other than UTF-8,C, you must specify a from_encoding and encoding. For encoding, specify UTF-8, and for from_encoding, specify the character encoding of the log file to be monitored. When an incorrect character code is specified, the intended character string cannot be extracted, or the content of garbled characters cannot be converted with the intended character code and is notified as a JP1 event..

(e) Recording the latest file read position

Tail records the last location read from the monitored file in pos_file. You must create pos_file for each monitor definition. Pos_file is created the first time tail is started. With JP1/IM - Agent defaults, the first time you read a log, tail starts reading from the log that was added after the log file was started, and creates pos_file to record the most recent read location. During startup, it reads the updated log and records the latest read position in pos_file.

On the second and subsequent invocations, tail reads the updated portion from the most recent read location recorded in pos_file.

If the contents of pos_file are corrupted while Fluentd is stopped, or if pos_file is erased, it is read from the end of the log file and updated (created) pos_file to record the latest read position in the same way as when the log file is started for the next time. In this case, the log added to the log file to be monitored during the stop is not monitored.

(f) Reading existing logs at initial startup

With JP1/IM - Agent defaults, tail starts reading from the end of the log file for the first log read. To read the logs that were already added at first startup, you must change the parameter read_from_head in the monitor definition file to true (defaults to false).

(g) Log parsing function (parse plug-in)

Purse plug-ins analyze the loaded logs and log them in the specified format.

The following log formats (type) can be specified with parse plug-in.

• none

  By default, newly added log one line is all read.

• regexp

  By specifying regexp for type, it is possible to parse and cut the log according to the format specified in the regular expression. Lines of the log that did not match the pattern specified in the regular expression are not read and a warning message is printed in Fluentd log. Refer to Text-formatted log file monitoring definition file (fluentd_@@trapname@@_tail.conf.template)" (2. Definition file) in JP1/Integrated Management 3-Manager Command, Definition File and API Reference manual for information about the warning messages that are output.

  When you cut out a log by the name of "time", you can use timezone to specify the time zone when the log was output. For more information, see Text-formatted log file monitoring definition file (fluentd_@@trapname@@_tail.conf.template)" (2. Definition file) in JP1/Integrated Management 3-Manager Command, Definition File and API Reference manual.

• multiline

  Multiline parse plug-in reads a multi-line log and parses the log. In [Input Settings] in text-formatted log file monitoring definition file, if the parameters "format_firstline" and "formatN" are set in regular expressions, you can use the regular expression named capture function to structure multi-line logging. For more information about the parameters to set, see Text-formatted log file monitoring definition file (fluentd_@@trapname@@_tail.conf.template)" (2. dDefinition file) in JP1/Integrated Management 3-Manager Command, Definition File and API Reference manual.

  The following shows a sample configuration for reading a stack trace log for a Java.

  • Definition file

  <parse>
    @type multiline
    format_firstline /\d{4}-\d{1,2}-\d{1,2}/
    format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
  </parse>

  • Log to be added to the monitored log file

  2013-3-03 14:27:33 [main] INFO  Main - Start
  2013-3-03 14:27:33 [main] ERROR Main - Exception
  javax.management.RuntimeErrorException: null
      at Main.main(Main.java:16) ~[bin/:na]
  2013-3-03 14:27:33 [main] INFO  Main - End

  • Result of parsing the log

  time:
  2013-03-03 14:27:33 +0900
  record:
  {
    "thread" :"main",
    "level"  :"INFO",
    "message":"  Main - Start"
  }
   
  time:
  2013-03-03 14:27:33 +0900
  record:
  {
    "thread" :"main",
    "level"  :"ERROR",
    "message":" Main - Exception\njavax.management.RuntimeErrorException: null\n    at Main.main(Main.java:16) ~[bin/:na]"
  }
   
  time:
  2013-03-03 14:27:33 +0900
  record:
  {
    "thread" :"main",
    "level"  :"INFO",
    "message":"  Main - End"
  }

  The following is a sample configuration for reading Rails logging.

  • Definition file

  <parse>
    @type multiline
    format_firstline /^Started/
    format1 /Started (?<method>[^ ]+) "(?<path>[^"]+)" for (?<host>[^ ]+) at (?<time>[^ ]+ [^ ]+ [^ ]+)\n/
    format2 /Processing by (?<controller>[^\u0023]+)\u0023(?<controller_method>[^ ]+) as (?<format>[^ ]+?)\n/
    format3 /(  Parameters: (?<parameters>[^ ]+)\n)?/
    format4 /  Rendered (?<template>[^ ]+) within (?<layout>.+) \([\d\.]+ms\)\n/
    format5 /Completed (?<code>[^ ]+) [^ ]+ in (?<runtime>[\d\.]+)ms \(Views: (?<view_runtime>[\d\.]+)ms \| ActiveRecord: (?<ar_runtime>[\d\.]+)ms\)/
  </parse>

  • Log to be added to the monitored log file

  Started GET "/users/123/" for 127.0.0.1 at 2013-06-14 12:00:11 +0900
  Processing by UsersController#show as HTML
    Parameters: {"user_id"=>"123"}
    Rendered users/show.html.erb within layouts/application (0.3ms)
  Completed 200 OK in 4ms (Views: 3.2ms | ActiveRecord: 0.0ms

  • Result of parsing the log

  time:
  1371178811 (2013-06-14 12:00:11 +0900)
   
  record:
  {
    "method"           :"GET",
    "path"             :"/users/123/",
    "host"             :"127.0.0.1",
    "controller"       :"UsersController",
    "controller_method":"show",
    "format"           :"HTML",
    "parameters"       :"{ \"user_id\":\"123\"}",
    ...
  }

• syslog

  Syslog parse plug-in analyzes the logging output by syslog. For more information about the parameters to set, see Text-formatted log file monitoring definition file (fluentd_@@trapname@@_tail.conf.template)" (2. Definition file) in JP1/Integrated Management 3-Manager Command, Definition File and API Reference manual.

  The following shows a sample setting for reading RFC-3164 formatted logs.

  • Definition file

  <parse>
    @type syslog
    time_format %b %d %H:%M:%S
    message_format rfc3164
    with_priority false
    parser_type string
    support_colonless_ident true
  </parse>

  • Log to be added to the monitored log file

  <6>Feb 28 12:00:00 192.168.0.1 fluentd[11111]: [error] Syslog test

  • Result of parsing the log

  time:
  1362020400 (Feb 28 12:00:00)
   
  record:
  {
    "pri"    : 6,
    "host"   : "192.168.0.1",
    "ident"  : "fluentd",
    "pid"    : "11111",
    "message": "[error] Syslog test"
  }

  The following shows a sample setting for reading RFC-5424 formatted logs.

  • Definition file

  <parse>
    @type syslog
    time_format %b %d %H:%M:%S
    rfc5424_time_format %Y-%m-%dT%H:%M:%S.%L%z
    message_format rfc5424
    with_priority false
    parser_type string
    support_colonless_ident true
  </parse>

  • Log to be added to the monitored log file

  <16>1 2013-02-28T12:00:00.003Z 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut="3" eventSource="Application" eventID="11211"] Hi, from Fluentd!

  • Result of parsing the log

  time:
  1362052800 (2013-02-28T12:00:00.003Z)
   
  record:
  {
    "pri"      : 16,
    "host"     : "192.168.0.1",
    "ident"    : "fluentd",
    "pid"      : "11111",
    "msgid"    : "ID24224",
    "extradata": "[exampleSDID@20224 iut=\"3\" eventSource=\"Application\" eventID=\"11211\"]",
    "message"  : "Hi, from Fluentd!"
  }

• csv

  Csv purse plug-in parses logging in csv format. For more information about the parameters to set, see Text-formatted log file monitoring definition file (fluentd_@@trapname@@_tail.conf.template)" (2. Definition file) in JP1/Integrated Management 3-Manager Command, Definition File and API Reference manual.

  The following is a sample configuration for reading csv format logging.

  • Definition file

  <parse>
    @type csv
    keys time,host,req_id,user
    time_key time
  </parse>

  • Log to be added to the monitored log file

  2013/02/28 12:00:00,192.168.0.1,111,-

  • Result of parsing the log

  time:
  1362020400 (2013/02/28/ 12:00:00)
   
  record:
  {
    "host"   : "192.168.0.1",
    "req_id" : "111",
    "user"   : "-"
  }

• tsv

  Tsv purse plug-in parses logging in tsv format. For more information about the parameters to set, see Text-formatted log file monitoring definition file (fluentd_@@trapname@@_tail.conf.template)"(2. Definition file) in JP1/Integrated Management 3-Manager Command, Definition File and API Reference manual.

  The following is a sample configuration for reading tsv format logging.

  • Definition file

  <parse>
    @type tsv
    keys time,host,req_id,user
    time_key time
  </parse>

  • Log to be added to the monitored log file

  2013/02/28 12:00:00\t192.168.0.1\t111\t-

  • Result of parsing the log

  time:
  1362020400 (2013/02/28/ 12:00:00)
   
  record:
  {
    "host"   : "192.168.0.1",
    "req_id" : "111",
    "user"   : "-"
  }

• ltsv

  Ltsv purse plug-in parses logging in ltsv format. For more information about the parameters to set, see Text-formatted log file monitoring definition file (fluentd_@@trapname@@_tail.conf.template)" (2. Definition file) in JP1/Integrated Management 3-Manager Command, Definition File and API Reference manual.

  The following is a sample configuration for reading ltsv format logging.

  • Definition file

  <parse>
    @type ltsv
    keys time,host,req_id,user
    time_key time
  </parse>

  • Log to be added to the monitored log file

  time:2013/02/28 12:00:00\thost:192.168.0.1\treq_id:111\tuser:-

  • Result of parsing the log

  time:
  1362020400 (2013/02/28/ 12:00:00)
   
  record:
  {
    "host"   : "192.168.0.1",
    "req_id" : "111",
    "user"   : "-"
  }

  The following is a sample configuration for using the delimiter_pattern.

  • Definition file

  <parse>
    @type ltsv
    delimiter_pattern /\s+/
    label_delimiter =
  </parse>

  • Log to be added to the monitored log file

  timestamp=1362020400 host=192.168.0.1  req_id=111 user=-

  • Result of parsing the log

  record:
  {
    "timestamp": "1362020400",
    "host"   : "192.168.0.1",
    "req_id" : "111",
    "user"   : "-"
  }

(a) Types of logs that can be monitored

The log type is the name of the log displayed in [Event Viewer] of Windows. If you specify a log type in channel parameter of conf file, you can monitor the log for that type. The windows_eventlog2 can read all channels except debug logs and analytics logs.

The following is an example of the types of logs that can be specified:

• Windows event log

  • application

  • system

  • setup

  • security

• Application and Service Logs

  HardwareEvents, etc.

Use the following procedure to check the types of logs that can be monitored.

1. Execute wevtutil command at the command prompt to check the list of logging types registered in the system.

   The following is an example of entering a command:

   >wevtutil el

2. Confirm the valid/invalid setting and type of "Log type" confirmed in step 1 for each "Log type".

   The following is an example of entering a command:

   >wevtutil gl Application
   name: Application
   enabled: true
   type: Admin
    :

   You can specify channel parameters only if all of the following are true:

   • Enabled is "true"

   • Type is "Admin" or "Operational"

(b) Log files that can be monitored

You can monitor Windows event log for supported language environments. For details about the supported language environments, see 9.3.2 Language settings.

(c) Recording the most recent read position of the event log (storage plug-in)

The windows_eventlog2 allows you to use storage plug-in to log the most recent read position of the log to local storage in JSON format.

(d) Notes

If an event is not added to the monitored Windows event log between the first start of Fluentd and the time it is stopped, the Windows event log added during the stop until the next start cannot be monitored. In this case, a warning message will be output at the next start-up. An example of the warning message that is output is as follows:

[warn]: #0 This stored bookmark is incomplete for using. Referring `read_existing_events` parameter to subscribe: <BookmarkList>
</BookmarkList>, channel:  (Name of the channel)

If no event logs are added and Fluentd is stopped again, the event log added during the stop cannot be monitored, and a similar warning message will be printed the next time it is started.

Similarly, if you add a new Windows event log monitoring definition to Fluentd or change the contents of the monitoring definition file to start monitoring channels that do not have the latest read location saved, if the event log was not added to the monitored Windows event log before the stop, The Windows event log added during the stop until the next start is not monitored, and a similar warning message is output.

If a new event log is added during the next boot, the event log will be monitored and subsequent event monitoring will work without any problems.

(a) Extraction condition specification

Specify the extraction condition in a regular expression. Within the <regexp> directive, you can set the criteria for a regular expression by specifying two parameters:

• key

  Name that identifies the pattern of the regular expression to be extracted

• pattern

  Regular expression to extract

You can create multiple extraction conditions and extract data by logical OR and logical AND as shown below.

• Specifying logical AND (AND) criteria

  If you want to extract logs that satisfy all of the regular expression patterns, place the <regexp> directive inside the <and> directive. Key of the <regexp> directive inside the <and> directive must differ. If key is duplicated, an error is issued at Fluentd startup and the system does not start.

• Specifying logical OR (OR) criteria

  If you want to extract logs that meet one of several regular expression patterns, place the <regexp> directive inside the <or> directive. Key of the <regexp> directive inside the <or> directive must differ. If key is duplicated, an error is issued at Fluentd startup and the system does not start.

(b) Specifying exclusion conditions

Can be specified in a conditional regular expression to be excluded. You can set the criteria for a regular expression by specifying the following two parameters in the <exclude> directive:

• key

  Name that identifies the pattern of the regular expression to be excluded

• pattern

  Regular expressions to exclude

You can create more than one exclusion conditions and exclude it by using the following logical OR and logical AND.

• Specifying logical AND (AND) criteria

  If you want to exclude logging that satisfies all of the regular expression patterns, place the <exclude> directive inside the <and> directive. Key of the <exclude> directive inside the <and> directive must differ. If key is duplicated, an error is issued at Fluentd startup and the system does not start.

• Specifying logical OR (OR) criteria

  If you want to exclude logging that satisfies one of several regular expression patterns, place the <exclude> directive inside the <or> directive. Key of the <exclude> directive inside the <or> directive must differ. If key is duplicated, an error is issued at Fluentd startup and the system does not start.

(c) Setting example

You can set the log extraction conditions or exclusion conditions using the directives enclosed in <> (DIRECTIVE in the following configuration rules).

"::=" indicates that the left-hand side is defined as the right-hand side. "|" indicates one of the items before or after this symbol.

CR ::= newline
KEY ::= key text
PATTERN ::= pattern /regular expressions/
 
DIRECTIVE ::= REGEXP | EXCLUDE | AND | OR
REGEXP ::= <regexp> KEY CR PATTERN</regexp>
EXCLUDE ::= <exclude> KEY CR PATTERN </exclude>
EXPRESSION ::= REGEXP | EXCLUDE 
EXPRESSIONS ::= EXPRESSION EXPRESSION  |  EXPRESSIONS EXPRESSION 
AND ::= <and> EXPRESSIONS </and>
OR ::= <or> EXPRESSIONS </or> |

The sum of the <regexp> or <exclude> directives that can be specified in a AND, OR for a log monitor definition is 256.

The following shows an example configuration.

• When extracting logs that contain the string "Error" in MESSAGE field.

@typeΔgrep
  <regexp>
    key MESSAGE
    pattern /Error /
  </regexp>

• When excluding logs that contain the string "Info" in MESSAGE field.

@typeΔgrep
  <exclude>
    key MESSAGE
    pattern /Info/
  </exclude>

• When extracting logs where VALUE field is numeric and RECORD_NAME starts with cpu_.

@typeΔgrep
  <and>
    <regexp>
      key VALUE
      pattern /[1-9]\d*/
    </regexp>
  
    <regexp>
      key RECORD_NAME
      pattern /^cpu_/
    </regexp>
  </and>

• When excluding logs where STATUS_CODE field is a three-digit number starting with 5 or the end of URL is.css

@typeΔgrep
  <or>
    <exclude>
      key STATUS_CODE
      pattern /^5\d\d$/
    </exclude>
  
    <exclude>
      key URL
      pattern /\.css$/
    </exclude>
  </or>

(a) Sending a metric

You can POST metric by specifying URL of the trend data write API in endpoint. Metric that you POST can be viewed as trend data from integrated operation viewer. You can also POST metric to create an IM management node.

You must specify the trend data write API.

The initial values are as follows.

endpoint http:// integrated agent host name:port/ima/api/v1/proxy/promscale/write

(b) Sending log data

You can POST log data by specifying URL of JP1 event converter API in endpoint. POST log data can be viewed as a JP1 event from integrated operation viewer.

You must specify JP1 event converter API.

The initial values are as follows.

endpoint http:// integrated agent host name:port/ima/api/v1/proxy/imdd/im/api/v1/events/transform

(c) Buffer function for log data to be sent (buffer)

Fluentd uses the buffering function when using output plug-ins. The buffer is where the acquired log data is stored. By using the buffer function, you can adjust the frequency of data transmission and temporarily store data that failed to be transmitted. The http plug-in accumulates the received log data in a buffer. The accumulated log data is POST to JP1/IM agent management base at regular intervals.

• Buffer file path setting

  The user must specify the absolute path of the directory containing the buffer. Path parameter in the <buffer> directive in the definition file enables you to specify the file that caches the buffer.

• Setting the transfer interval for accumulated log data

  If you set the flush_interval, you can specify how often the accumulated log data is transferred.

• Configuring the behavior of output plug-in when buffer cue is full

  If you set the overflow_action, you can specify from the throw_exception, block, drop_oldest_chunk three actions when the total size of the buffers reaches the size specified in the total_limit_size. For more information about each behavior, see Text-formatted log file monitoring definition file (fluentd_@@trapname@@_tail.conf.template)" (2. Definition file) in JP1/Integrated Management 3-Manager Command, Definition File and API Reference manual.

• Setting the retry interval

  Fluentd accesses JP1/IM agent control base and JP1/IM agent management base to connect to JP1/IM-Manager. If JP1/IM agent control base or JP1/IM agent management base, or JP1/IM - Manager is down, you may not be able to connect to JP1/IM - Manager.

  If you specify a retry_wait, you can specify how often to retry if a connection fails due to a temporary communication failure.

(a) Logging formatting function (format)

You can format the log and output it to a log file.

The output is as follows.

Time delimiter tag delimiter record newline

• Delimiter

  The delimiter is \t (tab).

• Line feed

  Line feeds are LF (except for Windows) or CRLF (for Windows).

• time

  The date and time formats are output in the following format:

  %Y-%m-%d %H:%M:%S %z

  For example, "2022-12-31 12:34:00 +900" is output when 12:34:0 on December 31, 2022 (Japan time).

  When the date and time in the log message are logged with the name "time" using the " (3) Text-format log file monitoring facility (tail plug-in) " (g) Log parsing function (parse plug-in) , the date and time are displayed. If "time" is not logged, the time of the hosted OS is set when Fluentd monitors that log message.

  When the time of the host OS is changed, "time" time is set to the new time. If the time zone is changed while Fluentd is stopped, the new time zone is output.

• tag

  Prints tag specified in the <source> directive.

  For the log that issued JP1 event, the message "Value of the specified tag.jp1event" is displayed. For the log that does not issue JP1 event, the message "tag value.outputlog" is displayed. To specify whether to issue JP1 events or to output log data to a log file without issuing JP1 events, use (12) Event forwarding function (rewrite_tag_filter plug-in) .

• record

  The log data that was analyzed in the [Input Settings] section and edited in the [Attributes Settings] section is output in the following JSON format.

  {
    "eventide" : "Event ID",
    "message" : "Message",
    "xsystem" : true,  
    "attrs" : {"Extended attribute name" : "Value",...}
  }

  The actual output is done without line breaks.

(a) Copy-destination specification function (store)

If you specified a copy plug-in in [Output Settings], specify the output plug-in (destination) to be used in the <store> directive. The <store> directive can be set more than once, allowing more than one plug-in to print the same log data.

The <store> directive is similar to what you specify in the <match> directive when you specify a single output plug-in.

(a) Specifying the log level

In the <system> section, you can specify the type of log by setting log_level. The log type applies to the entire definition file. The types of logs that can be specified are listed in descending order of severity.

• fatal

• error

• warn

• info

• debug

• trace

The default log level is info, which outputs info, warn, error, and fatal logs with a severity greater than or equal to info.

(a) Communication protocols and authentication methods

The following shows the communication protocols used by integrated agent and authentication methods.

#

ICMPv6 cannot be used.

(b) Network Configuration

Integrated agent can be used in a network configuration with only a IPv4 environment or in a network configuration with a mix of IPv4 and IPv6 environments. Only IPv4 communication is supported in a network configuration with a mix of IPv4 and IPv6 environments.

You can use integrated agent in the following configurations without a proxy server:

Integrated agent transmits the following:

#1

For more information, see the description of the message body for the request in the manual JP1/Integrated Management 3-Manager Command, Definition File and API Reference, "5.6.5 JP1 Event-Translation".

*2

For more information, see the description of the message body for the request in section "5.11.3 Write Trend Data" in JP1/Integrated Management 3-Manager Command, Definition File and API Reference manual.