Monitoring text-formatted log file definition file (fluentd_@@trapname@@_tail.conf.template)

Format

<worker 0>
## [Metric Settings]
<source>
  @type exec
  command "echo {}"
  <parse>
    @type json
  </parse>
  run_interval 60s
  tag jpc_ima_metrics.tail.log-monitoring-name
</source>
 
<filter jpc_ima_metrics.tail.log-monitoring-name>
  @type record_transformer
  enable_ruby true
  auto_typecast false
  <record>
    __name__ fluentd_logtrap_running
    instance host-name
    jp1_pc_nodelabel IM-management-node-label-name
    jp1_pc_category category-ID
    jp1_pc_logtrap_defname log-monitoring-name_tail
    jp1_pc_trendname fluentd
    job jpc_fluentd
    jp1_pc_nodelabel_fluentd Log trapper(Fluentd)
    jp1_pc_addon_program JPC Fluentd
  </record>
</filter>
</worker>
<worker worker id>
## [Input Settings]
<source>
  @type tail
  tag tail.log-monitoring-name
  path monitored-paths
  follow_inodes true
  refresh_interval 60
  skip_refresh_on_startup false
  read_from_head read-the-logs-to-be-monitored-when-Fluentd-is-started-for-the-first-time-from-the-beginning
#  encoding "Fluentd-character-code"
#  from_encoding "character-codes-of-monitored-logs"
  read_lines_limit 1000
  read_bytes_limit_per_second -1
  pos_file ../data/fluentd/tail/log-monitoring-name.pos
  path_key tailed_path
  rotate_wait 5s
  enable_watch_timer enable-additional-watch-timers
  flush-interval-for-multiline-logs
  enable_stat_watcher true
  open_on_every_update false
  emit_unmatched_lines false
  ignore_repeated_permission_error false
  <parse>
    @type log-format
    settings-depending-on-the-log-format
  </parse>
</source>
 
## [Attributes Settings]
<filter tail.log-monitoring-name>
  @type record_transformer
  enable_ruby true
  auto_typecast false
  renew_record true
  
  <record>
    ID event-ID
    MESSAGE ${record["message"]}
    JP1_SOURCEHOST host-ame
    JPC_LOG_TIME ${time.utc.to_i}
    PRODUCT_NAME /HITACHI/JP1/JPCCS2/LOGTRAP/IM-management-node-label-name
    PPNAME /HITACHI/JP1/JPCCS2
    SEVERITY severity
    PLATFORM ${ if RUBY_PLATFORM.downcase =~ /mswin(?!ce)|mingw|cygwin|bccwin/; 'NT'; else 'UNIX'; end }
    OBJECT_TYPE LOGFILE
    OBJECT_NAME ${record['tailed_path']}
    ROOT_OBJECT_TYPE LOGFILE
    ROOT_OBJECT_NAME ${record['tailed_path']}
    JP1_TRAP_NAME ${tag_parts[1]}
    JPC_NODELABEL IM-management-node-label-name
    any-attribute-name any-value
  </record>
</filter>
 
## [Inclusion Settings]
#<filter tail.log-monitoring-name>
#  @type grep
#  <regexp>
#    key attribute-name-of-JP1-event
#    pattern /regular-expression-of-logs-to-monitor/
#  </regexp>
#</filter>
 
## [Exclusion Settings]
#<filter tail.log-monitoring-name>
#  @type grep
#  <exclude>
#    key attribute-name-of-JP1-event
#    pattern /regular-expressions-for-logs-that-you-do-not-want-to-monitor/
#  </exclude>
#</filter>
 
## [Forward Settings]
<match tail.log-monitoring-name>
  @type rewrite_tag_filter
  <rule>
    key attribute-name-of-JP1-event
    pattern /regular-expression-for-logs-that-emit-JP1-events/
    tag ${tag}.jp1event
  </rule>
  <rule>
    key SEVERITY
    pattern /.*/
    tag ${tag}.outputlog
  </rule>
</match>
 
<filter /tail\.log-monitoring-name\.(jp1event|outputlog)/>
  @type record_transformer
  enable_ruby true
  auto_typecast true
  renew_record true
  <record>
    eventId ${record['ID']}
    xsystem true
    message ${record['MESSAGE']}
    attrs ${record}
  </record>
  remove_keys $.attrs.ID
  remove_keys $.attrs.MESSAGE
</filter>
</worker>

To Page Top

File

fluentd_@@trapname@@_tail.conf.template

fluentd_@@trapname@@_tail.conf.template.model (model file)

To Page Top

Storage directory

■Integrated agent host

In Windows:

• For a physical host

  Agent-path\conf\

• For a logical host

  shared-folder\jp1ima\conf\

In Linux:

• For a physical host

  /opt/jp1ima/conf/

• For a logical host

  shared-directory/jp1ima/conf/

To Page Top

Description

Definition File for monitoring text-formatted logging File.

Copy the template (fluentd_@@trapname@@_tail.conf.template) and change file name of Copy destination to fluentd_log-monitoring-name_tail.conf for use. File name must be unique within integrated agent host. For details on the location of fluentd_log-monitoring-name_tail.conf, see Appendix A.4(3) Integrated agent host (Windows) and Appendix A.4(4) Integrated agent host (Linux) in the JP1/Integrated Management 3 - Manager Overview and System Design Guide. log-monitoring-name must be between 1 and 30 characters long. Allowed characters are single-byte alphanumeric characters, "-" (hyphen), and "_" (underscore).

Create a monitor-definition file for each wrapped-around log file group that you want to monitor (or for each log file that does not wrap-around). JP1/IM - Agent creates a IM managed node for SID of target of monitoring according to the value specified in IM-management-node-label-name of the IM managed node in the monitoring definition file. If another monitoring definition file has the same IM-management-node-label-name, only one IM management node is created.

The text-based log file monitoring feature reads this definition file and analyzes the log that the application has written to the text-based log file. You can setup if you specify a condition for the analyzed information and the condition is met, the information to be converted to JP1 events or output to Fluentd logging file. For details about JP1 event to be issued, see 3.2.3(2) JP1 event issued that monitoring a textual log File.

Lines beginning with "#" are treated as Comment and do not affect the programming behavior.

The default definition in [Forward Settings] section is set to transform log data to JP1 event and transfer to JP1/IM - Manager when SEVERITY is worse than Warning.

When transforming log data to JP1 event and transferring it to JP1/IM - Manager, set SEVERITY so that its severity is equal or worse than Warning.

To Page Top

Character code

UTF-8 (without BOM)

To Page Top

Line feed code

In Windows: CR+LF

In Linux: LF

To Page Top

When the definitions are applied

This information is reflected in Fluentd operation when Fluentd service is restart.

If add or delete of definition files or value in [Metric Settings] section is changed, the change is reflected in tree view of the Integrated Operation Viewer window.

For details about application method, see 1.21.2(19) Creation and import of IM management node tree data (for Windows) (mandatory) in the JP1/Integrated Management 3 - Manager Configuration Guide.

To Page Top

Information that is specified

<worker> directive

See the description of <worker> directive in Log monitoring common definition file (jpc_fluentd_common.conf).

worker-id (optional)

Description	Changeability	What You Setup in Your JP1/IM - Agent	JP1/IM - Agent Defaults Value
Specifies the number of workers that Fluentd will start. Serves as an argument to the <worker N> directive. Valid values are integers from 1 to 128.	Can be changed	It must be specified so as not to duplicate the worker ID specified in the existing text log file monitoring definition file or the Windows event log monitoring definition file.	1

[Metric Settings] section

Setup Value of label of sample that you want to send to JP1/IM - Manager's Trend data Management Database.

log-monitoring-name (mandatory)

Specifies log-monitoring-name specified in the file name of copy destination as a string of 1 to 30 characters. Allowed characters are single-byte alphanumeric characters, "-" (hyphen), and "_" (underscore). The default value is "@@trapname@@".

Because you need to setup several locations in the file, use OS command/editor function to replace the "@@trapname@@" location with log-monitoring-name you want to specify.

If the specification is omitted, error occurs when Fluentd is started.

Note that log-monitoring-name must be setup as follows:

• All log-monitoring-name in the same file are the same.

• log-monitoring-name is unique for the monitoring text-formatted log file definition file and the Windows event log monitoring definition file

host-name (optional)

Specify the host name to be monitored using characters 1 to 255 other than control characters. The default value is setup by integrated agent installers.

If the specification is omitted, IM management node is not created.

You can also dynamically setup the canonical host name of the system by doing the following:

    instance ${Socket.gethostname}

IM-management-node-label-name (optional)

Specifies the character string that integrated operation viewer displays on IM management node label. This is not a control character. When URL is encoded, the character string must be between 1 and 234 bytes (the upper limit for multibyte characters is 26). The default value is "Application".

If the specified information is invalid or omitted, IM management node is not going to be created.

You can specify the same IM management node label name in different monitoring definition files. Then, only one IM management node is created, and JP1 events in both of monitor-defined files are Add to one IM management node.

category-ID (optional)

Specifies the category ID of IM management node corresponding to SID to be monitored for logging as a character 1 to 255 other than control characters. If the specification is omitted, "otherApplications" is assumed.

[Input Settings] section

Setup the path to the text-formatted log File that you want to monitor and the regular expressions that parse the log Message.

log-monitoring-name (mandatory)

Same as the section [Metric Settings] description.

monitored-paths (required)

Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
Specify the path to read. You can specify multiple paths by separated by commas. You can include * and strftime formats to dynamically add and delete the logging file you want to monitor. The list of log files is updated at refresh_interval intervals. For specification examples, see (3) Text-format log file monitoring facility (tail plug-in) in 3.15.3 Log monitoring function by JP1/IM - Agent in the JP1/Integrated Management 3 - Manager Overview and System Design Guide. If you specify an incorrect path, Log Files logging is not read. The following rules apply to programming: Specify as an absolute path Directories and File on network drives cannot be specified (for Windows) Specify "/" instead of "\" as the path delimiter (in Windows) Multiple paths can be specified. You can specify "*" (wildcard). You specify within 256 bytes. The following path names cannot be specified. - File with a leading "-" (hyphen) - Folder name, directory name, or File name containing environment-dependent characters - Directory name with spaces (for Linux)	Installation Required	Specifies Log Files path.	Not applicable

read-the-logs-to-be-monitored-when-Fluentd-is-started-for-the-first-time-from-the-beginning (optional)

Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
Specifies whether the log should start reading from the beginning, not the end, or from the last read position recorded in pos_file. You can specify true or false.	Can be changed	If you want to read a log that was already Add at startup, change it to true.	false

Fluentd-character-code (optional)

If character-codes-of-monitored-logs is C (ISO-8859-1), specify the default setup (handled as Comment). Specify UTF-8 if character-codes-of-monitored-logs is not C (ISO-8859-1). In the default Setup, since "#" is specified at the beginning of the line and it is handled as Comment, delete the "#".

Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
Specifies the encoding in which to read the line. By JP1/IM - Agent, in_tail outputs value of string in ASCII-8BIT encoding in default. You can change it with the following options: encoding changes the text to encoding. If both encoding and from_encoding are specified, in_tail attempts to convert the jj, string to a encoding.	Can be changed	In JP1/IM - Agent, you can specify the following Value: UTF-8	Not specified (Comment out) # encoding "UTF-8"

character-codes-of-monitored-logs (optional)

If character-codes-of-monitored-logs is C (ISO-8859-1), specify the default setup (handled as comment). If character-codes-of-monitored-logs is not C (ISO-8859-1), specify the character code. In the default setup, since "#" is specified at the beginning of the line and it is handled as comment, delete the "#". When monitoring UTF-8 log files, a warning message is output when the Fluentd service starts, but please ignore it. For the warning message, see 3.15.3(3)(d) Character code of the log file that can be monitored in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.

Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
See the explanation of Fluentd-character-code (optional).	Can be changed	Specifies the character encoding of log files. In JP1/IM - Agent, you can specify the following value: UTF-16LE UTF-16BE Shift_JIS Windows-31J GB18030	Not specified (Comment out) # encoding "Shift_JIS"

enable-additional-watch-timers

Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
Specify true or false. If false is specified for this parameter, the most recent log is not monitored when reading multiple lines of log. Therefore, if multiline is specified as the type of the parse plug-in, true is specified. Specifying false for this parameter significantly reduces CPU and I/O consumption when tailing a large number of files on systems that support inotify.	Can be changed	Specify true only if multiline is specified as the type of the parse plugin.	false

flush-interval-for-multiline-logs

Item Name	Description	Changeability	JP1/IM - What the user sets on the agent	JP1/IM - Initial value of Agent
multiline_flush_interval	Specify multiline_flush_interval item as flush interval for multiline logs. If this item is not specified, the latest log is not monitored when multiline logs are monitored. Therefore, when type of parsing plugin is set multiline, set it as following: multiline_flush_interval 5s	Changeable	Set 5s only when type of parsing plugin is multiline.	5s

log-format

Specifies the format for parsing the imported log.

The following formats can be specified:

type	Description
none (Default)	Read a one-line log as it is without parsing or structuring.
regexp	Reads a single-line log that matches the pattern specified by the regular expression.
multiline	Loads a multi-line log that matches the pattern specified by the regular expression.
syslog	Read the log output by syslog.
csv	Load logs in CSV format (comma delimited).
tsv	Loads logs in TSV format (tab-delimited).
ltsv	Import logs in LTSV format (labeled tab-delimited).

For examples of specifying logs in each format, see 3.15.3(3)(g) Log parsing function (parse plug-in) in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.

settings-depending-on-the-log-format

Specify the entries according to the log-format.

• If none

  <parse>
    @type none
    message_key message
    time_key time
    null_empty_string false
    estimate_current_event true
    keep_time_key false
  </parse>

• If regexp

  <parse>
    @type regexp
    expression regular-expressions-to-parse-logs
    time_key time
    null_empty_string false
    estimate_current_event true
    keep_time_key false
    items-for-parsing-date-and-time-of-logs
  </parse>

regular-expression-to-parse-log (required)

When using the Named Capture fieature to trim character strings, one of names is necessary to be "message". If character strings named "message" is not trimmed, MESSAGE of JP1 event will be empty.

Specifies a regular expression and parses the contents of one line of the log. Use the Named Capture feature to trim a string named "message" that Setup to Message of JP1 event. For example: The default Value contains a regular expression that trims the entire line in "message". You can also trim with another name and Setup to any property of JP1 event.

items-for-parsing-date-and-time-of-logs

When a Date/time in the logging Message is trimmed as the name "time", it is set as value of JPC_LOG_TIME of JP1 event. When you trim a Date/time in the logging Message as the name "time", it is necessery to define the items for parsing date and time of logs. When you do not trim a Date/time, or define the items for parsing date and time of logs are not defined, the value of JPC_LOG_TIME will be Date/time when Fluentd monitored the log message.

Item Name	Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
expression	specifies the regular expressions matches for logging. Regular expressions must be sandwiched between "/" (delimiters). If the delimiter is not used, an error is output. Regular expressions must specify at least one named capture (?<name>Regular expression for truncated logs). Regular expressions can have i and m suffixes. i(ignorecase) Ignores the case of the match. m (multi-line) Creates a regular expression as a multi-line mode. ". " matches to a line break. both Specify both i and m. If the log read does not Match the regular expression, the following Warning Message is printed in Fluentd log and the log is not going to be monitored. 2022-01-23 12:34:56 +0900 [warn]: #0 pattern not matched: "Error Message"	Can be changed	Setup according to Log Files logging format.	expression /^(?<message>.*)$/
time_type	Specify type of the date and time of log to be parsed.	Changeable	Specify type of time according to the format of the log file to be monitored. Available time zone format: unixtime Seconds from Epoch (e.g. 1510544815) string Use format specified by time_format	--
time_format	Specify the time format within 256 bytes. Used to trim logs with the name "time". Processes values according to the specified format. It is available if the time_type is string. The following formats are supported: %b Abbreviated month (Jan,Feb,...) %d Day (01~31) %H 24-hour clock (00~23) %M min (00~59) %m Month number (01~12) %S sec (00~60 (60 indicatesleap second)) %Y A number representing the year %N fractional seconds If you specify an incorrect value, a warning message similar to the one shown below may be output to the Fluentd log, and the log may not be monitored. 2022-09-08 17:15:10 +0900 [warn]: #0 invalid line found file="C:/fluentd/install/log/app1/20220906_log1_utf8.txt" line="2022/12/3 12:34:56 jpcagt0 00004864 00008904 agent.cpp 572 KAVL99999-E \xE3\x82\xA8\xE3\x83\xA9\xE3\x83\xBC\xE3\x83\xA1\xE3\x83\x83\xE3\x82\xBB\xE3\x83\xBC\xE3\x82\xB8(2022/09/0817:15:09.24) " error="invalid timeformat: value = 2022/12/3 12:34:56, error_class = ArgumentError, error =string doesn't match" If this parameter is omitted, the time set to JPC_LOG_TIME is the time when Fluentd detected the log message. If syslog is specified for type and this parameter is not specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.	Changeable	Specify the time format according to the format of the log file to be monitored.	--
localtime	Specify true because local time is used.	Not changeable	true	true
utc	Specify false because local time is used.	Not changeable	false	false
timezone	Date/time is parsed in specified timezone.	Changeable	Specify the timezone according to the format of the log file to be monitored. Available time zone format: [+-]HH:MM e.g. "+09:00" [+-]HHMM e.g. "+0900" When timezone is specified, time_format is must be specified.	--

(Legend) -: Not applicable

• For multiline

  <parse>
    @type multiline
    format_firstline regular-expression-to-parse-the-first-line-log
    formatN regular-expression-to-parse-logs
    time_key time
    null_empty_string false
    estimate_current_event true
    keep_time_key false
    items-for-parsing-date-and-time-of-logs
  </parse>

regular-expression-to-parse-the-first-line-log (required)

Specify a regular expression to parse the contents of one log line. If the specified regular expression matches the contents of the log, the matched log line is read as the first line of a multi-line log.

regular-expression-to-parse-logs (required)

Similar to the description in "For regexp". N can be an integer from 1 to 20, and the specified regular expression is used to parse the contents of a multi-line log as line N.

items-for-parsing-date-and-time-of-logs

Same as description of "If regexp".

Item Name	Description	Changeability	JP1/IM - What the user sets on the agent	JP1/IM - Initial value of Agent
format_firstline	Specify the first line of the log as a regular expression. The multiline parse plug-in parses multi-line logs. If multiline is specified as the type of the parse plug-in, formatN and format_firstline must be specified. The maximum number of bytes that can be specified in a regular expression is 1023 bytes (excluding delimiters). Regular expressions must be sandwiched between "/" (delimiters). If the delimiter is not used, an error is output.	Changeable	Specify the first line of the log as a regular expression according to the format of the log file to be monitored.	--
formatN	Specify each line of the log as a regular expression. Specifies the format of the multiline log. N is an integer from 1 to 20 that creates a list of regular expression formats. The maximum number of bytes that can be specified in a regular expression is 1023 bytes (excluding delimiters). Regular expressions must be sandwiched between "/" (delimiters). If the delimiter is not used, an error is output. If this parameter is not specified, an error is printed when Fluentd is invoked.	Changeable	Specify each line of the log as a regular expression according to the format of the log file to be monitored.	--

(Legend) -: Not applicable

• For syslog

  <parse>
    @type syslog
    time_type string
    time_format date-and-time-formats
    rfc5424_time_format syslog-date-and-time-format-in-RFC-5424-format
    message_format types-of-syslogs
    with_priority priority-prefix
    parser_type string
    support_colonless_ident presence-or-absence-of-ident-field
    time_key time
    null_empty_string false
    estimate_current_event true
    keep_time_key false
    localtime true
    utc false
  </parse>

date-and-time-formats (required)

Same as description of "If regexp". Specify a regular expression to parse the date and time in the log message. If auto is specified as types-of-syslogs, specify syslog-date-and-time-format-in-RFC-3164-format.

syslog-date-and-time-format-in-RFC-3164-format (optional)

Specify a regular expression to parse the date and time of the syslog in RFC-5424 format. Use this parameter only if types-of-syslogs is specified to auto.

types-of-syslogs (required)

Specify the type of syslog to be analyzed: rfc3164 (RFC-3164 format), rfc5424 (RFC-5424 format), or auto (both).

priority-prefix (required)

Indicates whether RFC-3164 formatted syslogs contain a priority prefix as true or false. false can be specified only when rfc3164 is specified as types-of-syslogs, otherwise it must be specified as true.

presence-or-absence-of-ident-field (required)

Specifies whether the RFC-3164-formatted syslog contains the IDENT field as true or false. false can be specified only when rfc3164 is specified as types-of-syslogs, otherwise it must be specified as true.

Item Name	Description	Changeability	JP1/IM - What the user sets on the agent	JP1/IM - Initial value of Agent
time_format	Same as description of "If regexp". If syslog is specified for type and auto is specified for message_format, specifies the RFC-3164 protocol time format. In this case, the RFC-5424 protocol time format is specified in rfc5424_time_format. The RFC-3164 protocol time format is "%b %d %H:%M:%S". If the output is time-stamped in seconds or less, change it to "%b %d %H:%M:%S.%N".	Changeable	Specify the time format as a regular expression according to the format of the log file to be monitored.	--
rfc5424_time_format	Specifies the RFC-5424 protocol time format, up to 256 bytes. The following formats are supported: %b:Abbreviated month (Jan,Feb,...) %d:Day (01~31) %H: 24-hour clock (00~23) %M:min(00~59) %m: Month number (01~12) %S:sec (00~60 (60 indicates leap second)) %Y: A number representing the year %N: fractional seconds If you specify an incorrect value, a warning message similar to the one shown below may be output to the Fluentd log, and the log may not be monitored. 2023-03-24 13:18:27 +0900 [warn]: #0 invalid line found file="/home/ec2-user/fluentd_test/input_log/20230315_log1.txt" line="<16>1 2023-03-24T13:18:27.31+0900 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut=\"3\" eventSource=\"Application\" eventID=\"11211\"] Hi, from Fluentd!" error="invalid time format: value = 2023-03-24T13:18:27.31+0900, error_class = ArgumentError, error = string doesn't match" Use this parameter only if the message_format is specified as AUTO. If not specified, the time is parsed and extracted according to the regular expression time format described in 3.15.3(3)(g)Log parsing function (parse plug-in) of the JP1/Integrated Management 3 - Manager Overview and System Design Guide.	Changeable	Specify the time format according to the format of the log file to be monitored.	--
message_format	Specifies the protocol format for syslog. You can specify RFC3164, RFC5424, or AUTO. The default is rfc3164. If the monitored syslog is output in RFC5424, specify RFC5424. Also, if the syslog to be monitored is logged using both RFC3164 and RFC5424 protocols, AUTO is specified. If auto is specified, the syslog parsing plug-in uses the message prefix to detect the format. If this parameter is not specified, or if an incorrect value is specified, an error is printed when Fluentd is started.	Changeable	Specify the log format according to the format of the log file to be monitored.	--
with_priority	Indicates whether RFC-3164 formatted syslogs contain a priority prefix as true or false. Specify true if the monitored log has a priority prefix such as [9]. If this parameter is not specified, the Fluentd log may display a warning message similar to the one shown below, and the log may not be monitored. 2023-03-24 14:15:01 +0900 [warn]: #0 pattern not matched: "Mar 24 14:15:01 192.168.0.1 fluentd[11111]: [error] Syslog test" If a value other than true or false is specified, an error is output when Fluentd is started.	Changeable	Specify according to the format of the log file to be monitored.	--
support_colonless_ident	Specifies whether RFC-3164 formatted syslogs contain the ident field as true or false. Used to monitor logs in RFC3164 format. Specifies false if the monitored log does not contain an ident field in the message. If this parameter is not specified, no error or warning messages are printed, and the monitored log may be parsed in the wrong format. Therefore, it is necessary to check whether JP1 events are issued in a normal format after starting Fluentd and adding logs. If a value other than true or false is specified, an error is output when Fluentd is started.	Changeable	Specify according to the format of the log file to be monitored.	--

(Legend) -: Not applicable

• For csv

  <parse>
    @type csv
    keys array-of-field-names-for-records
    delimiter ,
    parser_type types-of-internal-parsers
    time_key time
    null_empty_string false
    estimate_current_event true
    keep_time_key false
    items-for-parsing-date-and-time-of-logs
  </parse>

array-of-field-names-for-records (required)

Specifies the field names of the record in the form of an array. One of field names is necessary to be "message" in order to set to MESSAGE of JP1 event. If character strings named "message" is not trimmed, MESSAGE of JP1 event will be empty.

types-of-internal-parsers (required)

Specifies the type of internal parser that parses logs in CSV format.

Item Name	Description	Changeability	JP1/IM - What the user sets on the agent	JP1/IM - Initial value of Agent
keys	Specify an array of record item names within 256 bytes. If this parameter is not specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.	Changeable	Specify according to the format of the log file to be monitored.	--
parser_type	Specifies the type of internal parser for parsing log lines, either normal or fast. If normal is specified, the Ruby CSV.parse_line method is used. If fast is specified, Fluentd's own lightweight implementation is used. The parser you use is several times faster than usual, but supports only typical patterns. The following formats are supported: # non-quoted value1,value2,value3,value4,value5 # quoted "value1","val,ue2","va,lu,e3","val ue4","" # escaped "message","mes""sage","""message""",,"""""" # mixed message,"mes,sage","me,ssa,ge",mess age,"" If this parameter is not specified, or if an incorrect value is specified, an error is output when Fluentd is started.	Changeable	If the format of the log file to be monitored is in the following format, specify fast. # non-quoted value1,value2,value3,value4,value5 # quoted "value1","val,ue2","va,lu,e3","val ue4","" # escaped "message","mes""sage","""message""",,"""""" # mixed message,"mes,sage","me,ssa,ge",mess age,"" If the format of the log file to be monitored does not match the above format, normal is specified.	--

(Legend) -: Not applicable

items-for-parsing-date-and-time-of-logs

Same as description of "If regexp". Specify when array of field names for records has "time".

• For tsv

  <parse>
    @type tsv
    keys array-of-field-names-for-records
    delimiter "\t"
    time_key time
    null_empty_string false
    estimate_current_event true
    keep_time_key false
    items-for-parsing-date-and-time-of-logs
  </parse>

array-of-field-names-for-records (required)

Specifies the field names of the record in the form of an array.

Item Name	Description	Changeability	JP1/IM - What the user sets on the agent	JP1/IM - Initial value of Agent
keys	Specify an array of record item names within 256 bytes. If this parameter is not specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.	Changeable	Specify according to the format of the log file to be monitored.	--

(Legend) -: Not applicable

items-for-parsing-date-and-time-of-logs

Same as description of "If regexp". Specify when array of field names for records has "time".

• For ltsv

  <parse>
    @type ltsv
    delimiter-between-items delimiter-pattern-between-items
    label_delimiter delimiter-between-label-and-value
    time_key time
    null_empty_string false
    estimate_current_event true
    keep_time_key false
    items-for-parsing-date-and-time-of-logs
  </parse>

delimiter-between-items delimiter-pattern-between-items (required)

Specifies the delimiter between items. Specify one of the following:

- When the separator between items is a tab

    delimiter "\t"

- When the separator between items is one or more blanks

    delimiter_pattern /\s+/

delimiter-between-label-and-value (required)

Specifies the delimiter between the label and the value.

Item Name	Description	Changeability	JP1/IM - What the user sets on the agent	JP1/IM - Initial value of Agent
delimiter	Specifies the delimiter between items. The only delimiter that can be specified is double-quoted "\t". If either this parameter or delimiter_pattern is specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.	Changeable	Specify according to the format of the log file to be monitored.	--
delimiter_pattern	In an LTSV format file, this is specified when the separator between entries is one or more spaces. The only delimiter that can be specified is "/\s+/". If either this parameter or delimiter is not specified, or if an incorrect value is specified, no error or warning message is output, and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.	Changeable	Specify according to the format of the log file to be monitored.	--
label_delimiter	Specifies the delimiter between the label and the value within 256 bytes. If this parameter is not specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.	Changeable	Specify according to the format of the log file to be monitored.	--

(Legend) -: Not applicable

items-for-parsing-date-and-time-of-logs

Same as description of "If regexp". Specify when array of field names for records has "time".

[Attributes Settings] section

Serup Attributes of JP1 events to be issued and Attribute value.

log-monitoring-name (mandatory)

Same as description of [Metric Settings] section.

event-ID (optional)

Specifies Value to Setup for B. ID property of JP1 event. For details about Value that can be specified, see JP1/Base Operation Manual. The default Value is "00007601" (Event ID used for monitoring text-formatted log file definition file).

If this option is omitted, JP1 events are not issued.

Instead of specifying "ID event-ID", you can setup event ID according to the value of message property by specifying:

    ID "${
         if record['message'].match(/regex-1/)
           'event-ID1'
         elsif record['message'].match(/regex-2/)
           'event-ID2'
         elsif record['message'].match(/regex-3/)
           'event-ID3'
...
         else
           'event-ID4'
         end}"

The conditional branch of Ruby determines value of message property and turns setup event ID. In the above cases, if regex-1 is matched, event ID is set to the value as specified in event-ID-1. If it doesn't match, it will setup event-ID-2 if it matches regex-2. If none of the matches are compared by the number specified in elsif, else statement setup value specified in event-ID-4. to event ID. You can specify a maximum of 100 if and elsif statements.

Note: The behavior follows the specifications of Ruby's regular expressions. In Ruby, backslash (\) is used as an escape character, so when using backslash (\) in a regular expression, it must be specified as \\.

host-name (optional)

Same as description of [Metric Settings] section.

If the specification is omitted, the attribute value of JP1_SOURCEHOST is not setup and JP1 event is not add to the correct IM management node.

You can also dynamically setup the canonical host name of the system by doing the following:

    JP1_SOURCEHOST ${Socket.gethostname}

severity (optional)

Specifies the value to setup for E.SEVERITY property of JP1 event. For details about the value that can be specified, see the JP1/Base User's Guide. The default value is "Notice".

If this option is omitted, JP1 events are not issued.

Instead of specifying "SEVERITY severity", you can setup the severity according to the value of message property by specifying:

    SEVERITY "${
         if record['message'].match(/regex-1/)
           'Critical'
         elsif record['message'].match(/regex-2/)
           'Error'
         elsif record['message'].match(/regex-3/)
           'Warning'
...
         else
           'Notice'
         end}"

The conditional branch of Ruby determines value of message property and turns setup severity. In the above cases, "Critical" is setup to severity if the regex-1 is matched. If it doesn't match, it will setup "Error" if it matches against regex-2. Setup "Notice" in else construct to severity if elsif matches none of the specified numbers. You can specify up to 100 statements of if and elsif.

Note: The behavior follows the specifications of Ruby's regular expressions. In Ruby, backslash (\) is used as an escape character, so when using backslash (\) in a regular expression, it must be specified as \\.

IM-management-node-label-name (optional)

Same as description of [Metric Settings] section.

If this option is omitted, JP1 events are not issued.

any-attribute-name any-value (optional)

Specify this operand if you want to Add a JP1 event-attribute. For details about Attribute name that can be specified, see JP1/Base Operation Manual.

For Value, you can specify the captured name in the [Input Settings] section with the regular expressions to parse the logs.

For example, to capture with the name "NUMBER" and Setup to the property EXIT_CODE, you would specify:

EXIT_CODE ${record['NUMBER']}

You can Add more than one extended attribute, but no JP1 event is issued if the sum of the sizes of Value that Setup the extended attribute of JP1 event exceeds the limit.

For details about the upper limit of extended attributes, see 4.4.4(5)__transformEvent method.

[Inclusion Settings] section

Specifies the conditions of the log to be monitored in a regular expression. If not specified, all logs are monitored. If an unmonitored log is output, the log is not converted to a JP1 event and is not output to Fluentd log.

In the default Setup, "#" is specified at the beginning of the line and is handled as Comment, so when specifying it, "#" is going to be deleted.

log-monitoring-name (mandatory)

Same as description of [Metric Settings] section.

attribute-name-of-JP1-event (optional)

Specifies the attribute name of JP1 event. For example, "MESSAGE". If the specification is omitted, error occurs when Fluentd is started.

regular-expression-of-logs-to-monitor (optional)

Specifies a regular expression for the value of the attribute specified by the attribute name of JP1 event. If the value to be match is included, monitoring is performed.

If the specification is omitted, error occurs when Fluentd is started.

You can also specify a logical AND or OR condition for multiple regular expression patterns. For details about how to specify the log data, see 3.15.3(7) Log data extractor (grep plug-in) in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.

[Exclusion Settings] section

Specifies the conditions for logs that are not monitored, in regular expressions. If not specified, all logs are monitored. In the default Setup, "#" is specified at the beginning of the line and is handled as Comment, so when specifying it, "#" is going to be deleted.

log-monitoring-name (mandatory)

Same as description of [Metric Settings] section.

attribute-name-of-JP1-event (optional)

Specifies the attribute name of JP1 event. For example, "MESSAGE".

If the specification is omitted, error occurs when Fluentd is started.

regular-expression-of-logs-to-monitor (optional)

Specifies a regular expression for the value of the attribute specified by attribute-name-of-JP1-event. If value to be match is included, monitoring is not performed.

If the specification is omitted, error occurs when Fluentd is started.

You can also specify a logical AND or OR condition for multiple regular expression patterns. For details about how to specify the log data, see 3.15.3(7) Log data extractor (grep plug-in) in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.

[Forward Settings] section

Setup the regular expression of the log data to be converted into a JP1 event.

log-monitoring-name (mandatory)

Same as description of [Metric Settings] section.

attribute-name-of-JP1-event (optional)

Specifies the attribute name of JP1 event. The default value is "SEVERITY".

If the specification is omitted, error occurs when Fluentd is started.

regular-expression-for-logs-that-emit-JP1-events (optional)

Specifies the condition for regular expressions that issue JP1 events for the value of the attribute specified by attribute-name-of-JP1-event.

The default value is "Warning|Error|Critical|Alert|Emergency" and matches if the value of SEVERITY is greater than or equal to Warning.

If Value of the attribute contains a Value that Match the condition, the monitored log content is converted to JP1 events and Add to JP1/Base in Integrated manager host. The content of the monitored log is also output to Fluentd log. If you do not Match the condition, JP1 event is not issued and only logged in Fluentd.

If the specification is omitted, Error occurs when Fluentd is started.

In the [Attributes Settings] section, "Notice" is specified as the default SEVERITY. Therefore, the log monitoring result is not output as a JP1 event. It is output only in Fluentd log.

If you want to publish a log with a SEVERITY of "Notice" as a JP1 event, change the definition as shown in the underlined part:

    pattern /Notice|Warning|Error|Critical|Alert|Emergency/

To Page Top

Example definition

The following is an example of the condition and definitions for monitoring a textual logging File.

■Conditions

• Path of the logged file to monitor

  C:\Program Files (x86)\Hitachi\HNTRLib2\spool\*

• Logging message

  6027 2022/08/25 17:45:50.219     jbssessionmgr    000018EC 00000FCC KAVA1497-I            jp1admin user has Login

• Log messages to monitor

  Monitor the logging message where message ID starts with KAVA.

• Value to setup to MESSAGE

  Setup message ID or later text in the logging message.

• Value to setup to SEVERITY

  Setup value according to severity of message ID.

• Value to setup to any attribute name

  Setup the process-name (jbssessionmgr) contained in the log message to attribute name PROCESS_NAME.

■Definitions

<worker 0>
## [Metric Settings]
<source>
  @type exec
  command "echo {}"
  <parse>
    @type json
  </parse>
  run_interval 60s
  tag jpc_ima_metrics.tail.user_app_log
</source>
 
<filter jpc_ima_metrics.tail.user_app_log>
  @type record_transformer
  enable_ruby true
  <record>
    __name__ fluentd_logtrap_running
    instance hostA
    jp1_pc_nodelabel UserApplication
    jp1_pc_category applicationServer
    jp1_pc_logtrap_defname user_app_log_tail
    jp1_pc_trendname fluentd
    job jpc_fluentd
    jp1_pc_nodelabel_fluentd Log trapper(Fluentd)
    jp1_pc_addon_program JPC Fluentd
  </record>
</filter>
</worker>
<worker 1>
## [Input Settings]
<source>
  @type tail
  tag tail.user_app_log
  path C:/Program Files (x86)/Hitachi/HNTRLib2/spool/*
  follow_inodes true
  refresh_interval 60
  skip_refresh_on_startup false
  read_from_head false
  encoding "UTF-8"
  from_encoding "Shift_JIS"
  read_lines_limit 1000
  read_bytes_limit_per_second -1
  pos_file ../data/fluentd/tail/user_app_log.pos
  path_key tailed_path
  rotate_wait 5s
  enable_watch_timer false
  enable_stat_watcher true
  open_on_every_update false
  emit_unmatched_lines false
  ignore_repeated_permission_error false
  <parse>
    @type regexp
    expression /^([^ ]* +(?<time>[^ ]* [^ ]*) +(?<PROCESS>[^ ]*) +[^ ]* +[^ ]* +(?<message>.*))$/
    time_key time
    null_empty_string false
    estimate_current_event true
    keep_time_key false
    localtime true
    utc false
  </parse>
</source>
 
## [Attributes Settings]
<filter tail.user_app_log>
  @type record_transformer
  enable_ruby true
  auto_typecast true
  renew_record true
  
  <record>
    ID 00007601
    MESSAGE ${record["message"]}
    JP1_SOURCEHOST hostA
    JPC_LOG_TIME ${time.utc.to_i}
    PRODUCT_NAME /HITACHI/JP1/JPCCS2/LOGTRAP/UserApplication
    PPNAME /HITACHI/JP1/JPCCS2/LOGTRAP
    SEVERITY "${
         if record['message'].match(/^KAVA[1-9]*-E/)
           'Error'
         elsif record['message'].match(/^KAVA[1-9]*-W/)
           'Warning'
         elsif record['message'].match(/^KAVA[1-9]*-I/)
           'Information'
         else
           'Notice'
         end}"
    PLATFORM ${ if RUBY_PLATFORM.downcase =~ /mswin(?!ce)|mingw|cygwin|bccwin/; 'NT'; else 'UNIX'; end }
    OBJECT_TYPE LOGFILE
    OBJECT_NAME ${record['tailed_path']}
    ROOT_OBJECT_TYPE LOGFILE
    ROOT_OBJECT_NAME ${record['tailed_path']}
    JP1_TRAP_NAME ${tag_parts[1]}
    JPC_NODELABEL UserApplication
    PROCESS_NAME ${record['PROCESS']}
  </record>
</filter>
 
## [Inclusion Settings]
<filter tail.user_app_log>
  @type grep
  <regexp>
    key MESSAGE
    pattern /^KAVA[0-9]*-(I|W|E)/
  </regexp>
</filter>
 
## [Exclusion Settings]
#<filter tail.user_app_log>
#  @type grep
#  <exclude>
#    key 
#    pattern //
#  </exclude>
#</filter>
 
## [Forward Settings]
<match tail.user_app_log>
  @type rewrite_tag_filter
  <rule>
    key SEVERITY
    pattern /Warning|Error|Critical|Alert|Emergency/
    tag ${tag}.jp1event
  </rule>
  <rule>
    key SEVERITY
    pattern /.*/
    tag ${tag}.outputlog
  </rule>
</match>
 
<filter /tail\.user_app_log\.(jp1event|outputlog)/>
  @type record_transformer
  enable_ruby true
  auto_typecast true
  renew_record true
  <record>
    eventId ${record['ID']}
    xsystem true
    message ${record['MESSAGE']}
    attrs ${record}
  </record>
  remove_keys $.attrs.ID
  remove_keys $.attrs.MESSAGE
</filter>
</worker>

To Page Top