Monitoring text-formatted log file definition file (fluentd_@@trapname@@_tail.conf.template)

Format

<worker 0>
## [Metric Settings]
<source>
  @type exec
  command "echo {}"
  <parse>
    @type json
  </parse>
  run_interval 60s
  tag jpc_ima_metrics.tail.Log Monitoring Name
</source>
 
<filter jpc_ima_metrics.tail.Log Monitoring Name>
  @type record_transformer
  enable_ruby true
  auto_typecast false
  <record>
    __name__ fluentd_logtrap_running
    instance Hostname
    jp1_pc_nodelabel IM Management Node Label Name
    jp1_pc_category Category ID
    jp1_pc_logtrap_defname Log Monitoring Name_tail
    jp1_pc_trendname fluentd
    job jpc_fluentd
    jp1_pc_nodelabel_fluentd Log trapper(Fluentd)
    jp1_pc_addon_program JPC Fluentd
  </record>
</filter>
</worker>
<worker worker id>
## [Input Settings]
<source>
  @type tail
  tag tail.Log Monitoring Name
  path Monitored paths
  follow_inodes true
  refresh_interval 60
  skip_refresh_on_startup false
  read_from_head Read the logs to be monitored when Fluentd is started for the first time from the beginning
#  encoding "Fluentd character code"
#  from_encoding "Character Codes of Monitored Logs"
  read_lines_limit 1000
  read_bytes_limit_per_second -1
  pos_file ../data/fluentd/tail/Log Monitoring Name.pos
  path_key tailed_path
  rotate_wait 5s
  enable_watch_timer Enable additional watch timers
  enable_stat_watcher true
  open_on_every_update false
  emit_unmatched_lines false
  ignore_repeated_permission_error false
  <parse>
    @type Log format
    Settings depending on the log format
  </parse>
</source>
 
## [Attributes Settings]
<filter tail.Log Monitoring Name>
  @type record_transformer
  enable_ruby true
  auto_typecast false
  renew_record true
  
  <record>
    ID event ID
    MESSAGE ${record["message"]}
    JP1_SOURCEHOST Hostname
    JPC_LOG_TIME ${time.utc.to_i}
    PRODUCT_NAME /HITACHI/JP1/JPCCS2/LOGTRAP/IM Management Node Label Name
    PPNAME /HITACHI/JP1/JPCCS2
    SEVERITY Severity
    PLATFORM ${ if RUBY_PLATFORM.downcase =~ /mswin(?!ce)|mingw|cygwin|bccwin/; 'NT'; else 'UNIX'; end }
    OBJECT_TYPE LOGFILE
    OBJECT_NAME ${record['tailed_path']}
    ROOT_OBJECT_TYPE LOGFILE
    ROOT_OBJECT_NAME ${record['tailed_path']}
    JP1_TRAP_NAME ${tag_parts[1]}
    JPC_NODELABEL IM Management Node Label Name
    Any attribute name Any value
  </record>
</filter>
 
## [Inclusion Settings]
#<filter tail.Log Monitoring Name>
#  @type grep
#  <regexp>
#    key Attribute name of JP1 event
#    pattern /Regular expression of logs to monitor/
#  </regexp>
#</filter>
 
## [Exclusion Settings]
#<filter tail.Log Monitoring Name>
#  @type grep
#  <exclude>
#    key Attribute name of JP1 event
#    pattern /Regular expressions for logs that you do not want to monitor/
#  </exclude>
#</filter>
 
## [Forward Settings]
<match tail.Log Monitoring Name>
  @type rewrite_tag_filter
  <rule>
    key Attribute name of JP1 event
    pattern /Regular expression for logs that emit JP1 events/
    tag ${tag}.jp1event
  </rule>
  <rule>
    key SEVERITY
    pattern /.*/
    tag ${tag}.outputlog
  </rule>
</match>
 
<filter /tail\.Log-Monitoring-Name\.(jp1event|outputlog)/>
  @type record_transformer
  enable_ruby true
  auto_typecast true
  renew_record true
  <record>
    eventId ${record['ID']}
    xsystem true
    message ${record['MESSAGE']}
    attrs ${record}
  </record>
  remove_keys $.attrs.ID
  remove_keys $.attrs.MESSAGE
</filter>
</worker>

To Page Top

File

fluentd_@@trapname@@_tail.conf.template

fluentd_@@trapname@@_tail.conf.template.model (model file)

To Page Top

Storage directory

■Integrated agent host

In Windows:

• For a physical host

  Agent-path\conf\

• For a logical host

  shared-folder\jp1ima\conf\

In Linux:

• For a physical host

  /opt/jp1ima/conf/

• For a logical host

  shared-directory/jp1ima/conf/

To Page Top

Description

Definition File for monitoring text-formatted logging File.

Copy the template (fluentd_@@trapname@@_tail.conf.template) and change File name of Copy destination to "fluentd_log monitoring name_tail.conf" for use. File name must be unique within integrated agent host. For details on the location of "fluentd_log monitoring name_tail.conf", see Appendix A.4(3) Integrated agent host (Windows) and Appendix A.4(4) Integrated agent host (Linux) in the manual "JP1/Integrated Management 3 - Manager Overview and System Design Guide". The logging log monitoring name must be between 1 and 30 characters long. Allowed characters are single-byte alphanumeric characters, "-" (hyphen), and "_" (underscore).

Create a monitor-definition File for each wrapped-around log File group that you want to monitor (or for each log File that does not wrap-around). JP1/IM-Agent creates a IM managed node for SID of target of monitoring according to Value specified in the Label Name of the IM managed node in the monitoring definition File. If another monitor-definition File has the same IM management node labelname, only one IM management node is created.

The text-based log File monitoring feature reads this definition File and analyzes the log that the application has written to the text-based log File. You can setup if you specify a condition for the analyzed information and the condition is met, the information to be converted to JP1 events or output to Fluentd logging File. For details about JP1 event to be issued, see 3.2.3(2) JP1 event issued that monitoring a textual log File.

Lines beginning with "#" are treated as Comment and do not affect the programming behavior.

To Page Top

Character code

UTF-8 (without BOM)

To Page Top

Line feed code

In Windows: CR+LF

In Linux: LF

To Page Top

When the definitions are applied

This information is reflected in Fluentd operation when Fluentd serviceis Restart.

If Add or Delete of definition files or value in [Metric Settings] section is changed, the change is reflected in Tree view of integrated operation viewer.

For details about Application method, see 1.21.2(10) Creating and importing IM management node tree data (Windows) (mandatory) in JP1/Integrated Management 3 - Manager Configuration Guide.

To Page Top

Information that is specified

<worker> directive

See the description of <worker> directive in Log monitoring common definition file (jpc_fluentd_common.conf).

worker id (Optional)

Description	Changeability	What You Setup in Your JP1/IM - Agent	JP1/IM - Agent Defaults Value
Specifies the number of workers that Fluentd will start. Serves as an argument to the <worker N> directive. Valid values are integers from 1 to 128.	Can be changed	It must be specified so as not to duplicate the worker ID specified in the existing text log file monitoring definition file or the Windows event log monitoring definition file.	1

[Metric Settings] Section

Setup Value of label of sample that you want to send to JP1/IM-Manager's Trend data Management Database.

Logging Log file trap name (mandatory)

Specifies the logging Log file trap name specified in File name of Copy destination as a string of 1 to 30 characters. Allowed characters are single-byte alphanumeric characters, "-" (hyphen), and "_" (underscore). Default Value is "@@trapname@@".

Because you need to Setup several locations in File, use OS command/editor function to replace the "@@trapname@@" location with the logging Log file trap name you want to specify.

If the specification is omitted, Error occurs when Fluentd is started.

Note that the logging Log file trap name must be Setup as follows:

• All logging Log file trap name in the same File are the same.

• Logging log file trap name is unique for Monitoring Text-formatted log file definition file and monitoring Windows event-log definition file

Host name (optional)

Specify Host name to be monitored using characters 1 to 255 other than control characters. The default Value is Setup by integrated agent installers.

If the specification is omitted, IM management node is not created.

You can also dynamically Setup the canonical Host name of the system by doing the following:

    instance ${Socket.gethostname}

Label-name of IM management node (optional)

Specifies the character string that integrated operation viewer displays on IM management node label. This is not a control character. When URL is encoded, the character string must be between 1 and 234 bytes (the upper limit for multibyte characters is 26). The default Value is "Application".

If the specified information is invalid or omitted, IM management node is not going to be created.

Different monitor definitions File can specify the same IM management node label-name. Then, only one IM management node is created, and JP1 events in both of monitor-defined files are Add to one IM management node.

Category ID (optional)

Specifies the category ID of IM management node corresponding to SID to be monitored for logging as a character 1 to 255 other than control characters. If the specification is omitted, "otherApplications" is assumed.

[Input Settings] Section

Setup the path to the text-formatted log File that you want to monitor and the regular expressions that parse the log Message.

Logging Log file trap name (mandatory)

Same as the section [Metric Settings] description.

Path to be monitored (required)

Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
Specify the path to read. You can specify multiple paths by separated by commas. You can include * and strftime formats to dynamically Add and Delete the logging File you want to monitor. The list of Log Files is updated at refresh_interval intervals. For specification examples, see "(3) Monitoring function of log File (tail plug-in)" in "9.5.4 fluentd"" in the manual JP1/Integrated Management 3 - Manager Overview and System Design Guide. If you specify an incorrect path, Log Files logging is not read. The following rules apply to programming: Specify as an absolute path Directories and File on network drives cannot be specified (for Windows) Specify "/" instead of "\" as the path delimiter (in Windows) Multiple paths can be specified. You can specify "*" (wildcard). You specify within 256 bytes. The following path names cannot be specified. - File with a leading "-" (hyphen) - Folder name, directory name, or File name containing environment-dependent characters - Space-directory-name (for Linux)	Installation Required	Specifies Log Files path.	Not applicable

Read logs to be monitored at first startup of fluentd from the beginning (optional)

Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
Specifies whether the log should start reading from the beginning, not the end, or from the last read position recorded in pos_file. You can specify true or false.	Can be changed	If you want to read a log that was already Add at startup, change it to true.	false

Fluentd character encoding (optional)

If the character code of the log to be monitored is UTF-8 (handled as Comment), specify the default Setup (handled as Comment). Specify UTF-8 if the character code of the log to be monitored is not UTF-8 (handled as Comment). In the default Setup, since "#" is specified at the beginning of the line and it is handled as Comment, "#" is turned Delete.

Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
Specifies the encoding in which to read the line. By JP1/IM - Agent, in_tail outputs value of string in ASCII-8BIT encoding in default. You can change it with the following options: encoding changes the text to encoding. If both encoding and from_encoding are specified, in_tail attempts to convert the from_encoding string to a encoding.	Can be changed	In JP1/IM - Agent, you can specify the following Value: UTF-8	Not specified (Comment out) # encoding "UTF-8"

Character code of the log to be monitored (optional)

If the character code of the log to be monitored is UTF-8, specify the default Setup (handled as Comment). If the character code of the log to be monitored is not UTF-8, specify the character code. In the default Setup, since "#" is specified at the beginning of the line and it is handled as Comment, "#" is turned Delete.

Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
See the explanation of the character encoding in Fluentd(optional).	Can be changed	Specifies the character encoding of Log Files. In JP1/IM - Agent, you can specify the following Value: UTF-16LE UTF-16BE Shift_JIS Windows-31J GB18030	Not specified (Comment out) # encoding "Shift_JIS"

Enable additional watch timers

Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
Specify true or false. If false is specified for this parameter, the most recent log is not monitored when reading multiple lines of log. Therefore, if multiline is specified as the type of the parse plug-in, true is specified. Specifying false for this parameter significantly reduces CPU and I/O consumption when tailing a large number of files on systems that support inotify.	Can be changed	True only if multiline is specified as the type of the parse plugin.	false

Log format

Specifies the format for parsing the imported log.

The following formats can be specified:

type	Description
none (Default)	Read a one-line log as it is without parsing or structuring.
regexp	Reads a single-line log that matches the pattern specified by the regular expression.
multiline	Loads a multi-line log that matches the pattern specified by the regular expression.
syslog	Read the log output by syslog.
csv	Load logs in CSV format (comma delimited).
tsv	Loads logs in TSV format (tab-delimited).
ltsv	Import logs in LTSV format (labeled tab-delimited).

For examples of specifying logs in each format, see "9.5.4(3)(g)Log parsing function (parse plug-in)" in the manual JP1/Integrated Management 3 - Manager Overview and System Design Guide.

Settings depending on the log format

Specify the entries according to the log format.

• If none

  <parse>
    @type none
    message_key message
  </parse>

• If regexp

  <parse>
    @type regexp
    expression Regular expressions to parse logs
    time_key time
    null_empty_string false
    estimate_current_event true
    keep_time_key false
    localtime true
    utc false
  </parse>

Regular expression to parse log (Required)

Specifies a regular expression and parses the contents of one line of the log. Use the Named Capture feature to trim a string named "message" that Setup to Message of JP1 event. For example: The default Value contains a regular expression that trims the entire line in "message". You can also trim with another name and Setup to any property of JP1 event.

If you trim a Date/time in the logging Message as the name "time", Date/time is Setup as Value in JPC_LOG_TIME property of JP1 event. Date/time format is specified in time_format and the timezone is specified in timezone. If you do not trim time, JPC_LOG_TIME property Value will Setup Date/time for which Fluentd monitored its logging Message.

Item Name	Description	Changeability	What You Setup in Your JP1/IM - Agent	Default Value for JP1/IM - Agent
expression	specifies the regular expressions matches for logging. Regular expressions must be sandwiched between "/" (delimiters). If the delimiter is not used, an error is output. Regular expressions must specify at least one named capture (?<name>Regular expression for truncated logs). Regular expressions can have i and m suffixes. i(ignorecase) Ignores the case of the match. m (multi-line) Creates a regular expression as a multi-line mode. ". " matches to a line break. both Specify both i and m. If the log read does not Match the regular expression, the following Warning Message is printed in Fluentd log and the log is not going to be monitored. 2022-01-23 12:34:56 +0900 [warn]: #0 pattern not matched: "Error Message"	Can be changed	Setup according to Log Files logging format.	expression /^(?<message>.*)$/

• For multiline

  <parse>
    @type multiline
    format_firstline Regular expression to parse the first line log
    formatN Regular expression to parse logs
  </parse>

Regular expression to parse the first line log (required)

Specify a regular expression to parse the contents of one log line. If the specified regular expression matches the contents of the log, the matched log line is read as the first line of a multi-line log.

Regular expression to parse logs (required)

Similar to the description in "For regexp". N can be an integer from 1 to 20, and the specified regular expression is used to parse the contents of a multi-line log as line N.

Item Name	Description	Changeability	JP1/IM - What the user sets on the agent	JP1/IM - Initial value of Agent
format_firstline	Specify the first line of the log as a regular expression. The multiline parse plug-in parses multi-line logs. If multiline is specified as the type of the parse plug-in, formatN and format_firstline must be specified. The maximum number of bytes that can be specified in a regular expression is 1023 bytes (excluding delimiters). Regular expressions must be sandwiched between "/" (delimiters). If the delimiter is not used, an error is output.	Changeable	Specify the first line of the log as a regular expression according to the format of the log file to be monitored.	--
formatN	Specify each line of the log as a regular expression. Specifies the format of the multiline log. N is an integer from 1 to 20 that creates a list of regular expression formats. The maximum number of bytes that can be specified in a regular expression is 1023 bytes (excluding delimiters). Regular expressions must be sandwiched between "/" (delimiters). If the delimiter is not used, an error is output. If this parameter is not specified, an error is printed when Fluentd is invoked.	Changeable	Specify each line of the log as a regular expression according to the format of the log file to be monitored.	--

(Legend) -: Not applicable

• For syslog

  <parse>
    @type syslog
    time_format Date and time formats
    rfc5424_time_format Syslog date and time format in RFC-5424 format
    message_format Types of syslogs
    with_priority Priority prefix
    parser_type string
    support_colonless_ident Presence or absence of IDENT field
  </parse>

Date and time format (required)

Specify a regular expression to parse the date and time in the log message. If auto is specified as the syslog type, specifies the format of the date and time of the syslog in RFC-3164 format.

Syslog date and time format in RFC-3164 format (optional)

Specify a regular expression to parse the date and time of the syslog in RFC-5424 format. Use this parameter only if the syslog type is specified to auto.

Syslog type (required)

Specify the type of syslog to be analyzed: rfc3164 (RFC-3164 format), rfc5424 (RFC-5424 format), or auto (both).

Priority prefix (required)

Indicates whether RFC-3164 formatted syslogs contain a priority prefix as true or false. false can be specified only when rfc3164 is specified as the syslog type, otherwise it must be specified as true.

Presence or absence of IDENT field (required)

Specifies whether the RFC-3164-formatted syslog contains the IDENT field as true or false. false can be specified only when rfc3164 is specified as the syslog type, otherwise it must be specified as true.

Item Name	Description	Changeability	JP1/IM - What the user sets on the agent	JP1/IM - Initial value of Agent
time_format	Specifies the time format within 256 bytes. Used to crop logs with the name "time". Processes values according to the specified format. It can only be used if the time_type is string. The following formats are supported: %b:Abbreviated month (Jan,Feb,...) %d:Day (01~31) %H: 24-hour clock (00~23) %M:min(00~59) %m: Month number (01~12) %S:sec (00~60 (60 indicates leap second)) %Y: A number representing the year %N: fractional seconds If you specify an incorrect value, a warning message similar to the one shown below may be output to the Fluentd log, and the log may not be monitored. 2022-09-08 17:15:10 +0900 [warn]: #0 invalid line found file="C:/fluentd/install/log/app1/20220906_log1_utf8.txt" line="2022/12/3 12:34:56 jpcagt0 00004864 00008904 agent.cpp 572 KAVL99999-E \xE3\x82\xA8\xE3\x83\xA9\xE3\x83\xBC\xE3\x83\xA1\xE3\x83\x83\xE3\x82\xBB\xE3\x83\xBC\xE3\x82\xB8(2022/09/08 17:15:09.24) " error="invalid time format: value = 2022/12/3 12:34:56, error_class = ArgumentError, error = string doesn't match" If this parameter is omitted, the time set to JPC_LOG_TIME is the time when Fluentd detected the log message. If syslog is specified for type and this parameter is not specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format. If syslog is specified for type and auto is specified for message_format, specifies the RFC-3164 protocol time format. In this case, the RFC-4353 protocol time format is specified in rfc5424_time_format. The RFC-3164 protocol time format is "%b %d %H:%M:%S". If the output is time-stamped in seconds or less, change it to "%b %d %H:%M:%S.%N".	Changeable	Specify the time format as a regular expression according to the format of the log file to be monitored.	nil
rfc5424_time_format	Specifies the RFC-5424 protocol time format, up to 256 bytes. The following formats are supported: %b:Abbreviated month (Jan,Feb,...) %d:Day (01~31) %H: 24-hour clock (00~23) %M:min(00~59) %m: Month number (01~12) %S:sec (00~60 (60 indicates leap second)) %Y: A number representing the year %N: fractional seconds If you specify an incorrect value, a warning message similar to the one shown below may be output to the Fluentd log, and the log may not be monitored. 2023-03-24 13:18:27 +0900 [warn]: #0 invalid line found file="/home/ec2-user/fluentd_test/input_log/20230315_log1.txt" line="<16>1 2023-03-24T13:18:27.31+0900 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut=\"3\" eventSource=\"Application\" eventID=\"11211\"] Hi, from Fluentd!" error="invalid time format: value = 2023-03-24T13:18:27.31+0900, error_class = ArgumentError, error = string doesn't match" Use this parameter only if the message_format is specified as AUTO. If not specified, the time is parsed and extracted according to the regular expression time format described in "9.5.4(3)(g)Log parsing function (parse plug-in)" of the manual JP1/Integrated Management 3 - Manager Overview and System Design Guide.	Changeable	Specify the time format according to the format of the log file to be monitored.	--
message_format	Specifies the protocol format for syslog. You can specify RFC3164, RFC5424, or AUTO. The default is rfc3164. If the monitored syslog is output in RFC5424, specify RFC5424. Also, if the syslog to be monitored is logged using both RFC3164 and RFC5424 protocols, AUTO is specified. If auto is specified, the syslog parsing plug-in uses the message prefix to detect the format. If this parameter is not specified, or if an incorrect value is specified, an error is printed when Fluentd is started.	Changeable	Specify the log format according to the format of the log file to be monitored.	--
with_priority	Indicates whether RFC-3164 formatted syslogs contain a priority prefix as true or false. Specify true if the monitored log has a priority prefix such as [9]. If this parameter is not specified, the Fluentd log may display a warning message similar to the one shown below, and the log may not be monitored. 2023-03-24 14:15:01 +0900 [warn]: #0 pattern not matched: "Mar 24 14:15:01 192.168.0.1 fluentd[11111]: [error] Syslog test" If a value other than true or false is specified, an error is output when Fluentd is started.	Changeable	Specify according to the format of the log file to be monitored.	--
support_colonless_ident	Specifies whether RFC-3164 formatted syslogs contain the ident field as true or false. Used to monitor logs in RFC3164 format. Specifies false if the monitored log does not contain an ident field in the message. If this parameter is not specified, no error or warning messages are printed, and the monitored log may be parsed in the wrong format. Therefore, it is necessary to check whether JP1 events are issued in a normal format after starting Fluentd and adding logs. If a value other than true or false is specified, an error is output when Fluentd is started.	Changeable	Specify according to the format of the log file to be monitored.	--

(Legend) -: Not applicable

• For csv

  <parse>
    @type csv
    keys Array of field names for records
    delimiter ,
    parser_type Types of internal parsers
  </parse>

Array of field names for records (required)

Specifies the field names of the record in the form of an array.

Types of internal parsers (required)

Specifies the type of internal parser that parses logs in CSV format.

Item Name	Description	Changeability	JP1/IM - What the user sets on the agent	JP1/IM - Initial value of Agent
keys	Specify an array of record item names within 256 bytes. If this parameter is not specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.	Changeable	Specify according to the format of the log file to be monitored.	--
parser_type	Specifies the type of internal parser for parsing log lines, either normal or fast. If normal is specified, the Ruby CSV.parse_line method is used. If fast is specified, Fluentd's own lightweight implementation is used. The parser you use is several times faster than usual, but supports only typical patterns. The following formats are supported: # non-quoted value1,value2,value3,value4,value5 # quoted "value1","val,ue2","va,lu,e3","val ue4","" # escaped "message","mes""sage","""message""",,"""""" # mixed message,"mes,sage","me,ssa,ge",mess age,"" If this parameter is not specified, or if an incorrect value is specified, an error is output when Fluentd is started.	Changeable	If the format of the log file to be monitored is in the following format, specify fast. # non-quoted value1,value2,value3,value4,value5 # quoted "value1","val,ue2","va,lu,e3","val ue4","" # escaped "message","mes""sage","""message""",,"""""" # mixed message,"mes,sage","me,ssa,ge",mess age,"" If the format of the log file to be monitored does not match the above format, normal is specified.	--

(Legend) -: Not applicable

• For tsv

  <parse>
    @type tsv
    keys Array of field names for records
    delimiter \t,
  </parse>

Array of field names for records (required)

Specifies the field names of the record in the form of an array.

Item Name	Description	Changeability	JP1/IM - What the user sets on the agent	JP1/IM - Initial value of Agent
keys	Specify an array of record item names within 256 bytes. If this parameter is not specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.	Changeable	Specify according to the format of the log file to be monitored.	--

(Legend) -: Not applicable

• For ltsv

  <parse>
    @type ltsv
    Delimiter between items
    label_delimiter delimiter between label and value
  </parse>

Delimiter between items (required)

Specifies the delimiter between items. Specify one of the following:

- When the separator between items is a tab

    delimiter \t

- When the separator between items is one or more blanks

    delimiter_pattern /\s+/

delimiter between label and value (required)

Specifies the delimiter between the label and the value.

Item Name	Description	Changeability	JP1/IM - What the user sets on the agent	JP1/IM - Initial value of Agent
delimiter	Specifies the delimiter between items. The only delimiter that can be specified is "\t". If either this parameter or delimiter_pattern is specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.	Changeable	Specify according to the format of the log file to be monitored.	--
delimiter_pattern	In an LTSV format file, this is specified when the separator between entries is one or more spaces. The only delimiter that can be specified is "/\s+/". If either this parameter or delimiter is not specified, or if an incorrect value is specified, no error or warning message is output, and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.	Changeable	Specify according to the format of the log file to be monitored.	--
label_delimiter	Specifies the delimiter between the label and the value within 256 bytes. If this parameter is not specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.	Changeable	Specify according to the format of the log file to be monitored.	--

(Legend) -: Not applicable

[Attributes Settings] Section

Serup Attributes of JP1 events to be issued and Attribute value.

Logging Log file trap name (mandatory)

[Metric Settings] Same as the section description.

Event ID (optional)

Specifies Value to Setup for B. ID property of JP1 event. For details about Value that can be specified, see JP1/Base Operation Manual. The default Value is "00007601" (Event ID used for monitoring text-formatted log file definition file).

If this option is omitted, JP1 events are not issued.

Instead of specifying "ID Event ID", you can Setup Event ID according to Value of message property by specifying:

    ID "${
         if record['message'].match(/Regex 1/)
           'Event-ID1'
         elsif record['message'].match(/Regex 2/)
           'Event-ID2'
         elsif record['message'].match(/Regex 3/)
           'Event-ID3'
...
         else
           'Event-ID4'
         end}"

The conditional branch of Ruby determines Value of message property and turns Setup Event ID. In the above cases, if regular expression 1 is matched, Event ID is set to Value as specified in Event ID 1. If it doesn't match, it will Setup Event ID 2 if it matches the regular expression 2. If none of the matches are compared by the number specified in elsif, else statement Setup Value specified in Event ID 4 to Event ID. You can specify a maximum of 100 if and elsif statements.

Host name (optional)

[Metric Settings] Same as the section description.

If the specification is omitted, Attribute value of JP1_SOURCEHOST is not Setup and JP1 event is not Add to the correct IM management node.

You can also dynamically Setup the canonical Host name of the system by doing the following:

    JP1_SOURCEHOST ${Socket.gethostname}

Event level (optional)

Specifies Value to Setup for E.SEVERITY property of JP1 event. For details about Value that can be specified, see JP1/Base Operation Manual. The default Value is "Notice".

If this option is omitted, JP1 events are not issued.

Instead of specifying SEVERITY Event level, you can Setup Event level according to Value of message property by specifying:

    SEVERITY "${
         if record['message'].match(/Regex 1/)
           'Critical'
         elsif record['message'].match(/Regex 2/)
           'Error'
         elsif record['message'].match(/Regex 3/)
           'Warning'
...
         else
           'Notice'
         end}"

The conditional branch of Ruby determines Value of message property and turns Setup Event level. In the above cases, "Critical" is Setup to Event level if the regular expression 1 is matched. If it doesn't match, it will Setup "Error" if it matches against regular expression 2. Setup "Notice" in else construct to Event level if elsif matches none of the specified numbers. You can specify up to 100 statements of if and elsif.

Label-name of IM management node (optional)

[Metric Settings] Same as the section description.

If this option is omitted, JP1 events are not issued.

Any Attribute name Any Value (optional)

Specify this operand if you want to Add a JP1 event-attribute. For details about Attribute name that can be specified, see JP1/Base Operation Manual.

For Value, you can specify the captured name in the [Input Settings] section with the regular expressions to parse the logs.

For example, to capture with the name "NUMBER" and Setup to the property EXIT_CODE, you would specify:

EXIT_CODE ${record['NUMBER']}

You can Add more than one extended attribute, but no JP1 event is issued if the sum of the sizes of Value that Setup the extended attribute of JP1 event exceeds the limit.

For details about the upper limit of extended attributes, see 4.4.4(5)__transformEvent method.

[Inclusion Settings] Section

Specifies the conditions of the log to be monitored in a regular expression. If not specified, all logs are monitored. If an unmonitored log is output, the log is not converted to a JP1 event and is not output to Fluentd log.

In the default Setup, "#" is specified at the beginning of the line and is handled as Comment, so when specifying it, "#" is going to be deleted.

Logging Log file trap name (mandatory)

[Metric Settings] Same as the section description.

JP1 Events Attribute name (Optional)

Specifies Attribute name of JP1 event. For example, "MESSAGE". If the specification is omitted, Error occurs when Fluentd is started.

Regular expression of the log to monitor (optional)

Specifies a regular expression for Value of the attribute specified by Attribute name of JP1 event. If Value to be Match is included, monitoring is performed.

If the specification is omitted, Error occurs when Fluentd is started.

You can also specify a logical AND or OR condition for multiple regular expression patterns. For details about how to specify the log data, see "9.5.4(7) Log data extraction facility (grep plug-in)" in JP1/Integrated Management 3 - Manager Overview and System Design Guide.

[Exclusion Settings] Section

Specifies the conditions for logs that are not monitored, in regular expressions. If not specified, all logs are monitored. In the default Setup, "#" is specified at the beginning of the line and is handled as Comment, so when specifying it, "#" is going to be deleted.

Logging Log file trap name (mandatory)

[Metric Settings] Same as the section description.

JP1 Events Attribute name (Optional)

Specifies Attribute name of JP1 event. For example, "MESSAGE".

If the specification is omitted, Error occurs when Fluentd is started.

Regular expression of the log to monitor (optional)

Specifies a regular expression for Value of the attribute specified by Attribute name of JP1 event. If Value to be Match is included, monitoring is not performed.

If the specification is omitted, Error occurs when Fluentd is started.

You can also specify a logical AND or OR condition for multiple regular expression patterns. For details about how to specify the log data, see "9.5.4(7) Log data extraction facility (grep plug-in)" in JP1/Integrated Management 3 - Manager Overview and System Design Guide.

[Forward Settings] Section

Setup the regular expression of the log data to be converted into a JP1 event.

Logging Log file trap name (mandatory)

[Metric Settings] Same as the section description.

JP1 Events Attribute name (Optional)

Specifies Attribute name of JP1 event. The default Value is "SEVERITY".

If the specification is omitted, Error occurs when Fluentd is started.

regular expression for the log that publishes JP1 events (optional).

Specifies the condition for regular expressions that issue JP1 events for Attribute name specified in Value of the event.

The default Value is "Warning|Error|Critical|Alert|Emergency" and matches if Value of SEVERITY is greater than or equal to Warning.

If Value of the attribute contains a Value that Match the condition, the monitored log content is converted to JP1 events and Add to JP1/Base in Integrated manager host. The content of the monitored log is also output to Fluentd log. If you do not Match the condition, JP1 event is not issued and only logged in Fluentd.

If the specification is omitted, Error occurs when Fluentd is started.

To Page Top

Example definition

The following is an example of the condition and definitions for monitoring a textual logging File.

■Conditions

• Path of the logged File to monitor

  C:\Program Files (x86)\Hitachi\HNTRLib2\spool\*

• Logging Message

  6027 2022/08/25 17:45:50.219     jbssessionmgr    000018EC 00000FCC KAVA1497-I            jp1admin user has Login

• Log messages to monitor

  Monitor the logging Message where Message ID starts with KAVA.

• Value to Setup to MESSAGE

  Setup Message ID or later text in the logging Message.

• Value to Setup to SEVERITY

  Setup Value according to Event level of Message ID.

• Value to Setup to any Attribute name

  Setup the process-name (jbssessionmgr) contained in the log Message to Attribute name PROCESS_NAME.

■Definitions

<worker 0>
## [Metric Settings]
<source>
  @type exec
  command "echo {}"
  <parse>
    @type json
  </parse>
  run_interval 60s
  tag jpc_ima_metrics.tail.user_app_log
</source>
 
<filter jpc_ima_metrics.tail.user_app_log>
  @type record_transformer
  enable_ruby true
  <record>
    __name__ fluentd_logtrap_running
    instance hostA
    jp1_pc_nodelabel UserApplication
    jp1_pc_category applicationServer
    jp1_pc_logtrap_defname user_app_log_tail
    jp1_pc_trendname fluentd
    job jpc_fluentd
    jp1_pc_nodelabel_fluentd Log trapper(Fluentd)
    jp1_pc_addon_program JPC Fluentd
  </record>
</filter>
</worker>
<worker 1>
## [Input Settings]
<source>
  @type tail
  tag tail.user_app_log
  path C:/Program Files (x86)/Hitachi/HNTRLib2/spool/*
  follow_inodes true
  refresh_interval 60
  skip_refresh_on_startup false
  read_from_head false
  encoding "UTF-8"
  from_encoding "Shift_JIS"
  read_lines_limit 1000
  read_bytes_limit_per_second -1
  pos_file ../data/fluentd/tail/user_app_log.pos
  path_key tailed_path
  rotate_wait 5s
  enable_watch_timer false
  enable_stat_watcher true
  open_on_every_update false
  emit_unmatched_lines false
  ignore_repeated_permission_error false
  <parse>
    @type regexp
    expression /^([^ ]* +(?<time>[^ ]* [^ ]*) +(?<PROCESS>[^ ]*) +[^ ]* +[^ ]* +(?<message>.*))$/
    time_key time
    null_empty_string false
    estimate_current_event true
    keep_time_key false
    localtime true
    utc false
  </parse>
</source>
 
## [Attributes Settings]
<filter tail.user_app_log>
  @type record_transformer
  enable_ruby true
  auto_typecast true
  renew_record true
  
  <record>
    ID 00007601
    MESSAGE ${record["message"]}
    JP1_SOURCEHOST hostA
    JPC_LOG_TIME ${time.utc.to_i}
    PRODUCT_NAME /HITACHI/JP1/JPCCS2/LOGTRAP/UserApplication
    PPNAME /HITACHI/JP1/JPCCS2/LOGTRAP
    SEVERITY "${
         if record['message'].match(/^KAVA[1-9]*-E/)
           'Error'
         elsif record['message'].match(/^KAVA[1-9]*-W/)
           'Warning'
         elsif record['message'].match(/^KAVA[1-9]*-I/)
           'Information'
         else
           'Notice'
         end}"
    PLATFORM ${ if RUBY_PLATFORM.downcase =~ /mswin(?!ce)|mingw|cygwin|bccwin/; 'NT'; else 'UNIX'; end }
    OBJECT_TYPE LOGFILE
    OBJECT_NAME ${record['tailed_path']}
    ROOT_OBJECT_TYPE LOGFILE
    ROOT_OBJECT_NAME ${record['tailed_path']}
    JP1_TRAP_NAME ${tag_parts[1]}
    JPC_NODELABEL UserApplication
    PROCESS_NAME ${record['PROCESS']}
  </record>
</filter>
 
## [Inclusion Settings]
<filter tail.user_app_log>
  @type grep
  <regexp>
    key MESSAGE
    pattern /^KAVA[1-9]*-(I|W|E)/
  </regexp>
</filter>
 
## [Exclusion Settings]
#<filter tail.user_app_log>
#  @type grep
#  <exclude>
#    key 
#    pattern //
#  </exclude>
#</filter>
 
## [Forward Settings]
<match tail.user_app_log>
  @type rewrite_tag_filter
  <rule>
    key SEVERITY
    pattern /Warning|Error|Critical|Alert|Emergency/
    tag ${tag}.jp1event
  </rule>
  <rule>
    key SEVERITY
    pattern /.*/
    tag ${tag}.outputlog
  </rule>
</match>
 
<filter /tail\.user_app_log\.(jp1event|outputlog)/>
  @type record_transformer
  enable_ruby true
  auto_typecast true
  renew_record true
  <record>
    eventId ${record['ID']}
    xsystem true
    message ${record['MESSAGE']}
    attrs ${record}
  </record>
  remove_keys $.attrs.ID
  remove_keys $.attrs.MESSAGE
</filter>
</worker>

To Page Top