Monitoring text-formatted log file definition file (fluentd_@@trapname@@_tail.conf.template)
- Organization of this page
Format
<worker 0> ## [Metric Settings] <source> @type exec command "echo {}" <parse> @type json </parse> run_interval 60s tag jpc_ima_metrics.tail.log-monitoring-name </source> <filter jpc_ima_metrics.tail.log-monitoring-name> @type record_transformer enable_ruby true auto_typecast false <record> __name__ fluentd_logtrap_running instance host-name jp1_pc_nodelabel IM-management-node-label-name jp1_pc_category category-ID jp1_pc_logtrap_defname log-monitoring-name_tail jp1_pc_trendname fluentd job jpc_fluentd jp1_pc_nodelabel_fluentd Log trapper(Fluentd) jp1_pc_addon_program JPC Fluentd </record> </filter> </worker> <worker worker id> ## [Input Settings] <source> @type tail tag tail.log-monitoring-name path monitored-paths follow_inodes true refresh_interval 60 skip_refresh_on_startup false read_from_head read-the-logs-to-be-monitored-when-Fluentd-is-started-for-the-first-time-from-the-beginning # encoding "Fluentd-character-code" # from_encoding "character-codes-of-monitored-logs" read_lines_limit 1000 read_bytes_limit_per_second -1 pos_file ../data/fluentd/tail/log-monitoring-name.pos path_key tailed_path rotate_wait 5s enable_watch_timer enable-additional-watch-timers flush-interval-for-multiline-logs enable_stat_watcher true open_on_every_update false emit_unmatched_lines false ignore_repeated_permission_error false <parse> @type log-format settings-depending-on-the-log-format </parse> </source> ## [Attributes Settings] <filter tail.log-monitoring-name> @type record_transformer enable_ruby true auto_typecast false renew_record true <record> ID event-ID MESSAGE ${record["message"]} JP1_SOURCEHOST host-ame JPC_LOG_TIME ${time.utc.to_i} PRODUCT_NAME /HITACHI/JP1/JPCCS2/LOGTRAP/IM-management-node-label-name PPNAME /HITACHI/JP1/JPCCS2 SEVERITY severity PLATFORM ${ if RUBY_PLATFORM.downcase =~ /mswin(?!ce)|mingw|cygwin|bccwin/; 'NT'; else 'UNIX'; end } OBJECT_TYPE LOGFILE OBJECT_NAME ${record['tailed_path']} ROOT_OBJECT_TYPE LOGFILE ROOT_OBJECT_NAME ${record['tailed_path']} JP1_TRAP_NAME ${tag_parts[1]} JPC_NODELABEL IM-management-node-label-name any-attribute-name any-value </record> </filter> ## [Inclusion Settings] #<filter tail.log-monitoring-name> # @type grep # <regexp> # key attribute-name-of-JP1-event # pattern /regular-expression-of-logs-to-monitor/ # </regexp> #</filter> ## [Exclusion Settings] #<filter tail.log-monitoring-name> # @type grep # <exclude> # key attribute-name-of-JP1-event # pattern /regular-expressions-for-logs-that-you-do-not-want-to-monitor/ # </exclude> #</filter> ## [Forward Settings] <match tail.log-monitoring-name> @type rewrite_tag_filter <rule> key attribute-name-of-JP1-event pattern /regular-expression-for-logs-that-emit-JP1-events/ tag ${tag}.jp1event </rule> <rule> key SEVERITY pattern /.*/ tag ${tag}.outputlog </rule> </match> <filter /tail\.log-monitoring-name\.(jp1event|outputlog)/> @type record_transformer enable_ruby true auto_typecast true renew_record true <record> eventId ${record['ID']} xsystem true message ${record['MESSAGE']} attrs ${record} </record> remove_keys $.attrs.ID remove_keys $.attrs.MESSAGE </filter> </worker>
File
fluentd_@@trapname@@_tail.conf.template
fluentd_@@trapname@@_tail.conf.template.model (model file)
Storage directory
- ■Integrated agent host
-
In Windows:
-
For a physical host
Agent-path\conf\
-
For a logical host
shared-folder\jp1ima\conf\
In Linux:
-
For a physical host
/opt/jp1ima/conf/
-
For a logical host
shared-directory/jp1ima/conf/
-
Description
Definition File for monitoring text-formatted logging File.
Copy the template (fluentd_@@trapname@@_tail.conf.template) and change file name of Copy destination to fluentd_log-monitoring-name_tail.conf for use. File name must be unique within integrated agent host. For details on the location of fluentd_log-monitoring-name_tail.conf, see Appendix A.4(3) Integrated agent host (Windows) and Appendix A.4(4) Integrated agent host (Linux) in the JP1/Integrated Management 3 - Manager Overview and System Design Guide. log-monitoring-name must be between 1 and 30 characters long. Allowed characters are single-byte alphanumeric characters, "-" (hyphen), and "_" (underscore).
Create a monitor-definition file for each wrapped-around log file group that you want to monitor (or for each log file that does not wrap-around). JP1/IM - Agent creates a IM managed node for SID of target of monitoring according to the value specified in IM-management-node-label-name of the IM managed node in the monitoring definition file. If another monitoring definition file has the same IM-management-node-label-name, only one IM management node is created.
The text-based log file monitoring feature reads this definition file and analyzes the log that the application has written to the text-based log file. You can setup if you specify a condition for the analyzed information and the condition is met, the information to be converted to JP1 events or output to Fluentd logging file. For details about JP1 event to be issued, see 3.2.3(2) JP1 event issued that monitoring a textual log File.
Lines beginning with "#" are treated as Comment and do not affect the programming behavior.
The default definition in [Forward Settings] section is set to transform log data to JP1 event and transfer to JP1/IM - Manager when SEVERITY is worse than Warning.
When transforming log data to JP1 event and transferring it to JP1/IM - Manager, set SEVERITY so that its severity is equal or worse than Warning.
Character code
UTF-8 (without BOM)
Line feed code
In Windows: CR+LF
In Linux: LF
When the definitions are applied
This information is reflected in Fluentd operation when Fluentd service is restart.
If add or delete of definition files or value in [Metric Settings] section is changed, the change is reflected in tree view of the Integrated Operation Viewer window.
For details about application method, see 1.21.2(16) Creation and import of IM management node tree data (for Windows) (mandatory) in the JP1/Integrated Management 3 - Manager Configuration Guide.
Information that is specified
- <worker> directive
-
See the description of <worker> directive in Log monitoring common definition file (jpc_fluentd_common.conf).
- worker-id (optional)
-
Description
Changeability
What You Setup in Your JP1/IM - Agent
JP1/IM - Agent Defaults Value
Specifies the number of workers that Fluentd will start. Serves as an argument to the <worker N> directive. Valid values are integers from 1 to 128.
Can be changed
It must be specified so as not to duplicate the worker ID specified in the existing text log file monitoring definition file or the Windows event log monitoring definition file.
1
- [Metric Settings] section
-
Setup Value of label of sample that you want to send to JP1/IM - Manager's Trend data Management Database.
- log-monitoring-name (mandatory)
-
Specifies log-monitoring-name specified in the file name of copy destination as a string of 1 to 30 characters. Allowed characters are single-byte alphanumeric characters, "-" (hyphen), and "_" (underscore). The default value is "@@trapname@@".
Because you need to setup several locations in the file, use OS command/editor function to replace the "@@trapname@@" location with log-monitoring-name you want to specify.
If the specification is omitted, error occurs when Fluentd is started.
Note that log-monitoring-name must be setup as follows:
-
All log-monitoring-name in the same file are the same.
-
log-monitoring-name is unique for the monitoring text-formatted log file definition file and the Windows event log monitoring definition file
-
- host-name (optional)
-
Specify the host name to be monitored using characters 1 to 255 other than control characters. The default value is setup by integrated agent installers.
If the specification is omitted, IM management node is not created.
You can also dynamically setup the canonical host name of the system by doing the following:
instance ${Socket.gethostname}
- IM-management-node-label-name (optional)
-
Specifies the character string that integrated operation viewer displays on IM management node label. This is not a control character. When URL is encoded, the character string must be between 1 and 234 bytes (the upper limit for multibyte characters is 26). The default value is "Application".
If the specified information is invalid or omitted, IM management node is not going to be created.
You can specify the same IM management node label name in different monitoring definition files. Then, only one IM management node is created, and JP1 events in both of monitor-defined files are Add to one IM management node.
- category-ID (optional)
-
Specifies the category ID of IM management node corresponding to SID to be monitored for logging as a character 1 to 255 other than control characters. If the specification is omitted, "otherApplications" is assumed.
- [Input Settings] section
-
Setup the path to the text-formatted log File that you want to monitor and the regular expressions that parse the log Message.
- log-monitoring-name (mandatory)
-
Same as the section [Metric Settings] description.
- monitored-paths (required)
-
Description
Changeability
What You Setup in Your JP1/IM - Agent
Default Value for JP1/IM - Agent
Specify the path to read. You can specify multiple paths by separated by commas.
You can include * and strftime formats to dynamically add and delete the logging file you want to monitor. The list of log files is updated at refresh_interval intervals.
For specification examples, see (3) Text-format log file monitoring facility (tail plug-in) in 3.15.3 Log monitoring function by JP1/IM - Agent in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.
If you specify an incorrect path, Log Files logging is not read.
The following rules apply to programming:
-
Specify as an absolute path
-
Directories and File on network drives cannot be specified (for Windows)
-
Specify "/" instead of "\" as the path delimiter (in Windows)
-
Multiple paths can be specified.
-
You can specify "*" (wildcard).
-
You specify within 256 bytes.
-
The following path names cannot be specified.
- File with a leading "-" (hyphen)
- Folder name, directory name, or File name containing environment-dependent characters
- Space-directory-name (for Linux)
Installation Required
Specifies Log Files path.
Not applicable
-
- read-the-logs-to-be-monitored-when-Fluentd-is-started-for-the-first-time-from-the-beginning (optional)
-
Description
Changeability
What You Setup in Your JP1/IM - Agent
Default Value for JP1/IM - Agent
Specifies whether the log should start reading from the beginning, not the end, or from the last read position recorded in pos_file.
You can specify true or false.
Can be changed
If you want to read a log that was already Add at startup, change it to true.
false
- Fluentd-character-code (optional)
-
If character-codes-of-monitored-logs is UTF-8 or C (handled as Comment), specify the default setup (handled as Comment). Specify UTF-8 if character-codes-of-monitored-logs is not UTF-8 nor C (handled as Comment). In the default Setup, since "#" is specified at the beginning of the line and it is handled as Comment, "#" is turned Delete.
Description
Changeability
What You Setup in Your JP1/IM - Agent
Default Value for JP1/IM - Agent
Specifies the encoding in which to read the line.
By JP1/IM - Agent, in_tail outputs value of string in ASCII-8BIT encoding in default.
You can change it with the following options:
-
encoding changes the text to encoding.
-
If both encoding and from_encoding are specified, in_tail attempts to convert the jj, string to a encoding.
Can be changed
In JP1/IM - Agent, you can specify the following Value:
-
UTF-8
Not specified (Comment out)
# encoding "UTF-8"
-
- character-codes-of-monitored-logs (optional)
-
If character-codes-of-monitored-logs is UTF-8 or C, specify the default setup (handled as comment). If character-codes-of-monitored-logs is not UTF-8 nor C, specify the character code. In the default setup, since "#" is specified at the beginning of the line and it is handled as comment, "#" is turned delete.
Description
Changeability
What You Setup in Your JP1/IM - Agent
Default Value for JP1/IM - Agent
See the explanation of Fluentd-character-code (optional).
Can be changed
Specifies the character encoding of log files.
In JP1/IM - Agent, you can specify the following value:
-
UTF-16LE
-
UTF-16BE
-
Shift_JIS
-
Windows-31J
-
GB18030
Not specified (Comment out)
# encoding "Shift_JIS"
-
- enable-additional-watch-timers
-
Description
Changeability
What You Setup in Your JP1/IM - Agent
Default Value for JP1/IM - Agent
Specify true or false.
If false is specified for this parameter, the most recent log is not monitored when reading multiple lines of log. Therefore, if multiline is specified as the type of the parse plug-in, true is specified.
Specifying false for this parameter significantly reduces CPU and I/O consumption when tailing a large number of files on systems that support inotify.
Can be changed
Specify true only if multiline is specified as the type of the parse plugin.
false
- flush-interval-for-multiline-logs
-
Item Name
Description
Changeability
JP1/IM - What the user sets on the agent
JP1/IM - Initial value of Agent
multiline_flush_interval
Specify multiline_flush_interval item as flush interval for multiline logs.
If this item is not specified, the latest log is not monitored when multiline logs are monitored.
Therefore, when type of parsing plugin is set multiline, set it as following:
multiline_flush_interval 5s
Changeable
Set 5s only when type of parsing plugin is multiline.
5s
- log-format
-
Specifies the format for parsing the imported log.
The following formats can be specified:
type
Description
none (Default)
Read a one-line log as it is without parsing or structuring.
regexp
Reads a single-line log that matches the pattern specified by the regular expression.
multiline
Loads a multi-line log that matches the pattern specified by the regular expression.
syslog
Read the log output by syslog.
csv
Load logs in CSV format (comma delimited).
tsv
Loads logs in TSV format (tab-delimited).
ltsv
Import logs in LTSV format (labeled tab-delimited).
For examples of specifying logs in each format, see 3.15.3(3)(g) Log parsing function (parse plug-in) in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.
- settings-depending-on-the-log-format
-
Specify the entries according to the log-format.
-
If none
<parse> @type none message_key message time_key time null_empty_string false estimate_current_event true keep_time_key false </parse>
-
If regexp
<parse> @type regexp expression regular-expressions-to-parse-logs time_key time null_empty_string false estimate_current_event true keep_time_key false items-for-parsing-date-and-time-of-logs </parse>
regular-expression-to-parse-log (required)
When using the Named Capture fieature to trim character strings, one of names is necessary to be "message". If character strings named "message" is not trimmed, MESSAGE of JP1 event will be empty.
Specifies a regular expression and parses the contents of one line of the log. Use the Named Capture feature to trim a string named "message" that Setup to Message of JP1 event. For example: The default Value contains a regular expression that trims the entire line in "message". You can also trim with another name and Setup to any property of JP1 event.
items-for-parsing-date-and-time-of-logs
When a Date/time in the logging Message is trimmed as the name "time", it is set as value of JPC_LOG_TIME of JP1 event. When you trim a Date/time in the logging Message as the name "time", it is necessery to define the items for parsing date and time of logs. When you do not trim a Date/time, or define the items for parsing date and time of logs are not defined, the value of JPC_LOG_TIME will be Date/time when Fluentd monitored the log message.
Item Name
Description
Changeability
What You Setup in Your JP1/IM - Agent
Default Value for JP1/IM - Agent
expression
specifies the regular expressions matches for logging.
Regular expressions must be sandwiched between "/" (delimiters). If the delimiter is not used, an error is output. Regular expressions must specify at least one named capture (?<name>Regular expression for truncated logs).
Regular expressions can have i and m suffixes.
-
i(ignorecase)
Ignores the case of the match.
-
m (multi-line)
Creates a regular expression as a multi-line mode. ". " matches to a line break.
-
both
Specify both i and m.
If the log read does not Match the regular expression, the following Warning Message is printed in Fluentd log and the log is not going to be monitored.
2022-01-23 12:34:56 +0900 [warn]: #0 pattern not matched: "Error Message"
Can be changed
Setup according to Log Files logging format.
expression /^(?<message>.*)$/
time_type
Specify type of the date and time of log to be parsed.
Changeable
Specify type of time according to the format of the log file to be monitored.
Available time zone format:
-
unixtime
Seconds from Epoch (e.g. 1510544815)
-
string
Use format specified by time_format
--
time_format
Specify the time format within 256 bytes. Used to trim logs with the name "time". Processes values according to the specified format. It is available if the time_type is string.
The following formats are supported:
-
%b
Abbreviated month (Jan,Feb,...)
-
%d
Day (01~31)
-
%H
24-hour clock (00~23)
-
%M
min (00~59)
-
%m
Month number (01~12)
-
%S
sec (00~60 (60 indicatesleap second))
-
%Y
A number representing the year
-
%N
fractional seconds
If you specify an incorrect value, a warning message similar to the one shown below may be output to the Fluentd log, and the log may not be monitored.
2022-09-08 17:15:10 +0900 [warn]: #0 invalid line found file="C:/fluentd/install/log/app1/20220906_log1_utf8.txt" line="2022/12/3 12:34:56 jpcagt0 00004864 00008904 agent.cpp 572 KAVL99999-E \xE3\x82\xA8\xE3\x83\xA9\xE3\x83\xBC\xE3\x83\xA1\xE3\x83\x83\xE3\x82\xBB\xE3\x83\xBC\xE3\x82\xB8(2022/09/0817:15:09.24) " error="invalid timeformat: value = 2022/12/3 12:34:56, error_class = ArgumentError, error =string doesn't match"
If this parameter is omitted, the time set to JPC_LOG_TIME is the time when Fluentd detected the log message. If syslog is specified for type and this parameter is not specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.
Changeable
Specify the time format according to the format of the log file to be monitored.
--
localtime
Specify true because local time is used.
Not changeable
true
true
utc
Specify false because local time is used.
Not changeable
false
false
timezone
Date/time is parsed in specified timezone.
Changeable
Specify the timezone according to the format of the log file to be monitored. Available time zone format:
-
[+-]HH:MM
e.g. "+09:00"
-
[+-]HHMM
e.g. "+0900"
When timezone is specified, time_format is must be specified.
--
(Legend) -: Not applicable
-
For multiline
<parse> @type multiline format_firstline regular-expression-to-parse-the-first-line-log formatN regular-expression-to-parse-logs time_key time null_empty_string false estimate_current_event true keep_time_key false items-for-parsing-date-and-time-of-logs </parse>
regular-expression-to-parse-the-first-line-log (required)
Specify a regular expression to parse the contents of one log line. If the specified regular expression matches the contents of the log, the matched log line is read as the first line of a multi-line log.
regular-expression-to-parse-logs (required)
Similar to the description in "For regexp". N can be an integer from 1 to 20, and the specified regular expression is used to parse the contents of a multi-line log as line N.
items-for-parsing-date-and-time-of-logs
Same as description of "If regexp".
Item Name
Description
Changeability
JP1/IM - What the user sets on the agent
JP1/IM - Initial value of Agent
format_firstline
Specify the first line of the log as a regular expression.
The multiline parse plug-in parses multi-line logs. If multiline is specified as the type of the parse plug-in, formatN and format_firstline must be specified.
The maximum number of bytes that can be specified in a regular expression is 1023 bytes (excluding delimiters). Regular expressions must be sandwiched between "/" (delimiters). If the delimiter is not used, an error is output.
Changeable
Specify the first line of the log as a regular expression according to the format of the log file to be monitored.
--
formatN
Specify each line of the log as a regular expression.
Specifies the format of the multiline log. N is an integer from 1 to 20 that creates a list of regular expression formats.
The maximum number of bytes that can be specified in a regular expression is 1023 bytes (excluding delimiters). Regular expressions must be sandwiched between "/" (delimiters). If the delimiter is not used, an error is output.
If this parameter is not specified, an error is printed when Fluentd is invoked.
Changeable
Specify each line of the log as a regular expression according to the format of the log file to be monitored.
--
(Legend) -: Not applicable
-
For syslog
<parse> @type syslog time_type string time_format date-and-time-formats rfc5424_time_format syslog-date-and-time-format-in-RFC-5424-format message_format types-of-syslogs with_priority priority-prefix parser_type string support_colonless_ident presence-or-absence-of-ident-field time_key time null_empty_string false estimate_current_event true keep_time_key false localtime true utc false </parse>
date-and-time-formats (required)
Same as description of "If regexp". Specify a regular expression to parse the date and time in the log message. If auto is specified as types-of-syslogs, specify syslog-date-and-time-format-in-RFC-3164-format.
syslog-date-and-time-format-in-RFC-3164-format (optional)
Specify a regular expression to parse the date and time of the syslog in RFC-5424 format. Use this parameter only if types-of-syslogs is specified to auto.
types-of-syslogs (required)
Specify the type of syslog to be analyzed: rfc3164 (RFC-3164 format), rfc5424 (RFC-5424 format), or auto (both).
priority-prefix (required)
Indicates whether RFC-3164 formatted syslogs contain a priority prefix as true or false. false can be specified only when rfc3164 is specified as types-of-syslogs, otherwise it must be specified as true.
presence-or-absence-of-ident-field (required)
Specifies whether the RFC-3164-formatted syslog contains the IDENT field as true or false. false can be specified only when rfc3164 is specified as types-of-syslogs, otherwise it must be specified as true.
Item Name
Description
Changeability
JP1/IM - What the user sets on the agent
JP1/IM - Initial value of Agent
time_format
Same as description of "If regexp".
If syslog is specified for type and auto is specified for message_format, specifies the RFC-3164 protocol time format. In this case, the RFC-5424 protocol time format is specified in rfc5424_time_format. The RFC-3164 protocol time format is "%b %d %H:%M:%S". If the output is time-stamped in seconds or less, change it to "%b %d %H:%M:%S.%N".
Changeable
Specify the time format as a regular expression according to the format of the log file to be monitored.
--
rfc5424_time_format
Specifies the RFC-5424 protocol time format, up to 256 bytes.
The following formats are supported:
%b:Abbreviated month (Jan,Feb,...)
%d:Day (01~31)
%H: 24-hour clock (00~23)
%M:min(00~59)
%m: Month number (01~12)
%S:sec (00~60 (60 indicates leap second))
%Y: A number representing the year
%N: fractional seconds
If you specify an incorrect value, a warning message similar to the one shown below may be output to the Fluentd log, and the log may not be monitored.
2023-03-24 13:18:27 +0900 [warn]: #0 invalid line found file="/home/ec2-user/fluentd_test/input_log/20230315_log1.txt" line="<16>1 2023-03-24T13:18:27.31+0900 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut=\"3\" eventSource=\"Application\" eventID=\"11211\"] Hi, from Fluentd!" error="invalid time format: value = 2023-03-24T13:18:27.31+0900, error_class = ArgumentError, error = string doesn't match"
Use this parameter only if the message_format is specified as AUTO. If not specified, the time is parsed and extracted according to the regular expression time format described in 3.15.3(3)(g)Log parsing function (parse plug-in) of the JP1/Integrated Management 3 - Manager Overview and System Design Guide.
Changeable
Specify the time format according to the format of the log file to be monitored.
--
message_format
Specifies the protocol format for syslog. You can specify RFC3164, RFC5424, or AUTO. The default is rfc3164.
If the monitored syslog is output in RFC5424, specify RFC5424. Also, if the syslog to be monitored is logged using both RFC3164 and RFC5424 protocols, AUTO is specified.
If auto is specified, the syslog parsing plug-in uses the message prefix to detect the format.
If this parameter is not specified, or if an incorrect value is specified, an error is printed when Fluentd is started.
Changeable
Specify the log format according to the format of the log file to be monitored.
--
with_priority
Indicates whether RFC-3164 formatted syslogs contain a priority prefix as true or false.
Specify true if the monitored log has a priority prefix such as [9].
If this parameter is not specified, the Fluentd log may display a warning message similar to the one shown below, and the log may not be monitored.
2023-03-24 14:15:01 +0900 [warn]: #0 pattern not matched: "Mar 24 14:15:01 192.168.0.1 fluentd[11111]: [error] Syslog test"
If a value other than true or false is specified, an error is output when Fluentd is started.
Changeable
Specify according to the format of the log file to be monitored.
--
support_colonless_ident
Specifies whether RFC-3164 formatted syslogs contain the ident field as true or false. Used to monitor logs in RFC3164 format. Specifies false if the monitored log does not contain an ident field in the message.
If this parameter is not specified, no error or warning messages are printed, and the monitored log may be parsed in the wrong format. Therefore, it is necessary to check whether JP1 events are issued in a normal format after starting Fluentd and adding logs.
If a value other than true or false is specified, an error is output when Fluentd is started.
Changeable
Specify according to the format of the log file to be monitored.
--
(Legend) -: Not applicable
-
For csv
<parse> @type csv keys array-of-field-names-for-records delimiter , parser_type types-of-internal-parsers time_key time null_empty_string false estimate_current_event true keep_time_key false items-for-parsing-date-and-time-of-logs </parse>
array-of-field-names-for-records (required)
Specifies the field names of the record in the form of an array. One of field names is necessary to be "message" in order to set to MESSAGE of JP1 event. If character strings named "message" is not trimmed, MESSAGE of JP1 event will be empty.
types-of-internal-parsers (required)
Specifies the type of internal parser that parses logs in CSV format.
Item Name
Description
Changeability
JP1/IM - What the user sets on the agent
JP1/IM - Initial value of Agent
keys
Specify an array of record item names within 256 bytes.
If this parameter is not specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.
Changeable
Specify according to the format of the log file to be monitored.
--
parser_type
Specifies the type of internal parser for parsing log lines, either normal or fast.
If normal is specified, the Ruby CSV.parse_line method is used.
If fast is specified, Fluentd's own lightweight implementation is used. The parser you use is several times faster than usual, but supports only typical patterns. The following formats are supported:
# non-quoted
value1,value2,value3,value4,value5
# quoted
"value1","val,ue2","va,lu,e3","val ue4",""
# escaped
"message","mes""sage","""message""",,""""""
# mixed
message,"mes,sage","me,ssa,ge",mess age,""
If this parameter is not specified, or if an incorrect value is specified, an error is output when Fluentd is started.
Changeable
If the format of the log file to be monitored is in the following format, specify fast.
# non-quoted
value1,value2,value3,value4,value5
# quoted
"value1","val,ue2","va,lu,e3","val ue4",""
# escaped
"message","mes""sage","""message""",,""""""
# mixed
message,"mes,sage","me,ssa,ge",mess age,""
If the format of the log file to be monitored does not match the above format, normal is specified.
--
(Legend) -: Not applicable
items-for-parsing-date-and-time-of-logs
Same as description of "If regexp". Specify when array of field names for records has "time".
-
For tsv
<parse> @type tsv keys array-of-field-names-for-records delimiter "\t" time_key time null_empty_string false estimate_current_event true keep_time_key false items-for-parsing-date-and-time-of-logs </parse>
array-of-field-names-for-records (required)
Specifies the field names of the record in the form of an array.
Item Name
Description
Changeability
JP1/IM - What the user sets on the agent
JP1/IM - Initial value of Agent
keys
Specify an array of record item names within 256 bytes.
If this parameter is not specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.
Changeable
Specify according to the format of the log file to be monitored.
--
(Legend) -: Not applicable
items-for-parsing-date-and-time-of-logs
Same as description of "If regexp". Specify when array of field names for records has "time".
-
For ltsv
<parse> @type ltsv delimiter-between-items delimiter-pattern-between-items label_delimiter delimiter-between-label-and-value time_key time null_empty_string false estimate_current_event true keep_time_key false items-for-parsing-date-and-time-of-logs </parse>
delimiter-between-items delimiter-pattern-between-items (required)
Specifies the delimiter between items. Specify one of the following:
- When the separator between items is a tab
delimiter "\t"
- When the separator between items is one or more blanks
delimiter_pattern /\s+/
delimiter-between-label-and-value (required)
Specifies the delimiter between the label and the value.
Item Name
Description
Changeability
JP1/IM - What the user sets on the agent
JP1/IM - Initial value of Agent
delimiter
Specifies the delimiter between items. The only delimiter that can be specified is double-quoted "\t".
If either this parameter or delimiter_pattern is specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.
Changeable
Specify according to the format of the log file to be monitored.
--
delimiter_pattern
In an LTSV format file, this is specified when the separator between entries is one or more spaces. The only delimiter that can be specified is "/\s+/".
If either this parameter or delimiter is not specified, or if an incorrect value is specified, no error or warning message is output, and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.
Changeable
Specify according to the format of the log file to be monitored.
--
label_delimiter
Specifies the delimiter between the label and the value within 256 bytes.
If this parameter is not specified, or if an incorrect value is specified, no error or warning message is printed and the monitored log is parsed in the wrong format. Therefore, after starting Fluentd and adding logs, it is necessary to check whether JP1 events are issued in a normal format.
Changeable
Specify according to the format of the log file to be monitored.
--
(Legend) -: Not applicable
items-for-parsing-date-and-time-of-logs
Same as description of "If regexp". Specify when array of field names for records has "time".
-
- [Attributes Settings] section
-
Serup Attributes of JP1 events to be issued and Attribute value.
- log-monitoring-name (mandatory)
-
Same as description of [Metric Settings] section.
- event-ID (optional)
-
Specifies Value to Setup for B. ID property of JP1 event. For details about Value that can be specified, see JP1/Base Operation Manual. The default Value is "00007601" (Event ID used for monitoring text-formatted log file definition file).
If this option is omitted, JP1 events are not issued.
Instead of specifying "ID event-ID", you can setup event ID according to the value of message property by specifying:
ID "${ if record['message'].match(/regex-1/) 'event-ID1' elsif record['message'].match(/regex-2/) 'event-ID2' elsif record['message'].match(/regex-3/) 'event-ID3' ... else 'event-ID4' end}"
The conditional branch of Ruby determines value of message property and turns setup event ID. In the above cases, if regex-1 is matched, event ID is set to the value as specified in event-ID-1. If it doesn't match, it will setup event-ID-2 if it matches regex-2. If none of the matches are compared by the number specified in elsif, else statement setup value specified in event-ID-4. to event ID. You can specify a maximum of 100 if and elsif statements.
- host-name (optional)
-
Same as description of [Metric Settings] section.
If the specification is omitted, the attribute value of JP1_SOURCEHOST is not setup and JP1 event is not add to the correct IM management node.
You can also dynamically setup the canonical host name of the system by doing the following:
JP1_SOURCEHOST ${Socket.gethostname}
- severity (optional)
-
Specifies the value to setup for E.SEVERITY property of JP1 event. For details about the value that can be specified, see the JP1/Base User's Guide. The default value is "Notice".
If this option is omitted, JP1 events are not issued.
Instead of specifying "SEVERITY severity", you can setup the severity according to the value of message property by specifying:
SEVERITY "${ if record['message'].match(/regex-1/) 'Critical' elsif record['message'].match(/regex-2/) 'Error' elsif record['message'].match(/regex-3/) 'Warning' ... else 'Notice' end}"
The conditional branch of Ruby determines value of message property and turns setup severity. In the above cases, "Critical" is setup to severity if the regex-1 is matched. If it doesn't match, it will setup "Error" if it matches against regex-2. Setup "Notice" in else construct to severity if elsif matches none of the specified numbers. You can specify up to 100 statements of if and elsif.
- IM-management-node-label-name (optional)
-
Same as description of [Metric Settings] section.
If this option is omitted, JP1 events are not issued.
- any-attribute-name any-value (optional)
-
Specify this operand if you want to Add a JP1 event-attribute. For details about Attribute name that can be specified, see JP1/Base Operation Manual.
For Value, you can specify the captured name in the [Input Settings] section with the regular expressions to parse the logs.
For example, to capture with the name "NUMBER" and Setup to the property EXIT_CODE, you would specify:
EXIT_CODE ${record['NUMBER']}
You can Add more than one extended attribute, but no JP1 event is issued if the sum of the sizes of Value that Setup the extended attribute of JP1 event exceeds the limit.
For details about the upper limit of extended attributes, see 4.4.4(5)__transformEvent method.
- [Inclusion Settings] section
-
Specifies the conditions of the log to be monitored in a regular expression. If not specified, all logs are monitored. If an unmonitored log is output, the log is not converted to a JP1 event and is not output to Fluentd log.
In the default Setup, "#" is specified at the beginning of the line and is handled as Comment, so when specifying it, "#" is going to be deleted.
- log-monitoring-name (mandatory)
-
Same as description of [Metric Settings] section.
- attribute-name-of-JP1-event (optional)
-
Specifies the attribute name of JP1 event. For example, "MESSAGE". If the specification is omitted, error occurs when Fluentd is started.
- regular-expression-of-logs-to-monitor (optional)
-
Specifies a regular expression for the value of the attribute specified by the attribute name of JP1 event. If the value to be match is included, monitoring is performed.
If the specification is omitted, error occurs when Fluentd is started.
You can also specify a logical AND or OR condition for multiple regular expression patterns. For details about how to specify the log data, see 3.15.3(7) Log data extractor (grep plug-in) in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.
- [Exclusion Settings] section
-
Specifies the conditions for logs that are not monitored, in regular expressions. If not specified, all logs are monitored. In the default Setup, "#" is specified at the beginning of the line and is handled as Comment, so when specifying it, "#" is going to be deleted.
- log-monitoring-name (mandatory)
-
Same as description of [Metric Settings] section.
- attribute-name-of-JP1-event (optional)
-
Specifies the attribute name of JP1 event. For example, "MESSAGE".
If the specification is omitted, error occurs when Fluentd is started.
- regular-expression-of-logs-to-monitor (optional)
-
Specifies a regular expression for the value of the attribute specified by attribute-name-of-JP1-event. If value to be match is included, monitoring is not performed.
If the specification is omitted, error occurs when Fluentd is started.
You can also specify a logical AND or OR condition for multiple regular expression patterns. For details about how to specify the log data, see 3.15.3(7) Log data extractor (grep plug-in) in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.
- [Forward Settings] section
-
Setup the regular expression of the log data to be converted into a JP1 event.
- log-monitoring-name (mandatory)
-
Same as description of [Metric Settings] section.
- attribute-name-of-JP1-event (optional)
-
Specifies the attribute name of JP1 event. The default value is "SEVERITY".
If the specification is omitted, error occurs when Fluentd is started.
- regular-expression-for-logs-that-emit-JP1-events (optional)
-
Specifies the condition for regular expressions that issue JP1 events for the value of the attribute specified by attribute-name-of-JP1-event.
The default value is "Warning|Error|Critical|Alert|Emergency" and matches if the value of SEVERITY is greater than or equal to Warning.
If Value of the attribute contains a Value that Match the condition, the monitored log content is converted to JP1 events and Add to JP1/Base in Integrated manager host. The content of the monitored log is also output to Fluentd log. If you do not Match the condition, JP1 event is not issued and only logged in Fluentd.
If the specification is omitted, Error occurs when Fluentd is started.
In the [Attributes Settings] section, "Notice" is specified as the default SEVERITY. Therefore, the log monitoring result is not output as a JP1 event. It is output only in Fluentd log.
If you want to publish a log with a SEVERITY of "Notice" as a JP1 event, change the definition as shown in the underlined part:
pattern /Notice|Warning|Error|Critical|Alert|Emergency/
Example definition
The following is an example of the condition and definitions for monitoring a textual logging File.
■Conditions
-
Path of the logged file to monitor
C:\Program Files (x86)\Hitachi\HNTRLib2\spool\*
-
Logging message
6027 2022/08/25 17:45:50.219 jbssessionmgr 000018EC 00000FCC KAVA1497-I jp1admin user has Login
-
Log messages to monitor
Monitor the logging message where message ID starts with KAVA.
-
Value to setup to MESSAGE
Setup message ID or later text in the logging message.
-
Value to setup to SEVERITY
Setup value according to severity of message ID.
-
Value to setup to any attribute name
Setup the process-name (jbssessionmgr) contained in the log message to attribute name PROCESS_NAME.
■Definitions
<worker 0> ## [Metric Settings] <source> @type exec command "echo {}" <parse> @type json </parse> run_interval 60s tag jpc_ima_metrics.tail.user_app_log </source> <filter jpc_ima_metrics.tail.user_app_log> @type record_transformer enable_ruby true <record> __name__ fluentd_logtrap_running instance hostA jp1_pc_nodelabel UserApplication jp1_pc_category applicationServer jp1_pc_logtrap_defname user_app_log_tail jp1_pc_trendname fluentd job jpc_fluentd jp1_pc_nodelabel_fluentd Log trapper(Fluentd) jp1_pc_addon_program JPC Fluentd </record> </filter> </worker> <worker 1> ## [Input Settings] <source> @type tail tag tail.user_app_log path C:/Program Files (x86)/Hitachi/HNTRLib2/spool/* follow_inodes true refresh_interval 60 skip_refresh_on_startup false read_from_head false encoding "UTF-8" from_encoding "Shift_JIS" read_lines_limit 1000 read_bytes_limit_per_second -1 pos_file ../data/fluentd/tail/user_app_log.pos path_key tailed_path rotate_wait 5s enable_watch_timer false enable_stat_watcher true open_on_every_update false emit_unmatched_lines false ignore_repeated_permission_error false <parse> @type regexp expression /^([^ ]* +(?<time>[^ ]* [^ ]*) +(?<PROCESS>[^ ]*) +[^ ]* +[^ ]* +(?<message>.*))$/ time_key time null_empty_string false estimate_current_event true keep_time_key false localtime true utc false </parse> </source> ## [Attributes Settings] <filter tail.user_app_log> @type record_transformer enable_ruby true auto_typecast true renew_record true <record> ID 00007601 MESSAGE ${record["message"]} JP1_SOURCEHOST hostA JPC_LOG_TIME ${time.utc.to_i} PRODUCT_NAME /HITACHI/JP1/JPCCS2/LOGTRAP/UserApplication PPNAME /HITACHI/JP1/JPCCS2/LOGTRAP SEVERITY "${ if record['message'].match(/^KAVA[1-9]*-E/) 'Error' elsif record['message'].match(/^KAVA[1-9]*-W/) 'Warning' elsif record['message'].match(/^KAVA[1-9]*-I/) 'Information' else 'Notice' end}" PLATFORM ${ if RUBY_PLATFORM.downcase =~ /mswin(?!ce)|mingw|cygwin|bccwin/; 'NT'; else 'UNIX'; end } OBJECT_TYPE LOGFILE OBJECT_NAME ${record['tailed_path']} ROOT_OBJECT_TYPE LOGFILE ROOT_OBJECT_NAME ${record['tailed_path']} JP1_TRAP_NAME ${tag_parts[1]} JPC_NODELABEL UserApplication PROCESS_NAME ${record['PROCESS']} </record> </filter> ## [Inclusion Settings] <filter tail.user_app_log> @type grep <regexp> key MESSAGE pattern /^KAVA[1-9]*-(I|W|E)/ </regexp> </filter> ## [Exclusion Settings] #<filter tail.user_app_log> # @type grep # <exclude> # key # pattern // # </exclude> #</filter> ## [Forward Settings] <match tail.user_app_log> @type rewrite_tag_filter <rule> key SEVERITY pattern /Warning|Error|Critical|Alert|Emergency/ tag ${tag}.jp1event </rule> <rule> key SEVERITY pattern /.*/ tag ${tag}.outputlog </rule> </match> <filter /tail\.user_app_log\.(jp1event|outputlog)/> @type record_transformer enable_ruby true auto_typecast true renew_record true <record> eventId ${record['ID']} xsystem true message ${record['MESSAGE']} attrs ${record} </record> remove_keys $.attrs.ID remove_keys $.attrs.MESSAGE </filter> </worker>