Log monitoring common definition file (jpc_fluentd_common.conf)

Organization of this page

Format

@include jpc_fluentd_common_wevt_rendered.conf (Windows only)
@include jpc_fluentd_common_list.conf
 
## [System Settings]
<system>
  log_level log-level
  format text
  time_format %Y-%m-%d %H:%M:%S %z
  workers number-of-workers
</system>
 
<worker 0>
## [Remote Write Settings]
  <filter jpc_ima_metrics.**>
    @type record_transformer
    enable_ruby true
    auto_typecast true
    renew_record true
    
    <record>
      labels ${record}
      samples ${[[time.utc.to_i*1000,1]]}
    </record>
  </filter>
 
  <match jpc_ima_metrics.**>
    @type http
    headers {"accept":"application/json"}
    content_type application/json
    json_array false
    endpoint Trend data writing API of the integrated agent control base
    <buffer>
      flush_interval 60s
      disable_chunk_backup true
    </buffer>
  </match>
</worker>
 
<worker 1-worker id>
## [Output Settings]
  <match {tail.*.jp1event,winevt.*.jp1event}>
    @type copy
    copy_mode no_copy
    <store>
      @type http
      endpoint JP1 Event Conversion API of Unified Agent Control Platform
      headers {"accept":"application/json"}
      content_type application/json
      json_array true
      open_timeout 60
      read_timeout 60
      error_response_as_unrecoverable true
      retryable_response_codes [503]
      <buffer tag>
        @type file
        flush_interval flush-interval
        overflow_action Output plug-in behavior when the buffer queue is full
        retry_wait Retry interval
        path ../data/fluentd/buffer
        timekey_wait 600
        timekey_use_utc false
        timekey_zone local timezone
        chunk_limit_size 32MB
        total_limit_size Buffer size limit
        chunk_full_threshold 0.95
        queued_chunks_limit_size 1
        compress text
        flush_at_shutdown false
        flush_mode default
        flush_thread_count 1
        flush_thread_interval 1.0
        flush_thread_burst_interval 1.0
        delayed_commit_timeout 60
        retry_forever true
        retry_type periodic
        retry_wait 10s
        disable_chunk_backup false
      </buffer>
    </store>
    <store>
      @type relabel
      @label @STDOUT
    </store>
  </match>
 
## [Private Settings]
  <match {tail.*.outputlog,wevt.*.outputlog}>
    @type relabel
    @label @STDOUT
  </match>
 
  <label @STDOUT>
    <match {tail.**,wevt.**}>
      @type stdout
      <format>
        @type out_file
        time_type string
        time_format %Y-%m-%d %H:%M:%S %z
        localtime true
        utc false
      </format>
    </match>
  </label>
</worker>

To Page Top

File

jpc_fluentd_common.conf

jpc_fluentd_common.conf.model (model file)

To Page Top

Storage directory

■Integrated agent host

In Windows:

For a physical host

Agent-path\conf\
For a logical host

shared-folder\jp1ima\conf\

In Linux:

For a physical host

/opt/jp1ima/conf/
For a logical host

shared-directory/jp1ima/conf/

To Page Top

Description

A File for defining Common behavior in log monitoring function, such as HTTP POST request function and log output function.

Lines that begin with a "#" are treated as Comment and do not affect programming behavior.

To Page Top

Character code

UTF-8 (without BOM)

To Page Top

Line feed code

In Windows: CR+LF

In Linux: LF

To Page Top

When the definitions are applied

This information is reflected in Fluentd operation when Fluentd service is restarted.

To Page Top

Information that is specified

<worker> directive

By specifying the ID of the worker that Fluentd starts as an argument, the plug-in specified in the directive will be operated only on the worker with the specified ID.

worker id (optional): Specifies the ID of the worker that Fluentd will start. Serves as an argument to the worker N-M directive. N always specifies 1. The value specified for M specifies the integer specified for the number of workers minus one.

[System Settings] section

Performs a Setup that affects the operation of the entire log monitoring function.

Log Level (optional)

Specifies the log level for Fluentd.

Description	Changeability	What You Setup in Your JP1/IM - Agent	JP1/IM - Agent Defaults Value
The following log levels can be specified in ascending order of redundancy: fatal error warn info debug trace Default is "info". Fluentd outputs info,warn,error, and fatal logs in default.	Can be changed	Setup it according to the logs you want to check.	info

Description

Changeability

What You Setup in Your JP1/IM - Agent

JP1/IM - Agent Defaults Value

The following log levels can be specified in ascending order of redundancy:

fatal
error
warn
info
debug
trace

Default is "info". Fluentd outputs info,warn,error, and fatal logs in default.

Can be changed

Setup it according to the logs you want to check.

info

number-of-workers (optional)

Description	Changeability	What You Setup in Your JP1/IM - Agent	JP1/IM - Agent Defaults Value
Specifies the number of workers that Fluentd will start. Valid values are integers from 1 to 31.	Can be changed	Specify the total number of log monitoring common definition files, text log file monitoring definition files, Windows event log monitoring definition files, and log metrics definition files.	10

[Output Settings] section

Executes setup for outputting the log data monitored by log monitoring function.

If you are using the data delivery function to multiple manager hosts, additionally describe the secondary JP1/IM agent control base configuration. For setting examples, see Sample configuration for log monitoring common definition file (jpc_fluentd_common.conf) in 3.15.7(5)(f) Send settings for programs other than JP1/IM agent control base in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.

JP1/IM agent control base's JP1 event-translation API (optional)

Description	Changeability	What You Setup in Your JP1/IM - Agent	JP1/IM - Agent Defaults Value
Specifies the endpoint for HTTP request, up to 512 bytes. <Sample Setup> `# Use HTTP` `endpoint http://integrated-agent-host-name:20726/ima/api/v1/proxy/service/imdd/im/api_system/v1/events/transform` This parameter supports placeholders, so you can embed time, tags, and record fields. The <buffer> section is also required for the placeholders to work. <Sample Setup> `endpoint http://integrated-agent-host-name:20726/ima/api/v1/proxy/service/imdd/im/api_system/v1/events/transform` `${tag}-${key}` `<buffer tag,key>` `# buffer parameters` `</buffer>` Specifies the destination of the log data and metric in endpoint. For details, see (9) HTTP POST request function (http plug-in) of 3.15.3 Log monitoring function by JP1/IM - Agent in the JP1/Integrated Management 3 - Manager Overview and System Design Guide. If URL specified in endpoint is incorrect, Warning Message is issued and no metric or log data is sent. Therefore, after Fluentd starts, you must ensure that no Warnings appear in the log at POST of metric and log data.	Setup Required	Specifies JP1 event translation API's URL for JP1/IM agent control base to issue JP1 events. If you have changed the Port number of JP1/IM agent control base, you must change Setup.	`http://integrated-agent-host-name:20726/ima/api/v1/proxy/service/imdd/im/api_system/v1/events/transform`

Description

Changeability

What You Setup in Your JP1/IM - Agent

JP1/IM - Agent Defaults Value

Specifies the endpoint for HTTP request, up to 512 bytes.

# Use HTTP

endpoint http://integrated-agent-host-name:20726/ima/api/v1/proxy/service/imdd/im/api_system/v1/events/transform

This parameter supports placeholders, so you can embed time, tags, and record fields. The <buffer> section is also required for the placeholders to work.

endpoint http://integrated-agent-host-name:20726/ima/api/v1/proxy/service/imdd/im/api_system/v1/events/transform

${tag}-${key}

<buffer tag,key>

# buffer parameters

</buffer>

Specifies the destination of the log data and metric in endpoint. For details, see (9) HTTP POST request function (http plug-in) of 3.15.3 Log monitoring function by JP1/IM - Agent in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.

If URL specified in endpoint is incorrect, Warning Message is issued and no metric or log data is sent. Therefore, after Fluentd starts, you must ensure that no Warnings appear in the log at POST of metric and log data.

Setup Required

Specifies JP1 event translation API's URL for JP1/IM agent control base to issue JP1 events. If you have changed the Port number of JP1/IM agent control base, you must change Setup.

http://integrated-agent-host-name:20726/ima/api/v1/proxy/service/imdd/im/api_system/v1/events/transform

Flush interval (optional)

Description	Changeability	What You Setup in Your JP1/IM - Agent	JP1/IM - Agent Defaults Value
Specify how often to send log data stored in buffer to JP1/IM agent control base. Value that you can specify is 1s to 86,400s. The unit (s) is required. It also works without a specification.	Yes	Change the sending intervals.	60s

Description

Changeability

What You Setup in Your JP1/IM - Agent

JP1/IM - Agent Defaults Value

Specify how often to send log data stored in buffer to JP1/IM agent control base.

Value that you can specify is 1s to 86,400s. The unit (s) is required.

It also works without a specification.

Yes

Change the sending intervals.

60s

How output Plug-In Works When the Buffer Queue Is Full (optional)

Description	Changeability	What You Setup in Your JP1/IM - Agent	JP1/IM - Agent Defaults Value
Determines how output plug-in behaves when buffer queue is full. The following Values can be specified: throw_exception Prints Error to Fluentd logs. Even if a new log is output to the log File to be monitored, it is not added to the buffers. When the transmission is successful, the log output to the log File is trapped, but if it is wrapped and deleted, it is lost. drop_oldest_chunk Delete the oldest chunk to accept the new chunk. block Does not print errors to Fluentd logs. Even if a new log is output to the log file to be monitored, it is not added to the buffers. When the transmission becomes successful, the log output to the log file is trapped, but if it has been wrapped and deleted, it is lost.	Yes	Determines how output plug-in behaves when buffer queue is full.	throw_exception

Description

Changeability

What You Setup in Your JP1/IM - Agent

JP1/IM - Agent Defaults Value

Determines how output plug-in behaves when buffer queue is full. The following Values can be specified:

throw_exception

Prints Error to Fluentd logs. Even if a new log is output to the log File to be monitored, it is not added to the buffers. When the transmission is successful, the log output to the log File is trapped, but if it is wrapped and deleted, it is lost.
drop_oldest_chunk

Delete the oldest chunk to accept the new chunk.
block

Does not print errors to Fluentd logs. Even if a new log is output to the log file to be monitored, it is not added to the buffers. When the transmission becomes successful, the log output to the log file is trapped, but if it has been wrapped and deleted, it is lost.

Yes

Determines how output plug-in behaves when buffer queue is full.

throw_exception

Retry interval (optional)

Description	Changeability	What You Setup in Your JP1/IM - Agent	JP1/IM - Agent Defaults Value
Specifies the retry interval for chunks that have failed to be sent. Value that you can specify is 1s to 600s. The unit (s) is required. If Value is specified for this parameter in a blank or invalid format, the retry interval is about 2 seconds in Winsows environment and about 0.02 seconds in Linux environment.	Yes	Specify the retry interval.	10s

Description

Changeability

What You Setup in Your JP1/IM - Agent

JP1/IM - Agent Defaults Value

Specifies the retry interval for chunks that have failed to be sent.

Value that you can specify is 1s to 600s. The unit (s) is required.

If Value is specified for this parameter in a blank or invalid format, the retry interval is about 2 seconds in Winsows environment and about 0.02 seconds in Linux environment.

Yes

Specify the retry interval.

10s

Buffer size limit (optional)

Description	Changeability	What You Setup in Your JP1/IM - Agent	JP1/IM - Agent Defaults Value
Specifies the size limit for the instance of the buffer plugin. Possible values are 32MB~65536MB. If the total size of the buffers stored reaches this threshold, all add operations fail with an error and data is lost. The buffer uses the amount of disk space in the environment where JP1/IM - Agent is set up. The total size of the buffer is estimated by the following formula. "0.015MB * Number of JP1 events to buffer"	Yes	Specifies the size limit for the instance of the buffer plugin.	4096MB

Description

Changeability

What You Setup in Your JP1/IM - Agent

JP1/IM - Agent Defaults Value

Specifies the size limit for the instance of the buffer plugin.

Possible values are 32MB~65536MB.

If the total size of the buffers stored reaches this threshold, all add operations fail with an error and data is lost.

The buffer uses the amount of disk space in the environment where JP1/IM - Agent is set up. The total size of the buffer is estimated by the following formula.

"0.015MB * Number of JP1 events to buffer"

Yes

Specifies the size limit for the instance of the buffer plugin.

4096MB

[Remote Write Settings] section

Fluentd performs a Setup to send sample to Trend data Management Database of JP1/IM - Manager.

JP1/IM agent control base Trend Data Write API (optional)

Description	Changeability	What You Setup in Your JP1/IM - Agent	JP1/IM - Agent Defaults Value
Specifies the endpoint for HTTP request, up to 512 bytes. If you use HTTPS, use https prefix. <Sample Setup> `# Use HTTP` `endpoint http://example.com/api` `# Use HTTPS. You can set additional HTTPS parameters like tls_xxx` `endpoint https://example.com/api` This parameter supports placeholders, so you can embed time, tag, and record fields. The <buffer> section is also required for the placeholders to work. <Sample Setup> `endpoint http://example.com/api/${tag}-${key}` `<buffer tag,key>` `# buffer parameters` `</buffer>` Specifies the destination of the log data and metric in endpoint. For details, see (9) HTTP POST request function (http plug-in) of 3.15.3 Log monitoring function by JP1/IM - Agent in the JP1/Integrated Management 3 - Manager Overview and System Design Guide. If URL specified in endpoint is incorrect, Warning Message is issued and no metric or log data is sent. Therefore, after Fluentd starts, you must ensure that no Warnings appear in the log at POST of metric and log data.	Setup Required	Specifies URL of the trend data write API for JP1/IM agent control base to send metric to JP1/IM - Manager. If you have changed Port number of JP1/IM agent control base, you must change Setup.	`http://integrated-agent-host-name:20727/ima/api/v1/proxy/service/promscale/write`

Description

Changeability

What You Setup in Your JP1/IM - Agent

JP1/IM - Agent Defaults Value

Specifies the endpoint for HTTP request, up to 512 bytes. If you use HTTPS, use https prefix.

# Use HTTP

endpoint http://example.com/api

# Use HTTPS. You can set additional HTTPS parameters like tls_xxx

endpoint https://example.com/api

This parameter supports placeholders, so you can embed time, tag, and record fields. The <buffer> section is also required for the placeholders to work.

endpoint http://example.com/api/${tag}-${key}

<buffer tag,key>

# buffer parameters

</buffer>

Setup Required

Specifies URL of the trend data write API for JP1/IM agent control base to send metric to JP1/IM - Manager. If you have changed Port number of JP1/IM agent control base, you must change Setup.

http://integrated-agent-host-name:20727/ima/api/v1/proxy/service/promscale/write

[Private Settings] section

Cannot be edited.

To Page Top

Example definition

## [System Settings]
<system>
  log_level info
  format text
  time_format %Y-%m-%d %H:%M:%S %z
  workers 10
</system>
 
<worker 0>
## [Remote Write Settings]
  <filter jpc_ima_metrics.**>
    @type record_transformer
    enable_ruby true
    auto_typecast true
    renew_record true
  
    <record>
      labels ${record}
      samples ${[[time.utc.to_i*1000,1]]}
    </record>
  </filter>
  <match jpc_ima_metrics.**>
    @type http
    headers {"accept":"application/json"}
    content_type application/json
    json_array false
    endpoint http://integrated-agent-host-name:20727/ima/api/v1/proxy/service/promscale/write
    <buffer>
      flush_interval 60s
      disable_chunk_backup true
    </buffer>
  </match>
</worker>
 
<worker 1-9>
  ## [Output Settings]
  <match {tail.*.jp1event,wevt.*.jp1event}>
    @type copy
    copy_mode no_copy
    <store>
      @type http
      endpoint http://integrated-agent-host-name:20726/ima/api/v1/proxy/service/imdd/im/api_system/v1/events/transform
      headers {"accept":"application/json"}
      content_type application/json
      json_array true
      open_timeout 60
      read_timeout 60
      error_response_as_unrecoverable true
      retryable_response_codes [503]
      <buffer tag>
        @type file
        flush_interval 5s
        overflow_action throw_exception
        retry_wait 10s
        path ../data/fluentd/buffer
        timekey_wait 600
        timekey_use_utc false
        timekey_zone local timezone
        chunk_limit_size 32MB
        total_limit_size 4096MB
        chunk_full_threshold 0.95
        queued_chunks_limit_size 1
        compress text
        flush_at_shutdown false
        flush_mode default
        flush_thread_count 1
        flush_thread_interval 1.0
        flush_thread_burst_interval 1.0
        delayed_commit_timeout 60
        retry_forever true
        retry_type periodic
        retry_wait 10s
        disable_chunk_backup false
      </buffer>
    </store>
    <store>
      @type relabel
      @label @STDOUT
    </store>
  </match>
 
## [Private Settings]
  <match {tail.*.outputlog,wevt.*.outputlog}>
    @type relabel
    @label @STDOUT
  </match>
 
  <label @STDOUT>
    <match {tail.**,wevt.**}>
      @type stdout
      <format>
        @type out_file
        time_type string
        time_format %Y-%m-%d %H:%M:%S %z
        localtime true
        utc false
      </format>
    </match>
  </label>   
</worker>

To Page Top