1.3.2 Managing the log file size

One of the factors that can cause insufficient disk capacity is an increase in the size of log files.

In the case of JP1/IM and JP1/Base, if you estimate the log file size in advance, there is no need to consider the possibility of increasing the log file size. This is because JP1/IM and JP1/Base use a method that outputs log files by switching between multiple log files.

For the OS and other products on the same host, check their specifications and make sure that their log file size will not increase.

Organization of this subsection

(1) Checking the log output by Intelligent Integrated Management Database

(1) Checking the log output by Intelligent Integrated Management Database

Depending on the PostgreSQL configuration, the following Intelligent Integrated Management Database logs are output. For details about setting PostgreSQL, see Intelligent Integrated Management Database configuration file (postgresql.conf) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

Storage destination

In Windows

Storage-destination-of-Intelligent-Integrated-Management-Database-data-file^#\log

In UNIX

Storage-destination-of-Intelligent-Integrated-Management-Database-data-file^#/log

#

See 2.7.1(1)(d) Where related files are stored in the JP1/Integrated Management 3 - Manager Overview and System Design Guide.
File name

- When JP1/IM - Manager 13-50 or later is newly installed

postgresql-rotate.log.serial-number-of-sectors^#

#

The serial-number-of-sectors is the serial number according to the number of sectors and the rotation of the log file. The number of log file sectors can be set from 0 to 100 in the log2_rotation_count parameter of Intelligent Integrated Management Database configuration file. The default is 7. If 0 is set, no log file is output.

The size of the log file rotation can be set in the range of 1B to 10GB in the log2_rotation_max_size parameter of the Intelligent Integrated Management Database configuration file.

For details about Intelligent Integrated Management Database configuration file, see Intelligent Integrated Management Database configuration file (postgresql.conf) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

- Upgrading from a version earlier than JP1/IM - Manager 13-50

postgresql-weekday-name^#.log

#

The weekday-name contains three alphabetic characters (either Mon, Tue, Wed, Thu, Fri, Sat, or Sun) that represent the day of the week in GMT timezone.

The log file for each day of the week name is rotated at 0:00 local time in the time zone set by the log_timezone parameter of Intelligent Integrated Management Database configuration file. There is also no upper limit on the size of the log file for each day of the week name.

Important

If you want to use postgresql-rotate.log.serial-number-of-sectors as a log file, you must change the parameter settings in Intelligent Integrated Management Database configuration file. When the logging_collector parameter is on, the log file is output. When log2 parameter is on, the postgresql-rotate.log.serial-number-of-sectors is used as the log file. When log2 parameter is off, the postgresql-weekday-name.log is used as the log file. If a value other than on or off is specified for log2 parameter, off is assumed. If you are using the postgresql-rotate.log.serial-number-of-sectors as a log file, you can set the log2_rotation_count parameter for the log file sectors and the log2_rotation_max_size parameter for the size of the log file rotation.

For details about Intelligent Integrated Management Database configuration file, see Intelligent Integrated Management Database configuration file (postgresql.conf) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.
Contents of the output

The format of the log output line is as follows:

YYYY-MM-DD HH:MM:SS.FFF GMT [Process-ID] SQL-Statement Logging-Message

In the "YYYY-MM-DD HH:MM:SS.FFF GMT [Process-ID] SQL-Statement" part, the timestamp, process ID, and SQL statement of the GMT time zone are output as prefixes.
Estimating the size of the log file

- For the postgresql-rotate.log.serial-number-of-sectors

Estimate based on the values set in the Intelligent Integrated Management Database configuration file (postgresql.conf) for the number of log file sectors and the size to be rotated. For details about Intelligent Integrated Management Database configuration file, see Intelligent Integrated Management Database configuration file (postgresql.conf) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

- For the postgresql-weekday-name.log

There is no upper limit on the size of the log file.

The following is an estimate of the log output size in normal operation:

Log output size in normal operation (one week's)

The output size of the log for one day x 7 (day)

Log output size for one day

= Log output size per 1 sample^#1 x scrapes per day^#2 x samples collected from every Exporter by a scrape^#3

#1

Indicates the output size of the log output by writing trend data.

For JP1/IM - Agent, 50 bytes are assumed.

#2

For JP1/IM - Agent, calculate based on scrape interval specified in Prometheus configuration file (jpc_prometheus_server.yml) scrape_interval. For details, see the description of scrape_interval in Prometheus configuration file (jpc_prometheus_server.yml) in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

If the specified scrape interval is 1 m (1 minute), it will be 1440 times (60 minutes x 24 hours).

#3

Indicates the sum of the number of metrics specified in the metric definition file for each Exporter targeted by the monitoring agent.

For details about metric definition file of each Exporters supported by JP1/IM - Agent, see the description of metric definition file of each Exporters in Chapter 2. Definition Files in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

In the case of Linux, you can reduce the disk space of log files by creating the following script (compressing or deleting logs for a certain period of time) ^# and running it regularly (once per day).

# Place it in the "/etc/cron.daily" directory.

Example of a shell script that compresses logs on a daily basis and deletes them on a 30-day basis:
```
#!/usr/bin/bash
LOGDIR=/var/opt/jp1imm/database/imgndb/log
LOGSAVE=/var/opt/jp1imm/log/imgndb
COMPRESS_DAY=1
REMOVE_DAY=30
COMPRESS_CMD=gzip
 
COMPRESS_MTIME=`expr $COMPRESS_DAY - 1`
#Search for ".log" files whose modification date is "$COMPRESS_MTIME" or later
COMPRESS_FILE=`find $LOGDIR -name '*.log' -daystart -type f -mtime +$COMPRESS_MTIME`
 
#Add the date to the end of the file to be compressed and compress it.
if [ "$COMPRESS_FILE" != "" ]
then
    for i in $COMPRESS_FILE
    do
       if [ -f ${i} ]
       then
          mv ${i} ${i}.`date '+%Y%m%d'`
          $COMPRESS_CMD ${i}.`date '+%Y%m%d'`
       fi
    done
fi
 
#Search for ".gz" files
MV_FILE=`find $LOGDIR -name '*.gz'`
 
#Move  compressed logfiles to "$LOGSAVE"
if [ "$MV_FILE" != "" ]
then
    for i in $MV_FILE
    do
       if [ -f ${i} ]
       then
          mv ${i} $LOGSAVE
       fi
    done
fi
 
REMOVEMTIME=`expr $REMOVE_DAY - 1`
#Search for ".gz" files whose modification date is "$REMOVEMTIME " or later
REMOVE_FILE=`find $LOGSAVE -name 'postgresql-*.gz' -daystart -type f -mtime +$REMOVEMTIME`
 
#Delete the file to be deleted
if [ "$REMOVE_FILE" != "" ]
then
    for i in $REMOVE_FILE
    do
        if [ -f ${i} ]
        then
            rm -f ${i}
        fi
    done
fi
```
When the above script is executed and operated, the estimated disk space of the log file will be approximately 20GB^# when the number of monitored items is 500.

#

The sum of the sizes of each of the following log files:

- The size of the log file of the day: about 4GB

- Size of the log file of the previous day (before compression): about 4GB

- The size of the log file after compression (400MB) x 30 days: about 12GB

Intelligent Integrated Management Database log is extremely large, so the data collection tool^# cannot collect it. If your case corresponds to the case described in " 12.3.1(1)(b) JP1 information" or " 12.3.1(2)(b) JP1 information", it must be collected individually by hand.

#

For details, see jim_log.bat (Windows only) and jim_log.sh (UNIX only) in Chapter 1. Commands in the JP1/Integrated Management 3 - Manager Command, Definition File and API Reference.

To Page Top