For Linux(R) (x86) Systems HA Monitor Cluster Software

3000-9-201-40(E)

Notices

Relevant program products

For the Red Hat(R) Enterprise Linux(R) server 6(x86), Red Hat(R) Enterprise Linux(R) server 6(x86_64), Red Hat(R) Enterprise Linux(R) server 7(x86_64), Red Hat(R) Enterprise Linux(R)server 8(x86_64), Oracle Linux 7(x86_64)#, Oracle Linux 8(x86_64)# operating systems:

P-852C-E112 HA Monitor 01-74

For the Red Hat(R) Enterprise Linux(R) AS 3(x86), Red Hat(R) Enterprise Linux(R) AS 4(x86), Red Hat(R) Enterprise Linux(R) ES 3(x86), Red Hat(R) Enterprise Linux(R) ES 4(x86), Red Hat(R) Enterprise Linux(R) AS 3(AMD64 & Intel EM64T), Red Hat(R) Enterprise Linux(R) AS 4(AMD64 & Intel EM64T), Red Hat(R) Enterprise Linux(R) ES 4(AMD64 & Intel EM64T), Red Hat(R) Enterprise Linux(R) 5(x86), Red Hat(R) Enterprise Linux(R) 5(AMD/Intel 64), Red Hat(R) Enterprise Linux(R) 5 Advanced Platform(x86), Red Hat(R) Enterprise Linux(R) 5 Advanced Platform(AMD/Intel 64) operating systems:

P-9S2C-E211 HA Monitor Extension 01-00

This manual can be used for products other than the product shown above. For details, see the Release Notes.

#

This product is supported only in an Oracle Cloud Infrastructure (OCI) environment.

Trademarks

HITACHI，BladeSymphony，Cosminexus，HA Monitor，HiRDB，JP1，OpenTP1，ServerConductor，uCosminexus，Virtage are either trademarks or registered trademarks of Hitachi, Ltd. in Japan and other countries.

AIX is a trademark of International Business Machines Corporation, registered in many jurisdictions worldwide.

Amazon Web Services, the “Powered by AWS” logo, and AWS are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.

AMD is a trademark (or registered trademark) of Advanced Micro Devices, Inc.

IBM is a trademark of International Business Machines Corporation, registered in many jurisdictions worldwide.

Intel is a trademark of Intel Corporation or its subsidiaries.

Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

Microsoft is a trademark of the Microsoft group of companies.

Microsoft, Windows are trademarks of the Microsoft group of companies.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Red Hat is a registered trademark of Red Hat, Inc. in the United States and other countries.

Red Hat Enterprise Linux is a registered trademark of Red Hat, Inc. in the United States and other countries.

UNIX is a trademark of The Open Group.

Other company and product names mentioned in this document may be the trademarks of their respective owners.

Restrictions

Information in this document is subject to change without notice and does not represent a commitment on the part of Hitachi. The software described in this manual is furnished according to a license agreement with Hitachi. The license agreement contains all of the terms and conditions governing your use of the software and documentation, including all warranty rights, limitations of liability, and disclaimers of warranty.

Material contained in this document may describe Hitachi products not available or features not available in your country.

No part of this material may be reproduced in any form or by any means without permission in writing from the publisher.

Printed in Japan.

Edition history

Dec. 2021: 3000-9-201-40(E)

Copyright

All Rights Reserved. Copyright (C) 2012, 2021, Hitachi, Ltd.

Preface

This manual describes HA Monitor functions, settings, and methods of operation. After reading this manual, the reader can expect to be able to use HA Monitor to design, configure, and correctly operate a hot-standby configuration.

This manual covers both HA Monitor, a product that increases system availability by providing hot standby operation, and HA Monitor Extension, an optional product of HA Monitor.

Intended readers

This manual is intended for system administrators, operators, and programmers. The manual assumes that system administrators will perform system installation, design, and configuration; that operators will perform the day-to-day operations on a configured system; and that programmers will create user programs.

Readers of this manual must have:

A basic knowledge of the applicable hardware

A basic knowledge of the applicable operating system

A basic knowledge of a program for setting up system switchover configurations

A basic knowledge of C-language syntax for coding user programs

If you use OpenTP1 or HiRDB for setting up your system switchover configuration, we recommend that you first read the relevant OpenTP1 or HiRDB manuals.

Conventions: Diagrams

This manual uses the following conventions in diagrams:

Conventions: Fonts and symbols

The following table explains the fonts used in this manual:

Font

Convention

Bold

Bold type indicates text on a window, other than the window title. Such text includes menus, menu options, buttons, radio box options, or explanatory labels. For example:

From the File menu, choose Open.

Click the Cancel button.

In the Enter name entry box, type your name.

Italics

Italics are used to indicate a placeholder for some actual text to be provided by the user or system. For example:

Write the command as follows:

copy source-file target-file

The following message appears:

A file was not found. (file = file-name)

Italics are also used for emphasis. For example:

Do not delete the configuration file.

Code font

A code font indicates text that the user enters without change, or text (such as messages) output by the system. For example:

At the prompt, enter dir.

Use the send command to send mail.

The following message is displayed:

The password is incorrect.

The following table explains the symbols used in this manual:

Symbol

Convention

|

In syntax explanations, a vertical bar separates multiple items, and has the meaning of OR. For example:

A|B|C means A, or B, or C.

{ }

In syntax explanations, curly brackets indicate that only one of the enclosed items is to be selected. For example:

{A|B|C} means only one of A, or B, or C.

[ ]

In syntax explanations, square brackets indicate that the enclosed item or items are optional. For example:

[A] means that you can specify A or nothing.

[B|C] means that you can specify B, or C, or nothing.

...

In coding, an ellipsis (...) indicates that one or more lines of coding are not shown for purposes of brevity.

In syntax explanations, an ellipsis indicates that the immediately preceding item can be repeated as many times as necessary. For example:

A, B, B, ... means that, after you specify A, B, you can specify B as many times as necessary.

Conventions: Functional differences depending on platform

HA Monitor supports a number of platforms. Depending on the platform used, there might be differences in the configuration of and the names used for hardware items. This manual uses the conventions shown below to indicate text that applies only to a particular platform. In explanations that apply to all, a generic term is used for a type of hardware. When you read this manual, replace this generic term with the name of the actual hardware that you are using.

Convention

Meaning

Hardware name used

For BladeSymphony or (BladeSymphony)

Used for the BladeSymphony Integrated Service Platform series

SVP

Failure management processor (general term)

For HA8000 or (HA8000)

Used for the Hitachi Advanced Server HA8000 series

Model xK or earlier

SVP

Models xL to xM

BMC

Model xN or later

(incl. HA8000V)

Management server or BMC

Conventions: Version numbers

The version numbers of Hitachi program products are usually written as two sets of two digits each, separated by a hyphen. For example:

Version 1.00 (or 1.0) is written as 01-00.

Version 2.05 is written as 02-05.

Version 2.50 (or 2.5) is written as 02-50.

Version 12.25 is written as 12-25.

The version number might be shown on the spine of a manual as Ver. 2.00, but the same version number would be written in the program as 02-00.

Important notes on this manual

The following functions are explained but are not supported:

HA Booster

hab_discnt_atend

hab_gid

TP1/EE

Linux Tough Dump

OS panic

Control group

HABST

server_type

HA Monitor Power Off AlertTransfer

RV3000

HA Monitor in public cloud environments

Part 1: Overview

1 Overview of HA Monitor

This chapter describes the purpose of HA Monitor; provides an overview of hot standby, its main function; and describes the types of hot standby. It also explains the system modes for employing HA Monitor and the operating environments.

1.1 Purpose of HA Monitor

The purpose of HA Monitor is to improve system reliability and availability by enabling switching of systems, including programs.

System reliability and availability are especially important when a UNIX system is used for a key business activity. A conventional system achieves reliability by deploying dual resources, such as disk devices or LANs. HA Monitor enables you to achieve not only dual resources but also dual server systems.

If a server performing job processing fails while you are using HA Monitor, switchover to a standby server system can occur immediately and automatically. This enables you to improve system reliability and availability without requiring intervention by the operator.

The following figure provides an overview of HA Monitor.

Figure 1‒1: Overview of HA Monitor

In this figure, server systems 1 and 2 constitute a hot-standby configuration. If an error occurs in the program running on server system 1, HA Monitor switches over to the program on server system 2. This minimizes the amount of time jobs are suspended because of the error.

Depending on the functions, some hardware is required to use HA Monitor, such as a monitoring path and a reset path. For details about the hardware and software requirements for HA Monitor, see 1.6 Operating environment for HA Monitor.

1.2 Hot standby

HA Monitor uses two systems to protect against system failures. One system performs job processing while the other system stands by. These systems are called hosts. The programs that execute jobs in these systems (hosts) are called servers. Hot standby is a function that transfers jobs to the standby host or server in the event the host executing the jobs or the server fails.

1.2.1 Hosts

A host is a collective name for the entire system, including the hardware required for job processing, the programs that are executed, and the communication devices.

The available combinations of hosts are the active and standby systems and the primary and secondary systems.

(1) Active system and standby system

The host that performs job processing on a server (host containing the running server) is called the active system for that server, and the host that is standing by in case of a failure (host containing the standby server) is called the standby system for that server.

The server currently performing job processing is called the active server, and the server standing by in case of a failure is called the standby server.

(2) Primary system and secondary system

To identify the two separate hosts during system configuration and environment setup, the host that first starts a server as the active system is called the primary system for that server, and the host that starts as the standby system is called the secondary system for that server. This relationship between primary and secondary systems remains unchanged, even when control is passed between the active and standby systems after system startup.

1.2.2 Flow of hot standby

This subsection explains the operation flow from performance of the hot standby operation to restarting the jobs after HA Monitor has detected a failure.

When hot standby occurs because of a failure, the standby system inherits job processing and becomes the active system. The active system where the failure occurred becomes the standby system that will be available in the event of a subsequent failure. Thereafter, each time a failure occurs, hot standby occurs to repeat job execution and standby so that job processing can always be performed on one of the hosts.

The following figure shows the hot standby operation.

Figure 1‒2: Hot standby operation

1.2.3 Server

HA Monitor performs the hot standby operation in units of servers. The term server is used in the meaning of a program.

(1) Types of servers

Servers are classified into the following two types, depending on the program:

Program with an HA Monitor interface

This type includes programs that have an HA Monitor interface, such as OpenTP1 (TP1/Server Base) and HiRDB.

Program without an HA Monitor interface

This type refers to all programs other than the programs with an HA Monitor interface. Examples of this type of program are JP1, Cosminexus, Oracle, and UAPs.

(2) Server operating methods

The server operating methods include the server mode and the monitor mode.

The following table compares the server mode and the monitor mode.

Table 1‒1: Comparison of server mode and monitor mode

Comparison item

Server mode

Monitor mode

OpenTP1 (TP1/Server Base) or HiRDB

Failures detected by HA Monitor

Server failures

Host failures

Server failures#1

Host failures

UAP failures#2

Server monitoring

Performed by HA Monitor

Performed by the user#1

Unsupported HA Monitor functions

None

Detection of server slowdown#3

Host pair shutdown#3

Specification of hot standby type during grouped-system switchover

Some user commands

Acquisition of server monitoring history

Advanced startup of the server in the standby system

...

Required settings

Environment settings for server support

Environment settings for server support

Environment settings for programs in the monitor mode#2

Creation of a server start command

(If necessary) creation of server termination and server monitoring commands

Creation of program restart commands (when necessary)#2

Including APIs in UAPs#2

Unit of one server

One program

One or multiple programs

#1: If a server monitoring command has been created by the user in advance, the user can then monitor the server in the monitor mode and detect server failures. When the monitor-mode program management function is used, HA Monitor monitors UAPs. For details about the program management function, see 3.6 Function for controlling programs (monitor mode).

#2: This applies only to programs that use the monitor-mode program management function.

#3: If a monitor-mode server is monitored by specifying the ptrlcmd_ex operand in the server environment definition, this function can be used (this event can be detected).

For programs with an HA Monitor interface

You can choose either the server mode or the monitor mode as the server operating method. We recommend that you choose the server mode when you use a program with an HA Monitor interface.

A server that runs in the server mode is also called a server with an HA Monitor interface. However, in this manual, such a server is called a server in the server mode.

For programs without an HA Monitor interface

Programs without an HA Monitor interface run in the monitor mode. No other server operating method is supported for them.

A server that runs in the monitor mode is also called a server without an HA Monitor interface. However, in this manual, such a server is called a server in the monitor mode.

1.2.4 Failures detected by HA Monitor

This subsection explains the failures that are detected by HA Monitor.

HA Monitor performs hot-standby switchover when detecting server failures that occur on servers, host failures that occurs on hosts, or resource failures that occur on resources (LANs or disks).

(1) Range of monitoring performed by HA Monitor

To detect server failures, host failures, and resource failures, HA Monitor monitors the following items:

Servers

Hosts

Resources (LANs and disks#)

#: System disks and disks for business use are monitored.

The following figure shows the range of monitoring performed by the HA Monitors in the system configuration.

Figure 1‒3: Range of monitoring performed by HA Monitor

(2) Server failures

Server failures can be classified as those that the server itself detects and those that the server cannot detect.

(a) Failures that the server itself detects

Failures that the server detects include the following:

Server's own logical errors

Any status that disables server operation because of a failure in a resource (such as a disk device)

In the event of a failure that the server itself detects, the server notifies HA Monitor of the failure and terminates abnormally. The HA Monitor that receives the failure notification executes hot standby. During hot standby processing, the standby server uses its own recovery function based on the inherited information to restart operation and inherits job processing.

(b) Failures that the server cannot detect

Failures that the server cannot detect include server slowdown.

HA Monitor detects the failures that the server itself cannot detect. When HA Monitor detects such a failure, it terminates the erroneous server and then executes hot standby.

(c) Difference in detectable server failures depending on the server operating mode

Detectable failures depend on whether the server is running in the server mode or the monitor mode. The following table shows the server failure detection methods and their availability in each server operating mode.

Table 1‒2: Availability of server failure detection methods in each server operating mode

Detection method

Server mode

Monitor mode

The server detects the failure and notifies HA Monitor.

Y

Y

(Creation of a server monitoring command is required#1)

HA Monitor monitors the server and detects failures.

Y

N#2

Legend:

Y: Can be detected

N: Cannot be detected

#1

In the monitor mode, in order to have HA Monitor perform the hot standby operations automatically in the event of a failure that the server itself can detect, you must create a server monitoring command. For details, see 3.2.1 Monitoring a server in the monitor mode (for the active server). Note that when you use the monitor-mode program management function to monitor UAPs, there is no need to implement UAP monitoring processing in the server monitoring commands. For details, see 3.6.1 Monitoring UAPs.

#2

You can use the monitor-mode program management function to monitor UAPs.

(3) Host failures

Host failures occur on the portion of the host that is not the server. HA Monitor assumes that a host failure occurred upon disruption of alive messages. Host failures include the following:

Host's hardware failures or power failures

If dual hardware (such as processors and disks) is employed by using functionalities such as OS functions, reduced operation of a single hardware unit is not treated as a host failure.

Kernel failures

HA Monitor failures

Failures on all monitoring paths

Host slowdown

If any of the preceding failures disable communications through all monitoring paths or are reported by the failure management processor, HA Monitor detects a host failure, terminates the erroneous host, and then executes hot-standby switchover.

(4) Resource failure

A failure that occurs on a hardware component in the system is called a resource failure. HA Monitor performs a hot-standby switchover when detecting any of the following failures:

LAN failure

Disk failure

When HA Monitor detects a LAN failure or disk failure, it switches the active host by planned hot-standby switchover or by host pair shutdown, in the same way as when a server failure or host failure is detected.

(a) LAN failure

A failure that occurs on a LAN is called a LAN failure. HA Monitor can detect a LAN failure by monitoring whether communication is possible over the LAN. For details, see 3.4.1 LAN monitoring and automatic hot standby in the event of a failure.

(b) Disk failure

A failure that occurs on a disk is called a disk failure. HA Monitor can detect a disk failure by monitoring whether a write to the disk is possible. For details, see 3.4.2 Disk monitoring.

1.3 Types of hot standby

The following types of hot-standby switchover can be used with HA Monitor:

Automatic hot-standby switchover: HA Monitor automatically performs hot-standby switchover.

Planned hot-standby switchover: The operator intentionally performs hot-standby switchover.

Forced hot-standby switchover: The operator forcibly performs hot-standby switchover.

1.3.1 Automatic hot standby

The function in which HA Monitor performs hot standby automatically when a failure occurs in the active system is called automatic hot standby.

The user must specify in advance for each host the operating environment, such as server and shared resource names, that is to be subject to automatic hot standby. In the event of a failure, HA Monitor controls the specified servers and shared resources on the basis of the environment settings and executes hot standby automatically.

The following figure provides an overview of automatic hot standby.

Figure 1‒4: Overview of automatic hot standby

1.3.2 Planned hot standby

The function in which the operator performs hot standby for a reason such as host maintenance is called planned hot standby. To perform planned hot-standby switchover, the operator executes the server hot-standby switchover command (monswap command) from the active system.

HA Monitor uses for planned hot standby the same environment settings as for automatic hot standby.

The following figure provides an overview of planned hot standby.

Figure 1‒5: Overview of planned hot standby

Note that HA Monitor might perform planned hot standby as automatic hot standby.

1.3.3 Forced hot-standby switchover

Forced hot-standby switchover is a function that allows the operator to forcibly switch the host if login to the OS of the active system is impossible or attempted hot-standby switchover does not finish. To perform forced hot-standby switchover, the operator executes the server hot-standby switchover command (monswap command) from the standby system.

HA Monitor uses the same environment settings regardless of whether automatic hot-standby switchover or forced hot-standby switchover is performed.

The following figure provides an overview of forced hot-standby switchover.

Figure 1‒6: Overview of forced hot-standby switchover

1.4 Hot-standby switchover methods

The following hot-standby switchover methods can be used with HA Monitor:

Hot-standby switchover based on hybrid fencing

Hot-standby switchover based on a host reset

Hot-standby switchover based on SCSI reservation for shared disk

Hot-standby switchover based on the availability of LAN communications

For HA Monitor, we recommend hot-standby switchover based on hybrid fencing. This method has the following advantages:

Even if host reset cannot be performed due to, for example, a failure of the failure management processor (SVP or VMware ESXi), the system can continue to perform hot-standby switchover by using a SCSI-reserved shared disk. Because the reliability of automatic hot-standby switchover in the event of a host failure is higher than other methods, system availability can be improved.

The failed host can be stopped and isolated completely by resetting the host, which prevents the same server from starting on multiple hosts as active servers. Thus, safe hot-standby switchover can be performed.

A dump can be obtained if an OS slowdown occurs.

If hot-standby switchover based on hybrid fencing cannot be selected, hot-standby switchover based on host reset is recommended. Hot-standby switchover based on host reset has the following advantages:

The failed host can be stopped and isolated completely by resetting the host, which prevents the same server from starting on multiple hosts as active servers. Therefore, safe hot-standby switchover can be performed.

A dump can be obtained if OS slowdown occurs.

Note, however, that the hot-standby switchover methods that can be selected differ depending on the system configuration and requirements. Select a hot-standby switchover method based on the information in the following table.

Table 1‒3: Selecting a hot-standby switchover method

System configuration (server model)

Hot standby method

BladeSymphony or HA8000

In a virtualization environment in which access to VMware and the failure management processor is permitted:

Select either of the following methods:

• Hot-standby switchover based on hybrid fencing

• Hot-standby switchover based on host reset

In a virtualization environment in which access to VMware and the failure management processor is prohibited:

Hot-standby switchover based on SCSI reservation for shared disk

In a non-virtualization environment (physical environment):

Select either of the following methods:

• Hot-standby switchover based on hybrid fencing

• Hot-standby switchover based on host reset

Non-Hitachi PC/AT-compatible system

In an environment that uses a shared disk:

Hot-standby switchover based on SCSI reservation for shared disk

In an environment that does not use a shared disk:

Hot-standby switchover based on the availability of LAN communications

1.5 System modes with HA Monitor deployed

This section describes the system modes when HA Monitor is deployed. Using HA Monitor improves system reliability and availability.

1.5.1 C/S system mode

The C/S system mode consists of client and server systems only. No host computer is used. This mode connects multiple systems via a LAN so that the systems are linked with each other and function as a single overall job processing system.

If you install HA Monitor in the server systems, you can reduce the amount of down time that is caused by server failures, thereby reducing adverse effects on the client's job processing.

The following figure provides an example of the C/S system mode.

Figure 1‒7: Example of C/S system mode

1.5.2 FEP mode

In the FEP mode, server systems called FEPs are placed between the host computer and distributed computers and terminals. Typically, the host computer and FEPs are connected via a high-speed LAN. FEPs perform for the host computer some of the communication protocol processing and job processing that are normally performed by the host computer.

Installing HA Monitor on the FEPs enables you to reduce downtime at the FEPs that is caused by failures, minimize the amount of time the center's job processing is suspended, and improve reliability at the host computer.

The following figure shows an example of the FEP mode.

Figure 1‒8: Example of FEP mode

1.5.3 Distributed processing system mode

In the distributed processing system mode, server systems called distributed systems that are placed at locations such as business stores and branch offices are connected to the host computer. The role of distributed systems includes distributed processing of host computer jobs and management of distributed data.

Installing HA Monitor in distributed systems enables you to reduce downtime at the distributed systems that is caused by server failures, thereby minimizing the amount of time processing is suspended at the locations.

The following figure shows an example of the distributed processing system mode.

Figure 1‒9: Example of distributed processing system mode

1.6 Operating environment for HA Monitor

This section describes the hardware and software requirements for HA Monitor.

1.6.1 Required hardware

This subsection describes the hardware required in order to run HA Monitor. For details about the hardware configuration, see 5.5 Evaluation of hardware configuration.

The following figure shows an example of a hardware configuration.

Figure 1‒10: Example of a hardware configuration using BladeSymphony, or model HA8000xM or earlier

Figure 1‒11: Example of a hardware configuration using model HA8000xN or later in which a management server is used as a failure management processor

Figure 1‒12: Example of a hardware configuration using model HA8000xN or later in which a management server is not used as a failure management processor

(1) Processors

You use one server machine in each of the active and standby systems to execute actual jobs.

Processor requirements for using HA Monitor are as follows:

BladeSymphony Integrated Service Platform servers

If you use BladeSymphony, you can install multiple processors on a single chassis. HA Monitor enables you to apply a hot-standby configuration to all the processors in the same chassis.

Hitachi Advanced Server HA8000 series servers

PC/AT-compatible systems from other companies

If you will be resetting hosts, you must not use BladeSymphony or HA8000 together with PC/AT-compatible systems from other companies.

For details about the functions supported by each shared disk data protection method and the function for controlling hot standby based on the availability of LAN communications, see 3.1 List of functions supported by HA Monitor.

Note

If you use BladeSymphony with Hitachi server virtualization (Virtage), you can use logical partitions (LAPRs) in the processor as hot-standby configuration units.

Regardless of the model of server used, you can run multiple virtual systems on a single processor by using VMware ESXiTM. In this case, you can define each virtual system as a hot-standby switchover configuration unit.

Note

With BladeSymphony and HA8000, a shared disk is protected from data corruption by host reset or SCSI reservation for shared disk. In PC/AT-compatible systems from other companies, a shared disk is protected from data corruption by SCSI reservation for shared disk. If neither host reset nor SCSI reservation for shared disk is available, the function for controlling hot standby based on the availability of LAN communications is used. For details about host reset, see 2.3.5 Host reset. For details about SCSI reservation for shared disk, see 2.3.6 SCSI reservation for shared disk. For details about the function for controlling hot standby based on the availability of LAN communications, see 2.3.7 Function for controlling hot standby based on the availability of LAN communications.

(2) Monitoring path

The monitoring path is used to connect the processors in the active and standby systems and for the systems to monitor each other and exchange information required for hot standby.

You can use a TCP/IP LAN as a LAN dedicated as the monitoring path (monitoring LAN). You must use a LAN that supports broadcast communication because HA Monitor uses a TCP/IP LAN to broadcast. Although you can use a TCP/IP LAN in any hot-standby configuration, you need a dedicated IP address and port number.

If you use HA Monitor Extension to change the maximum number of hosts in a single hot-standby configuration, use a LAN that is capable of multicasting. For details about HA Monitor Extension, see 3.7 Functions of HA Monitor Extension.

(3) Reset path

The reset path is used to send host input/output reset instructions to a failure management processor in the event of a failure in the active system. A reset path is not needed if you do not perform host reset. A failure management processor is a hardware device that enables you to specify the system configuration and to control the CPU. It is an independent hardware device that is installed in the system separately from the CPU. Its name depends on the system you use, such as an SVP, BMC, or management server, but its functions are the same. In this manual, this hardware is referred to collectively as a failure management processor.

For notes about the reset path, see 5.5.4 Reset path configuration.

(4) Shared disk

A shared disk is used to store data that is shared by the active and standby system servers. A shared disk is mandatory when you use SCSI reservation for shared disk. Data on the shared disk is inherited during hot standby. All systems that use data stored on the shared disk are connected to the shared disk. The shared disk is controlled by using LVM .

Supported shared disks include Hitachi disk array systems and shared datastores built with VMware vSAN.

HA Monitor does not handle disk and I/O path failures. To handle them, you must use some method other than HA Monitor to prevent a shared disk from becoming a single point of failure. For details about preventing a single point of failure, see 5.5.1 Shared disk configuration.

To use SCSI reservation for data protection, the following conditions must be met:

The storage device supports SCSI-3 Persistent Reservation and has a RAID controller.

A host-base RAID with individual RAID controller is not supported. Such a RAID can be accessed from the standby system, even if it is reserved in the active system.

To use a shared datastore built with VMware vSAN, make sure that the datastore is accessed via the vSAN iSCSI target service.

(5) Local disk

A local disk is used as a host-specific disk that is not shared with other hosts. You store on the local disk OS files and HA Monitor's various files. You start HA Monitor from the local disk, not the shared disk.

For the local disk also, we recommend that you employ a configuration that can prevent a single point of failure, in the same manner as with a shared disk.

In the event of a shared disk or local disk failure, applications can no longer be run because files cannot be accessed. In such a case, the server and OS will not terminate abnormally but hot standby might fail. To prevent this, consider generating an OS panic by using OS and driver functions to perform hot standby in the event of any I/O error on shared or local disks.

(6) Console

You use the console to execute HA Monitor commands and to check messages that are displayed.

(7) LAN

LANs are used to connect the active and standby systems, as well as to connect workstations and PCs that are clients. Use of a LAN is mandatory when you use the function for controlling hot standby based on the availability of LAN communications. Among the LANs that support TCP/IP communications, you can use any Ethernet LAN supported by the OS.

HA Monitor does not directly process LAN failures in HUB and LAN adaptors. For these failures, use some other method and employ a configuration that can prevent a LAN from becoming a single point of failure. For details about preventing a LAN from becoming a single point of failure, see 5.5.2 LAN configuration.

(8) Management server

To perform host reset with model HA8000xN or later, management servers with HCSM installed are required, depending on the number of monitored hosts. The connection topology of management servers or the number of monitored hosts differs depending on the HA Monitor operating environment. Examples of possible configurations are as follows:

One management server is connected to a hot-standby switchover configuration. One management server is used both for central server management and as a failure management processor for HA Monitor.

A management server dedicated to HA Monitor is connected to a hot-standby switchover configuration, in addition to a management server for central server management.

One management server is connected to multiple hot-standby switchover configurations.

One management server is connected to each hot-standby switchover configuration.

1.6.2 Required software

This subsection describes the software required to run HA Monitor. The following figure shows an example of a software configuration.

Figure 1‒13: Example of a software configuration using BladeSymphony, or model HA8000xM or earlier

Figure 1‒14: Example of a software configuration using model HA8000xN (or later) in which a management server is used as a failure management processor

#1

The HA Monitor Option configuration includes HA Monitor Connector.

#2

HCSM is used only if the model is HA8000xN or HA8000xN1.

Figure 1‒15: Example of a software configuration using model HA8000xN or later in which a management server is not used as a failure management processor

#

An example of HA Monitor Option is HA Monitor Connector.

(1) Kernel

Linux (x86)

This is the key operating system for the system.

(2) Hot-standby control

HA Monitor

Controls hot standby.

HA Monitor Extension

This is an HA Monitor optional product that controls hot standby in a large system configuration.

To use HA Monitor Extension, you must use host reset in BladeSymphony or HA8000.

For details about HA Monitor Extension, see 3.7 Functions of HA Monitor Extension.

(3) Operations management

JP1

Manages events that occur in the system.

(4) Programs

In a hot-standby configuration that uses HA Monitor, the range of functions supported by HA Monitor depends on whether the program has an HA Monitor interface. The subsections below describe the programs supported by HA Monitor.

(a) Programs with an HA Monitor interface

OpenTP1

This is a transaction processing monitor that achieves distributed transaction processing.

HiRDB

This is a relational database management system that can achieve a scalable system configuration.

(b) Programs without an HA Monitor interface

Cosminexus

This is a program that provides a platform for performing business-related tasks. Among other products, it includes an application server, an SOA infrastructure product, an OLTP product, and a report-related product.

JP1

This is an integrated system operations management product that supports stable IT system operation. It includes integrated management products, job management products, asset and distribution management products, and network management products.

Other servers

These include servers other than the above, such as UAPs.

(5) Management server

HA Monitor Option

Software that controls host reset

HCSM

Software that centrally manages server machines. This is used only if the model is HA8000xN or HA8000xN1.

2 Hot Standby Using HA Monitor

This chapter explains the hot standby operation that is performed by HA Monitor after a failure. The information provided here is intended for users who are familiar with the concept of hot standby and focuses on how HA Monitor performs the hot standby operation.

It explains the hot-standby configurations supported by HA Monitor, outlines the operation from failure detection to hot standby, describes how HA Monitor monitors servers and hosts, and explains how shared resources used by a server are inherited to a standby system.

2.1 Hot-standby configurations

HA Monitor enables you to choose from the hot-standby configurations described below. This section explains examples of the configurations. For details about the characteristics of each configuration, see 5.2 Evaluation of hot-standby configuration.

Note that if you use SCSI reservation for shared disk, reset paths cannot be used.

2.1.1 1-to-1 switchover configuration

In this configuration, there is a one-to-one correspondence between the primary system and the secondary system. If the primary system fails, the secondary system takes over.

The following figure shows an example of a 1-to-1 switchover configuration.

Figure 2‒1: Example of a 1-to-1 switchover configuration

2.1.2 Mutual hot-standby configuration

In this configuration, each server system runs as a primary system and also functions as a secondary system for another system. If one system fails, the other system takes over.

The following figure shows an example of a mutual hot-standby configuration.

Figure 2‒2: Example of a mutual hot-standby configuration

2.1.3 2-to-1 switchover configuration

In this configuration, one secondary system is provided for two primary systems. Two servers stand by in the secondary system. If one of the primary systems fails, it (the failed system) becomes a secondary system.

The following figure shows an example of a 2-to-1 switchover configuration.

Figure 2‒3: Example of a 2-to-1 switchover configuration

Note

There can be two or more primary systems.

2.1.4 Multi-standby configuration

In this configuration, multiple secondary systems are provided for a single primary system so that multiple standby servers are assigned to a server that executes a particular type of job processing. If a failure occurs on the secondary system before the primary system has been recovered, the secondary system can be switched over to another secondary system because there are multiple standby servers. This configuration improves system availability because multiple systems are available to handle system failures.

The following figure shows an example of a multi-standby configuration.

Figure 2‒4: Example of a multi-standby configuration

You must use the multi-standby function to employ this configuration. For details about the multi-standby function, see 4.5 Managing servers and hosts when using the multi-standby function.

2.1.5 Cluster hot-standby configuration

This subsection explains the cluster hot-standby configuration that combines multiple hot-standby configurations including the 1-to-1 switchover configuration.

In the cluster hot-standby configuration, multiple server systems connected in a cluster configuration run as primary systems while they function as each other's secondary systems. A maximum of 32 server systems can be connected in a cluster hot-standby configuration.

The cluster hot-standby configuration can be achieved by running multiple servers, such as a server performing job processing and a server that stands by, in a single server system. You can also assign multiple standby servers to a server that executes a particular type of job processing.

(1) Example of a configuration in which a server performing job processing and a standby server have a one-to-one correspondence

The following figure shows an example of a cluster hot-standby configuration in which a server that performs job processing and a standby server have a one-to-one correspondence.

Figure 2‒5: Example of a cluster hot-standby configuration (a server that performs job processing and a standby server have a one-to-one correspondence)

(2) Example of a configuration in which multiple servers stand by for a server that performs job processing

The figure below shows an example of a cluster hot-standby configuration in which multiple servers stand by for a server that performs job processing. In this example, system failures can be handled even before the primary system has been recovered from a failure because multiple servers stand by in all systems for the server that performs job processing.

Figure 2‒6: Example of a cluster hot-standby configuration (multiple servers stand by for an executing server)

You must use the multi-standby function to employ this configuration. For details about the multi-standby function, see 4.5 Managing servers and hosts when using the multi-standby function.

2.2 Operation from failure detection to hot standby

HA Monitor detects two principal types of failures, server failures and host failures. The processing and flow of hot standby operations depend on the type of failure.

In the event of a server failure, HA Monitor removes the erroneous server as a shared resource and switches over to the server in the standby system. In the event of a host failure, HA Monitor disconnects shared resources from the failed host using a method such as host reset, and then switches over to a standby system server. This section provides an overview of the processing flow from failure detection to hot standby. For details about the processing flow, see 4.8 Processing flows.

2.2.1 Hot standby operation in the event of a server failure

In the event of a server failure, the HA Monitor in the active system resulting in the failure removes shared resources from the active system to prepare for hot standby. When hot standby preparations, such as prohibiting access to the shared disk and LAN disconnection, are completed, HA Monitor sends a notification to that effect to the standby system. The standby system then starts hot standby processing.

The following figure provides an overview of hot standby by HA Monitor in the event of a server failure.

Figure 2‒7: Overview of hot standby in the event of a server failure

2.2.2 Hot standby operation in the event of a host failure

In the event of a host failure, the hot standby operation to be performed depends on whether the host is reset, SCSI reservation for shared disk is used, or the function for controlling hot standby based on the availability of LAN communications is used.

(1) Resetting hosts

If a host failure occurs when host reset is to be performed, the HA Monitor in the standby system issues a request to reset the active system. It issues this request to the failure management processor on the failed host via the reset path. The failure management processor that manages the failed host then performs reset processing, such as termination of input/output operations. When the reset processing is completed, the failed host sends a notification to that effect to the standby system. The standby system then starts hot standby processing.

The following figure provides an overview of HA Monitor's hot standby operation in the event of a host failure.

Figure 2‒8: Overview of the hot standby operation in the event of a host failure (when host reset is used)

(2) Using SCSI reservation for shared disk

If a host failure occurs when SCSI reservation for shared disk is used, a reservation is forcibly obtained by the target host, and then the active server is started so that the shared disk is protected from data corruption that might result from concurrent write operations from multiple hosts.

If the hot standby operation occurs due to a monitoring path failure and the failed host detects a loss of reservation, planned termination is performed on the server.

The following figure provides an overview of HA Monitor's hot standby operation in the event of a host failure.

Figure 2‒9: Overview of the hot standby operation in the event of a host failure (when SCSI reservation for shared disk is used)

The following figure provides an overview of hot standby in the event of a monitoring path failure when SCSI reservation for shared disk is used.

Figure 2‒10: Hot standby in the event of a host failure (monitoring path failure when SCSI reservation for shared disk is used)

(3) Using the function for controlling hot standby based on the availability of LAN communications

If a host failure occurs while the function for controlling hot standby based on the availability of LAN communications is being used, whether LAN communication is available is checked. If the standby system confirms that LAN communication is available, the hot standby operation is performed. In the event of a monitoring path failure, HA Monitor ensures that a server in the system in which LAN communication is available will perform job processing.

The following figure provides an overview of HA Monitor's hot standby operation in the event of a host failure.

Figure 2‒11: Overview of the hot standby operation in the event of a host failure (when the function for controlling hot standby based on the availability of LAN communications is used)

The following figure provides an overview of the hot standby operation in the event of a monitoring path failure when the function for controlling hot standby based on the availability of LAN communications is used.

Figure 2‒12: Hot standby operation in the event of a host failure (monitoring path failure when the function for controlling hot standby based on the availability of LAN communications is used)

2.3 Failure detection by HA Monitor

When a failure occurs in a system that uses HA Monitor, HA Monitor detects the failure. HA Monitor then performs the hot standby operation if it is able to do so. If hot standby is not possible, HA Monitor does not perform it.

Both server failures and host failures can occur in the system. The failure detection method depends on the failure type. This section explains how failures are detected for each failure type. It also explains the conditions under which HA Monitor performs the hot standby operation and the methods used to achieve hot standby, including the mechanism of host reset, SCSI reservation, and the function for controlling hot standby based on the availability of LAN communications.

2.3.1 Conditions under which HA Monitor performs the hot standby operation

Several conditions must be satisfied before HA Monitor can perform the hot standby operation. The status in which these conditions have been satisfied is called the hot-standby enabled state.

(1) Conditions under which the hot standby operation is performed in the event of a server failure

The following are the conditions under which the hot standby operation is performed in the event of a server failure:

Communication with the HA Monitor in the standby system is available via a monitoring path.

The standby server corresponding to the active server is on standby.

(2) Conditions under which the hot standby operation is performed in the event of a host failure

The following are the conditions under which the hot standby operation is performed in the event of a host failure:

Communication with the HA Monitor in the standby system is available via a monitoring path.

The standby server corresponding to the active server is on standby.

A reset path is available (applicable when host reset is performed)#.

SCSI reservation for shared disk is available (applicable when SCSI reservation for shared disk is used).

Communication is available via LAN communications used by business applications (if the function for controlling hot standby based on the availability of LAN communications is used).

#

If the shared disk can be SCSI-reserved, hot-standby switchover can be performed even when a failure has occurred on the reset path. For details, see 2.3.8 Hybrid fencing.

2.3.2 Detecting server failures (in the server mode)

When the server is in the server mode, HA Monitor monitors the server and performs the hot standby operation in the event of a server failure. This subsection explains how HA Monitor detects server failures and HA Monitor's processing after detection of a failure.

(1) How server failures are detected

Server failures can be classified into two types, those that can be detected by the server itself and those that cannot.

When a failure that can be detected by the server itself occurs:

HA Monitor receives a failure notification from the server and detects the server failure.

When a failure that cannot be detected by the server occurs:

On the active server, HA Monitor detects the failure. The server monitoring method depends on the program being used.

If TP1/Server Base or HiRDB is being used:

HA Monitor detects a failure by monitoring operation reports from the server. To monitor operation reports from a server, specify the server failure monitoring time for the patrol operand in the server environment definition. The server produces operation reports on a regular basis, and HA Monitor checks at one-second intervals whether there is an operation report from the server. If HA Monitor does not receive any operation reports from a server within the specified server failure monitoring time, HA Monitor assumes that a server failure has occurred. To monitor operation reports from a server, specify A for the server_type operand in the server environment definition. Alternatively, omit the server_type operand.

Note that HA Monitor does not detect failures on the standby server. The user must restart and terminate the standby server, if necessary.

(2) HA Monitor processing after detection of a server failure

When HA Monitor detects a server failure on the active server and hot standby is possible, HA Monitor performs one of the types of processing listed below. The type of processing to be performed must have been specified by the user in advance in the switchtype operand in the server environment definition.

Performs the hot standby operation

HA Monitor forcibly terminates the active server resulting in the failure, and then performs the hot standby operation to switch jobs over to the standby server in the standby system.

Waits for a restart of the active server; if restart fails, performs the hot standby operation

HA Monitor terminates the active server resulting in the failure and waits for its restart. If restart of the active server fails, HA Monitor performs the hot standby operation to switch over to the standby system.

Waits for a restart of active server; if restart fails, waits for operator intervention

HA Monitor terminates the active server resulting in the failure and waits for its restart. If restart of the active server fails, HA Monitor terminates the active server and waits for the operator to act.

Note that when you use this setting, HA Monitor will not regard a slowdown of the active server as a server failure. When HA Monitor detects slowdown of the active server, it issues a message and continues monitoring the server.

The following subsections provides details of each of these types of processing.

(a) Performs the hot standby operation

When HA Monitor detects a server failure, the active system's HA Monitor forcibly terminates the active server and performs the hot standby operation to switch over to the standby system. The following figure provides an example of the processing that is performed by HA Monitor when no operation report is received and a server failure is detected.

Figure 2‒13: HA Monitor processing when a server failure is detected (when hot standby operation is performed)

(b) Waits for a restart of the active server; if restart fails, performs the hot standby operation

When HA Monitor detects a server failure, it waits for a restart of the active server resulting in the server failure. The active server's status until it is restarted is called the restart wait state.

If the active server restarts successfully, there is no change in the host used to execute jobs after the failure because HA Monitor does not perform the hot standby operation. If restart of the active server fails, the server program retries the restart for a specified number of times. If the restart still fails and HA Monitor detects the active server restart limit, HA Monitor performs the hot standby operation to switch over to the standby system. HA Monitor processing during hot standby is the same as in (a) Performs the hot standby operation.

The active server's restart limit is the maximum number of attempts to be made to restart the active server. HA Monitor detects the following values as the active server's restart limit:

Program's retry count limit

This is the maximum number of retry attempts. This value is specified in the program.

Active server's restart monitoring time limit

This is the amount of monitoring time before the active server is restarted after a server failure at the active server has been detected. This value is specified in the server environment definition.

(c) Waits for a restart of the active server; if restart fails, waiting for operator intervention

This subsection explains HA Monitor processing when it involves waiting for a restart of active server after detection of a server failure and then waiting for an operator to take action if the restart fails.

The active server restart processing is the same as in (b) Waits for a restart of the active server; if restart fails, performs the hot standby operation. Note that when HA Monitor is to wait for operator intervention, you can specify only the program's retry count as the active server's restart limit.

If restart of the active server fails and HA Monitor detects the active server's restart limit, HA Monitor terminates the active server in the active system. In the standby system, HA Monitor terminates the standby server that corresponds to the active server resulting in the failure and then restarts the standby server to place it in the active server start wait state. The user must either start the standby server in the active server start wait state as the active server or terminate it. For details about how to operate a server that is in wait state, see 7.4.1 Starting a server in the wait state and then restarting jobs.

2.3.3 Detecting server failures (in the monitor mode)

When the server is in the monitor mode, how server failures are detected depends on whether APIs are issued.

(1) When APIs are not issued

HA Monitor cannot detect server failures because it does not monitor servers. HA Monitor detects only host failures.

If a server failure occurs in the monitor mode, the user must take the following action:

If hot standby switchover to the standby system is necessary

Stop the active server and execute the server host hot-standby switchover command (monswap command) in the active system to perform planned hot-standby switchover.

If hot standby switchover to the standby system is not necessary

Restart the server in the active system

If you use the server monitoring command beforehand to create processing for monitoring for server failures, HA Monitor can perform the hot standby operation in the event of a server failure. For details about the server monitoring command, see 3.2.1 Monitoring a server in the monitor mode (for the active server).

(2) When APIs are issued

If the monitor-mode program management function is used to issue APIs from the UAP, HA Monitor monitors the UAP to detect server slowdown.

HA Monitor detects server failures caused by the UAP by monitoring the UAP's processes and operation reports. You can choose to monitor both the UAP's processes and operation reports or you can choose to monitor only the UAP's processes.

In the event of a UAP failure, HA Monitor can also restart the UAP by issuing a program restart command. The program restart command must have been created in advance by the user.

For details about monitoring UAPs, see 3.6.1 Monitoring UAPs.

2.3.4 Detecting host failures

This subsection describes monitoring between hosts, and the reset and monitoring paths.

(1) Monitoring between hosts

The HA Monitors in the active and standby systems monitor hosts by checking the alive messages that are reported mutually at a specified interval. This subsection explains monitoring between hosts by using alive messages.

(a) Detecting host failures by using alive messages

HA Monitor detects host failures when the alive messages stop. You must specify the amount of time that is to pass since the last alive message was received before HA Monitor is to detect a host failure (host failure monitoring time). This specification is made in the patrol operand in the HA Monitor environment settings. You also specify the transmission interval for the alive messages in the alive_interval operand in the HA Monitor environment settings. If you set the host failure monitoring time to a small value, we strongly recommend that you change the alive message transmission interval from its default setting.

Transmission of alive messages starts when the HA Monitor in the remote system is contacted successfully and the hot standby operation becomes enabled. The alive messages are transferred via the monitoring path. If no alive message is received from the remote system within the specified host failure monitoring time, HA Monitor determines that a host failure has occurred in the remote system.

By default, the HA Monitors in the active and standby systems check mutually for alive messages at intervals of one second. You can change this alive message checking interval to 100 milliseconds by specifying the patrol_100ms operand in the HA Monitor environment settings. If you set the host failure monitoring time to a small value, we recommend that you set checking for alive messages to the interval of 100 milliseconds.

(b) Alive message transmission method

HA Monitor uses the unicast method to send and receive alive messages. This method sends and receives as many alive messages as there are hosts connected per monitoring path.

If you use HA Monitor Extension to set the maximum number of hosts to 33 or greater, HA Monitor uses the multicast method to send alive messages. The multicast method sends and receives one alive message per monitoring path. This method can reduce the number of messages that are sent and received as well as the overhead involved in sending and receiving the messages, thereby minimizing the machine's CPU usage and the loading on LANs used as monitoring paths. For details about changing the maximum number of hosts, see 3.7.2 Changing the maximum number of hosts.

(2) Health check of reset paths

To determine whether host reset is enabled in the event of a host failure, HA Monitor performs a health check on the status of all failure management processors in all remote systems that are connected. A health check starts when startup of the active and standby servers is completed and hot-standby switchover between the local and remote systems becomes possible.# Thereafter, a health check is performed every two minutes. You can change this interval by specifying the resetpatrol operand in the HA Monitor environment settings.

A health check is performed from both the active system and the standby system.

#

In either of the following cases, HA Monitor starts a health check when a connection is established between hosts. HA Monitor continuously performs health checks while the hosts are connected.

The multi-standby function is used.

The reset_type operand is set to host in the HA Monitor environment settings.

If a failure is detected in the failure management processor or in the reset path, HA Monitor issues one of the following failure detection messages, and then suspends the reset path health check for the corresponding host:

KAMN399-E

KAMN624-E

If HA Monitor detects a failure in a failure management processor or on a reset path, it issues a failure detection message and suspends health checking on the reset path for the corresponding host. For details about how to handle the message, see the manual HA Monitor Cluster Software Messages. You can also use the reset path status display command (monrp command) to check the status of the failure management processor.

For details about the command, see 9. Commands. Note that after the host has been recovered from a failure, you can use the reset path status display command (monrp command) to restart the health check that was suspended.

If the resetpath_retry operand is set to use or is omitted in the HA Monitor environment settings, HA Monitor automatically checks the status at regular intervals. When the reset path is restored, HA Monitor outputs the KAMN646-I message, and restarts the health check that was suspended.

Note

If you are using BladeSymphony, you must specify the resetpatrol_mode operand in the HA Monitor environment settings. For details, see 8.3.1 HA Monitor environment settings (sysdef).

(3) Health check of monitoring paths

HA Monitor can perform health checks on the status of the monitoring paths at a specified interval. The health check interval is specified in the pathpatrol operand in the HA Monitor environment settings. If a failure occurs on a monitoring path, HA Monitor issues a communication error message.

Once HA Monitor has started, it checks the status of the monitoring paths each time the health check interval specified in the environment settings has elapsed. The health check is performed on all monitoring paths connected to the host and terminates when HA Monitor is terminated. This health checking occurs only where a monitoring path is connected between local and remote systems and HA Monitor is running.

The user can see the status of the monitoring paths by using the monitoring path status display command (monpath command).

When a failure is detected on a monitoring path and the pathpatrol_retry operand was specified in the HA Monitor environment settings, the monitoring path's status is rechecked. If the recheck still results in detection of the failure, HA Monitor issues one of the following messages:

KAMN609-W

KAMN635-E

KAMN640-E

KAMN641-W

Once the KAMN641-W message is issued for any reason, or if the monitoring path status display command (monpath command) detects a failure, this message will not be issued again until after the system recovers from the failure.

For details about how to handle the messages, see the information provided in the manual HA Monitor Cluster Software Messages. Note that if the monitoring path status display command (monpath command) is executed while the monitoring path's status is being rechecked, the KAMN393-E message is output, and the status of monitoring paths cannot be checked.

(4) Retrying transmission of query response messages

HA Monitor transfers via the monitoring path not only alive messages but also other messages with the remote system. For example, the monitoring path is also used for transfer of query response messages for determining whether the same server is already running in the remote system.

If transmission of such a query response message fails, HA Monitor retries transmission of the message every three seconds until the message is sent successfully. The user can change this message transmission retry interval by specifying the message_retry operand in the HA Monitor environment settings.

If no monitoring is being performed between hosts, such as before transmission of alive messages begins, and a query response message cannot be sent successfully within 60 seconds,# HA Monitor determines that a host failure has occurred.

#

If the value of the message_retry operand in the HA Monitor environment settings is greater than 60 seconds and a query response message cannot be sent successfully within that amount of time, HA Monitor determines that a host failure has occurred.

(5) Recognizing a slowdown of the local system

This subsection explains HA Monitor's processing when HA Monitor cannot operate for more than the host failure monitoring time specified in the patrol operand in the HA Monitor environment settings for reasons such as high system load (slowdown of host).

If a slowdown occurs in the active system, HA Monitor performs hot standby switching to the standby system.

On the other hand, if a slowdown occurs in the standby system, the HA Monitor in the active system determines that a host failure has occurred in the standby system. In this case, when the standby system recovers from the slowdown, a status conflict occurs between the hosts because the HA Monitor in the standby system has not detected a host failure in the standby system.

Therefore, the HA Monitor in the standby system reconnects hosts when the standby system has recovered from slowdown, and then restarts the standby server. This automatically resolves the status conflict between the hosts and restarts monitoring of both active and standby systems.

2.3.5 Host reset

Host reset means forcibly terminating the system in order to shut down I/O operations on a host that has experienced a failure. A host reset prevents the same active server from running on multiple hosts.

You must decide whether to use this function. For details about how to decide, see 1.4 Hot-standby switchover methods and 3.1 List of functions supported by HA Monitor.

Host reset is achieved by linking HA Monitor and a failure management processor. If VMware ESXi-based virtualization is used when host reset is performed, VMware ESXi is linked with a failure management processor.

HA Monitor monitors for host failures and issues a host reset request when it detects a host failure and the hot standby operation has been enabled. Normally, the standby system resets the active system on which the failure occurred (resetting the active system). Depending on the type of hot-standby configuration, the active system can reset the standby system (resetting the standby system).

Note

For details about what to do if the host reset fails, see 7.5.3 Handling host reset errors.

(1) Resetting the active system

If the HA Monitor in the standby system detects a host failure in the active system, it issues an active system reset request to the failure management processor in the active system. It does this by using a reset command via the reset path. If a system dump can be obtained during the reset operation, a system dump of the reset system is obtained by an OS function. When the reset operation is completed, HA Monitor starts the hot standby operation. If the reset operation fails, the user must reset the system manually and then obtain a system dump.

You must have Linux Tough Dump installed to obtain a system dump successfully. For details about Linux Tough Dump, contact a Hitachi sales representative.

The following figure shows host status monitoring and host reset.

Figure 2‒14: Host status monitoring and host reset

HA Monitor monitors the remote system by sending alive messages. If a host failure occurs in the active system, the HA Monitor in the standby system issues a reset request to the failure management processor# because it cannot receive an alive message from the active system. The failure management processor# then sends a host reset instruction.

#

If the server model is HA8000xN or later, HA Monitor links with the failure management processor on the management server.

If VMware ESXi-based virtualization is used, VMware ESXi is used, not the failure management processor.

Because host reset is achieved by linking with hardware, you must use the address operand in the HA Monitor environment settings to specify a unique host address for each host. This host address is neither an IP address in TCP/IP nor a MAC address in OSI. The HA Monitor administrator can specify any address.

If the system being used is BladeSymphony, the same processing is performed by HA Monitor and the failure management processor, although the standby system that issues a reset instruction might be in the same chassis.

If VMware ESXi-based virtualization is used and a failure occurs in the virtual machine where the HA Monitor in the active system is running, the HA Monitor in the standby system resets only the virtual machine resulting in the failure. The HA Monitor in the standby system issues a reset request to the VMware ESXi that manages HA Monitor's virtual machine, so that the virtual machine resulting in the failure is reset. Therefore, when VMware ESXi-based virtualization is used, you must use the HA Monitor environment setup command (monsetup command) to specify in the HA Monitor environment settings the IP address of the VMware ESXi and the virtual machine name, in addition to the normal settings.

If host reset fails in a virtualization environment that uses Hitachi server virtualization (Virtage) or VMware ESXi, you can use the physical partition reset function to reset the entire processor. For details about the physical partition reset function, see 3.3.6 Physical partition reset function in a virtualization environment.

(2) Resetting the standby system

Depending on the type of hot-standby configuration, the standby system might reset the active system or another standby system.

(a) 1-to-1 switchover configuration

If no alive message is received from the standby system within the host failure monitoring time specified in the patrol operand in the HA Monitor environment settings, the HA Monitor in the active system determines that a host failure has occurred in the standby system. In this case, the HA Monitor in the active system does nothing because the active server can continue job processing.

If a failure occurs in the active system while the standby system is in host failure status, hot standby switching to the standby system is not possible. Therefore, you can have the HA Monitor in the active system reset the standby system when a host failure is detected in the standby system. You use the standbyreset operand in the HA Monitor environment settings to specify whether the standby system is to be reset in the event of a host failure. Do not perform standby system reset in any configuration other than the 1-to-1 switchover configuration.

If the standby system is reset in the event of a host failure in the standby system, the operator must determine the cause of the host failure and take appropriate action.

(b) Multi-standby configuration

In a multi-standby configuration, there are multiple standby systems per active system. If a host failure occurs in one of the standby systems, another standby system might reset the standby system that failed. For details about host reset in a multi-standby configuration, see 4.2.3 Host reset when there are multiple standby systems.

2.3.6 SCSI reservation for shared disk

SCSI reservation for shared disk is a lock function for a shared disk in which the active system reserves the shared disk so that only the active system can access the shared disk. This protects the shared disk from data corruption that might result from concurrent write operations from multiple systems.

If hot standby has been enabled and HA Monitor detects a failure, HA Monitor uses SCSI reservation for shared disk and then starts the target server.

You must decide whether to use this function. For details about how to decide, see 1.4 Hot-standby switchover methods and 3.1 List of functions supported by HA Monitor.

You use SCSI reservation for shared disk by specifying the following HA Monitor environment settings:

fence_reset operand: nouse

fence_scsi operand: use

For details about these operands, see 8.3.1 HA Monitor environment settings (sysdef). If a specified setting is invalid, the KAMN977-W message is issued, but processing continues.

(1) Reservation during server start and termination

The active server is started after the shared disk to be used by the server has been reserved. The reservation is obtained before the shared disk is connected.

When the active server is terminated, the reservation obtained when the active server was started is released once the server's termination processing has been completed. The reservation is released after the shared disk is disconnected.

There is no reservation-related processing when the standby server is started or terminated.

The following table lists and describes the messages that are issued when reservation has failed and describes the subsequent processing.

Table 2‒1: Messages that are issued when reservation has failed and the subsequent processing

Case

Message that is issued

Subsequent processing

Reservation of the portion of the shared disk to be used by the server has failed due to a shared disk path failure.

KAMN725-W, KAMN726-E

Server startup processing continues.

Reservation has failed because the portion of the shared disk to be used by the server has already been reserved by another system.

KAMN725-W, KAMN726-E

Server startup processing is canceled.

Reservation of the entire shared disk to be used by the server has failed.

KAMN725-W, KAMN726-E, KAMN498-E

Server startup processing is canceled.

Release of reservation has failed for all or part of the shared disk.

KAMN725-W, KAMN726-E

Processing continues.

(2) Reservation during the hot standby operation

Reservation is handled as explained below during the hot standby operation.

In the event of a server failure or during planned hot standby:

After the source host releases reservation, the target host forcibly obtains reservation and then starts the standby server as the active server. If release of resources has failed on the target host, an OS panic is generated on the source host.

In the event of a host failure due to an OS or HA Monitor failure:

Because the source host does not release the reservation, the target host forcibly obtains reservation and then starts the standby server as the active server. At this time, in the event of a process failure on the HA Monitor on the source host, the KAMN066-E message is output and an OS panic is generated. This prevents contention for shared resources.

If the switching-source server is standing by again before the system switching-destination server is stopped, it outputs the KAMN728-W message, and then stops the server without releasing the reservation. When the alive message is blocked due to the slowdown of the OS in the active server, and the reservation is released after the shared disk is updated in the switched system, the system recovers from the slowdown later and might update the shared disk and destroy the data.

If the server is stopped without releasing the reservation, make sure that there is no active server on another host, release the reservation, and then start the active server.

In the event of a host failure due to a monitoring path failure:

The HA Monitor in the standby system detects the host failure in the active system and performs the hot standby operation. As a result, the reservation obtained by the active system is assumed forcibly by the standby system.

The HA Monitor in the active system detects the loss of reservation, issues the KAMN666-E message, forcibly terminates the server in the active system, and then disconnects the shared resources. The methods used for server termination and shared disk processing during the hot standby operation are the same as when the hot standby operation is performed.

Note that HA Monitor does not recognize monitoring path failures and OS slowdown in the active system as host failures. If these failures occur and the target server needs to be terminated, HA Monitor issues the KAMN728-W message and terminates the server without releasing reservation, in the same manner as when a host failure has occurred due to an OS failure or HA Monitor failure. If the monitoring path is restored and the source server is placed in standby status again before the target server has terminated, HA Monitor releases the reservation when the target server is terminated and does not issue the KAMN728-W message.

The following figure shows the processing flow in the event of a monitoring path failure.

Figure 2‒15: Processing flow in the event of a monitoring path failure

If a failure has occurred on the disk path from the standby system, the HA Monitor in the standby system assumes that reservation has failed and terminates the standby server without reserving any part of the shared disk needed by the server. The active system continues active server operation because it maintains the reservation.

The following figure shows the processing flow in the event of a monitoring path failure when a failure has occurred in some of the disk paths in the standby system.

Figure 2‒16: Processing flow in the event of a monitoring path failure when a failure has occurred in some of the disk paths in the standby system

The following are details of the processing flow shown in the figure. The numbers correspond to the numbers in the figure.

A failure has occurred on one of the shared disk paths in the standby system.

A monitoring path failure has occurred.

The standby system detects the host failure and performs the hot standby operation, but cannot obtain reservation because the path failure occurred in a part of the shared disk. As a result, the hot standby operation fails.

The active system continues job processing because it maintains the reservation.

(3) Checking the reservation status

To be prepared to perform the hot standby operation, HA Monitor periodically checks the reservation status. This subsection describes the checking of reservation status from the active and standby systems.

(a) Checking the reservation status from the active system

From the active system, HA Monitor periodically checks the reservation status on all shared disks that are used by the server. Specifically, it checks for a forced reservation from the remote system for the purpose of performing hot standby switching to the standby system due to a monitoring path failure.

You use the scsi_check operand in the HA Monitor environment settings to specify the check interval. The default value is five seconds.

(b) Checking the shared disk path status from the standby system

HA Monitor monitors the disk path status by obtaining the shared disk status from the standby system. Hot standby errors can be avoided by checking for any abnormality in advance and then notifying the user. The hot standby operation fails if reservation of a portion or all of the shared disk cannot be obtained.

You use the scsi_pathcheck operand in the HA Monitor environment settings to specify the check interval. The default value is one minute.

If HA Monitor detects a disk path error, it issues the KAMN725-W and KAMN726-E messages.

If HA Monitor detects recovery of the disk path, it issues the KAMN727-I message.

(4) When there is a server that does not use a shared disk

A shared disk is required in order to obtain a reservation. If the active server does not use a shared disk, it might not be able to detect the hot standby operation that results from a failure, such as a monitoring path failure.

However, reservation can be obtained in a system configuration that consists both of servers that use a shared disk and of servers that do not use a shared disk. Therefore, if there are servers that do not use a shared disk, you must specify either of the settings in the following table.

Table 2‒2: Setting to be specified if there are servers that do not use a shared disk

Setting

Advantage

Disadvantage

Provide a shared disk (for reservation use only) for the servers that do not use a shared disk, and define the provided shared disk for the scsi_device or dmmp_device operand in the server environment definition.

Because these servers do not have a parent-child relation with other servers, they are not affected by order control.

A new shared resource that will not be used for actual operation must be added as a shared disk for reservation use only.

Make a group of the servers that use a shared disk and a group of the servers that do not and then make sure that both groups of servers are run together.

Specific examples of this setting are as follows:

Put all shared disks on the resource server and define each server as a child of the resource server.

Specify a server that uses a shared disk as a parent server of a server that does not use a shared disk. Start the parent server and then the child server in this order, and terminate the child server and then the parent server in this order.

A shared disk for reservation use only is not necessary.

The servers that use a shared disk and the servers that do not use a shared disk have a parent-child relation. Therefore, order control produces the following effects:

When hot-standby switchover occurs, the parent server starts, and then the child servers start. Therefore, the time required for switchover increases.

If the parent server fails to start, the child servers also fail to start.

(5) Notes

In the following cases, the reservation that was obtained cannot be released:

When the OS is shut down while an active server is running

When the reservation fails to release when an active server terminates

When the active server is terminated after hot standby processing was performed because of a host failure but before the failed host and standby server are started

When a host failure occurs on the active system while hot standby is not possible, such as when the standby server is inactive

The reservation that has been obtained might not be released in the following case:

When the reservation was obtained by a user operation

If you try to start an active server without the reservation having been released, the active server fails to start. In this case, make sure that there is no active server on another host, release the reservation, and then start the active server. For instructions on checking and releasing reservations, see 7.5.6 Handling device failures on a shared disk (while the active server is starting) (using SCSI reservation for shared disk) and 7.5.8 Handling device failures on a shared disk (while the active server is terminating) (using SCSI reservation for shared disk).

These procedures for releasing reservations and the procedure described in 7.5.7 Handling device failures on a shared disk (while the active server is running) (using SCSI reservation for shared disk) use the sg_persist command. If the sg_persist command is not installed, install the sg3_utils and sg3_utils-libs packages provided with Linux.

If you release reservations without using the reservation release command (monscsiclr command), you must execute the sg_persist command multiple times. To be able to release reservations with a single procedure, we recommend that you create a shell for releasing reservations in advance. For details about creating a shell for releasing reservations in advance, see 7.5.11 Dealing with the server that does not release the reservation (using SCSI reservation for shared disk).

2.3.7 Function for controlling hot standby based on the availability of LAN communications

The function for controlling hot standby based on the availability of LAN communications is used to control operation in the event of a host failure in such a manner that the active server runs only on a host on which LAN communication is available.

You must decide whether to use this function. For details about how to decide, see 1.4 Hot-standby switchover methods and 3.1 List of functions supported by HA Monitor.

When this function is used, a business application LAN that is used to check the availability of communication must be configured in such a manner that it is also used as a monitoring path.

For examples of configuring the environment in this way, see (10) Example of environment settings when using the function for controlling hot standby based on the availability of LAN communications.

(1) Hot standby operation when a host failure is detected

This subsection explains hot standby operation when the function for controlling hot standby based on the availability of LAN communications is used.

If the multi-standby function is not used

If a host failure is detected in the primary system or the secondary system, HA Monitor checks whether LAN communication is available. HA Monitor then runs the server as the active server on a host on which LAN communication is available and terminates the server on the host on which LAN communication is not available.

The following figure shows the general flow of operations when a host failure is detected due to a monitoring path failure, communication is not available via a business application LAN in the primary system, and communication is available via a business application LAN in the secondary system.

Figure 2‒17: Hot standby operation when communication is not available via a business application LAN in the primary system and communication is available via a business application LAN in the secondary system

The following figure shows the general flow of operations when a host failure is detected due to a monitoring path failure, communication is available via a business application LAN in the primary system, and communication is not available via a business application LAN in the secondary system.

Figure 2‒18: Hot standby operation when communication is available via a business application LAN in the primary system and communication is not available via a business application LAN in the secondary system

If the multi-standby function is used

See (3) Function for controlling hot standby based on the availability of LAN communications.

(2) Operation when the primary system has recovered from a slowdown

If hot standby occurred because the primary system slowed down and this was detected by the secondary system as a host failure, but then the primary system recovered from the slowdown, HA Monitor in the primary system will detect that the active server is running in the secondary system. HA Monitor then performs planned termination on the active server in the primary system.

In a configuration that inherits IP addresses, the IP addresses might overlap temporarily between the primary and secondary systems until planned termination takes place on the active server that has recovered from the slowdown that occurred.

(3) OS panic in the event of a process failure on HA Monitor

If a process failure is detected on HA Monitor, the KAMN066-E message is output and an OS panic is generated to avoid contention for shared resources.

(4) Hot standby operation or planned hot standby operation in the event of an active server failure

In the event of an active server failure, HA Monitor performs a hot standby operation regardless of the status of business application LANs in the same manner as when the function for controlling hot standby based on the availability of LAN communications is not used.

If the shared resource (IP address) disconnection process fails on the source host, HA Monitor generates an OS panic on the source host, and then continues with the hot standby operation. This prevents contention for shared resources.

(5) Conditions of use

The function for controlling hot standby based on the availability of LAN communications can be used when both the following conditions are satisfied:

There are no shared disks in the configuration.

Business application LANs are being used for checking the availability of communication and the following conditions are all satisfied:

The node specified to check the LAN status has an IP address for communication with the monitored LAN in the same network, and a response to ICMP echo packets can be sent.

The node specified to check the LAN status is already running when HA Monitor is started.

The business application LANs are shared with HA Monitor's monitoring path.

(6) Required environment settings

Specify the settings as shown in the following table, as appropriate for the type of business application LANs used for checking the availability of communication.

Table 2‒3: Environment settings depending on the type of business application LANs

Required environment settings

Types of business application LANs#2

hbonding#1

bonding

ethernet

active-backup

mode

802.3ad

mode

HA Monitor environment settings

fence_lan

Y

Y

Y

Y

lanfailswitch

Y

Y

Y

Y

lancheck_patrol

O

O

O

O

hbond_lacp

N

O

N

N

lancheck_mode

O

O

O

O

Server environment definition

switch_judge

Y

Y

Y

Y

switchbyfail

Y

Y

Y

Y

Files used for monitoring LANs

LAN monitor definition file

Y

Y

Y

Y

LAN monitoring script

O

Y

Y

Y

Legend:

Y: Specify.

O: Specify if necessary.

N: Cannot be specified.

#1

The monitoring interval of a slave interface monitored by hbonding must be shorter than the host failure monitoring interval (specified in the patrol operand in the HA Monitor environment settings).

#2

A tagged VLAN can also be used by specifying an interface name that does not include a period (.) for the device name.

For details about the settings related to the monitoring of LANs, see 3.4.1 LAN monitoring and automatic hot standby in the event of a failure, (1) Specifying a LAN monitor definition file, and (2) Specifying a LAN monitoring script.

Note that when a LAN monitoring script (lanpatrol.sh) is used, the time required for a hot standby operation in the event of a host failure includes the time needed for LAN monitoring script execution. Adjust the LAN monitoring script execution time according to the requirements for the hot standby processing time.

If you monitor the number of packets received, specify the host failure monitoring time (patrol operand in the HA Monitor environment settings) as follows:

Host failure monitoring time ≥ lancheck_patrol × 2

(7) Correspondence between server configuration and business application LANs

You must choose one business application LAN to be used for checking the availability of communication for each ungrouped server and for each group.

If there is a server that does not use a business application LAN, either set up a business application LAN to be used for checking the availability of communication, or group that server with other servers that use a business application LAN.

(8) Notes about the configuration of LANs

Note the following about the configuration of LANs:

A maximum of five business application LANs (networks) can be used for checking the availability of communication during a hot standby operation.

In the event of a monitoring path failure, only one of the hosts must be able to communicate via a business application LAN for each server program or group.

2.3.8 Hybrid fencing

The hybrid fencing function is a combination of host reset and SCSI reservation for shared disk.

If hot-standby switchover is performed based on only host reset and host reset fails, the host will be placed in the hot-standby wait state. By contrast, if hot-standby switchover is performed based on hybrid fencing and host reset fails, the system can continue switchover by performing SCSI reservation for shared disk. Therefore, the hot-standby switchover method using hybrid fencing has the following advantage over other methods:

Because the reliability of automatic hot-standby switchover in the event of a host failure is high, system availability can be improved.

Note that if host reset is successful, SCSI reservation for shared disk is not performed.

You must decide whether to use this function. For details about how to decide, see 1.4 Hot-standby switchover methods and 3.1 List of functions supported by HA Monitor.

The following figure shows the processing flow in the case where this function is used.

Figure 2‒19: Processing flow in the case where hot-standby switchover is performed based on hybrid fencing

(1) Required environment settings

Specify the following settings in the HA Monitor environment settings:

Specify use for the fence_reset and fence_scsi operands in the HA Monitor environment settings.

Specify a device for the scsi_device or dmmp_device operand in the server environment definition.

In cases in which the settings in (4) When there is a server that does not use a shared disk under 2.3.6 SCSI reservation for shared disk are not specified, if host reset fails, the servers that do not use a shared disk are placed in the hot-standby wait state.

(2) Time to perform SCSI reservation for shared disk

HA Monitor issues a reset request to the hosts with different times so that the hosts are not reset at the same time. Furthermore, HA Monitor performs SCSI reservation for shared disk after the reset end times of all hosts have passed to prevent hosts from attempting SCSI reservation at the same time. The type of failure in host reset differs depending on the environment as follows.

Basic environment: Failure in a physical reset

Virtualization environment (where the physical partition reset function is not used): Failure in a logical reset

Virtualization environment (where the physical partition reset function is used): Failure in logical reset and physical reset

(3) Behavior when hot-standby switchover is performed based on SCSI reservation for shared disk

If hot-standby switchover is performed based on SCSI reservation for shared disk, system operation continues with a shared disk reserved. HA Monitor releases reservation for shared disks in the following cases:

After the switching-source server (prior active server) is restored as the standby server, the hot-standby switchover destination server (current active server) stops.

After the switching-source server (prior active server) is restored as the standby server, the hot-standby switchover destination (current active server) fails, and host reset is successful.

After HA Monitor is notified that hardware was turned off, the hot-standby switchover destination server stops.

A server in the hot-standby wait state is started as the active server by executing the wait-state server startup command (monact command).

If any of the following events occur before HA Monitor releases reservation of a shared disk, reservation is not released:

The active server stops (in this case, the KAMN728-W message is output).

The OS is shut down when the active server is running.

A host failure occurs in the active system.

If reservation is not released, the next time an attempt is made to start the active server, the attempt fails. Make sure that there is no active server on the other host, release the reservation, and then start the active server. For details about the procedures for checking the reservation status and releasing reservation, see 7.5.6 Handling device failures on a shared disk (while the active server is starting) (using SCSI reservation for shared disk) and 7.5.8 Handling device failures on a shared disk (while the active server is terminating) (using SCSI reservation for shared disk).

(4) Checking the status of SCSI reservation for shared disk (for a host on which the active server is operating)

In the active system in which the active server is operating, HA Monitor periodically checks whether reservation is obtained by the other host. If hot-standby switchover occurs and the standby system reserves the shared disk, the active system detects it, and stops the server.

For details about the check, see (a) Checking the reservation status from the active system in (3) Checking the reservation status under 2.3.6 SCSI reservation for shared disk.

(5) Checking the path to a shared disk (for a host on which the standby server is operating)

HA Monitor obtains the status of the shared disk from the standby system to check the status of the disk path. If host reset fails, the host might be placed in the hot-standby wait state because the shared disk cannot be reserved. To prevent this, HA Monitor checks whether the shared disk can be reserved. If the shared disk cannot be reserved, HA Monitor notifies the user accordingly so that the user can take measures beforehand.

For details about the check, see (b) Checking the shared disk path status from the standby system in (3) Checking the reservation status under 2.3.6 SCSI reservation for shared disk.

(6) Behavior if both host reset and SCSI reservation for shared disk fail

If both host reset and SCSI reservation for shared disk fail, the server is placed in the hot-standby wait state. The following figure shows the system behavior.

Figure 2‒20: Behavior if both host reset and SCSI reservation for shared disk fail

(7) Results (success/failure) of hot-standby switchover of grouped servers

For grouped servers, whether hot-standby switchover is successful depends on the server configuration in the group (whether the servers are under order control and whether they use a shared disk), and the result of SCSI reservation for shared disk. The following describes the results of hot-standby switchover in the preceding conditions.

(a) If the servers are not under order control

In the case where servers use a shared disk

As shown in the following figure, hot-standby switchover occurs according to whether reservation of each server succeeds or fails.

Figure 2‒21: Results of hot-standby switchover for grouped servers that are not under order control (in the case where servers use a shared disk)

In the case where servers do not use a shared disk

As shown in the following figure, these servers are placed in the hot-standby wait state.

Figure 2‒22: Results of hot-standby switchover for grouped servers that are not under order control (in the case where servers do not use a shared disk)

(b) If the servers are under order control

In the case where the child server uses a shared disk and the parent server succeeded in reservation

As shown in the following figure, hot-standby switchover occurs according to whether reservation of each server succeeds or fails.

Figure 2‒23: Results of hot-standby switchover for grouped servers that are under order control (in the case where the child server uses a shared disk and the parent server succeeded in reservation)

In the case where the child server uses a shared disk and the parent server failed in reservation

As shown in the following figure, the result of reservation by the parent server affects the child server, and both the parent server and child server are placed in the hot-standby wait state.

Figure 2‒24: Results of hot-standby switchover for grouped servers that are under order control (in the case where the child server uses a shared disk and the parent server failed in reservation)

In the case where the child server uses no shared disk and the parent server succeeded in reservation

As shown in the following figure, the result of reservation by the parent server affects the child server, and hot-standby switchover occurs on both the parent server and child server.

Figure 2‒25: Results of hot-standby switchover for grouped servers that are under order control (in the case where the child server uses no shared disk and the parent server succeeded in reservation)

In the case where the child server uses no shared disk and the parent server failed in reservation

As shown in the following figure, the result of reservation by the parent server affects the child server, and both the parent server and child server are placed in the hot-standby wait state.

Figure 2‒26: Results of hot-standby switchover for grouped servers that are under order control (in the case where the child server uses no shared disk and the parent server failed in reservation)

(8) Notes

The sg_persist command might be used when SCSI reservation for shared disk is released. If the sg_persist command has not been installed, install the sg3_utils and sg3_utils-libs packages that are bundled with Linux.

2.4 Inheriting shared resources

This section describes inheritance of shared resources for hot standby using HA Monitor.

Shared resources are the resources that are shared by the active and standby systems. During the hot standby operation, the shared resources are disconnected from the system resulting in a failure and then connected to the standby system. This processing involving disconnection and connection is called inheriting. For details about the HA Monitor processing that takes place when shared resources are connected and disconnected, see 4.3 Managing shared resources.

2.4.1 Resources that can be inherited and their units

HA Monitor enables the following types of resources to be inherited:

Shared disks

File systems

LANs

In addition to the above types of resources, other resources can also be inherited by HA Monitor if you create a user command beforehand.

Unit of inheritance

Shared resources are inherited for each server. To share a single shared resource among multiple servers, you can provide a dedicated server called a resource server.

Prevention of conflict on shared resources

If separation of a shared resource fails in the active system during the hot standby operation, a conflict might occur in accesses to the shared resource, resulting in corruption of the shared resource and system shutdown. To prevent both hosts from conflicting with each other for a shared resource, HA Monitor forcibly stops the active system to perform hot-standby switchover in the following cases:

If disconnection of a shared resource fails during the processing of hot-standby switchover

If disconnection processing times out with the timeout value specified for the dev_timelimit and dev_offlimit operands in the server environment definition

The method used to forcibly stop the active system differs depending on the condition in which hot-standby switchover is performed if a host failure occurs.

If host reset is to be performed: Host is reset.

If SCSI reservation for shared disk is used or the function for controlling hot standby based on the availability of LAN communications is used: The host that failed to separate the shared resource shuts down the OS by generating a kernel panic.

2.4.2 Inheriting a shared disk

HA Monitor uses an LVM to inherit a shared disk. While the shared disk is being used by the active server, HA Monitor applies a logical lock to the shared disk so that it cannot be accessed from the standby server.

When LVM is used, the shared disk is inherited in units of volume groups. LVM controls the connection between HA Monitor and the shared disk.

2.4.3 Inheriting a file system

HA Monitor inherits a file system on the shared disk by mounting and unmounting it. The mount and unmount operations are controlled by HA Monitor.

Connecting the file system

HA Monitor connects the shared disk and then connects the file system. The file system is mounted when the active server starts so that the active server can use the file system. When the file system is mounted, its consistency is checked by using the fsck command of the OS (or the xfs_repair command if the file system of the shared disk is XFS in RHEL7 or a later version).

Disconnecting the file system

In the event of a failure, HA Monitor inherits the file system by unmounting it from the host resulting in the failure before separating the shared disk and then mounting the file system from the host that inherits job processing. If there are processes that are using the file system when it is unmounted, HA Monitor uses the fuser OS command to forcibly terminate those processes.

2.4.4 Inheriting a LAN

HA Monitor inherits alias IP addresses.

An alias IP address is an alias assigned to an existing IP address. Alias IP addresses enable you to assign multiple IP addresses to a single interface. HA Monitor switches LANs by inheriting these alias IP addresses from the active system to the standby system during the hot standby operation.

Stationary IP addresses and alias IP addresses

A unique IP address must be assigned to each host for each LAN interface. This IP address is called a stationary IP address. Stationary IP addresses are not inherited by the remote system during the hot standby operation. Therefore, stationary IP addresses are used for servers that do not inherit IP addresses, communication for programs that are not subject to hot standby, and monitoring paths.

In addition to the stationary IP address, you can assign one or more additional IP addresses to a server. Such an IP address is called an alias IP address. HA Monitor inherits alias IP addresses. Therefore, alias IP addresses are used for servers that inherit IP addresses and communication for programs that are subject to the hot standby operation. If alias IP addresses are used, the client can use the same IP addresses for communication even after the hot standby operation has occurred. Note that for a TCP-base application, recovery processing such as reconnection is required because connection is released.

You can assign alias IP addresses only to an interface for which a stationary IP address has already been specified.

Inheriting alias IP addresses

An alias IP address assigned to a server is added to the LAN interface when the active server starts and is deleted from the LAN interface when the active server terminates. During the hot standby operation, an alias IP address is inherited by deleting the alias IP address from the active system and adding it to the standby system.

HA Monitor does not inherit MAC addresses. However, when inheriting alias IP addresses, HA Monitor executes a command for broadcasting an ARP request and reports a new mapping of IP addresses and MAC addresses.

The following figure shows an example of inheriting alias IP addresses.

Figure 2‒27: Example of inheriting alias IP addresses

In this example, the IP addresses (a.b.c.d and e.f.g.h) assigned to the LAN adapter eth1 are stationary IP addresses. The IP addresses (s.t.u.v and w.x.y.z) assigned to the server are alias IP addresses. If hot standby switching occurs from host 1 to host 2, the alias IP addresses are inherited to host 2.

Part 2: Description

3 Functions Supported by HA Monitor

This chapter describes the hot standby functions supported by HA Monitor. It provides an overview of the functions and explains the required environment settings for servers, hosts, and resources.

3.1 List of functions supported by HA Monitor

The following table shows the functions supported by HA Monitor depending on the hardware that is used.

Table 3‒1: Functions supported by HA Monitor depending on the hardware that is used

Function

Hardware

Classification

Function name

BladeSymphony, HA8000, or RV3000

PC/AT-compatible system from another company

Shared disk data protection method

Function for controlling hot standby based on the availability of LAN communications

Shared disk data protection method

Function for controlling hot standby based on the availability of LAN communications

Host reset

SCSI reservation

Hybrid fencing

SCSI reservation

Server control

Monitoring servers in the monitor mode

Y

Y

Y

Y

Y

Y

Monitoring in the standby system in the monitor mode

Y

Y

Y

Y

Y

Y

Grouped-system switchover of grouped servers

Y

Y

Y

Y

Y

Y

Server switchover order control

Y

Y

Y

Y

Y

Y

Multi-standby function allowing deployment of multiple standby systems

Y

Y

--

Y

Y

Y

Locking servers to prevent processing load concentration after hot-standby switchover

Y

Y

Y

Y

Y

Y

Performing start retries during hot standby processing

Y

Y

Y

Y

Y

Y

System operation in linkage with JP1

Y

Y

Y

Y

Y

Y

Hot-standby switchover in the online serverless mode

Y

Y

Y

Y

Y

Y

Host control

Preventing the hosts from being reset at the same time

Y

--

Y

--

--

--

Preventing concurrent resets between multiple hosts

Y

--

Y

--

--

--

Preventing a host from being reset twice

Y

--

Y

--

--

--

Preventing a host from being reset

Y

--

--

--

--

--

Host pair shutdown for each host in the event of a server failure

Y

--

Y

--

--

--

Host pair shutdown performed for each host in the event of a user command error

Y

Y

Y

Y

Y

Y

Host pair shutdown for each host in the event of a server termination command error

Y

Y

Y

Y

Y

Y

Physical partition reset function in a virtualization environment

Y

--

Y

--

--

--

Hot-standby switchover in an N+M cold standby configuration

Y#

--

--

--

--

--

Shared resource control

Sharing shared resources using a resource server

Y

Y

Y

Y

Y

Y

Issuing a command when the status of a server or HA Monitor changes

Y

Y

Y

Y

Y

Y

Canceling server startup in the event of a shared resource connection error

Y

Y

Y

Y

Y

Y

Timeout for inheritance of shared resources

Y

Y

Y

Y

Y

Y

Specifying the order in which shared resources are disconnected

Y

Y

Y

Y

Y

Y

Dynamically changing shared resources

Y

--

--

--

--

--

Resource monitoring

Monitoring of the LAN and automatic hot-standby switchover in the event of a failure

Y

Y

Y

Y

Y

Y

Disk monitoring

Y

Y

Y

Y

Y

Y

Program control

Monitoring UAPs

Y

Y

Y

Y

Y

Y

HA Monitor Extension

Changing the maximum number of servers

Y

--

--

--

--

--

Changing the maximum number of hosts

Y

--

--

--

--

--

Legend:

Y: Supported.

--: Not supported.

#: Unavailable with an RV3000

3.2 Functions for server control

This section describes the functions that you can use on the servers in a hot-standby configuration.

3.2.1 Monitoring a server in the monitor mode (for the active server)

This subsection explains the function for monitoring servers in the monitor mode. This function enables the hot standby operation to be performed automatically in the event of a server failure on a server in the monitor mode. Such a hot standby operation is performed as a planned hot standby.

If you use the monitor-mode program management function to monitor UAPs, also see 3.6 Function for controlling programs (monitor mode).

(1) How to monitor servers

On a server in the monitor mode, the method to detect an application or task failure differs depending on the specifications of the application or task. Therefore, to notify HA Monitor that an application or task failure occurred, the user must create a server monitoring command.

A server monitoring command can be specified in either of the following ways:

Specifying a monitoring command for the ptrlcmd_ex operand in the server environment definition

Specifying a monitoring command for the patrolcommand operand in the server environment definition

In the following cases, we recommend that you specify a monitoring command by using the ptrlcmd_ex operand in the server environment definition:

Hot-standby switchover needs to be performed when HA Monitor detects an application or task slowdown.

Monitoring of servers needs to be temporarily stopped and restarted.

The following describes the flow of failure monitoring processing in the case where the ptrlcmd_ex and patrolcommand operands are specified in the server environment definition.

(a) Specifying a monitoring command for the ptrlcmd_ex operand in the server environment definition

If a monitoring command is specified for the ptrlcmd_ex operand in the server environment definition, when startup of the server finishes, HA Monitor executes the command at regular intervals to determine the command execution result (EXIT code). The interval is specified in the ptrlcmd_ex_inter operand in the server environment definition. HA Monitor can detect a server failure from the EXIT code. If the ptrlcmd_ex_tmout operand is specified in the server environment definition and command execution does not end within the time specified for that operand, HA Monitor assumes that a slowdown occurred. In this case, HA Monitor determines that a server failure occurred on the server.

The following figure shows an overview of monitoring a failure on a server in the monitor mode (in the case where a monitoring command is specified for the ptrlcmd_ex operand in the server environment definition).

Figure 3‒1: Monitoring a failure on a server in the monitor mode (in the case where a monitoring command is specified for the ptrlcmd_ex operand in the server environment definition)

Important

If a monitoring command is specified for the ptrlcmd_ex operand in the server environment definition, the following items cannot be specified:

Setting of a monitoring command specified for the patrolcommand operand in the server environment definition

Monitor-mode program management function

For details about using HA Monitor to control the server monitoring command, see 4.1.4 Controlling the server monitoring command.

The monchange command allows you to change the settings and operations while HA Monitor and servers are running. You can use this command to temporarily stop and restart server monitoring. For details, see 9.6 monchange (changes settings and operations while HA Monitor and servers are running).

(b) Specifying a monitoring command for the patrolcommand operand in the server environment definition

If a monitoring command is specified for the patrolcommand operand in the server environment definition, HA Monitor automatically starts the command when startup of the server is completed. HA Monitor then checks for the command's process at 1-second intervals. When HA Monitor detects termination of the server monitoring command, it determines that a failure has occurred on the server.

The following figure shows an overview of monitoring a failure on a server in the monitor mode (in the case where a monitoring command is specified for the patrolcommand operand in the server environment definition).

Figure 3‒2: Monitoring a failure on a server in the monitor mode (in the case where a monitoring command is specified for the patrolcommand in the server environment definition)

For details about using HA Monitor to control the server monitoring command, see 4.1.4 Controlling the server monitoring command.

(2) HA Monitor processing after detection of a server failure

When HA Monitor detects a server failure, it performs one of the following operations:

Restarts the active server.

HA Monitor restarts the active server on the host resulting in the server failure. You can use the servexec_retry operand in the server environment definition to specify the number of start retries.

If the restarted server has been operating normally for a certain period of time, HA Monitor assumes that the server operation is stable, and resets the number of start retries to 0. The time required for HA Monitor to assume stable operation differs depending on the operand for which the monitoring command is specified, as follows:

If the monitoring command is specified for the ptrlcmd_ex operand in the server environment definition:

(The interval time specified for the ptrlcmd_ex_inter operand in the server environment definition) × (the count specified for the ptrlcmd_ex_retry operand in the server environment definition)

Note, however, the monitoring command must continue to return an EXIT code of 0 until the preceding time period expires.

If the monitoring command is specified for the patrolcommand operand in the server environment definition:

The time period specified for the retry_stable operand in the server environment definition

Performs the hot standby operation.

In the following cases, HA Monitor stops the active server and begins to switch to the standby server.

The value 0 has been specified in the servexec_retry operand in the server environment definition.

The number of server failures that have been detected has reached the value specified in the servexec_retry operand in the server environment definition.

The monitoring command is specified for the ptrlcmd_ex operand in the server environment definition, and the server monitoring command terminates with an EXIT code of 20 to 29.

Note that if hot standby processing cannot be performed for a reason such as that there is no available standby server, the server status remains under execution processing in the HA Monitor. Restart the active server or perform hot standby processing, as appropriate for your operation.

Performs host pair shutdown.

If the monitoring command is specified for the ptrlcmd_ex operand in the server environment definition, and the pair shutdown function is used (use is set for the pairdown operand in the server environment definition), pair shutdown is performed.

For details about the processing flow for starting and terminating the server monitoring command, and for monitoring, see 4.1.4 Controlling the server monitoring command.

(3) Required environment settings

The user must create a server monitoring command. The user also must specify the created server monitoring command for the ptrlcmd_ex or patrolcommand operand in the server environment definition.

Some programs themselves provide a command for monitoring the program. For details about how to create a server monitoring command, see 6.12.3 Creating a server monitoring command.

3.2.2 Monitoring a server in monitor mode (for the standby server)

This subsection describes a function that enables monitoring in the standby system in the monitor mode. With this function, you can start processes, and can monitor processes and resources, in the standby system. For example, this function can be used in the following cases:

When creating a hot-standby switchover configuration in which the processes of an application run on both the active system and standby system

A feature of the function that enables monitoring in the standby system in the monitor mode is that the user can start and monitor applications even in the standby system. For example, this function can be applied to an application whose processes run in both the active system and standby system to synchronize their data.

The following figure shows a hot-standby switchover configuration that uses an application whose processes run in both the active system and standby system.

Figure 3‒3: Hot-standby switchover configuration that uses an application whose processes run in both the active system and standby system

When detecting an error by monitoring processes and resources (such as LAN and disk paths) in the standby system

The function that enables monitoring in the standby system in the monitor mode can be used to monitor processes and resources (such as LAN and disk paths) in the standby system. If a failure is detected in the standby system, you can select either of the following types of system behavior:

HA Monitor stops the standby server so that hot-standby switchover cannot be performed.

HA Monitor attempts to restart the standby server so that hot-standby switchover can be performed. After the standby server is restarted successfully, if the monitoring command does not return a failure for a certain period of time, HA Monitor assumes that the standby server is running stably, and resets the number of start retries.

The preceding behavior can prevent task operation from discontinuing if hot-standby switchover occurs before the standby system is recovered from a failure.

The following figure shows an example of monitoring a resource (path to the shared disk) in the standby system.

Figure 3‒4: Example of monitoring a resource (path to the shared disk) in the standby system

(1) Method of monitoring

During monitoring in the standby system in the monitor mode, any commands can be executed to start, stop, and monitor the standby server#. The standby server is stopped or restarted according to the command execution result.

#

In a multi-standby configuration, all standby servers are subject to the command regardless of priority. In this case, the standby server monitoring command is executed on all standby servers. Therefore, make sure that the active server is not affected when, for example, resources are being monitored.

The monchange command allows you to change the settings and operations while HA Monitor and servers are running. You can use this command to temporarily stop and restart server monitoring. For details, see 9.6 monchange (changes settings and operations while HA Monitor and servers are running).

(2) Required environment settings

To perform monitoring in the standby system in the monitor mode, use the following procedure to specify the environment settings:

Create commands that will be used to start, stop, and monitor the standby server.

For details about how to create the commands, see 6.13 Creating commands that will be used to start, stop, and monitor the standby server (in the case where monitoring is performed in the standby system).

Specify the commands created in step 1 for the following operands:

Server environment definition:

sby_actcommand

sby_termcommand

sby_ptrlcmd_ex

sby_ptrlcmd_ex_inter

sby_ptrlcmd_ex_retry

sby_ptrlcmd_ex_tmout

sby_start_timeout

sby_stop_timeout

sby_servexec_retry

3.2.3 Grouped-system switchover achieved by grouping servers

This subsection explains the function for using the batch mode to switch over multiple servers all at once in the event of a failure. This function, which is called grouped-system switchover, is used when multiple servers are running in the active system. Grouped-system switchover is achieved by grouping multiple servers. A grouping of servers is called a server group. When at least two servers in a server group become available for hot-standby switchover, the server group is ready for grouped-system switchover. If a job uses multiple servers and you group its servers into a server group, you can switch all them in the batch mode when a failure occurs on one of the servers. A maximum of 128 servers can be placed in a group per host.

(1) Overview of grouped-system switchover

If a failure occurs on one of the servers in a server group in the active system, all servers in that group are switched over to the standby system. Planned hot standby is performed for the servers that have not failed.

Grouped-system switchover supports two types of hot-standby switchover: automatic hot-standby switchover and planned hot-standby switchover. With the former type, the host is switched automatically if a failure occurs. With the latter type, the host is switched by executing the server hot-standby switchover command (monswap command) with the -g option specified.

The following figure provides an overview of grouped-system switchover.

Figure 3‒5: Overview of grouped-system switchover

If a host failure occurs in the active system, or if a server failure occurs on one of the servers in a server group, or when the operator performs the planned hot standby operation, HA Monitor switches all the servers in the server group over to the standby system. For servers in the server mode, you can specify individually for each server in the server group either to perform grouped-system switchover if this server fails or to continue job processing on the server group without the hot standby operation being performed, even if this server fails. Because this function enables you to decide whether or not switchover is to be performed for the server group depending on the importance of individual servers, you can prevent the hot standby operation from occurring unnecessarily. For details about the settings required in order to use this function, see 4.4.1 Server switchover type during grouped-system switchover.

Normally, grouped servers are switched over individually. For details about how to control the switchover order for the servers in a group, see 3.2.4 Controlling the server switchover order.

If you want to share resources among multiple servers, include a resource server in the server group. A maximum of 64 resource servers can be placed in groups per host. For details about resource servers, see 3.5.1 Using a resource server to share shared resources.

If use is specified in the servcomplete_msg operand in the HA Monitor environment settings, the KAMN496-I message is output when the grouped-system switchover is completed.

(2) Required environment settings

In order to group servers, you must specify the group operand in the server environment definition. This operand is required for all servers that are to be run in a group.

If you run the servers in a server group in the monitor mode, you must create a server termination command and then specify that command in the termcommand operand in the server environment definition (in addition to specifying the group operand). Additionally, if you create a server monitoring command (and specify the command in the ptrlcmd_ex or patrolcommand operand in the server environment definition), you can perform grouped-system switchover in the event of a server failure occurs in a server in the monitor mode.

Before HA Monitor can perform a grouped-system switchover, the conditions listed below must be satisfied. Specify the system configuration and environment settings in such a manner that these conditions will be satisfied:

The same group name is specified for each active server and its corresponding standby server.

The hot-standby target servers must be on the same host for all the servers in the same server group.

The same start type must be specified for all the servers in the same server group.

Each and every server in the server group is a member of a pair (and the standby server corresponding to the active server has been started).

Grouping of servers is determined when the servers are started. If a single host contains both active and standby servers, only the active servers can be grouped. On a host in which servers have already been grouped, if a new server is started that does not satisfy the conditions, that server will not be subject to grouping.

3.2.4 Controlling the server switchover order

You can control the order in which the servers in a server group are to be terminated during a hot standby operation, which will also be the order in which the servers are started on the target host. To control these orders, you group multiple servers together and then define parent-child relationships among those servers that correspond to the desired order.

Server switchover order control is applied only after a server failure or host failure has occurred and when a grouped-system switchover is to be performed during a planned hot standby operation. Server switchover order cannot be controlled during normal server start and termination (the servers are started and terminated regardless of a specified start or termination order).

Note that if a server in the monitor mode is using the program management function, the switchover order among the UAPs and programs managed by the server cannot be controlled.

(1) Correspondence between server start order and parent-child relationships

This subsection explains how to define parent-child relationships for purpose of determining a server start order.

A server required in order to start another server is called a parent server. A server that is started after the parent server has started is called a child server.

Servers are started in order starting with the parent server. When the parent server has been started, its child servers are started. The termination order is the reverse of the start order. A parent server's termination processing begins when all its child servers have been terminated.

The figure below shows the parent-child relationships and start order for a server group. For example, if you want to start server1, server2, server3, and server4 in this order, their parent-child relationships are as shown in the figure.

Figure 3‒6: Parent-child relationships and start order of a server group

In this example, server1 is the parent server of server2, and server2 is the parent server of both server3 and server4.

Take the following into account with respect to server parent-child relationships:

A child server can have only one parent server. You must not define more than one parent server for each child server. On the other hand, a parent server can have multiple child servers, so you can define multiple child servers for a single parent server.

You can assign to a server group servers for which no order relationship is defined.

If the server group contains a resource server, the resource server is the highest parent server.

While you can control the start and termination orders for the servers in a group, you cannot define order control among groups when there are multiple groups.

For details about the processing flow for controlling server switchover order, see 4.1.5 Processing flow for order control of server switchover.

(2) Required environment settings

You must define the server grouping settings in the group operand in the server environment definition beforehand.

When you define parent-child relationships for multiple servers in a group, be sure to take into account the desired server start order. You specify the parent operand in the server environment definition to define a parent-child relationship. There is no requirement to specify a parent operand for all servers.

You use the command for displaying the server order control status (monodrshw command) to check the defined start order (by checking the parent-child relationships within the group).

3.2.5 Multi-standby for providing multiple standby systems

The multi-standby function is used to provide multiple standby servers for a single active server. Use of multiple standby servers protects against system failures while the active system is being recovered. For example, if a failure occurs in the standby system during the hot standby operation, this function enables switchover to be made to another standby system.

The multi-standby function is required in order to use a multi-standby configuration. This subsection explains hot standby using the multi-standby function.

(1) How to determine the hot standby target

When you use the multi-standby function, you must define the priority order for choosing the hot standby target because there are multiple eligible standby systems. In the event of a failure, HA Monitor determines the hot standby target based on the specified priority order.

(2) In the event of a failure in the active system

When the multi-standby function is being used and a failure occurs on the active server, HA Monitor performs the hot standby operation automatically according to the specified priority order. The following figure shows the hot standby operation in the event of a failure in the active system when the multi-standby function is used.

Figure 3‒7: Hot standby operation when the multi-standby function is used

If a failure occurs in the active server, the server is switched over to the available standby server with the highest priority. You must specify in advance in the server environment definition the priorities of the standby servers.

(3) In the event of a failure in the target standby system during hot standby operation

If a failure occurs in the target standby system during the hot standby operation and another standby system is running, HA Monitor switches over to that standby system.

The following figure shows HA Monitor processing in the event of a failure in the target standby system during the hot standby operation.

Figure 3‒8: HA Monitor processing in the event of a failure in the target standby system during the hot standby operation

If a failure occurs in standby system 1 while hot standby switching to standby system 1 is underway, HA Monitor switches to standby system 2, because it has the next highest priority. This enables job processing to continue even when a failure occurs in the standby system.

(4) For planned hot standby

When the operator uses an HA Monitor command to perform the hot standby operation, the hot standby target will be the standby system with the highest priority.

If you want to start a standby server on a desired host, terminate the active server and then start the standby server on the other host or terminate the standby servers with a higher priority than the priority of the desired switchover target standby server, and then perform planned hot standby.

(5) For grouped-system switchover

When you use the multi-standby function, you can also perform a grouped-system switchover that switches over multiple servers in the batch mode. In this case, HA Monitor performs the hot standby operation on the system that has the highest priority among all standby systems.

Note the following when you use the multi-standby function, because there are multiple hot standby targets:

The server start type and priority of all standby servers within the same host must be the same.

All standby servers must be running. If any standby servers are not running, the switchover targets of the servers in the server group are not consolidated.

If no_exchange is specified as the switchover type, only the standby server in each of the standby systems with the highest priority is placed in grouped-system switchover wait state.

For details about the server start type and about grouped-system switchover wait state for grouped-system switchover, see 4.4.1 Server switchover type during grouped-system switchover.

(6) Required environment settings

You specify the multistandby operand in the HA Monitor environment settings to use the multi-standby function. You use the standbypri operand in the server environment definition to specify priorities. You must specify a different standby server priority value for each host. Zero (0), which is the highest priority, is set automatically for the active server.

(7) Hardware configuration

When you employ a hot-standby configuration that uses the multi-standby function, your hardware configuration must satisfy the following conditions:

All hosts must be connected to the monitoring path.

All hosts must be connected to the reset path.

Shared disks and LANs must be accessible from all hosts that use them.

3.2.6 Locking servers to prevent concentration of the processing load after the hot standby operation

When one of the standby servers provided on a host becomes the active server, HA Monitor can terminate the other standby servers. Such a terminated server is called a locked server. While this function can be applied in a 1-to-1 switchover configuration, we recommend that you use it in an n-to-1 switchover configuration (n: 2 or more).

In a configuration that places multiple standby servers in a single secondary system, such as a 2-to-1 switchover configuration, multiple jobs are executed on a single host after the hot standby operation. If processing performance, such as memory usage and server performance, is important after the hot standby operation, the server lock function enables you to prevent multiple active servers from being run on the same host.

(1) Overview of server lock

In order to use the server lock function, you must specify locked servers in the environment settings beforehand. When the active server is started on a host, HA Monitor terminates the other standby servers running on the same host. Note that HA Monitor does not terminate the standby servers that belong to the same group as the server that was started as the active server. Because these standby servers are specified in the same group, HA Monitor performs grouped-system switchover.

To start a locked server that has been terminated by the user, you must either terminate the active server running on the same host or perform hot standby switching to another host.

The figure below shows the hot standby operation when server lock is used. This example uses a 2-to-1 switchover configuration and defines standby server 2 as a locked server.

Figure 3‒9: Hot standby operation when server lock is used

(2) Required environment settings

To use the server lock function, you must specify environment settings for locked servers. For details about the environment settings for locked servers, see 8.4.2 Locked server environment settings (servers_opt).

3.2.7 Performing start retries during hot standby processing

If resource connection and server startup fail on the target host during hot standby processing and there is no available switchover host, a hot standby error occurs and the job stops.

HA Monitor enables you to retry restarting the server on the target host, thereby avoiding a hot standby error due to a transient cause.

The following describes the servers that can use the start retry function and the start retry range:

Server-mode servers other than TP1/EE

In the event of a hot standby error, restart is retried either until hot standby processing is completed or for as many times as specified.

Servers in the monitor mode and resource servers

During hot standby processing, restart is retried either until hot standby is enabled or for as many times as specified.

If the server on the target host fails to start, server restart is attempted in the following cases:

Server in the server mode failed to start.

Server in the monitor mode failed to start or a server monitoring command detected a failure.

The function for canceling server startup in the event of a shared resource connection error is being used and the shared resource connection processing failed.

The function for canceling server startup in the event of a shared resource inheritance timeout error is being used and the shared resource connection processing timed out.

SCSI reservation for shared disk is being used as a shared disk data protection method and the reservation failed.

If another host's active server starts while start retries are underway on the local host, the server on the local host will be started as a standby server.

In a grouped-system switchover configuration, start retries are performed for each server. If a server group contains a server engaged in start retry processing and another server in the same server group successfully switches over to the remote host, the start reties are canceled.

(1) Overview of start retries during hot standby processing

If the server on the target host fails to start, HA Monitor uses the following procedure to retry server startup:

HA Monitor places the server in the start retry state (>ONL) and waits for the server to be restarted.

The server is restarted (the server must be set to be restarted).

When HA Monitor accepts the server restart, it starts the server as the active server.

When start retries are performed, the flow of processing, including messages, user commands, and JP1 events, is the same as during normal active server startup, not during hot standby processing.

The following figure shows the flow of start retry processing.

Figure 3‒10: Flow of start retry processing

The following figure shows the flow of start retry processing when the multi-standby function is used.

Figure 3‒11: Flow of start retry processing when the multi-standby function is used

(2) Required environment settings

You must specify the switch_error operand in the server environment definition to perform start retries during hot standby processing.

In server mode, TP1/Server Base and HiRDB must also be set up. For details, see the switch_error operand in 8.4.1 Server environment definition (servers).

For a server in the monitor mode or a resource server, you can specify a retry count in the switch_retry operand in the server environment definition.

3.2.8 System operation linked with JP1

HA Monitor can be linked to a JP1 system to achieve central management of the systems in a hot-standby configuration. When changes occur in the systems in a hot-standby configuration in which HA Monitor is running, HA Monitor issues JP1 events. You can view the issued events by using a program such as the integrated JP1 management window.

If you link other application systems as well as the systems in the hot-standby configuration to a JP1 system, you can centrally manage operations in all the systems from the JP1 system.

(1) Correctly using user commands and JP1 events

You can use JP1 events and HA Monitor's user commands to operate a system in which HA Monitor is running. You must use them correctly as appropriate to the purpose, as described below.

If you want to perform some processing or operation automatically when a message is output:

Use JP1 events. JP1 events are useful when there is no need to synchronize with a change in server status, such as turning on a patrol lamp in the event of an error.

JP1 events provide the following benefits:

Because JP1 events are issued when messages are issued, more JP1 events are issued than user commands.

There is no need for the user to create events because events that have already been prepared are used.

JP1 events provide more detailed information about failures than messages do.

For details about the event IDs of the JP1 events that are issued by HA Monitor, see A. Event IDs Used by HA Monitor.

If you want to perform some processing or operation automatically when the server or HA Monitor status changes:

Create user commands. When the server or HA Monitor status changes, HA Monitor issues a user command created by the user. User commands are issued when shared resources are started and terminated and when inheritance processing is performed. User commands are useful for automatically processing resources that are not supported by HA Monitor.

For details about user commands, see 3.5.2 Issuing commands when the server and HA Monitor status changes.

(2) Required environment settings

To run your system linked with a JP1 system, you must install the appropriate JP1 product and set up its environment. For HA Monitor to be able to link with a JP1 system, JP1/Base's physical host must be running on the host on which HA Monitor is running. You must start JP1/Base before you start HA Monitor.

Note also that you must specify the jp1_event operand in the HA Monitor environment settings.

3.2.9 Hot-standby switchover in the online serverless mode

The online serverless mode is a function that allows only standby servers to start in advance on a remote host on which no active server exists. If you use this function in a system with a disaster recovery configuration consisting of an original site and a subsite, the time required for starting the active server in the subsite is reduced.

Notes

For servers in the monitor mode, the time required for starting the active server cannot be reduced by using the online serverless mode. However, the following operations are available during hot-standby switchover:

- Group the servers in the monitor mode with servers in the server mode.

- Control the server switchover order of the servers in the monitor mode and those in the server mode.

You cannot use the online serverless mode if you are using the multi-standby function that allows deployment of multiple standby systems.

The online serverless mode is applicable when hot-standby switchover is performed for sites in the following conditions:

The active server and standby server are running in the original site, and they are ready for hot-standby switchover.

No active server exists in the subsite.

When you use the online serverless mode in such conditions, you can start the standby servers in the subsite before switching over the site.

(1) Flow of tasks for hot-standby switchover in the online serverless mode

The following describes the flow of tasks for performing planned hot-standby switchover for sites by using the online serverless mode in the following three steps:

Setting the online serverless mode

Hot-standby switchover in the online serverless mode

Releasing the online serverless mode

Setting the online serverless mode

The following figure shows the flow of tasks for setting the online serverless mode.

Figure 3‒12: Flow of tasks for setting the online serverless mode

The following describes the flow of tasks for setting the online serverless mode. The numbers correspond to the numbers in the figure.

Execute the HA Monitor start command (monstart command) for the subsite.

HA Monitor in the subsite starts.

Execute the command for changing the settings and operations while HA Monitor and servers are running (monchange command) for the subsite.

HA Monitor is placed in the online serverless mode.

Execute the server start command for the subsite.

The standby servers in the subsite start.

#1

All servers run in the online serverless mode no matter whether the server mode or monitor mode is set.

#2

In the online serverless mode, you can start a standby server on each host regardless of the value of the initial operand in the server environment definition. When startup of a standby server finishes, the status of the server changes to +SBY+ (startup completed as a standby server in the online serverless mode).

Hot-standby switchover in the online serverless mode

The following figure shows the flow of tasks for hot-standby switchover in the online serverless mode.

Figure 3‒13: Flow of tasks for hot-standby switchover in the online serverless mode

The following describes the flow of tasks for hot-standby switchover in the online serverless mode. The numbers correspond to the numbers in the figure.

Stop synchronization of the shared disks, and then terminate the active server and standby server in the original site if necessary.

Execute the wait-state server start command (monact command) for the subsite.

The active server starts in the primary system of the subsite.

Hot-standby switchover becomes possible in the subsite, and site switchover is complete.

#1

To start the active server in the +SBY+ state (startup completed as a standby server in online serverless mode), execute the wait-state server start command (monact command). To normally terminate the standby server in the +SBY+ state, execute the standby server termination command (monsbystp command).

#2

Hot-standby switchover processing is performed, which allows the active server to be started in the +SBY+ state by using the wait-state server start command (monact command). For the following operands, the same processing as for hot-standby switchover is performed, rather than the processing to be performed when the active server starts.

- usrcommand operand in the HA Monitor environment settings

- termcmd_at_abort operand in the HA Monitor environment settings

- servcomplete_msg operand in the HA Monitor environment settings

- parent operand in the server environment definition

- uoc_pairdown operand in the server environment definition

- dev_timelimit operand in the server environment definition

- dev_onlimit operand in the server environment definition

#3

The online serverless mode allows a standby server to start on each host. This enables a configuration in which a standby server exists on each of the primary system and the secondary system in the subsite. A server started as the active server in the primary system is ready for hot-standby switchover within the site.

Releasing the online serverless mode

The following figure shows the flow of tasks for releasing the online serverless mode.

Figure 3‒14: Flow of tasks for releasing the online serverless mode

The following describes the flow of tasks for releasing the online serverless mode. The numbers correspond to the numbers in the figure.

Execute the command for changing the settings and operations while HA Monitor and servers are running (monchange command) for the subsite.

The online serverless mode is released.

For details about the status transition of a standby server in the online serverless mode, see (3) Standby server's status transitions in the online serverless mode under 4.1.3 Server status transitions.

(2) Operating the online serverless mode

The following describes the operation of the online serverless mode.

Important

To use the online serverless mode in the server mode, note the following.

To start a server standing by in the online serverless mode as the active server, make sure that the server can restart in the same manner as for hot-standby switchover. To perform switchover to the subsite for which the online serverless mode is enabled in a system with a disaster recovery configuration, take either of the following actions:

Stop synchronization of the shared disks from the original site to the subsite, and then switch over the site.

Forcibly terminate the server in the original site, and then switch over the site.

Unless either of these actions is taken, execution of the wait-state server start command (monact command) fails. If the monact command fails, restarting a server in the switchover destination site fails and hot-standby switchover for the server is not possible.

Releasing the online serverless mode

When hot-standby switchover has finished, release the online serverless mode by using the monchange command, which changes the settings and operations while HA Monitor and servers are running.

If the online serverless mode is not released after all servers started in the online serverless mode are placed in either of the following statuses, a KAMN549-W message is output every hour:

Startup of the server as the active server is complete.

The server in the online serverless mode has terminated.

Operation after the active server failed to start in the online serverless mode

Resolve the cause of the failure in starting the active server, and then take either of the following actions:

Retry to start the active server on the host on which the failure occurred.

Restart the standby server, and then execute the wait-state server start command (monact command) to start the active server.

Start the active server on another host on which the failure did not occur.

Terminate all servers on the host on which the failure occurred, and then release the online serverless mode. Then, start the active server by executing the wait-state server start command (monact command) on another host on which a standby server is running in the online serverless mode.

(3) Required environment settings

To use the online serverless mode, you must execute the monchange command while HA Monitor is running. This command changes the settings and operations while HA Monitor and servers are running. For details about the monchange command, see 9.6 monchange (changes settings and operations while HA Monitor and servers are running).

3.3 Functions for host control

In the event of a host failure, HA Monitor performs the hot standby operation and transfers jobs to the standby system. This section explains the functions needed to complete the hot standby operation successfully. In the case of the functions related to host reset, you can achieve successful hot standby operation by using the default settings. We recommend that you read the corresponding subsections before you change the default settings.

Note that the functions that are supported depend on the method for protecting data on a shared disk and whether the function for controlling hot standby based on the availability of LAN communications is used. For details, see 3.1 List of functions supported by HA Monitor.

3.3.1 Preventing concurrent host resets

Concurrent host resets means that the active and standby systems both perform host reset at the same time. HA Monitor prevents concurrent host resets from occurring, because if concurrent host resets do occur, both hosts are terminated and job processing is suspended. This subsection explains how to prevent concurrent host resets.

(1) Configurations in which concurrent host resets might occur

Depending on the hot-standby configuration and the functions in use, the active and standby systems might both perform reset at the same time, resulting in termination of jobs.

Especially in the hot-standby configurations described below, the active and standby systems might both be reset at the same time. Therefore, you must determine which host is to be reset first. If both hosts detect host failures at the same time, the host that issues a reset request first is called the host that has reset priority.

Mutual hot-standby configuration

The reset priority must be determined because both hosts are active systems and constitute each other's standby systems. If the system being used is BladeSymphony and SVP supports the cluster management function, you can also choose a method for preventing concurrent host resets without having to determine the reset priority. For details about the cluster management function, see the hardware-related documentation.

Resetting the standby system in a 1-to-1 switchover configuration

If the standby system is to be reset in the event of a failure in the standby system, both hosts might issue a reset at the same time. Therefore, you must determine the reset priority beforehand. For details about standby system reset, see (2) Resetting the standby system.

The following figure shows a configuration in which concurrent host resets might occur.

Figure 3‒15: Configuration in which concurrent host resets might occur

In the configuration shown in the figure above, if two hosts detect the host failure at the same time, both hosts issue a reset request at the same time. As a result, both hosts might be reset simultaneously.

(2) Required environment settings

You must determine the host that is to have reset priority in order to prevent concurrent host resets. HA Monitor identifies the host that has reset priority on the basis of the information specified in the cpudown operand in the HA Monitor environment settings. By default, a reset request from the active system has priority over a reset request from a standby system.

If you want to prevent concurrent host resets without having to determine if the host has reset priority, specify system in the cpudown operand in the HA Monitor environment settings. This specification is applicable only when a mutual hot-standby configuration is used, the system being used is BladeSymphony, and the SVP supports the cluster management function. If VMware ESXi-based virtualization is used, system cannot be specified.

For details about the HA Monitor processing for preventing concurrent host resets, see 4.2.2 Hot standby operation when both hosts detect a failure at the same time.

3.3.2 Preventing concurrent resets between multiple hosts

This subsection explains how to prevent concurrent host resets from occurring in a multi-standby configuration or a cluster hot-standby configuration.

In a configuration in which the multi-standby function is used to provide multiple standby servers for a single active server, concurrent resets on multiple hosts must be avoided.

You can also suppress resets if doing so reduces the number of hosts to below a certain minimum. For details about suppressing resets, see 3.3.4 Suppressing host resets.

(1) Configuration in which concurrent host resets might occur on multiple hosts

In a configuration consisting of three or more hosts, if the hosts are divided due to a failure in a monitoring path network component, a host failure might be detected in each of the network segments and all the hosts might be reset.

To avoid this possibility, you must define a unique reset issuance order for all hosts that belong to the hot-standby configuration. This reset issuance order is called the reset priority. You specify the reset priority in the HA Monitor environment settings.

Concurrent host resets might occur among multiple hosts in the following configurations:

Configuration in which all hosts attempt reset

If all hosts detect a host failure at the same time due to a failure on the monitoring path, all hosts might issue a reset request at the same time and all the hosts might be reset.

Configuration in which the multi-standby function is used

In a configuration that uses the multi-standby function to provide multiple standby servers for a single active server, all the standby servers running on multiple hosts detect a failure in the active system. This might result in concurrent host resets.

The following figure shows a configuration in which all hosts attempt reset.

Figure 3‒16: Configuration in which all hosts attempt reset

In this configuration, if a failure occurs on all monitoring paths, all hosts are reset. If you specify a reset priority beforehand, HA Monitor determines the hosts to be reset and the order based on the reset priority.

(2) Required environment settings

You must specify host in the reset_type operand in the HA Monitor environment settings. If the multi-standby function is used, HA Monitor prevents concurrent resets based on the reset priority regardless of whether this operand is specified.

For details about the reset priority, see 4.2.3 Host reset when there are multiple standby systems.

3.3.3 Preventing multiple resets on a host

Multiple resets on a host means performing reset on the same host more than once. This subsection explains how to prevent multiple resets on a host.

(1) Configurations in which multiple resets on a host might occur

If the hot standby operation is performed due to a host failure, recovery of the active system might be delayed depending on the hot-standby configuration because multiple standby systems issue host resets.

The following figure shows a configuration in which multiple resets on a host might occur.

Figure 3‒17: Configuration in which multiple resets on a host might occur

In the configuration shown in the figure above, standby system 1 and standby system 2 both issue a reset request for a host failure on the active system, thereby delaying recovery of the active system.

(2) Required environment settings

There are no required environment settings.

To prevent multiple resets on a host, there must be only one standby system that resets the active system. This standby system is called the reset issuing host. The reset issuing host is determined automatically by HA Monitor when the server is started.

For details about the reset issuing host, see 4.2.1 How to determine a reset issuing host.

3.3.4 Suppressing host resets

In the case of a configuration in which more than two hosts are running and the hosts monitor each other, if one of the hosts detects a host failure (such as a monitoring path failure on another host), operation might be switched over to one host and all the other hosts might be reset, as shown in the figure below. In such a circumstance, jobs might be terminated if the system requires at least a certain number of active hosts.

Figure 3‒18: Example in which operation is switched over to one host and all the other hosts are reset

This example assumes that no more than three active servers can be run concurrently on a single host because of memory size limitations. If an attempt is made in this system to run five active servers on host 1, jobs will be terminated because the system cannot function.

In a system that requires at least a certain number of active hosts, HA Monitor can suppress hosts from being reset , thus protecting jobs from being terminated. The minimum number of hosts required for a system to run jobs is called the minimum number of active hosts. This subsection explains the operation and the environment settings when host resets are suppressed.

(1) Operation when host resetting is suppressed

This subsection explains the HA Monitor processing when host resetting is suppressed under each of the following scenarios:

A host failure occurs on one host, the host is reset, and then jobs are continued.

A monitoring path failure occurs on one host, host resetting is suppressed, and jobs continue.

A host failure occurs in an N:2 configuration, the host is reset, and then jobs continue.

A host failure occurs while three hosts are running in reduced operation mode, host resetting is suppressed, and jobs are terminated.

These examples assume that the minimum number of active hosts is three and a maximum of two active servers can be run per host.

(a) A host failure occurs on one host, the host is reset, and then jobs continue

The following figure shows the processing when a host failure occurs on one host, the host is reset, and then jobs continue.

Figure 3‒19: Processing when a host failure occurs on one host, the host is reset, and then jobs continue

The following explains the processing, where the step numbers correspond to the numbers in the figure.

Because a host failure occurred on host 4, alive message transmission stops. As a result, hosts 1, 2, 3, and 5 detect a failure on host 4.

Hosts 1, 2, 3, and 5 transmit failure detection notifications among themselves.

Hosts 1, 2, 3, and 5 count the received failure detection notifications and check if the minimum number of active hosts is satisfied. As a result, host 1, which has the highest reset priority, resets host 4.

The active server running on host 4 performs hot standby processing. In this example, operation is switched over to the standby server running on host 5.

In this example, jobs continue because the minimum number of active hosts is satisfied after hot standby processing and no more than two active servers are running on one host.

(b) A monitoring path failure occurs on one host, host resetting is suppressed, and jobs continue

The following figure shows the processing when a monitoring path failure occurs on one host, host resetting is suppressed, and jobs continue.

Figure 3‒20: Processing when a monitoring path failure occurs on one host, host resetting is suppressed, and jobs continue

The following explains the processing, where the step numbers correspond to the numbers in the figure.

Because a monitoring path failure occurred on host 1, alive message transmission stops. As a result, isolated host 1 detects failures on hosts 2 through 5. At the same time, hosts 2 through 5 detect a failure on host 1.

Host 1 is placed in the reception wait state because it cannot transmit failure detection notifications due to the monitoring path failure. At the same time, hosts 2 through 5 transmit failure detection notifications among themselves.

Host 1 cannot check if the minimum number of active hosts are running because it cannot receive failure detection notifications. Therefore, this host suppresses resetting of hosts 2 through 5.

Hosts 2 through 5 count the received failure detection notifications and check if the minimum number of active hosts is satisfied. As a result, host 2, which has the highest reset priority, resets host 1 after 10 seconds have passed.

The active server running on host 1 performs hot standby processing. In this example, operation is switched over to the standby server running on host 2.

In this example, jobs continue because the minimum number of active hosts is satisfied after hot standby processing and no more than two active servers are running on one host.

(c) A host failure occurs in an N:2 configuration, the host is reset, and then jobs continue

The following figure shows the processing when a host failure occurs in an N:2 configuration, the host is reset, and then jobs continue.

Figure 3‒21: Processing when a host failure occurs in an N:2 configuration, the host is reset, and then jobs continue

The following explains the processing, where the step numbers correspond to the numbers in the figure.

Because a host failure occurred on host 1, alive message transmission stops. As a result, hosts 2 through 5 detect a failure on host 1.

Hosts 2 through 5 transmit failure detection notifications among themselves.

Hosts 2 through 5 count the received failure detection notifications and check if the minimum number of active hosts is satisfied. As a result, host 2, which has the highest reset priority, resets host 1.

The active server running on host 1 performs hot standby processing. In this example, operation is switched over to the server running on host 4.

In this example, jobs continue because the minimum number of active hosts is satisfied after hot standby processing and only one active server is running on each host.

(d) A host failure occurs while three hosts are running in reduced operation mode, host resetting is suppressed, and jobs are terminated

The following figure shows the processing when a host failure occurs while three hosts are running in reduced operation mode, host resetting is suppressed, and jobs are terminated.

Figure 3‒22: Processing when a host failure occurs while three hosts are running in reduced operation mode, host resetting is suppressed, and jobs are terminated

The following are details of the processing flow shown in the figure. The numbers correspond to the numbers in the figure.

Because a host failure occurred on host 3, alive message transmission stops. Hosts 1 and 2 detect a failure that occurred on host 3 from disruption of alive messages.

Hosts 1 and 2 transmit failure detection notifications with each other.

Hosts 1 and 2 suppress resetting of host 3 because the numbers of failure detection notifications received by these hosts do not satisfy the minimum number of active hosts.

If host resetting is suppressed on all hosts,# hosts 1 and 2 cannot check the status of host 3. To prevent multiple active servers from running, the server statuses change as follows:

Standby server whose server priority is lower than the standby server on host 3: Terminated (SBY → inactive).

This is because the status of the standby server on host 3 might change if a failure occurs on host 1 or 2.

Standby server that has the highest server priority among all hosts: Placed in hot-standby wait state (SBY → ONL??).

This is because the active server might be running on host 3.

#

You can use the following formula to determine the time required to suppress host resets on all hosts: total number of hosts making up the group × 10 seconds (varies depending on the system). For example, if there are 32 hosts, the time required for suppressing host resets on all hosts is as follows:

32 × 10 = 320 seconds = 5 minutes, 20 seconds

In this example, jobs are terminated because no active server is running on host 3, in which case the minimum number of active hosts (3) is no longer satisfied.

Depending on the availability of standby servers that can be switched over to a remote host, you might need to perform some action, such as executing the monact command. For details, see 7.2.1 Starting if the server is terminated, or see 7.4.1 Starting a server in the wait state and then restarting jobs if the server is placed in hot-standby wait state (ONL??).

If a host reset is suppressed due to a monitoring path failure that was detected as a host failure, when the monitoring path is restored, use the monshow -c command to check the connection between HA Monitors. If there are unconnected hosts, use the monlink command to connect them.

(2) Failures for which host resetting can be suppressed

Host resetting can be suppressed only for those host failures that can be detected because alive message transmissions stop.

If a pair shutdown is notified from HA Monitor running on a remote host or if shared disk disconnection processing fails during hot standby processing, HA Monitor resets the remote host without waiting for a failure detection notification from the remote host and then performs hot standby processing. If HA Monitor receives a failure notification from hardware, or if a server failure occurs, HA Monitor performs hot standby processing without waiting for a failure detection notification.

(3) Conditions under which host resetting is suppressed and examples

Host resetting can be suppressed only when all the following conditions are satisfied:

The minimum number of active hosts is greater than half of the total number of hosts that are connected.

The multi-standby function is being used (use is specified in the multistandby operand in the HA Monitor environment settings) or host is specified in the reset_type operand.

host is specified in the patrol_type operand or the following conditions are satisfied on all hosts:

The servers have started.

All hosts that connect to the local host contain a server that is to be paired with the server on the local host.

The following figures show examples of when host resetting can be suppressed.

Figure 3‒23: Example 1 of when host resetting can be suppressed

In this example, host resetting can be suppressed because the following conditions are satisfied:

The minimum number of active hosts is greater than half of the total number of hosts that are connected.

The multi-standby function is being used.

The servers have started and all hosts that connect to the local host contain a server that is to be paired with the server on the local host.

Figure 3‒24: Example 2 of when host resetting can be suppressed

In this example, host resetting can be suppressed because the following conditions are satisfied:

The minimum number of active hosts is greater than half of the total number of hosts that are connected.

host is specified in the reset_type operand.

host is specified in the patrol_type operand.

Note

If you set the minimum number of active hosts to a value that is less than or equal to half of the total number of hosts that are connected, when a network split-brain condition occurs, the minimum number of active hosts might be exceeded within the split configuration, resulting in hosts resetting each other. Therefore, you must set the minimum number of active hosts to a value that greater than half of the total number of hosts that are connected.

(4) Required environment settings

To suppress host resets, specify the suppress_reset operand in the HA Monitor environment settings.

3.3.5 Hot-standby switchover caused by a host pair shutdown

If any of the following events occurred, HA Monitor enables you to forcibly terminate the switching-source host and continue hot-standby switchover. This is called host pair shutdown.

Server failure

A user command error occurred on the switching-source host.

A server termination command error occurred on the switching-source host.

(1) Host pair shutdown for each host in the event of a server failure

If the active server that is ready for hot-standby switchover fails, HA Monitor forcibly terminates the host on which the failure occurred and then performs switchover to the standby host.

Use this function in the following cases:

Hot-standby switchover is to be promptly performed by forcibly terminating the host without disconnecting individual resources on the host on which the failure occurred.

Hot-standby switchover is to be promptly performed by forcibly terminating the host without terminating other servers in the same group.

You can use this function on the following servers:

Servers in the server mode

Servers in the monitor mode (being monitored by specifying the ptrlcmd_ex operand in the server environment definition)

The active server failures that result in pair shutdown include failures detected by the server itself and server slowdown. If TP1/Server Base or HiRDB is used as a server in the server mode, you can also specify an option that uses only detection of server slowdown as a trigger for pair shutdown.

The active system is reset by the HA Monitor in the standby system that has received a message from the HA Monitor to be terminated. This reset occurs immediately without having to determine the host that has reset priority or the reset issuing host.

Required environment settings

Use the pairdown operand in the server environment definition to specify the pair shutdown function.

(2) Host pair shutdown performed for each host in the event of a user command error

You can also shut down a host pair if a user command that is executed on the source host fails during hot standby processing resulting from a server failure (online -a start) or during planned hot standby processing (online -w start).

Use this function if all the following conditions apply:

An attempt was made to disconnect the resource of the switching-source host by using a user command.

After the attempt to disconnect the resource of the switching-source host failed, you want to release the resource by forcibly terminating that host to continue hot-standby switchover.

A user command error means any of the following:

The user command's termination code is not 0.

No user commands can be executed.

A user command was executed, but it does not return control.

Note that you can handle the preceding errors after the processing times out as specified by using the dev_timelimit or dev_offlimit operand in the server environment definition.

Required environment settings

Specify use for the uoc_pairdown operand in the server environment definition. Check user commands to be executed during hot-standby switchover due to a server failure (online -a start) or during planned hot-standby switchover (online -w start). Make sure that these commands are defined to return a termination code other than 0 in the event of abnormal termination.

(3) Host pair shutdown for each host in the event of a server termination command error

If a server termination command error occurs when hot-standby switchover is performed, you can choose to perform host pair shutdown.

Use this function if you want to forcibly terminate the server and continue hot-standby switchover by forcibly terminating the host after a server termination command error occurred.

A server termination command error means any of the following:

The server termination command's termination code is not 0.

The server termination command cannot be executed.

The server termination command was executed, but it does not return control.

Note that you can handle the preceding errors after the processing times out as specified by using the stop_timeout or stop_timeout_sw operand in the server environment definition.

Required environment settings

Specify use for the term_pairdown operand in the server environment definition.

3.3.6 Physical partition reset function in a virtualization environment

HA Monitor enables you to use Hitachi server virtualization (Virtage) or VMware ESXi to perform the hot standby operation in a virtualization environment.

In BladeSymphony with Hitachi server virtualization (Virtage) installed, you can divide the processor into logical partitions (LPARs) and perform the hot standby operation in units of LPARs. An environment that uses Hitachi server virtualization to run LPARs is called the LPAR mode, and an environment that does not use Hitachi server virtualization is called the Basic mode. In this manual, a BladeSymphony environment in which Hitachi server virtualization is not installed is generally referred to as the Basic mode.

If you use VMware ESXi, you can use a single processor to run multiple virtual machines and perform the hot standby operation in units of virtual machines.

This subsection explains the physical partition reset function that can be used as an option in these virtualization environments in comparison with the normal hot standby operation.

(1) Normal hot standby operation

HA Monitor detects host failures for each LPAR in the LPAR mode and each virtual machine when VMware ESXi is used. By default, if a host failure occurs in the active system, the HA Monitor for the standby system resets only the LPAR or virtual machine where the HA Monitor for the active system is running. The figure below shows the normal hot standby operation in the event of a host failure in the active system in the LPAR mode. When VMware ESXi is used, the normal hot standby operation in the event of a host failure in the active system is the same as for the LPAR mode. Simply replace LPAR with virtual machine.

Figure 3‒25: Hot standby operation in the LPAR mode

(2) Operation when the physical partition reset function is used

Normally if HA Monitor's host reset operation fails, the target host is placed in a wait state that requires user intervention.

If a reset operation on a host (LPAR or virtual machine) fails, the physical partition reset function determines that one of the failures listed below has occurred and resets the entire processor containing the erroneous host (LPAR or virtual machine):

HVM system failure (a failure in Hitachi server virtualization, which is a type of host failure)

VMware ESXi failure

Processor failure

VMware ESXi communication error caused by a failure such as a network outage

When this function is used, HA Monitor performs the reset operation regardless of whether another LPAR or virtual machine is running in the processor.

Resetting of the entire processor occurs when the host resulting in a failure and the host detecting the failure are located in different processors. If the host resulting in a failure and the host detecting the failure are located in the same processor, the entire processor is not reset, in which case the target host will wait for user intervention.

When the physical partition reset function is used, HA Monitor gives precedence to performing the hot standby operation for an LPAR or virtual machine resulting in a failure over continuing operation in other LPARs or virtual machines in the same processor. As a result, the standby server on the target host starts as the active server without having to wait for user intervention.

The figure below shows the operation when the physical partition reset function is used in the LPAR mode. When the physical partition reset function is used with VMware ESXi, operation is the same as for the LPAR mode. Simply replace LPAR with virtual machine.

Figure 3‒26: Operation using the physical partition reset function

If the hot standby operation is performed by using the physical partition reset function, the entire processor is reset. This means that all other LPARs (virtual machines) in the same processor are reset at the same time.

Before using this function, you must check that the processor subject to the reset operation does not contain any LPAR or virtual machine that is more important in terms of job processing than hot standby. If such an LPAR or virtual machine is running in the processor, do not use this function. For details about the HA Monitor processing when the physical partition reset function is not used and a host reset has failed, see 4.2.4 Processing in the event a host reset has failed.

(3) Required environment settings

You use the partition_reset operand in the HA Monitor environment settings to specify whether the physical partition reset function is to be used.

3.3.7  Hot standby in an N+M cold standby configuration

HA Monitor supports hot standby in an N+M cold standby configuration provided by JP1/ServerConductor or HCSM. If a failure occurs on the active server module in an N+M cold standby configuration, HA Monitor performs the hot standby operation. After that, JP1/ServerConductor or HCSM passes the settings and local disks of the active server module to the standby server module. Therefore, you can use standby server modules as standby systems.

(1) Hot standby operation in an N+M cold standby configuration

The following figure shows the hot standby operation in an N+M cold standby configuration.

Figure 3‒27: Hot standby operation in an N+M cold standby configuration

If a host failure occurs in active server module 1, HA Monitor detects the host failure and switches over to active server module 2. As a result of this hot standby processing, the active server module 2 becomes the active system.

After that, JP1/ServerConductor or HCSM inherits settings to the standby server module, including the various settings for active server module 1 and the local disks. As a result, the standby server module becomes the standby system.

Similarly, if a host failure occurs on active server module 2, the standby server module becomes the standby system.

(2) Required environment settings

To enable hot-standby switchover in an N+M cold standby configuration, you must set a reset path and specify SVP settings. For details, see 6.6.1 Information required for configuration (BladeSymphony) and (6) N+M cold standby configuration.

For details about the N+M cold standby configuration, see the manual in the following list that is appropriate for your system:

For JP1/ServerConductor

JP1/ServerConductor/Blade Server Manager Series System Administrator's Guide

For HCSM

HCSM documentation

3.4 Functions for monitoring resources

HA Monitor monitors resources to detect failures that occurred on the resources. This section describes the functions for monitoring resources.

3.4.1 LAN monitoring and automatic hot standby in the event of a failure

HA Monitor enables you to monitor the communication status of the LAN used by the active server and to perform the hot standby operation automatically in the event of a failure. This hot standby operation is performed as a planned hot standby. This subsection explains the processing flow for LAN monitoring.

(1) Processing flow for monitoring a LAN

HA Monitor monitors a LAN by monitoring ping responses to a specified address or the number of packets received by a monitored LAN adapter. We recommend that you use the method in which HA Monitor monitors ping responses to a specified address. Use the lancheck_mode operand in the HA Monitor environment settings to specify the monitoring method.

HA Monitor periodically checks the status of the LAN of the active system, and if HA Monitor detects either of the following statuses, it assumes that a LAN failure occurred and performs hot-standby switchover:

If HA Monitor is set to monitor ping responses to a specified address: No response arrives until ping execution is retried the number of times specified in the LAN monitoring script.

If HA Monitor is set to monitor the number of packets received by a monitored LAN adapter: The number of received packets does not change from the number checked at the previous monitoring cycle.

The following figure shows the processing flow for monitoring a LAN.

Figure 3‒28: Processing flow for monitoring a LAN: If HA Monitor monitors ping responses to a specified address

Figure 3‒29: Processing flow for monitoring a LAN: If HA Monitor monitors the number of packets received by a monitored LAN adapter

If HA Monitor detects a LAN failure, it searches for the server specified as the hot standby target and then performs the hot standby operation if that server is available for switchover. If the corresponding server is grouped, HA Monitor performs the hot standby operation for the group. If the target server is not ready for the hot standby operation, HA Monitor performs the hot standby operation when the server is ready. If multiple LANs are monitored, HA Monitor monitors each LAN independently and asynchronously for each interface.

(2) Operating environment

To monitor ping responses or the number of received packets, the HA Monitor operating environment must satisfy the following conditions:

The node specified to check the LAN status has the IP address for communicating with the monitored LAN in the same network, and a response to ICMP echo packets can be sent.

The node specified to check the LAN status is already running when HA Monitor is started.

(3) Devices monitored by HA Monitor

HA Monitor can monitor the following three LAN interfaces: hbonding, bonding, and ethernet. In a virtualization environment, HA Monitor can monitor a virtual LAN interface. In a non-virtualization environment, HA Monitor can monitor a physical LAN interface. hbonding can be monitored in the two modes active-backup and 802.3ad. The monitoring target is selected based on settings in the LAN monitor definition file. The following are the failure monitoring methods for each mode:

active-backup mode: MII, ARP, or path monitoring method

802.3ad mode: LACP monitoring method

Note that tagged VLANs can also be monitored. To monitor a tagged VLAN, specify an interface name that does not include a period (.).

(4) Required environment settings

The following table shows the environment settings required for each monitoring target.

Table 3‒2: Environment settings required for each monitoring target

Required environment settings

Monitoring target#3

hbonding

bonding

ethernet

active-backup mode

802.3ad mode

HA Monitor environment settings

lanfailswitch

Y

Y

Y

Y

lancheck_patrol#1

O

O

O

O

lancheck_mode

O

O

O

O

hbond_lacp

N

O

N

N

Server environment definition

switchbyfail

Y

Y

Y

Y

Files required for monitoring LANs

LAN monitor definition file#2

Y

Y

Y

Y

LAN monitoring script#1

O

O

Y

Y

Legend:

Y: Specify.

O: Specify if necessary.

N: Cannot be specified. If the setting is specified, an error results.

#1: For details about the lancheck_patrol setting and LAN monitoring script, see (2) Specifying a LAN monitoring script.

#2: For details about the LAN monitor definition file, see (1) Specifying a LAN monitor definition file.

#3: To use a tagged VLAN, specify an interface name that does not include a period (.) for the device name.

(a) Environment settings required for monitoring a LAN

Specify in the lanfailswitch operand in the HA Monitor environment settings whether a LAN is to be monitored.

Specify a LAN monitoring interval in the lancheck_patrol operand in the HA Monitor environment settings.

Specify the name of the LAN interface to be monitored in the switchbyfail operand in the server environment definition.

To monitor hbonding with the LACP monitoring method in the 802.3ad mode, specify the hbond_lacp operand in the HA Monitor environment settings.

To monitor hbonding with the LACP monitoring method in the 802.3ad mode without specifying this operand, HA Monitor must monitor ping responses or the number of received packets.

(b) Files required for monitoring a LAN

Specify the LAN monitor definition file and LAN monitoring script. If the monitoring target is hbonding, in addition to using the method that monitors only the hbonding status, you can also use the method that monitors ping responses or the number of received packets. With this file, you can specify whether to use the additional monitoring method.

For details, see 6.19.1 Settings in the files required for monitoring a LAN.

(5) Notes about monitoring a LAN

This subsection provides notes about monitoring a LAN for each of the following cases:

Monitoring hbonding

While a LAN is being monitored (HA Monitor is running), do not release a slave interface bound to the hbonding interface nor delete the hbonding interface.

If the monitoring target is hbonding in active-backup mode, in addition to using the method that monitors only the hbonding status, you can also use the method that monitors ping responses or the number of received packets. Normally, select the method for monitoring only the hbonding status. However, if you want to specify multiple monitoring targets in addition to the one specified in the hbonding settings, you must also use the method that monitors ping responses or the number of received packets.

When monitoring hbonding with the LACP monitoring method in the 802.3ad mode, failures are detected up to the connected network switch. To detect failures beyond the network switch, you must also use the method that monitors ping responses or the number of received packets.

If you will be monitoring hbonding with the LACP monitoring method in the 802.3ad mode, check the channel group settings for the network switch beforehand. Invalid settings might be detected as failures. While LANs are being monitored (HA Monitor is running), do not change the channel group settings in the network switch.

Performing grouped-system switchover

Configuration in which server switchover order is not controlled

Specify the switchbyfail operand in the server environment definition for all servers in the same group. Specify the same LAN interface name for all the servers.

Configuration in which server switchover order is controlled

We recommend that you specify the switchbyfail operand in the server environment definition for the server that is started last among the servers subject to server switchover order control. For this operand specified for the server to be started last, specify all LAN interface names to be used by the servers in the group. If the switchbyfail operand is specified for a server that is not the last one to be started and if a LAN failure is detected when the server with the switchbyfail operand specified starts, HA Monitor performs a grouped-system switchover. This might prevent the hot-standby switchover from being performed on the lower-order servers concurrently.

To use HA Monitor in a virtualization environment

If you use HA Monitor in a virtualization environment, we recommend that you use the method that monitors ping responses. If you use the method that monitors the variation in the number of received packets, when the number of packets received over a virtual LAN interface does not change, HA Monitor performs hot-standby switchover, assuming that a failure occurred. This means that no hot-standby switchover occurs as long as the virtual machines on a physical machine communicate normally over a virtual LAN interface even if a failure occurs on a physical LAN interface. The following figure shows an example in which no hot-standby switchover occurs even if a failure occurs on a physical LAN interface. Note that in the figure, NICs (network interface cards) are LAN interfaces.

Figure 3‒30: Example in which no hot-standby switchover occurs even if a failure occurs on a physical LAN interface

In monitoring the variation in the number of received packets, if you want hot-standby switchover to occur also when a failure occurs on a physical LAN interface, set up the configuration as follows:

Use LAN interfaces not subject to monitoring for communications between multiple virtual machines.

Set packet filtering so that virtual LAN interfaces subject to monitoring do not communicate with other virtual machines.

3.4.2 Disk monitoring

With disk monitoring of HA Monitor, hot-standby switchover can occur when access to the following disks fails:

System disk

Space of a disk used by the system.

Disk for business use

Space of a disk used by servers. Note that a shared disk is a type of disk for business use.

This section describes monitoring of system disks and disks for business use.

(1) System disk monitoring

If the system disk becomes inaccessible when system disk monitoring is not used, hot-standby switchover does not occur while the OS and HA Monitor is operating. If system disk monitoring is used, hot-standby switchover can occur when the system disk becomes inaccessible, regardless of whether the OS and HA Monitor can operate.

(a) System disk monitoring method

In system disk monitoring, a file created on the system disk is used as the monitoring target. HA Monitor periodically checks whether a write is possible to the monitoring-target file. Note that the monitoring target must be a file in a file system. For details about how to set the monitoring target, see (c) Required environment settings.

The following describes the times when HA Monitor monitors a system disk and describes the processing of monitoring.

Times when HA Monitor performs monitoring

The following shows when HA Monitor starts monitoring and when it stops monitoring.

Monitoring start time

• When HA Monitor startup is completed

• When system disk monitoring is restarted#

Monitoring end time

• When HA Monitor stops

• When system disk monitoring is temporarily stopped#

#

For details about temporarily stopping and restarting system disk monitoring, see 9.6 monchange (changes settings and operations while HA Monitor and servers are running).

Processing of monitoring

An attempt is made to write to the monitoring-target file at set intervals (the check interval). If a write to the monitoring-target file is successful, HA Monitor judges that the system disk is healthy and continues monitoring. The following figure shows the processing that is performed if a write to the monitoring-target file is successful.

Figure 3‒31: Processing performed if a write to the monitoring-target file is successful

If an attempt to write to the monitoring-target file fails, HA Monitor retries the attempt at retry intervals. If a write attempt fails a certain number of times (retry count), HA Monitor judges that access to the system disk failed. The following figure shows the processing that is performed if a write to the monitoring-target file fails. This figure is an example in which the retry count is set to 1.

Figure 3‒32: Processing performed if a write to the monitoring-target file fails (when the retry count is 1)

If the system disk does not respond to a write to the monitoring-target file, HA Monitor waits for a response until the period calculated by retry interval × (retry count + 1) elapses. If the system disk does not respond until the period calculated by retry interval × (retry count + 1) elapses, HA Monitor judges that access to the system disk failed. The following figure shows the processing that is performed if the system disk does not respond to a write to the monitoring-target file. This figure is an example in which the retry count is set to 1.

Figure 3‒33: Processing performed if the system disk does not respond to a write to the monitoring-target file (when the retry count is 1)

(b) Processing performed if access to the system disk fails

The processing that HA Monitor performs differs depending on the type of the node for which system disk access failed and whether hot-standby switchover is possible. The following shows possible patterns of processing performed when system disk access fails.

If the primary system fails to access the system disk when hot-standby switchover is possible

Processing is performed as follows:

The primary system notifies the secondary system that access failed.

The primary system stops.

If the method of hot-standby switchover triggered by host reset is selected, host reset occurs. If another method is selected, an OS panic is caused.

The secondary system performs hot-standby switchover based on the selected method.

The following figure shows the preceding operation.

Figure 3‒34: Processing performed if the primary system fails to access the system disk when hot-standby switchover is possible

If the secondary system fails to access the system disk when hot-standby switchover is possible

Processing is performed as follows:

The secondary system notifies the primary system that access failed.

The secondary system stops.

If the method of hot-standby switchover triggered by a host reset is selected, a host reset occurs. Note that the host reset occurs independently of the setting of the standbyreset operand in the HA Monitor environment settings. If a method other than the method triggered by host reset is selected, an OS panic is caused.

The following figure shows the preceding operation.

Figure 3‒35: Processing performed if the secondary system fails to access the system disk when hot-standby switchover is possible

If the primary system fails to access the system disk when hot-standby switchover is impossible

HA Monitor does not perform hot-standby switchover and continues monitoring of the system disk. If hot-standby switchover becomes possible after the failure occurred, HA Monitor behaves in the same way as in the case where the primary system fails to access the system disk when hot-standby switchover is possible.

If the secondary system fails to access the system disk when hot-standby switchover is impossible

HA Monitor continues system disk monitoring. If hot-standby switchover becomes possible after the failure occurred, HA Monitor behaves in the same way as in the case where the secondary system fails to access the system disk when hot-standby switchover is possible.

(c) Required environment settings

This subsection describes the environment settings that must be specified to monitor the system disk.

Create a monitoring definition file for the system disk, and then specify use for the disk_ptrl operand in the HA Monitor environment settings.

For details about how to specify the settings in the monitoring definition file for the system disk, see (1) Settings in the files required for monitoring the system disk in 6.19.2 Settings in the files required for disk monitoring.

(2) Monitoring of a disk for business use

If HA Monitor is not monitoring a disk for business use and either of the following cases applies, HA Monitor does not perform hot-standby switchover even when the disk for business use becomes inaccessible:

The active server does not fail.

HA Monitor is not set to perform server monitoring.

In the preceding situation, if HA Monitor is set to monitoring a disk for business use, hot-standby switchover can occur when the disk becomes inaccessible.

If a failure in a disk for business use causes a server failure, the server specification allows HA Monitor to perform hot-standby switchover by monitoring servers. In this case, monitoring disks for business use is not necessary.

(a) Method of monitoring a disk for business use

In monitoring of a disk for business use, a file created on the disk for business use is used as the monitoring target. HA Monitor periodically checks whether a write is possible to the monitoring-target file.

Note that the monitoring target must be a file in a file system. Character special files cannot be used as the monitoring target. HA Monitor can monitor a disk for business use when the monitoring target is a file system on the disk for business use on which character special files are configured. For details about how to set the monitoring target, see (c) Required environment settings.

The following describes the times when HA Monitor monitors a disk for business use and describes the processing of monitoring.

Times when HA Monitor performs monitoring

The following shows when HA Monitor starts monitoring and when it stops monitoring.

Monitoring start time

• When startup of the active server is completed

• When the monitoring of a disk for business use is restarted#

Monitoring end time

• When the active server stops

• When the monitoring of a disk for business use is temporarily stopped#

Note that disks for business use are not monitored on a server that is in the restart wait state.

#

For details about temporarily stopping and restarting the monitoring of a disk for business use, see 9.6 monchange (changes settings and operations while HA Monitor and servers are running).

Processing of monitoring

A write is attempted once to the monitoring-target file every certain length of time (check interval). If a write to the monitoring-target file is successful, HA Monitor judges that the disk for business use is healthy and continues monitoring. The following figure shows the processing that is performed if a write to the monitoring-target file is successful.

Figure 3‒36: Processing performed if a write to the monitoring-target file is successful

If an attempt to write to the monitoring-target file fails, HA Monitor retries the attempt at retry intervals. If a write attempt fails a certain number of times (retry count), HA Monitor judges that access to the disk for business use failed. The following figure shows the processing that is performed if a write to the monitoring-target file fails. This figure is an example in which the retry count is set to 1.

Figure 3‒37: Processing performed if a write to the monitoring-target file fails (when the retry count is 1)

If the system disk does not respond to a write to the monitoring-target file, HA Monitor waits for a response until the period calculated by retry interval × (retry count + 1) elapses. If the system disk does not respond until the period calculated by retry interval × (retry count + 1) elapses, HA Monitor judges that access to the disk for business use failed. The following figure shows the processing that is performed if the system disk does not respond to a write to the monitoring-target file. This figure is an example in which the retry count is set to 1.

Figure 3‒38: Processing performed if the system disk does not respond to a write to the monitoring-target file (when the retry count is 1)

(b) Processing performed if access to a disk for business use fails

If access to a disk for business use fails, HA Monitor performs one of the following operations based on the setting of the disk_ptrl_act operand in the server environment definition:

Performs planned hot-standby switchover for servers.

Causes host pair shutdown to perform hot-standby switchover as a host failure.

Continues monitoring of the disk for business use without performing hot-standby switchover.

Planned hot-standby switchover for servers and host pair shutdown can be performed only when hot-standby switchover for servers is possible. However, even in a case where hot-standby switchover for servers is impossible when access to the disk for business use fails, if hot-standby switchover becomes possible later, hot-standby switchover is performed.

If servers are grouped, servers for which grouped-system switchover can be performed are switched over on a server group basis. If a server group includes servers for which grouped-system switchover could not be performed, start those servers on the host that has become active.

The following figure shows a typical example of processing performed if access to a disk for business use fails when hot-standby switchover is possible. Note that this figure shows an example in a case where planned hot-standby switchover for servers is performed.

Figure 3‒39: Processing performed if access to a disk for business use fails when hot-standby switchover is possible (in a case where planned hot-standby switchover for servers is performed)

(c) Required environment settings

This subsection describes the environment settings that must be specified to monitor disks for business use.

For each monitoring-target disk for business use, create a monitoring definition file for the disk for business use. Then, specify use for the disk_ptrl operand in the server environment definition.

For details about how to specify the settings in a monitoring definition file for the disk for business use, see (2) Settings in the files required for monitoring disks for business use in 6.19.2 Settings in the files required for disk monitoring.

3.5 Functions for shared resource control

This section describes the functions used to connect and disconnect shared resources that are used by servers.

HA Monitor supports automatic switching of shared disks and LANs as shared resources. To switch other shared resources, the user must create a command. For details, see 3.5.2 Issuing commands when the server and HA Monitor status changes.

Note that the functions that are supported depend on the shared disk data protection method and whether the function for controlling hot standby based on the availability of LAN communications is used. For details, see 3.1 List of functions supported by HA Monitor.

3.5.1 Using a resource server to share shared resources

Normally, you must provide shared resources for each server. If you use a resource server, shared resources can be shared among multiple servers. You use a resource server when the available shared resources are limited.

(1) Configuration using a resource server

A resource server controls shared resources. No program runs in a resource server. You always group a resource server with other servers. When you use a resource server, you can control shared resources for a server group.

The figure below shows a configuration that includes a resource server. In this example, servers 1 and 2 share the same shared disk and alias IP address.

Figure 3‒40: Example of configuration using a resource server

If servers are grouped together to perform grouped-system switchover but no resource server is used, the shared resources are controlled for each server. Therefore, the same shared resources cannot be shared among multiple servers.

(2) Required environment settings

The following environment settings are required in order to use a resource server:

Use the resource definition statement in the server environment definition to specify the resource server settings.

Environment settings for a resource server are required because the resource server is added to the host.

Use the group operand in the server environment definition to specify the settings for grouping servers.

This specification is required for all servers that are to run in the same group.

Use the parent operand in the server environment definition to specify the settings for server order control.

Specify the resource server as the parent server for all servers that are not the resource server and that are to run in the same group.

3.5.2 Issuing commands when the server and HA Monitor status changes

HA Monitor enables you to issue a user-created command automatically whenever the status of the server and HA Monitor changes. This command is called a user command. Creation of user commands is optional

(1) Advantages of using user commands

If you create user commands, you can have HA Monitor automatically perform user-specific processing when HA Monitor processing occurs, such as server startup, hot standby, and server failure handling.

You can also share any resources among hosts by specifying in a user command processing on the shared resources that are not handled by HA Monitor.

For details about the timing of user command issuance and the parameters passed from HA Monitor to user commands, see 6.18 Creating user commands.

(2) Required environment settings

You create a user command, and then specify it in the usrcommand operand in the HA Monitor environment settings. For details about how to create user commands and for coding examples, see 6.18 Creating user commands.

3.5.3 Canceling server startup in the event of a shared resource connection error

In the event of a connection error on a shared resource and a user command error, you can cancel startup of the active server and the hot standby operation. You can specify whether server startup is to be canceled for each resource.

If a connection error occurs on a shared resource, HA Monitor continues connection processing on the remaining shared resources, cancels startup of the active server, and then disconnects all shared resources. If the hot standby operation is to be canceled, HA Monitor terminates the standby server forcibly.

If the start retry function is used during hot standby processing, HA Monitor waits until the server is restarted.

(1) Shared resources subject to this processing

The following types of shared resources are subject to this processing:

Shared disks

File systems

LANs (IP address settings in the server-alias-name.up file)

User commands

(2) Required environment settings

This subsection describes how to specify the settings.

(a) Specifying shared disk settings

Specify use in the vg_neck operand in the server environment definition. You can change the vg_neck operand value with the HA Monitor command for dynamically changing a server's shared resources (mondevice command).

(b) Specifying file system settings

Specify use in the fs_neck operand in the server environment definition.

(c) Specifying LAN settings

Specify use in the ip_neck operand in the server environment definition.

You must also create a server-alias-name.up file, which is a LAN status settings file in HA Monitor. Use the sample file provided by HA Monitor. For details, see 6.11.2 Specifying LAN status settings files.

(d) Specifying user command settings

By specifying use in the uoc_neck operand in the server environment definition, you can cancel server startup whenever the execution result of the user command is not 0.

Note that HA Monitor checks for errors only for user commands that are executed at the start of server startup processing or at the start of server hot standby. The results of user commands executed at any other times are ignored.

For details about user commands, see 6.18 Creating user commands.

3.5.4 Shared resource inheritance timeout

This subsection explains the function for monitoring the inheritance of shared resources (disconnection and connection) during the hot standby operation to see whether the inheritance processing is completed within a time limit. The timeout is monitored only during the hot standby operation. It is not monitored during server startup or termination.

(1) Processing when a timeout is detected

A time limit is set for the entire processing of disconnecting or connecting the shared resources. If the disconnection or connection processing is not completed within this time limit, processing on the shared resources is cancelled.

If the function for canceling server startup in the event of a shared resource connection error is used and the shared resource connection processing results in a timeout, the server's startup is canceled.

If the start retry function is used during hot standby processing, HA Monitor waits until the server is restarted.

(2) Monitoring range

Monitoring for the time limit is supported for either of the ranges described below. HA Monitor monitors each instance of processing to see whether it is completed within the specified time limit.

From the time disconnection of shared resources begins on the hot-standby target host to the time execution of the user command at the start of server failure processing is completed.

From the time connection of shared resources begins on the hot-standby target host to the time execution of the user command at the start of server hot standby processing is completed

(3) Required environment settings

Set timeout values related to switching (disconnection and connection) of shared resources for the following operands:

The dev_timelimit operand in the server environment definition

Specify the timeout value for disconnection and connection of shared resources.

The dev_offlimit operand in the server environment definition

Specify the timeout value for disconnection of shared resources.

The dev_onlimit operand in the server environment definition

Specify the timeout value for connection of shared resources.

To specify different timeout values for disconnection and connection of shared resources, specify the disconnection timeout value for the dev_offlimit operand, and the connection timeout value for the dev_onlimit operand. Note that if you specify the dev_timelimit operand, you can specify neither the dev_offlimit operand nor the dev_onlimit operand.

The following table describes how the behavior of timeout monitoring changes according to the settings of the dev_timelimit, dev_offlimit, and dev_onlimit operands in the server environment definition.

Operand specification

Behavior of timeout monitoring

dev_timelimit

dev_onlimit

dev_offlimit

Not specified

Not specified

Not specified

No monitoring is performed.

Specified

The timeout for disconnection of shared resources is monitored.

Specified

Not specified

The timeout for connection of shared resources is monitored.

Specified

The timeout for disconnection and connection of shared resources is monitored.

Specified

Not specified

Not specified

The timeout for disconnection and connection of shared resources is monitored.

Specified

This combination is not possible.

Specified

Not specified

This combination is not possible.

Specified

This combination is not possible.

3.5.5 Specifying the shared resource separation order

This subsection explains specification of the processing order for disconnecting shared resources. Shared resources are disconnected automatically at times such as when the server is terminated and when the hot standby operation is performed.

You can specify either of the following shared resource disconnection order options:

Disconnecting shared resources in the same order as their connection order

Disconnecting shared resources in the reverse order of their connection order

If you disconnect shared resources in the reverse order of their connection order, you can implement processing that depends on the shared resources within the user-specific termination processing in a user command because the shared resources are disconnected after the user command has been issued. When shared resources are connected, a user command is issued after the shared resources have been connected.

For details about the processing when shared resources are disconnected in reverse order, see 4.8.7 Processing flow for disconnecting shared resources in the reverse order from which they were connected.

(1) Required environment settings

Specify the deviceoff_order operand. By default, HA Monitor disconnects shared resources in the same order as their connection order.

The environment settings to be specified depend on the range to which disconnection in the reverse order is to be applied. If this operand is specified in both the HA Monitor environment settings and the server environment definition, the specification in the HA Monitor environment settings takes effect.

To apply the operand to all servers registered in HA Monitor:

Specify the operand in the HA Monitor environment settings.

To apply the operand only to specific servers:

Specify the operand in the server environment definitions.

3.5.6 Dynamic change of shared resources

You can dynamically change the shared resources that are to be subject to the hot standby operation, even if the active server cannot be terminated due to 24-hour continuous operation.

(1) Shared resources that can be changed dynamically

You can change the following shared resources:

Shared disks

LANs

(2) How to change shared resources dynamically

The method depends on the type of resource.

(a) Shared disk

Use the command for dynamically changing a server's shared resources (mondevice command). This command enables you to add and delete shared resources in units specified in the server environment definition.

The figure below provides an overview of dynamically changing shared resources by using the command for dynamically changing a server's shared resources (mondevice command). This example adds a shared disk.

Figure 3‒41: Overview of dynamically changing shared resources by using the HA Monitor command mondevice

When the user uses the command for dynamically changing a server's shared resources (mondevice command), HA Monitor performs the following processing:

Controls the status of the shared resources and performs server connection and disconnection processing (when shared resources are to be added).

Automatically changes the server environment definition according to the change made to the shared resources for both the active and the standby servers that share the resources (when shared resources are to be added, changed, or deleted).

(b) LAN

The user adds or deletes information about the LAN to be added or deleted, respectively, to and from the LAN status settings file.

3.6 Function for controlling programs (monitor mode)

In the monitor mode, you can assign multiple programs, such as Cosminexus, JP1, and UAPs, to a single server. You can also start, monitor, and stop processing for each UAP. The function for controlling processing in units of UAPs is called the monitor-mode program management function.

To use this program management function to monitor UAPs, you must issue APIs from the UAPs.

This section explains the program management function.

3.6.1 Monitoring UAPs

To monitor server programs in the monitor mode, the user normally creates a server monitoring command. By using the monitor-mode program management function, HA Monitor can monitor UAPs. You do not need to include the UAP monitoring process in the server monitoring command. If you issue APIs for UAP monitoring from UAPs, HA Monitor can monitor the UAPs and detect server failures caused by the UAPs.

The following figure shows the flow of UAP monitoring when the program management function is used.

Figure 3‒42: Flow of UAP monitoring when the program management function is used

To use the program management function to monitor UAPs, the user must include APIs in the UAPs. Therefore, UAPs can be monitored only if the UAPs can be modified to include APIs. For details about including APIs in UAPs and the conditions under which APIs can be included in UAPs, see 6.15.1 Including APIs in UAPs.

(1) How to monitor UAPs

HA Monitor uses the two methods described below to monitor UAPs and detect UAP failures. For details about these methods, see 4.7.1 UAP monitoring control.

Monitoring UAP processes

HA Monitor uses this method to determine the existence of a UAP process.

Monitoring UAP operation reports

HA Monitor uses this method to monitor the operation reports sent from UAPs.

If HA Monitor detects a UAP failure, it restarts the UAP and server. For details about the HA Monitor processing that occurs after detection of a UAP failure, see (3) HA Monitor processing after detection of a UAP failure. For details about the processing during UAP restart, see 4.7.2 Program restart control.

(2) Required environment settings

To monitor UAPs, you must specify the following environment settings:

Environment settings that are needed to use the program management function:

To use the program management function

Specify use in the program operand in the server environment definition.

Maximum number of programs

Specify in the pgmmax operand in the HA Monitor environment settings the maximum number of programs that can run concurrently on a single host and that will be using the program management function. Because the default value is 0, make sure that you specify a pgmmax operand value.

Program management function settings

Specify the monitor-mode program environment definition. For details about the monitor-mode program environment definition, see 8.5.1 Monitor-mode program environment definition (programs).

Creating a program restart command:

If you want to restart automatically an individual UAP, rather than an individual server, in the event of a UAP failure, you must create a program restart command. You specify your program restart command in the restartcommand operand in the monitor-mode program environment definition. For details about creating program restart commands, see 6.14 Creating a program restart command (when the program management function is used).

Specifying UAP environment variables:

Specify the environment variables required for executing UAPs. For details about specifying UAP environment variables, see 6.15.2 Specifying UAP environment variables.

3.7 Functions of HA Monitor Extension

This section explains the functions provided by HA Monitor Extension.

HA Monitor Extension is an optional HA Monitor product that is used to handle hot standby control in a large system for such purposes as increasing the maximum number of hosts to be controlled by HA Monitor.

Note that HA Monitor Extension cannot be used if SCSI reservation for shared disk or the function for controlling hot standby based on the availability of LAN communications is used.

The following table lists and describes the functions of HA Monitor Extension.

Table 3‒3: Functions of HA Monitor Extension

Function

When HA Monitor Extension is used

When HA Monitor Extension is not used

Changing the maximum number of servers

You can change to more than 128 the maximum number of servers that can run concurrently on a single host.

The maximum number of servers that can run concurrently on a single host is 128.

Changing the maximum number of hosts

You can change to more than 32 the maximum number of hosts in a hot-standby configuration.

The maximum number of hosts in a hot-standby configuration is 32.

The subsections below provide the details of these functions.

Important

To use HA Monitor Extension, you must perform the following tasks on all the hosts in the hot-standby configuration:

Install HA Monitor Extension.

Specify the environment settings for HA Monitor Extension in the definition file.

Execute the HA Monitor environment setup command (monsetup -ex command) to enable the functions of HA Monitor Extension.

3.7.1 Changing the maximum number of servers

The default for the maximum number of servers that can run concurrently on a single host is 128. When you use HA Monitor Extension, you can increase this number to more than 128, which enables you to apply one hot-standby configuration to a large system in which more than 128 servers run concurrently.

(1) Maximum number of servers that can be specified

When you use HA Monitor Extension, you can specify any value from 128 to 512 as the maximum number of servers that can run concurrently on a single host.

(2) Required environment settings

You use the servmax operand in the HA Monitor environment settings to specify the maximum number of servers that can run concurrently on a single host.

You must also execute the HA Monitor environment setup command (monsetup -ex command) to enable the functions of HA Monitor Extension.

3.7.2 Changing the maximum number of hosts

The default for the maximum number of hosts in a hot-standby configuration is 32. When you use HA Monitor Extension, you can increase this number to more than 32, which enables you to apply one hot-standby configuration to a large system in which more than 32 hosts run concurrently.

(1) Maximum number of hosts that can be specified

When you use HA Monitor Extension, you can specify any value from 32 to 256 as the maximum number of hosts in a hot-standby configuration.

(2) Operation when the maximum number of hosts is set to more than 32

When the maximum number of hosts is set to more than 32, alive messages are transmitted by multicasting on all monitoring paths. When one alive message is sent by multicasting, all HA Monitors (other than the local HA Monitor) receive the alive message. For details about alive messages, see (1) Monitoring between hosts.

(3) Required environment settings

To change the maximum number of hosts, you must specify the following environment settings:

Specify the maximum number of hosts in a single hot-standby configuration in the hostmax operand in the HA Monitor environment settings.

If the hot-standby configuration contains any host whose maximum count has been increased, you must specify this operand on all hosts in the hot-standby configuration.

Note that when you set the maximum number of hosts to more than 32, the alive message communication method is set automatically to multicasting. You can also use the alive_multicast operand in the HA Monitor environment settings to specify the multicast communication method.

If you want to specify a desired multicast group ID, specify it and the host name corresponding to that multicast group ID in the /etc/hosts file. You must also specify the host name that you specified in the /etc/hosts file in the multicast_lan operand in the HA Monitor environment settings.

If the multicast_lan operand is omitted, 239.0.0.1 will be assumed as the multicast group ID.

Execute the HA Monitor environment setup command (monsetup -ex command) to enable the functions of HA Monitor Extension.

The following figure shows an example of the environment settings for changing the maximum number of hosts.

Figure 3‒43: Example of environment settings for changing the maximum number of hosts

This example uses hosts 1 through 32 as primary systems and host 33 as the secondary system. For hosts 1 through 32, the number of hosts to be connected is 2 including the local host because they all connect to host 33. For host 33, the number of hosts to be connected is 33 including the local host because host 33 connects to hosts 1 through 32. Therefore, for host 33, 33 is specified in the hostmax operand. For hosts 1 through 32, although the number of hosts to be connected is 2 for each of them, 33 is specified in the hostmax operand because each of these hosts connects to host 33 for which the maximum number of hosts is set to 33 or greater.

4 System Management

This chapter describes how HA Monitor manages servers, hosts, and shared resources. It provides the details of HA Monitor's operation.

4.1 Managing servers

HA Monitor runs in both active and standby systems. The two HA Monitors control server status by communicating with each other regarding the startup and termination statuses of the active and standby servers. This section explains how HA Monitor controls servers.

For servers in the monitor mode, you can achieve program-level control by using the program management function. For details about managing programs, see 4.7 Managing programs (monitor mode).

4.1.1 HA Monitor's control of server startup

The user starts a server in the server mode by executing the start command provided by the program, and starts a server in the monitor mode by executing the monitor-mode server start command (monbegin command).

When a server is started, HA Monitor checks the server environment definition and the server status on the remote host to determine whether the server is to be started as the active server or a standby server. This subsection explains how HA Monitor determines the active server or standby server. It also explains the HA Monitor processing that occurs when the server status on the remote host cannot be checked due to a monitoring path failure.

(1) Determining the active server or standby server during normal operation

Normally, HA Monitor determines whether a server is to be started as the active server or a standby server on the basis of the start type specified in the initial operand in the server environment definition and the status of the remote host. The following table describes how this determination is made.

Table 4‒1: Determining whether the active server or a standby server

Status of remote host

Server environment definition

Defined as an active server

Defined as a standby server#1

Active server exists.

Startup processing is underway.

Start as a standby server.

Start as a standby server.

Execution processing is underway.

Start as a standby server.

Start as a standby server.

Termination processing is underway.

Cannot be started.#2

Cannot be started.#2

Standby server exists.

Startup processing is underway.

Start as the active server.

Cannot be started.#2

Execution processing is underway.

Start as the active server.

Cannot be started.#2

Termination processing is underway.

Start as the active server.

Wait until the active server starts.#3

Waiting for grouped-system switchover.#4

Start as the active server.

Start as the active server.

Hot standby switching from standby server to active server is underway.

Start as a standby server.

Start as a standby server.

Retrying startup.#5

Start as the active server.

Start as a standby server.

No server is found (HA Monitor is active).

Start as the active server.

Wait until the active server starts.#3

No server is found (HA Monitor is inactive).

Start as the active server.#7

Wait until the active server starts.#3

The server status cannot be checked.

Retry processing.#6

Wait until the active server starts.#3

#1: If the status is Retrying startup, the operation is the same as for Defined as an active server.

#2: Cancels the server startup processing.

#3: Places startup of the standby server on hold until startup of the active server on the remote host is confirmed (active server start wait state). After start of the active server startup processing is confirmed, HA Monitor starts the server as the standby server.

#4: For details about the grouped-system switchover wait state, see 4.4.1 Server switchover type during grouped-system switchover.

#5: For details about the retrying startup status, see 3.2.7 Performing start retries during hot standby processing.

#6: If HA Monitor is unable to check the server status on the remote host due to a host failure or a monitoring path failure on the remote host, it retries for only one minute. If the status still cannot be checked after one minute of retries, HA Monitor determines that no server is available on the remote host due to a failure and places the server in the active server start wait state.

#7: If it is impossible to check the status of a host for which pair is specified in the connection configuration file, HA Monitor places the server in the active server start wait state.

The method used to determine the active server or standby server is different when the server defined as the active server is restarted following a failure. For details, see (2) Start type when the server is restarted.

For details about starting and terminating a server in the wait state as the active server, see 7.4.1 Starting a server in the wait state and then restarting jobs.

(2) Start type when the server is restarted

When a server is restarted after a failure, an active server might already be running on the remote host. Therefore, HA Monitor does not start the server as an active server, even if it is defined as an active server, until the server status on the remote host can be verified. Once the server status on the remote host has been verified, HA Monitor determines whether the server is to be started as the active server or a standby server.

The following table describes how the start type is determined when a server defined as the active server is restarted.

Table 4‒2: Determining the start type when the server is restarted

Status of remote host

Server start type#1

Active server exists.

Startup processing is underway.

Start as a standby server.

Execution processing is underway.

Start as a standby server.

Termination processing is underway.

Cannot be started.#2

Standby server exists.

Startup processing is underway.

Start as the active server.

Execution processing is underway.

Cannot be started.#2

Termination processing is underway.

Wait until the active server starts.#3

Waiting for grouped-system switchover.

Start as the active server.

Hot standby switching from standby server to active server is underway.

Start as a standby server.

No server is found (or HA Monitor is inactive).

Wait until the active server starts.#3

The server status cannot be checked.

Wait until the active server starts.#3

#1: The start type specified in the initial operand in the server environment definition is ignored.

#2: Cancels the server startup processing.

#3: Places startup of the standby server on hold until startup of the active server on the remote host is confirmed (active server start wait state). After start of the active server startup processing is confirmed, HA Monitor starts the server as a standby server.

For details about starting and terminating a server in the wait state as the active server, see 7.4.1 Starting a server in the wait state and then restarting jobs.

4.1.2 HA Monitor's control of server termination

HA Monitor terminates a server in the event of a failure or when the operator executes a command. The conditions under which HA Monitor terminates a server depend on the server's mode (server mode or monitor mode).

(1) Server in the server mode

The principal circumstances under which HA Monitor terminates a server are as follows:

The operator has executed a command:

If the operator has terminated the active server normally or has performed planned or forced shutdown of the server by entering a command provided by the program, HA Monitor terminates the standby server in the standby system that corresponds to the active server.

When the operator has executed a forced termination command provided by the program on the active server, HA Monitor performs the same processing as for planned termination. When the operator has executed a forced termination command provided by the program on the standby server, HA Monitor performs the same processing as for abnormal termination.

When the operator has executed the server hot-standby switchover command (monswap command), HA Monitor terminates the active server before performing hot-standby switchover.

When the operator has executed the standby server termination command (monsbystp command), HA Monitor terminates the specified standby server.

A failure has occurred:

If HA Monitor detects a server failure, it performs one of the following operations, depending on settings specified in the server environment definition:

If execution of the hot standby operation is specified, HA Monitor terminates the active server before performing the hot standby operation.

If a restart of the active server is specified, HA Monitor changes the active server's status to active server restart wait state and then waits for the active server to be restarted.

The following table describes the events that result in server termination and the HA Monitor processing that occurs after the server has been terminated.

Table 4‒3: Events that result in server termination and processing that occurs after the server has been terminated (server mode)

Event

Server terminated by HA Monitor

HA Monitor processing after termination

Event on the server

Normal termination of active server#1

Standby server

--

Abnormal termination of active server#2

--

Automatic hot standby#3

Normal termination of standby server

--

--

Abnormal termination of standby server#4

--

--

Event on HA Monitor

Detection of an active server failure

Active server#3

Automatic hot standby#3

Standby server termination command (monsbystp command)

Standby server

--

Server hot-standby switchover command (monswap command)#5

Active server

Planned hot standby

Legend: --: HA Monitor does nothing.

#1: Includes planned and forced server termination.

#2: Occurs when the server itself has detected a failure.

#3: If there is no standby server to which hot standby switching is to be performed, HA Monitor does nothing. If restart or manual is specified in the switchtype operand in the server environment definition, HA Monitor places the active server on restart wait state without terminating it. HA Monitor waits for the active server to be restarted without performing the hot standby operation.

#4: Occurs when a forced server termination has occurred or the server itself has detected a failure.

#5: If there is no standby server to which hot standby switching is to be performed, HA Monitor does nothing.

(2) Server in the monitor mode

The user uses HA Monitor commands to switch over and terminate a server in the monitor mode.

The following table describes the events that result in server termination and the HA Monitor processing that occurs after the server has terminated.

Table 4‒4: Events that result in server termination and processing that occurs after the server has terminated (monitor mode)

Event

Server terminated by HA Monitor

HA Monitor processing after termination

Event on the server

Normal termination of active server

--

--

Abnormal termination of active server

--

Planned or automatic hot standby#1, #2

Normal termination of standby server

--

--

Abnormal termination of standby server

--

--

Event on HA Monitor

Monitor-mode server termination command (monend command)

Active server#1 and standby server

--

Standby server termination command (monsbystp command)

Standby server

--

Server hot-standby switchover command (monswap command)

Active server#3

Planned hot standby#3

Legend: --: HA Monitor does nothing.

#1: If a server termination command is specified in the termcommand operand in the server environment definition, HA Monitor executes the server termination command and terminates the active server specified in the server termination command.

#2: The operator requests the processing by executing the server hot-standby switchover command (monswap command) from the active system. If a server monitoring command is specified for the ptrlcmd_ex or patrolcommand operand in the server environment definition, HA Monitor performs restart or automatic hot-standby switchover. If there is no standby server to which hot standby switching is to be performed, HA Monitor does nothing.

#3: The operator performs planned termination of the active server, and then executes the server hot-standby switchover command (monswap command) from the active system. If a server termination command is specified in the termcommand operand in the server environment definition, the monswap command can be used to terminate the active server. If there is no standby server to which hot standby switching is to be performed, HA Monitor does nothing.

4.1.3 Server status transitions

HA Monitor controls server status. This subsection explains the server status transitions.

The server transition status depends on whether the server is started as the active server or a standby server.

(1) Server status transitions when the server is started as the active server

The following figure shows the server status transitions when the server is started as the active server.

Figure 4‒1: Active server's status transitions

#1: The status changes only if the server belongs to a server group to which server switchover order control is applied.

#2: Indicates a connection error on a shared resource used by the server, such as a volume group, LAN, or file system. The status changes only if an operand such as vg_neck or ip_neck is specified in the server environment definition.

#3: Indicates that a user command issued when the server startup processing started or the hot standby operation started has failed. The status changes only if the uoc_neck operand is specified in the server environment definition.

#4: The status changes only when SCSI reservation for shared disk is used.

#5: During normal startup processing, the status changes only if all shared disk reservations have failed. During the hot standby operation, the status changes if any of the shared disk reservations has failed.

#6: The status changes only on the server to which the KAMN310-I message is output.

(2) Server status transitions when the server is started as a standby server

The following figure shows the server status transitions when the server is started as a standby server.

Figure 4‒2: Standby server's status transitions

#1: Indicates that a user command issued when the server startup processing started or the hot standby operation started has failed. The status changes only if the uoc_neck operand is specified in the server environment definition.

#2: The status changes if the server belongs to a server group to which server switchover order control is applied.

#3: The status might change only in a multi-standby configuration.

#4: The status changes only if there is no standby server whose priority level is higher than that of the local system.

#5: The status changes only if the multi-standby function is used and host resets are suppressed.

(3) Standby server's status transitions in the online serverless mode

The following figure shows the status transitions of a standby server in the online serverless mode.

Figure 4‒3: Standby server's status transitions in the online serverless mode

#: Indicates that a user command issued when the server startup processing started has failed. The status changes only if the uoc_neck operand is specified in the server environment definition.

4.1.4 Controlling the server monitoring command

The server monitoring command is a command created by the user that is used to monitor servers in the monitor mode. This subsection explains the flow for starting and terminating the server monitoring command and for performing monitoring processing.

(1) Starting and stopping the server monitoring command

The timing of starting the monitoring command, the timing of stopping the monitoring command, the execution privilege, and the behavior when startup fails differ depending on whether the monitoring command is specified for the ptrlcmd_ex or patrolcommand operand in the server environment definition. The following table shows the differences that arise according to the specification of the monitoring command.

Table 4‒5: Differences arising according to the specification of the monitoring command

Item

Operand for which the monitoring command is specified

The ptrlcmd_ex operand in the server environment definition

The patrolcommand operand in the server environment definition

Timing of starting the monitoring command

After the active server has been started successfully, HA Monitor executes the monitoring command at regular intervals (the interval is specified for the ptrlcmd_ex_inter operand in the server environment definition).

When the active server is started successfully, HA Monitor starts the monitoring command.

Timing of stopping the monitoring command

The monitoring command is stopped at the following times:

Each time the server monitoring command is executed, it returns the result (EXIT code) and terminates.

If the monitoring command does not terminate within the time (number of seconds) specified for the ptrlcmd_ex_tmout operand in the server environment definition, HA Monitor stops the monitoring command.

When the active server is terminated or switched by planned hot-standby switchover, HA Monitor stops the monitoring command.

The monitoring command is stopped at the following times:

The server monitoring command detects a server failure and stops.

When the active server is terminated or switched by planned hot-standby switchover, HA Monitor stops the monitoring command.

Monitoring command execution privilege

Superuser

Superuser

Behavior when startup of the monitoring command fails

The system behaves according to the termination result (EXIT code) of 1 to 9.

Note that in the following case, although the subsequent monitoring processing is not performed, server processing normally continues.

After the active server is started successfully, the monitoring command is not found when the command is executed for the first time, or the monitoring command cannot be executed because the privilege to execute it is not given.

Although the monitoring processing is not performed, server processing continues normally.

(2) Processing flow

The following figure shows the processing flow for using the server monitoring command to monitor a server in the monitor mode.

Figure 4‒4: Processing flow for monitoring a server in the monitor mode (in the case where the monitoring command is specified for the ptrlcmd_ex operand in the server environment definition)

Figure 4‒5: Processing flow for monitoring a server in the monitor mode (in the case where the monitoring command is specified for the patrolcommand operand in the server environment definition)

4.1.5 Processing flow for order control of server switchover

If you organize multiple servers in a group, you can control the order in which the servers in the group are switched over. This subsection explains planned hot standby, the processing flow for controlling the order in which servers are switched over in the event of a server failure and in the event of a host failure, and the HA Monitor processing that occurs if the hot standby operation fails.

(1) Planned hot standby and the processing flow for controlling the switchover order after a server failure in the monitor mode

The figure below shows the order in which the servers are terminated during planned hot standby and the order in which the servers are started on the target host. For the parent-child relationships among the servers shown in this figure, see (1) Correspondence between server start order and parent-child relationships.

Figure 4‒6: Processing flow for controlling the order in which servers are switched over during a planned hot standby operation

The following are details of the processing flow shown in the figure. The numbers correspond to the numbers in the figure.

Active servers 1 to 4 have started in the active system and standby servers 1 to 4 have started in the standby system. All of them are set up for grouped-system switchover.

The server hot-standby switchover command (monswap command) is executed to start planned hot-standby switchover.

HA Monitor starts termination processing on servers 3 and 4 that are at the bottom of the parent-child relationship chain. Servers 1 and 2 are placed in termination wait state.

If you use the server and host status display command (monshow command), ONL is displayed as the status of servers 1 and 2. ONL means that active server startup processing has been completed, but the servers themselves are actually in termination wait state. In other words, the status displayed by the command is not the same as the actual server status.

HA Monitor performs termination processing on servers 3 and 4 in parallel. During this termination processing, HA Monitor first terminates the servers, and then disconnects the shared resources. When the shared resources have been disconnected, HA Monitor notifies the standby system to prepare for hot standby. Next, servers 3 and 4 are placed on wait state until the required server is started because servers 3 and 4 must wait for completion of the hot standby operation for server 2, which is their parent server.

If you use the server and host status display command (monshow command), ONL is displayed as the status of servers 3 and 4. ONL means that the active servers are running, but the servers themselves are actually on wait state until the required server has started. In other words, the status displayed by the command is not the same as the actual server status.

When termination processing is completed on both servers 3 and 4, HA Monitor starts termination processing on server 2, which is their parent server.

When termination processing is completed on server 2, HA Monitor starts termination processing on server 1, which is server 2's parent server.

When termination processing is completed on the highest-level server (server 1), HA Monitor notifies the standby system and then starts the hot standby operation first on server 1.

When server 1 has started as the active server pursuant to the hot standby operation, HA Monitor starts the hot standby operation on its child server (server 2).

When the hot standby operation has been completed on server 2, HA Monitor starts the hot standby operation on its child servers (servers 3 and 4) in parallel. When the hot standby operation has been completed on servers 3 and 4, grouped-system switchover is complete.

(2) Processing flow for controlling switchover after a server failure in the server mode

This subsection explains the order in which the servers are terminated after a server failure and the order in which the servers are started on the target host for the following two cases:

A failure occurred on a server that has both parent and child servers

A failure occurred on the lowest-level server in a chain of servers with parent-child relationships

For the parent-child relationships among the servers shown in this figure, see (1) Correspondence between server start order and parent-child relationships.

(a) For a failure on a server that has both parent and child servers

The following figure shows the processing flow for controlling the switchover order after a failure on a server that has both parent and child servers.

Figure 4‒7: Processing flow for controlling the switchover order after a server failure (when the failure has occurred on a server that has both parent and child servers)

The following explains the processing flow shown in the figure, where the item numbers correspond to the numbered sections in the figure:

Active servers 1 to 4 have started in the active system and standby servers 1 to 4 have started in the standby system. All of them are set up for grouped-system switchover.

If a failure occurs on active server 2, a grouped-system switchover starts.

HA Monitor starts termination processing on servers 3 and 4 that are at the bottom of the parent-child relationship chain when active server 2 resulted in the failure. Server 1 is placed in termination wait state.

If you use the server and host status display command (monshow command), ONL is displayed as the status of server 1. ONL means that active server startup processing has been completed, but the server itself is actually in termination wait state. In other words, the status displayed by the command is not the same as the actual server status.

HA Monitor performs termination processing on servers 2 through 4. During this termination processing, HA Monitor first terminates the servers, and then disconnects the shared resources. When the shared resources have been disconnected, HA Monitor notifies the standby system that that it can prepare for hot standby. Next, the servers 2 through 4 are placed on wait state until the required server has started.

If you use the server and host status display command (monshow command), ONL is displayed as the status of servers 2 through 4. ONL means that the active servers are running, but the servers themselves are actually on wait state until the required server has started. In other words, the status displayed by the command is not the same as the actual server status.

When termination processing is completed on servers 2 through 4, HA Monitor starts termination processing on server 1, which is the highest-level server.

When termination processing is completed on server 1, HA Monitor notifies the standby system and then starts the hot standby operation first on server 1

When the hot standby operation is completed on server 1, HA Monitor starts the hot standby operation on its child server (server 2).

When the hot standby operation is completed on server 2, HA Monitor starts the hot standby operation on its child servers (servers 3 and 4) in parallel. When the hot standby operation is completed on servers 3 and 4, grouped-system switchover is complete.

(b) For a failure on the lowest-level server in a chain of servers with parent-child relationships

The following figure shows the processing flow for controlling the switchover order after a failure on the lowest-level server in a chain of servers with parent-child relationships.

Figure 4‒8: Processing flow for controlling the switchover order after a server failure (when the failure has occurred on the lowest-level server in a chain of servers with parent-child relationships)

The following explains the processing flow shown in the figure, where the item numbers correspond to the numbered sections in the figure:

Active servers 1 to 4 have started in the active system and standby servers 1 to 4 have started in the standby system. All of them are set up for grouped-system switchover.

If a failure occurs on active server 3, grouped-system switchover starts.

HA Monitor starts termination processing on servers 3 and 4 that are at the bottom of the parent-child relationship chain. Servers 1 and 2 are placed in termination wait state.

If you use the server and host status display command (monshow command), ONL is displayed as the status of servers 1 and 2. ONL means that active server startup processing is completed, but the servers themselves are actually in termination wait state. In other words, the status displayed by the command is not the same as the actual server status.

HA Monitor performs termination processing on servers 3 and 4 in parallel. During this termination processing, HA Monitor first terminates the servers, and then disconnects the shared resources. When the shared resources have been disconnected, HA Monitor notifies the standby system so that it can prepare for hot standby. Next, servers 3 and 4 are placed on wait state until the required server is started because servers 3 and 4 must wait for completion of the hot standby operation for server 2, which is their parent server.

If you use the server and host status display command (monshow command), ONL is displayed as the status of servers 3 and 4. ONL means that the active servers are running, but the servers themselves are actually on wait state until the required server has started. In other words, the status displayed by the command is not the same as the actual server status.

When termination processing is completed on both servers 3 and 4, HA Monitor starts termination processing on server 2, which is their parent server.

When termination processing is completed on server 2, HA Monitor starts termination processing on server 1, which is its parent server.

When termination processing is completed on the highest-level server (server 1), HA Monitor notifies the standby system, and then starts the hot standby operation first on server 1.

When the hot standby operation has been completed on server 1, HA Monitor starts the hot standby operation on its child server (server 2).

When the hot standby operation has been completed on server 2, HA Monitor starts the hot standby operation on its child servers (servers 3 and 4) in parallel. When the hot standby operation has been completed on servers 3 and 4, grouped-system switchover is complete.

(3) Processing flow for controlling the switchover order after a host failure

The figure below shows the order in which the servers are terminated after a host failure and the order in which the servers are started on the target host. For the parent-child relationships among the servers shown in this figure, see (1) Correspondence between server start order and parent-child relationships.

Figure 4‒9: Processing flow for controlling the switchover order after a host failure

The following explains the processing flow shown in the figure, where the item numbers correspond to the numbered sections in the figure:

Active servers 1 to 4 have started in the active system and standby servers 1 to 4 have started in the standby system. All of them are set up for grouped-system switchover.

If a failure occurs on an active server, HA Monitor resets the active system and starts the hot standby operation.

HA Monitor starts the hot standby operation on server 1, the highest-level server in the parent-child relationships.

When the hot standby operation has been completed on server 1, HA Monitor starts the hot standby operation on its child server (server 2).

When the hot standby operation has been completed on server 2, HA Monitor starts the hot standby operation on its child servers (servers 3 and 4) in parallel. When the hot standby operation has been completed on servers 3 and 4, grouped-system switchover is complete.

(4) Processing when the hot standby operation fails

If the hot standby operation fails on a parent server, all of its child servers result in a hot standby error, in which case the standby servers of the corresponding child servers are terminated forcibly. If there is no parent server at the switchover destination, HA Monitor resumes the hot standby operation.

If the start retry function is used during hot standby processing, the parent server's hot standby processing fails, the parent server is placed in the start retry state, and the child server is placed in the wait state until the required server is started. If the parent server is placed in the start retry state due to a failure after hot standby processing has been completed, the child server does not retry startup processing.

(5) About servers on the source host

When grouped-system switchover is performed on a server group that uses the order control function, any server on which grouped-system switchover cannot be performed during startup processing will be terminated forcibly.

On the switchover source host, none of the servers belonging to the server group subject to grouped-system switchover processing can be started until all the applicable servers have been terminated during the grouped-system switchover processing.

If a target server fails to start during grouped-system switchover processing and there is no standby server available for the hot standby operation in the same server group, grouped-system switchover is not performed even if active server startup processing fails.

(6) Notes on performing order control for TP1/Server Base and HiRDB

If all of the following conditions are met, specify the settings so that TP1/Server Base and HiRDB do not automatically restart when they terminate abnormally:

TP1/Server Base and HiRDB are server-mode servers.

Order control is performed for TP1/Server Base and HiRDB.

The value switch is specified for the switchtype operands of TP1/Server Base and HiRDB.

Note

Be careful if switch is specified for the switchtype operand in the server environment definition on a server of a server group for which order control is enabled. In this case, to prevent TP1/Server Base or HiRDB from being automatically restarted when they terminate abnormally, specify MANUAL2 for the following operand:

In TP1/Server Base: mode_conf operand in the system environment definition

For HiRDB: pd_mode_conf operand in the system common definition

For details about these operands, see the manual OpenTP1 System Definition or the manual HiRDB System Definition.

4.2 Managing hosts

If a host reset is to be performed, HA Monitor performs a hot standby operation after it resets the host on which the error occurred.

This section explains HA Monitor processing related to host resets. For details about the operations by the user, see 7. System Operation.

4.2.1 How to determine a reset issuing host

In HA Monitor, if a host failure occurs on the active system, its standby system performs a host reset on the active system and then performs the hot standby operation. Depending on the hot-standby configuration, multiple standby systems might issue a host reset, thereby delaying recovery of the active system. For this reason, you must designate in advance the standby system that is to perform host reset. The standby system designated to perform host reset is called the reset issuing host. For details about the configuration in which multiple standby systems might issue a host reset, see 3.3.3 Preventing multiple resets on a host.

This subsection explains how to designate the reset issuing host and the processing performed by the reset issuing host in the event of a host failure.

(1) How to designate the reset issuing host

At the time of system startup, the operator starts the active server in the active system. In the standby system, the operator starts the standby server corresponding to the active server. When the first combination (pair) of active and standby systems is established, that standby system becomes the reset issuing host for the paired active system. If a host failure occurs in the active system, the first standby system paired with the active system (reset issuing host) resets the active system. Even if the same active system is paired with another standby system, only the first standby system paired with that active system can reset the active system as the reset issuing host.

If the servers are terminated and the pair relationship is no longer valid between the active system and the first standby system that became the reset issuing host, the active system is paired with a new standby system and that standby system becomes the reset issuing host. If multiple pairs are established between active and standby systems, the standby system with the lowest host address becomes the reset issuing host thereafter.

The following figure shows how HA Monitor designates the reset issuing host.

Figure 4‒10: How HA Monitor designates the reset issuing host

In this figure, when active server 1 is terminated and the pair relationship between this active system and standby system 1 is no longer valid, one of the standby systems that are paired later, standby system 2 or standby system 3, becomes the reset issuing host. In this example, standby system 2 becomes the reset issuing host because its host address is smaller than that of standby system 3.

(2) Processing by the reset issuing host in the event of a host error

The reset issuing host resets the active system and then notifies the other standby systems of completion of the reset operation. Upon receiving the reset completion message, a standby system that is not the reset issuing host switches the active server in the active system that was reset over to the standby server on its local host.

If a standby system that is not the reset issuing host detects a host failure in the active system, the standby system waits without doing anything until it receives a reset completion message from the reset issuing host. If the standby system does not receive a reset completion message within 40 seconds (80 seconds when VMware ESXi-based virtualization is used) for a reason such as a failure in the monitoring LAN, it determines that the reset from the reset issuing host has failed, in which case the standby system that detected the host failure in the active system resets the active system.

The following figure shows the processing by the reset issuing host in the event of a host failure.

Figure 4‒11: Processing by the reset issuing host in the event of a host failure

The following provides the details of the processing by the reset issuing host in the event of a host failure, where the item numbers correspond to the numbers in the figure:

Send a reset instruction.

When the standby system (reset issuing host) detects a host failure, it resets the active system.

Send a reset completion message.

When the reset operation has been completed, the reset issuing host sends a reset completion message to the other standby systems.

Perform the hot standby operation.

When the reset completion message is received, each active server in the active system is switched over to the corresponding standby system.

The following figure shows the processing when a standby system that is not the reset issuing host has detected a host failure in the active system.

Figure 4‒12: Processing when a standby system that is not the reset issuing host has detected a host failure in the active system

The following provides the details of the processing when a standby system that is not the reset issuing host has detected a host failure in the active system, where the item numbers correspond to the numbers in the figure:

No reset completion message has been received.

If a host that is not the reset issuing host detects a host failure in the active system, it waits for a reset completion message from the reset issuing host.

Send a reset instruction.

If no reset completion message is received, for a reason such as a failure in the monitoring LAN, the host that detected the failure resets the active system.

The amount of time the host waits for the reset completion message is 40 seconds (80 seconds if VMware ESXi-based virtualization is used).

4.2.2 Hot standby operation when both hosts detect a failure at the same time

If the active and standby systems both detect a failure at the same time, both systems might issue a host reset, resulting in termination of both systems. To prevent this, HA Monitor provides a function for determining the host that is to perform a host reset first (host that has reset priority). For details about configurations in which active and standby systems might both be terminated, see 3.3.1 Preventing concurrent host resets.

This subsection explains how HA Monitor performs the hot standby operation when the active and standby systems both detect a failure at the same time. The following figure shows the HA Monitor processing flow in the event both hosts detect a host failure at the same time.

Figure 4‒13: Processing flow when both hosts detect a host failure at the same time

#1

This setting is specified with the standbyreset operand in the HA Monitor environment settings.

#2

This setting is specified with the cpudown operand in the HA Monitor environment settings. If online is specified, the active system becomes the host that has reset priority. If standby is specified, the standby system becomes the host that has reset priority. If system is specified, the host that has reset priority is not determined.

#3

This setting is specified with the address operand in the HA Monitor environment settings.

#4

If VMware ESXi-based virtualization is used, the system waits for 40 seconds.

The following subsections explain the differences in the hot standby operation depending on whether the user specifies the host that has reset priority as indicated in the figure.

(1) Hot standby operation when the user specifies the host that has reset priority

If the HA Monitor environment settings provide that the user specifies the host that has reset priority and both hosts detect a failure at the same time, the host that has reset priority resets the remote host. A host that does not have reset priority waits for 20 seconds (40 seconds if VMware ESXi-based virtualization is used) after it has detected a failure and then issues a reset request. Note that even if a host has been defined as having reset priority, its reset priority might change depending on whether the local host has more running servers than the remote host.

(2) Hot standby operation when the user does not specify the host that has reset priority

If the HA Monitor environment settings provide that the user does not specify the host that has reset priority and both hosts detect a failure at the same time, HA Monitor issues a reset request from both hosts. In this case, concurrent host resets are prevented because only the first reset request accepted by SVP is processed. In the operation mode in which the host that has reset priority is not specified, the time required for the hot standby operation is reduced because neither host waits for issuance of a reset.

You can choose this operation mode when you employ a mutual hot-standby configuration, use BladeSymphony, and SVP supports the cluster management function. For details about the cluster management function, see the hardware-related documentation. The user must specify the host that has reset priority in the case where you want to perform hot standby switching to one of the hosts in the event of a failure on all monitoring paths.

If the user does not specify the host that has reset priority, they must specify the reset path and SVP settings. For details, see 6.6.1 Information required for configuration (BladeSymphony) and (7) Mutual hot-standby configuration (when a host with reset priority is not specified).

4.2.3 Host reset when there are multiple standby systems

This subsection explains host reset when there are multiple standby systems, such as in a multi-standby configuration.

(1) How to determine reset priorities

If there are multiple standby systems, you must specify reset priorities in order to determine a unique reset issuance order in all hosts belonging to the same hot-standby configuration.

When the local host is connected to the remote hosts at startup, HA Monitor determines the local host's reset priority. Among all the connected hosts, the one with the lowest host address has the highest priority. The reset priority ranges from 0 to 31, where 0 is the highest priority.

The figure below shows the relationship between host address and reset priority. In this example, if a failure occurs on host 3, the reset is issued by host 1 because it has the lowest host address.

Figure 4‒14: Relationship between host address and reset priority

If a host failure is detected, the host with the highest reset priority issues the reset. Each other host waits for a specific period of time before it issues a reset, depending on its reset priority. This wait time for each host is determined by the formula host's reset priority × 10 (seconds).

Note that the value of 10 (seconds) in the formula might differ, depending on the system being used.

(2) Host reset when reset priorities are specified

This subsection explains how each host functions during a host reset when reset priorities have been specified.

A host that has detected a host failure issues a reset to the host on which the failure occurred according to its (the local host's) reset priority. That host then issues a reset issuance request to all the hosts that are connected. A host that has received the reset issuance request issues a reset to the failed host according to the local host's reset priority.

If a reset is successful:

If a reset is successful, the host that issued the reset notifies all the hosts of the success (reset success notification). If there is any standby server to be switched over, the host then performs the hot standby operation.

A host that has received the reset success notification stops waiting if it has been on reset issuance wait state for the failed host. If there is any standby server to be switched over, the host performs the hot standby operation.

If a reset fails:

If a reset fails, the host that issued the reset waits for a reset success notification from some other host.

(3) Processing if a host reset from a standby system has failed

If a host reset from a standby system fails, the hot standby operation can be performed automatically because there are multiple standby systems. This subsection explains by way of example the reset processing when a host reset from a standby system has failed. In this example, a failure has occurred on host 2, a reset from host 1 has failed, and a reset is then issued from host 3 after the reset issuance wait time has elapsed. The following figure shows the host reset processing when a host reset has failed in a hot-standby configuration where there are multiple standby systems.

Figure 4‒15: Host reset processing when a host reset has failed in a hot-standby configuration where there are multiple standby systems

The following provides the details of reset issuance when reset priorities are specified, where the item numbers correspond to the numbers in the figure:

Send a reset instruction and a reset request.

Host 1, which has the highest reset priority, detects a host failure on host 2 and issues a reset instruction to host 2. Host 1 also sends a reset request to the other hosts.

Host 3 detects a host failure on host 2 and waits for issuance of a host 2 reset.

Reset has failed.

If the reset instruction from host 1 has failed, host 1 waits for a reset success notification from another host.

Send a reset instruction.

If there is no reset success notification from host 1 and the reset wait time based on the reset priority has elapsed, host 3 sends a reset instruction to host 2.

Send a reset success notification and perform the hot standby operation.

If the reset is successful, host 3 sends a reset success notification to the other hosts and performs the hot standby operation (if there is a standby server available for the hot standby operation).

4.2.4 Processing in the event a host reset has failed

If a host reset has failed, HA Monitor starts the standby server in the standby system as the active server even if the active system has not been reset. In this case, instances of the same active server might be running in both the active system and the standby system. If you terminate the standby server in the standby system abnormally to prevent multiple active servers from starting, the hot standby operation can no longer be performed immediately in the event of a failure. Therefore, after switching the standby server in the standby system over to the active server, HA Monitor places the active server in the hot-standby wait state before actually starting it as the active server.

The following figure shows the processing in the event host reset has failed.

Figure 4‒16: Processing in the event host reset has failed

For details about how to start and terminate a server in the hot-standby wait state as the active server, see 7.4.1 Starting a server in the wait state and then restarting jobs.

Note

If a reset fails on a host in the LPAR mode or in a virtual machine, you can prevent the host from being placed in the hot-standby wait state by using the physical partition reset function. For details about the physical partition reset function, see (2) Operation when the physical partition reset function is used.

4.3 Managing shared resources

This section explains how HA Monitor manages resources used by servers.

Servers can also use other resources than the shared resources described here. However, you must create user commands for controlling shared resources. For details about how to create user commands, see 6.18 Creating user commands.

4.3.1 Managing shared disks

If servers share a disk (shared disk), HA Monitor uses OS commands to control connection with and switching of the shared disk. HA Monitor controls the switching of a shared disk in units of volume groups.

The following table shows the shared disk control methods and the formats of the commands used by HA Monitor.

Table 4‒6: Shared disk control methods and formats of commands executed by HA Monitor

Control method

Format of command to be executed

Connection for referencing and updating

vgchange -a y path-name-of-volume-group

Disconnection

vgchange -a n path-name-of-volume-group

To enable use of the vgchange command with options for referencing and updating specified during connection, the vg_on_opt operand must have been specified in the server environment definition.

(1) Connecting

The kernel connects a shared disk.

When HA Monitor is connected to a shared disk, the kernel performs logical locking in units of volume groups so that the shared disk cannot be accessed from a standby server while the active server is using it.

The following figure shows the processing flow for connecting a shared disk.

Figure 4‒17: Processing flow for connecting a shared disk

(2) Switching

When a shared disk is switched, locking is performed in such a manner that the shared disk cannot be accessed from a server on a host resulting in a failure, but can be accessed from a server on a host that inherits job processing. When a volume group is disconnected, the fuser OS command is used to forcibly terminate all processes using the shared disk.

The processing flow for shared disk switching in the event of a server failure differs from that for host failures. The following figure shows the processing flow for shared disk switching in the event of a server failure.

Figure 4‒18: Processing flow for shared disk switching in the event of a server failure

The following figure shows the processing flow for shared disk switching in the event of a host failure.

Figure 4‒19: Processing flow for shared disk switching in the event of a host failure

(3) Log files

When volume groups are connected and disconnected, the execution results are output for each volume group with the file name volume-group-name.vglog under /opt/hitachi/HAmon/spool/. If an error occurs, check these log files to determine the cause of the error.

If the size of a log file exceeds 65,535 bytes when log data is acquired, HA Monitor creates a backup file named volume-group-name.vglog_old, clears the log file, and then collects log data again.

4.3.2 Managing file systems

HA Monitor switches file systems on a shared disk by mounting and unmounting them. The following table describes how to control switching of file systems and HA Monitor processing.

Table 4‒7: How to control switching of file systems and HA Monitor processing

Control method

HA Monitor processing

Mounting

After connecting a shared disk, HA Monitor connects a file system by performing the following processing:

A file system is checked by using the fsck command of the OS (or the xfs_repair command if the file system of the shared disk is XFS in RHEL7 or a later version). Then, the file system is mounted.

If the connection operation fails, HA Monitor issues an error message and resumes processing.

File systems are mounted in the order specified in the fs_name operand in the server environment definition.

Unmounting

Before disconnecting a shared disk, HA Monitor disconnects the file systems by performing the processing described below. If the active server is terminated, HA Monitor always unmounts the file systems regardless of the cause of termination.

The file system is unmounted.

If the unmount operation fails, HA Monitor uses the fuser OS command to forcibly terminate all processes that are accessing the file system. Then, HA Monitor retries an unmount attempt once a second before the retry count limit specified in the server environment definition is reached. If the retry count limit is reached, HA Monitor issues an error message and resumes processing.

File systems are unmounted in the reverse order of the specification of the fs_name operand in the server environment definition.

File systems are mounted in the order in which they are specified for the fs_name operand in the server environment definition. The file systems are unmounted in reverse order.

(1) Connecting

When the active server starts, HA Monitor unmounts the file system from the standby server and then mounts it from the active system so that the active server can use the file system. If there are processes that are using the file system when it is unmounted, HA Monitor uses the fuser OS command to forcibly terminate those processes. When the file system is mounted, its consistency is checked by using the fsck command of the OS (or the xfs_repair command if the file system of the shared disk is XFS in RHEL7 or a later version).

The following figure shows the processing flow for connecting a file system.

Figure 4‒20: Processing flow for connecting a file system

(2) Switching

HA Monitor switches file systems by unmounting the file system from the host resulting in a failure and mounting the file system from the host that inherits job processing. If there are processes that are using the file system when it is unmounted, HA Monitor uses the fuser OS command to forcibly terminate those processes. The processing flow for file system switching in the event of a server failure differs from that for host failures.

The following figure shows the processing flow for file system switching in the event of a server failure.

Figure 4‒21: Processing flow for file system switching in the event of a server failure

The following figure shows the processing flow for file system switching in the event of a host failure.

Figure 4‒22: Processing flow for file system switching in the event of a host failure

(3) Log files

When HA Monitor switches file systems, it outputs the execution results to a file named /opt/hitachi/HAmon/spool/server-alias-name.fslog for each server. In the event of an error such as a switching error, you can determine the cause of the error by referencing this log file.

You can specify a maximum size for a log file in the fs_log_size operand in the HA Monitor environment settings. If a log file's size exceeds this value during log collection, HA Monitor backs up the file under the name server-alias-name.fslog_old, clears the file, and then collects log data again.

4.3.3 Managing a LAN

HA Monitor assigns an alias IP address to a server and switches a LAN by adding and deleting that alias IP address to and from the LAN adapter.

(1) Connecting

When HA Monitor sets a network interface status, it adds an alias IP address to the LAN adapter. The following figure shows the processing flow for connecting with a LAN.

Figure 4‒23: Processing flow for connecting with a LAN

(2) Switching

When HA Monitor sets the network interface status, it deletes the alias IP address.

The following figure shows the processing flow for switching a LAN in the event of a server failure.

Figure 4‒24: Processing flow for switching a LAN in the event of a server failure

The following figure shows the processing flow for switching a LAN in the event of a host failure.

Figure 4‒25: Processing flow for switching a LAN in the event of a host failure

4.3.4 List of shared resource statuses

This subsection explains the events that occur in HA Monitor and on the servers and how HA Monitor controls the shared resource status when each event occurs.

The following table describes the shared resource statuses that are controlled by HA Monitor.

Table 4‒8: Shared resource statuses controlled by HA Monitor

Event

Shared resource

Shared disk

File system

LAN

Volume group

Event on the server

Start of the active server has started.

Connected for referencing and updating

Mounted

Connected

Start of the active server has been completed.

--

--

--

Start of the active server failed

Disconnected

Unmounted

Disconnected

Start of the active server failed (when the hot standby operation is performed in a multi-standby configuration).

Disconnected

Unmounted

Disconnected

The active server restart limit was detected.#1

Disconnected

Unmounted

Disconnected

The active server was terminated normally.#2

Disconnected

Unmounted

Disconnected

Abnormal termination of active server

#3, #4

Disconnected

Unmounted

Disconnected

Start of the standby server has started.

Disconnected

Unmounted

Disconnected

Start of the standby server was completed.

--

--

--

Start of the standby server failed.

--

--

--

The standby server was terminated normally.#2

--

--

--

The standby server was terminated abnormally.

--

--

--

Event in HA Monitor

Detection of an active server failure

#4, #5

Disconnected

Unmounted

Disconnected

An active system failure was detected.

Input/output reset#6

Input/output reset#6

Input/output reset#7

A standby system failure was detected.

--

--

--

Hot standby switching to the standby system was completed.

Connected for referencing and updating

Mounted

Connected

Hot standby switching to the standby system failed.

Disconnected

Unmounted

Disconnected

Monitor-mode server startup command

(monbegin command)#8

Active system

Connected for referencing and updating

Mounted

Connected

Standby system

Disconnected

Unmounted

Disconnected

Monitor-mode server termination command

(monend command)

Active system

Disconnected

Unmounted

Disconnected

Standby system

--

--

--

Standby server termination command

(monsbystp command)

--

--

--

Server hot-standby switchover command

(monswap command)

Active system

Disconnected

Unmounted

Disconnected

Standby system

Connected for referencing and updating

Mounted

Connected

Command for adding shared resources

(mondevice -a command)

Active system

Connected for referencing and updating

N

N

Standby system

Disconnected

N

N

Command for deleting shared resources

(mondevice -d command)

Active system

--

N

N

Standby system

--

N

N

Legend:

--: Remains unchanged.

N: Not subject to command processing.

#1: Already disconnected if the active server is in the start retry state.

#2: Includes planned and forced server termination.

#3: This is when the server itself has detected a failure.

#4: The status remains unchanged when the active server is placed in the restart wait state.

#5: The status remains unchanged if there is no standby server subject to the hot standby operation.

#6: This resource is forcibly reserved if SCSI reservation for shared disk is used.

#7: The status remains unchanged if SCSI reservation for shared disk is used.

#8: After server startup has been completed or has failed, the shared resource statuses are the same as for similar events on the server.

4.4 Managing servers when servers are grouped

If you group multiple servers together and a failure occurs on one server in the server group, you can perform grouped-system switchover to perform the hot standby operation on the entire server group. This section explains HA Monitor processing for performing grouped-system switchover.

4.4.1 Server switchover type during grouped-system switchover

For servers in the server mode, you can specify for each server in a server group whether grouped-system switchover is to be performed in the event of a failure on that server. This specification is called the server switchover type. This subsection explains the relationship between the specification of the server switchover type and HA Monitor processing.

If you specify for the server switchover type for a server that grouped-system switchover is to be performed if a server failure occurs on that server, HA Monitor will perform grouped-system switchover in the event of a server failure on the server. If you specify that grouped-system switchover is not to be performed, HA Monitor will terminate abnormally that server if it suffers a server failure and will place the corresponding standby server in grouped-system switchover wait state.

If you use TP1/Server Base or HiRDB as a server in the server mode, you can specify the settings so that grouped-system switchover is not performed even when a failure occurs on the server. In this case, you can select how the system behaves if the server slows down from the following two types: 1) abnormally terminate the server and wait for grouped-system switchover, and 2) continue server monitoring without abnormally terminating the server.

You specify the server switchover type in the group operand in the server environment definition. Specify exchange to perform grouped-system switchover in the event of a server failure and no_exchange to not perform grouped-system switchover. Specify cancel to terminate the server abnormally without performing grouped-system switchover in the event of a server slowdown and no_cancel to continue monitoring even if a server slowdown is detected.

Notes

Even if no_exchange is specified, grouped-system switchover is performed in the following case:

A server group includes a resource server and all the resource server's child servers fail.

If you do not want grouped-system switchover to occur when a server failure occurs even if no_exchange is specified, use either of the following server configurations:

Do not use a resource server. Instead, use a monitor-mode server for which only resources are defined.

Define a monitor-mode server as a child server of the resource server in the server group.

In the environment definition for the monitor-mode server, specify either of the following settings:

• Set no server monitoring command.

• Set 9999 for the servexec_retry operand in the server environment parameter so that an attempt to restart the server can be performed indefinitely if a failure occurs.

If the server group already includes a monitor-mode server for which the preceding settings have been specified, you do not need to add a new server.

If either of the preceding server configurations is used, grouped-system switchover occurs only when a host failure occurs or when server planned hot-standby switchover is performed (monswap -g).

The figure below shows the relationship between the specification of the server switchover type and grouped-system switchover. In this example, there are two server groups, A and B.

Figure 4‒26: Relationship between the specification of the server switchover type and grouped-system switchover

In server group groupA, HA Monitor performs grouped-system switchover because a failure occurred on active server 1 for which exchange is specified. Active servers 1 and 2 continue job processing in the standby system.

In server group groupB, HA Monitor does not perform switchover because a failure occurred on active server 4 for which no_exchange is specified. Active server 3 continues job processing as is in the active system.

4.4.2 Hot standby control on grouped servers

When servers are grouped, hot standby control depends on the combination of the server switchover types specified for the individual servers in the server group.

Note that a server switchover type cannot be specified for a server in the monitor mode. If a server group contains a server in the monitor mode, it is assumed that exchange is specified as its server switchover type. If you want to perform the hot standby operation in the event of a server failure or slowdown on a server in the monitor mode, you must create a server monitoring command.

The following table shows the relationship between the server switchover type and HA Monitor's hot standby control for each event that triggers the hot standby operation.

Table 4‒9: Relationship between server switchover type and HA Monitor's hot standby control

Event that triggers the hot standby operation

Combination of server switchover types

Event classification

Server on which the event has occurred

exchange only

exchange and no_exchange

no_exchange only

Is there any server running in the group?

Is there any server running in the group?

Yes

No

Yes

No

Server failure has occurred.

Server for which exchange is specified.

Performs grouped- system switchover.

Performs grouped- system switchover.

Performs the hot standby operation independently.#1

--

--

Server for which no_exchange is specified.

--

Waits for grouped- system switchover.#2

Performs the hot standby operation independently.#1

Waits for grouped- system switchover.#2, #3

Performs the hot standby operation independently.#1

Server slowdown has occurred.

Server for which exchange is specified.

Performs grouped- system switchover.

Performs grouped- system switchover.

Performs the hot standby operation independently.#1

--

--

Server for which no_exchange is specified.

--

Depends on another setting.#4

Performs the hot standby operation independently.#1

Depends on another setting.#4

Performs the hot standby operation independently.#1

Host failure has occurred.

Server for which exchange is specified.

Performs grouped- system switchover.

Performs grouped- system switchover.

Performs the hot standby operation independently.#1

--

--

Server for which no_exchange is specified.

--

Performs grouped- system switchover.

Performs the hot standby operation independently.#1

Performs grouped- system switchover.

Performs the hot standby operation independently.#1

Planned hot standby is to be performed.

Server for which exchange is specified.

Performs grouped- system switchover.

Performs grouped- system switchover.

Performs the hot standby operation independently.#1

--

--

Server for which no_exchange is specified.

--

Performs grouped- system switchover.

Performs the hot standby operation independently.#1

Performs grouped- system switchover.

Performs the hot standby operation independently.#1

Legend:

--: Invalid combination

#1:

If the standby system contains a standby server that belongs to the same server group and that is on grouped-system switchover wait state, HA Monitor starts that standby server as the active server at the same time.

#2:

The standby server has started in the standby system and is waiting for grouped-system switchover on the server group. For details about the grouped-system switchover wait state, see 7.4.1 Starting a server in the wait state and then restarting jobs.

#3:

HA Monitor performs grouped-system switchover if a failure occurs on all servers.

#4:

If cancel is specified, HA Monitor terminates the server abnormally. The standby server is placed on grouped-system switchover wait state. If no_cancel is specified, HA Monitor continues monitoring server status. The standby server is not placed on grouped-system switchover wait state.

4.5 Managing servers and hosts when using the multi-standby function

The multi-standby function provides multiple standby servers for a single active server. By providing multiple standby servers, you can protect against system failures while the primary system is being recovered from a failure.

When the multi-standby function is used, the server start and termination conditions, as well as host reset, differ from when the multi-standby function is not used. This section describes the differences depending on whether the multi-standby function is used.

4.5.1 Detection of host failures and host reset (multi-standby)

This subsection describes some of the differences between when the multi-standby function is used compared to when it is not used, with respect to detection of a host failure and host reset, detection of a host failure and SCSI reservation, and the function for controlling hot standby based on the availability of LAN communications.

In a hot-standby configuration using the multi-standby function, multiple standby servers are run in multiple standby systems. Therefore, in the event of a failure, such as a monitoring path failure, multiple standby systems might perform the hot standby operation at the same time. When the multi-standby function is used to prevent such concurrent hot standby operations, HA Monitor monitors hosts even among standby systems.

(1) Host reset

If a host failure is detected in a standby system, the host that detected the host failure resets the host on which is running a standby server with a higher priority than the local host's standby server. The following figure shows the processing flow for monitoring host status and resetting hosts when the multi-standby function is used.

Figure 4‒27: Processing flow for monitoring host status and resetting hosts (when the multi-standby function is used)

In the figure above, standby system 2 that has a standby server with priority 2 resets standby system 1 because the priority of the standby server resulting in the failure is 1.

If a reset attempt from standby system 2 fails or the reset operation fails, the actual host that will issue a reset differs. For details about the processing flow for issuing a reset in the event a reset attempt from a standby system has failed, see 4.2.3 Host reset when there are multiple standby systems.

(2) SCSI reservation

If a configuration is split into multiple configuration segments due to a monitoring path failure, hot standby processing is started for each configuration segment. The host that will continue with jobs as the active system is the host that contains the server with the highest priority within the configuration segment that contains the server with the lowest priority.

The following figure shows the flow of hot standby processing when failures occur on multiple hosts at the same time.

Figure 4‒28: Hot standby processing when failures occur on multiple hosts at the same time (when the multi-standby function is used)

In this figure, HA Monitor detects a failure on the active server and switches from host 1 in the active system to host 2, which has the highest priority. HA Monitor then switches from host 2 to host 3 when a failure is detected on the host that has the highest priority including the active server.

(3) Function for controlling hot standby based on the availability of LAN communications

If a host failure is detected on the active system or the standby system, HA Monitor checks whether LAN communication is available. HA Monitor then runs the highest-priority standby server as the active server on a host on which LAN communication is available, leaving the other standby servers on standby. In addition, it terminates the server on any host on which LAN communication is not available.

The following figure shows the general flow of operations when a host failure is detected due to a monitoring path failure, and communication is not available via a business application LAN in the active system, but communication is available via a business application LAN in the standby system.

Figure 4‒29: Hot standby operation when communication is not available via a business application LAN in the active system, but communication is available via a business application LAN in the standby system

The following figure shows the general flow of operations when a host failure is detected due to a monitoring path failure, and communication is available via a business application LAN in the active system, but communication is not available via a business application LAN in the standby system.

Figure 4‒30: Hot standby operation when communication is available via a business application LAN in the active system, but communication is not available via a business application LAN in the standby system

4.5.2 Processing in the event a host reset has failed (multi-standby)

This subsection explains the processing that occurs when a host reset fails in a hot-standby configuration using the multi-standby function.

(1) If reset of an active system has failed

If a failure occurs on an active system and reset of that active system has failed, HA Monitor places in hot-standby wait state only the standby server with the highest priority among all standby systems.

The following figure shows the processing that occurs when reset of an active system has failed in a hot-standby configuration using the multi-standby function.

Figure 4‒31: Processing that occurs when reset of an active system has failed (when using the multi-standby function)

In this example, an operator intervention is required. For details about how to start a server in the hot-standby wait state as an active server, see 7.4.1 Starting a server in the wait state and then restarting jobs.

(2) If reset of a standby system has failed

If reset of a standby system has failed, HA Monitor can no longer determine the priority in a configuration where multiple standby systems are used, in which case multiple active servers might be running. In order to prevent multiple active servers from running, HA Monitor terminates the standby servers whose priority is lower than that of the standby server that is running in the standby system resulting in the failure.

The following figure shows the processing that occurs when a reset of a standby system has failed in a hot-standby configuration using the multi-standby function.

Figure 4‒32: Processing that occurs when reset of a standby system has failed (when using the multi-standby function)

In this example, an operator intervention is required. For details about how to start a standby server, see 7.2.1 Starting.

4.5.3 Controlling the server start and termination (multi-standby)

This subsection explains when HA Monitor starts and terminates servers when the multi-standby function is used.

This subsection describes only the differences between when the multi-standby function is used and when it is not used.

(1) Server start control by HA Monitor

A difference from when the multi-standby function is not used is that there are multiple standby systems when the multi-standby function is used. Therefore, how to determine an active server or a standby server and the start type to be used for restarting a server differ from when the multi-standby function is not used.

(a) Determining the active server or standby server

When the multi-standby function is used, HA Monitor determines the start type based not only on the status of the active server in the remote host but also on the combination of active server status and standby server status. The following table describes how this determination in made when the multi-standby function is used.

Table 4‒10: Determining the active server or standby server (when using the multi-standby function)

Status of remote host

Server environment definition

Defined as active server

Defined as a standby server#1

Active server exists.

Startup processing is underway.

Start as standby server.

Start as standby server.

Execution processing is underway.

Start as standby server.

Start as standby server.

Termination processing is underway.

Cannot be started.#2

Cannot be started.#2

Hot standby operation is underway.

Wait until the active server starts.#3

Wait until the active server starts.#3

There is no active server.

Start as active server.

Wait until the active server starts.#3

Retrying startup#4

Start as active server.

Start as a standby server.

No server is found (or HA Monitor is inactive).

Start as active server.

Wait until the active server starts.#3

The server status cannot be checked.

Retry processing.#5

Wait until the active server starts.#3

#1: If the status is Retrying startup, the operation is the same as for Defined as active server.

#2: Cancels the server startup processing.

#3: Places startup of the standby server on hold until startup of the active server on the remote host is confirmed. After start of the active server startup processing is confirmed, HA Monitor starts the server as the standby server (active server start wait state).

#4: For details about the retrying startup status, see 3.2.7 Performing start retries during hot standby processing.

#5: If HA Monitor is unable to check the server status on the remote host due to a host failure or a monitoring path failure on the remote host, it retries for only one minute. If the status still cannot be checked after one minute of retries, HA Monitor determines that no server is available on the remote host due to a failure and places the server in the active server start wait state.

For details about the user operation when a server is placed in the active server start wait state, see 7.4.1 Starting a server in the wait state and then restarting jobs.

(b) Start type for restarting a server

When a server is restarted after a failure, an active server might already be running on the remote host. Therefore, HA Monitor does not start the server as an active server, even if it is defined as an active server, until the server status on the remote host can be verified. Once the server status on the remote host has been verified, determine whether the server is to be started as the active server or a standby server.

The following table describes how the start type is determined for restarting a server that has been defined as the active server when the multi-standby function is used.

Table 4‒11: Determining the start type for restarting a server (when using the multi-standby function)

Status of remote host

Server start type

Active server exists.

Startup processing is underway.

Start as standby server.

Execution processing is underway.

Start as standby server.

Termination processing is underway.

Cannot be started.#1

Hot standby switching from standby server to active server is underway.

Start as standby server.

No server is found (or HA Monitor is inactive).

Wait until the active server starts.#2

The server status cannot be checked.

Wait until the active server starts.#2

#1: Cancels the server startup processing.

#2: Places startup of the standby server on hold until startup of the active server on the remote host is confirmed (active server start wait state). After start of the active server startup processing is confirmed, HA Monitor starts the server as the standby server.

For details about the user operation when a server is placed in the active server start wait state, see 7.4.1 Starting a server in the wait state and then restarting jobs.

(2) Server termination control by HA Monitor

When the multi-standby function is used, there are multiple standby systems. Therefore, if an active server is terminated, HA Monitor terminates all the corresponding standby servers.

If the operator terminates a standby server with the standby server termination command (monsbystp command), HA Monitor terminates only the standby servers on the host on which the command was executed.

4.6 Managing a resource server

A resource server is a server used for controlling shared resources. If you do not use a resource server, you must use a separate shared resource for each server. When you use a resource server, you can share a single shared resource among multiple servers. A resource server is useful when the number of available shared resources is limited.

This section explains how HA Monitor controls a resource server. The following subsections explain the hot standby operation when a resource server is used, the processing flow for connecting and disconnecting shared resources, and how to determine a resource server's status.

4.6.1 Hot standby when a resource server is used

A resource server is used to share a single shared resource among multiple servers. Therefore, a resource server is always part of a group of servers. A resource server is subject to switchover as a member of the server group.

The following figure shows the hot standby processing when a resource server is used.

Figure 4‒33: Hot standby processing when a resource server is used

A resource server is switched over as part of the server group when a failure occurs on one of the servers in the server group that uses the resource server.

4.6.2 Processing flow for connecting and disconnecting shared resources when a resource server is used

When a resource server is used, shared resources are connected when the resource server is started and disconnected when the resource server is terminated. HA Monitor controls the order in which the servers and resource server defined as a group are started so that the servers can share the resources.

This subsection explains the order in which the servers and resource server are started, the processing flow for connecting and disconnecting shared resources, and the order in which the servers and resource server are terminated.

How the active server or standby server is determined when a resource server is started is the same as when other servers are started. For details about how to determine an active server or a standby server, see 4.1.1 HA Monitor's control of server startup.

(1) Order in which the servers and resource server are started

HA Monitor treats the servers and resource server in a group as having parent-child relationships and starts them sequentially beginning with the parent. The following figure shows the parent-child relationships and the start order using as an example a server group consisting of server 1, server 2, and a resource server.

Figure 4‒34: Parent-child relationships and start order of a server group that includes a resource server

To use this resource server, the user must specify it as the parent server of servers 1 and 2. A resource server must always be specified at the top of the hierarchy as the parent server. HA Monitor controls the start order based on the specified parent-child relationships in such a manner that the servers are started sequentially starting with the parent server.

(2) Processing flow for connecting and disconnecting shared resources

This subsection explains the processing flow for connecting and disconnecting shared resources when a resource server is used. The explanation covers the period from the time the servers are started to the time the hot standby operation is performed in the event of a failure. For details about the parent-child relationships of the server group explained here, see (1) Order in which the servers and resource server are started. For details about HA Monitor and server processing, see 4.8 Processing flows.

The following figure shows the processing flow for connecting and disconnecting shared resources when a resource server is used.

Figure 4‒35: Processing flow for connecting and disconnecting shared resources (when using a resource server)

The following describes the steps of the connection flow in the preceding figure. The numbers correspond to the numbers in the figure.

When the operator starts server 1 or 2, HA Monitor starts the resource server. At the same time, HA Monitor connects the shared resources. Until the resource server's startup processing is completed, HA Monitor keeps servers 1 and 2 in the active system in the resource server start wait state so that they cannot be started.

Once the shared resources have been connected in the active system, HA Monitor completes startup processing for servers 1 and 2.

If the operator starts server 1 or 2 in the standby system, HA Monitor starts the resource server.

If a server failure occurs, HA Monitor performs a grouped-system switchover. In this example, HA Monitor terminates servers 1 and 2 before terminating the resource server because a server failure has occurred on server 1.

When servers 1 and 2 in the active system have terminated, HA Monitor terminates the resource server. At the same time, HA Monitor disconnects the shared resources.

When the shared resources have been disconnected in the active system, the resource server in the standby system starts the hot standby operation. When the hot standby operation starts, the shared resources are connected. In this case, servers 1 and 2 in the standby system are placed in the hot-standby completion wait state.

If you use the server and host status display command (monshow command), (ONL) is displayed as the status of servers 1 and 2. (ONL) means that the active servers are engaged in startup processing, but the servers themselves are actually waiting for completion of the hot standby operation on the resource server. In other words, the status displayed by the command is not the same as the actual server status.

After connecting the shared resources, HA Monitor starts servers 1 and 2 in the standby system as active servers. If the operator restarts server 1 or 2 in the active system, the resource server and servers 1 and 2 are started as standby servers.

(3) Order in which the servers and resource server are terminated

A resource server is terminated automatically when all the servers for whom the resource server is the parent server have terminated. The resource server is not terminated automatically when a server in the same group has been terminated if its parent server is not the resource server. If a resource server is terminated in the active system, the resource server in the corresponding standby system is also terminated automatically.

4.6.3 Determining a resource server's status

This subsection explains how HA Monitor determines the status of a resource server.

A resource server is used only to control shared resources. A resource server is always grouped with other servers. Therefore, HA Monitor determines a resource server's status in the same manner as for the other servers in the same group.

A resource server's status is either running or standby. If the group contains a server running as the active server, the resource server is in running status. If the group contains a server running as a standby server, the resource server is in standby status.

For details about determining server status, see 4.1.1 HA Monitor's control of server startup.

4.7 Managing programs (monitor mode)

This section explains program management when the server operating method is monitor mode and the monitor-mode program management function is used.

For an overview of the monitor-mode program management function and details about the required environment settings, see 3.6 Function for controlling programs (monitor mode).

To use the monitor-mode program management function, you must specify a monitor-mode program environment definition and issue APIs.

4.7.1 UAP monitoring control

When you modify a UAP in such a manner that it can issue HA Monitor's APIs, you can use HA Monitor to monitor the UAP without having to create a UAP monitoring process in the server monitoring command.

The two types of APIs to be issued are for UAP monitoring start (hamon_patrolstart function) and for UAP monitoring stop (hamon_patrolstop function). For details about the APIs, see 10. APIs.

HA Monitor monitors a UAP by monitoring UAP processes and operation reports. You can choose to monitor both UAP processes and operation reports or only UAP processes. To specify the monitoring method, you use the patrol operand in the monitor-mode program environment definition and the hamon_patrolstart function. Select the monitoring method as appropriate for the types of failures you wish to have detected.

The following table lists the types of detectable failures and the available monitoring methods.

Table 4‒12: Detectable failures and available monitoring methods

Detectable failure

Monitoring processes

Monitoring operation reports

UAP abnormal termination

Y

Y

UAP slowdown

System call hang

N

Y

Page-in wait

N

Y

Scheduling wait

N

Y

Legend:

Y: Can be detected.

N: Cannot be detected.

The following subsections explain HA Monitor processing for monitoring UAP processes and operation reports.

(1) Monitoring UAP processes

HA Monitor checks for a UAP process by connecting to the UAP when the hamon_patrolstart function is issued, and then by monitoring that connection. If the connection is lost, HA Monitor detects the loss of the process and determines that a UAP failure has occurred. HA Monitor monitors a UAP connection at one-second intervals.

If a program restart command has been created when HA Monitor detects a UAP failure, HA Monitor restarts the UAP; otherwise, HA Monitor performs necessary operations including restarting the server and hot standby processing. For details about HA Monitor processing that occurs after detection of a UAP failure, see (3) HA Monitor processing after detection of a UAP failure.

The following figure shows the processing flow when HA Monitor monitors UAP processes.

Figure 4‒36: Processing flow when HA Monitor monitors UAP processes

HA Monitor starts monitoring UAP processes once it has received the hamon_patrolstart function issued from the UAP. If there is no monitoring request from the UAP (the hamon_patrolstart function is not issued), HA Monitor does not detect UAP failures. If the start_timeout operand has been specified in the server environment definition, a timeout occurs when the monitor-mode server is started.

(2) Monitoring UAP operation reports

HA Monitor can detect failures, including UAP slowdowns, that are caused by the events listed below and that cannot be detected by the UAP itself. It does this by monitoring the operation reports that are sent to HA Monitor at specific intervals by APIs issued from the UAP.

System call hangs

Page-in waits

Scheduling waits

You must specify a UAP operation report monitoring interval (program slowdown monitoring interval) in the patrol operand in the monitor-mode program environment definition. If no operation report is sent from the UAP within the specified program slowdown monitoring interval, HA Monitor determines that a UAP failure has occurred.

When HA Monitor detects a UAP failure, it terminates forcibly the process that issued the hamon_patrolstart function. If a program restart command has been created, HA Monitor restarts the UAP; otherwise, HA Monitor performs necessary operations including restarting the server and hot standby processing. For details about HA Monitor processing that occurs after detection of a UAP failure, see (3) HA Monitor processing after detection of a UAP failure.

If the UAP generates a process, you must specify in the program restart command and the server termination command the processing needed to terminate the generated process.

The following figure shows the processing flow when HA Monitor monitors UAP operation reports.

Figure 4‒37: Processing flow when HA Monitor monitors UAP operation reports

HA Monitor starts monitoring UAP operation reports once it has received the hamon_patrolstart function issued from the UAP. If there is no monitoring request from the UAP (the hamon_patrolstart function is not issued), HA Monitor does not detect UAP failures. If the start_timeout operand has been specified in the server environment definition, a timeout occurs when the monitor-mode server is started.

(3) HA Monitor processing after detection of a UAP failure

The HA Monitor processing that occurs after a UAP failure has been detected depends on the program and server definitions and on the server status.

The following figure shows the HA Monitor processing after detection of a UAP failure.

Figure 4‒38: HA Monitor processing after detection of a UAP failure

#1

Specify in the restartcommand operand in the monitor-mode program environment definition.

#2

Specify in the exec_retry operand in the monitor-mode program environment definition.

#3

Specify in the servexec_retry or switch_error operand in the server environment definition.

#4

Specify in the servexec_retry or switch_retry operand in the server environment definition.

(4) Monitoring a UAP consisting of multiple processes

If one UAP consists of multiple processes, HA Monitor can group those processes together and monitor them as a single UAP (a UAP defined in the monitor-mode program environment definition). HA Monitor monitors the individual processes that have issued the hamon_patrolstart function. If a failure occurs in one of the monitored processes, HA Monitor treats the failure as a UAP failure.

Note that the same program name must be set in the HAMON_UAPNAME environment variable for all the processes that issue the hamon_patrolstart function.

The following figure shows an example of monitoring the UAP operation reports of a UAP that consists of multiple processes.

Figure 4‒39: Example of monitoring the UAP operation reports of a UAP that consists of multiple processes

4.7.2 Program restart control

The monitor-mode program management function enables you to restart individual programs after detection of a program failure by executing a program restart command. The user must create the program restart command. The user then specifies the created program restart command in the restartcommand operand in the monitor-mode program environment definition.

For details about how to create a program restart command, see 6.14 Creating a program restart command (when the program management function is used).

(1) Unit of restart processing

You can select server or program as the unit of program restart processing. This means that if a UAP failure occurs, you can choose to restart only the UAP resulting in the UAP failure or you can restart the server that contains the UAP where the failure occurred.

The unit of restart processing is determined by whether the restartcommand operand is specified in the monitor-mode program environment definition. When this operand is specified, HA Monitor restarts the individual program; when specification of this operand is omitted, HA Monitor restarts the server.

(2) Processing flow of program restart

This subsection explains the processing flow when restart processing is performed for the program.

If a program fails, HA Monitor attempts to restart the program as many times as specified in program restart limit. The program restart limit means the maximum number of times program restart is to be attempted. You specify the program restart limit in the exec_retry operand in the monitor-mode program environment definition. You can also specify a timeout value for program restart processing. When the program restart command times out, the program restart command is re-executed. You specify a timeout value for program restart processing in the restart_timeout operand in the monitor-mode program environment definition.

In the event of a program failure, HA Monitor repeats the following processing until the program restart limit is reached:

If the program restart command is already executing, forcibly terminates the program restart command.

If the UAP being monitored consists of multiple processes, forcibly terminates all processes that issued the API.

Executes the program restart command.

(3) Handling of restart errors

If HA Monitor detects a restart limit resulting in program termination and hot standby processing can be performed, HA Monitor performs hot standby processing. If hot standby processing cannot be performed, the user must restart the program manually.

4.7.3 Program status transitions

Specifying the -u option in the server and host status display command (monshow command) enables you to check the status of programs that are using the monitor-mode program management function. For details about the server and host status display command (monshow command) and the program statuses, see 9.23 monshow (displays the status of servers and hosts) in 9. Commands.

The figure below shows the status transitions for a program that is using the monitor-mode program management function. The program shown in this figure belongs to the active server. The standby server's program is inactive.

Figure 4‒40: Program status transitions

#: The server start command's return value is not 0.

4.7.4 Processing flow of UAP monitoring

When UAPs are monitored, the processing flow during server start and termination processing and in the event of a server failure differs from the normal situation. This subsection explains the differences from the processing flow described in 4.8 Processing flows when APIs are issued from UAPs. This subsection also explains the processing flow in the event of a UAP failure.

(1) Flow of server startup processing (when UAPs are monitored)

When UAPs are monitored, the processing flow from the beginning of active server startup processing to the completion of active server startup processing differs from that described in (2) Processing flow for starting a server (in the case where the active server is monitored in the monitor mode). The following figure shows the flow of server startup processing when UAPs are monitored.

Figure 4‒41: Flow of server startup processing (monitor mode: UAPs are monitored)

(2) Flow of server termination processing (when UAPs are monitored)

When UAPs are monitored, the processing flow from the beginning of active server termination processing to the completion of active server normal termination processing differs from that described in (2) Processing flow for terminating a server (in the case where the active server is monitored in the monitor mode). The following figure shows the flow of server termination processing when UAPs are monitored.

Figure 4‒42: Flow of server termination processing (monitor mode: UAPs are monitored)

(3) Flow of hot standby processing in the event of a server failure (when UAPs are monitored)

When UAPs are monitored, the processing flow from the occurrence of a server failure to completion of active server hot standby processing differs from that described in (2) Processing flow for hot-standby switchover if a server failure occurs (in the case where the active server is monitored in the monitor mode). The following figure shows the flow of hot standby processing in the event of a server failure when UAPs are monitored.

Figure 4‒43: Flow of hot standby processing in the event of a server failure (monitor mode: UAPs are monitored)

(4) Flow of hot standby processing in the event of a UAP failure (when UAPs are monitored)

The following figure shows the flow of hot standby processing in the event of a UAP failure. In this example, a failure occurs in UAP2. As a result, the server in the primary system is terminated and operation is switched over to the server in the secondary system.

Figure 4‒44: Processing flow in the event of a UAP failure (monitor mode: UAPs are monitored)

4.8 Processing flows

In 4.3 Managing shared resources, the processing flow for connecting and switching a specific shared resource are explained. This section explains the flow of processing for HA Monitor processing for all shared resources.

The descriptions of processing flows in this section include the message IDs issued by HA Monitor and the event IDs that are issued. You can use this information to determine the timing of message ID and event issuance. The processing flow descriptions also provide the timing of user command execution and the arguments that are passed to the user commands. Use this information as a reference when you create your user commands.

The following figure shows the correspondence between the information provided as an operation flow and the information provided as a processing flow in a hot-standby configuration.

Figure 4‒45: Correspondence between the information provided as an operation flow and the information provided as a processing flow in a hot-standby configuration

4.8.1 Flow of processing for server startup processing

This subsection explains the HA Monitor processing flow when servers are started. The details of this processing flow depend on whether the servers are used in the server mode or the monitor mode.

(1) Processing flow for starting a server (in the server mode)

The following figure shows the processing that HA Monitor performs when servers are started in the server mode.

Figure 4‒46: Processing flow for starting a server (in the server mode)

(2) Processing flow for starting a server (in the case where the active server is monitored in the monitor mode)

The following figure shows the flow of processing that HA Monitor performs when starting a server in the case where the active server is monitored in the monitor mode.

Figure 4‒47: Processing flow for starting a server (in the case where the active server is monitored in the monitor mode)

Note

In the case where APIs are issued from a UAP (when the monitor-mode program management function is used), the processing flow from the beginning of active server startup processing to the completion of active server startup processing differs from the flow shown here. For details, see (1) Flow of server startup processing (when UAPs are monitored).

(3) Processing flow for starting a server (in the case where monitoring is performed in the standby system in the monitor mode)

The following figure shows the flow of processing that HA Monitor performs when starting a server in the case where monitoring is performed in the standby system in the monitor mode.

Figure 4‒48: Processing flow for starting a server (in the case where monitoring is performed in the standby system in the monitor mode)

4.8.2 Flow of server termination processing

This subsection explains the HA Monitor processing flow when servers are terminated. The details of this processing flow depend on whether the servers are used in the server mode or the monitor mode.

The processing flow explained here assumes that the shared resources are disconnected in the same order in which they were connected. For details about the HA Monitor processing that occurs when shared resources are disconnected in the reverse order from which they were connected, see 4.8.7 Processing flow for disconnecting shared resources in the reverse order from which they were connected.

Note

If you want to access shared resources by using a user command, you must disconnect the shared resources in the reverse order from which they were connected. For details about the specification method, see 3.5.5 Specifying the shared resource separation order.

(1) Processing flow for terminating a server (in the server mode)

In the server mode, there are two types of server termination processing:

Terminating standby servers when active server termination processing is completed

Terminating standby servers when active server termination processing is started

This subsection explains the flow of HA Monitor processing when servers in the server mode are terminated.

(a) Terminating standby servers when active server termination processing is completed

The following figure shows the flow of HA Monitor processing when standby servers are terminated at the time that active server termination processing is completed.

Figure 4‒49: Processing flow for server termination (server mode: terminating standby servers when active server termination processing is completed)

(b) Terminating standby servers when active server termination processing is started

The following figure shows the flow of HA Monitor processing when standby servers are terminated at the time that active server termination processing is started.

Figure 4‒50: Processing flow for server termination (server mode: terminating standby servers when active server termination processing is started)

(2) Processing flow for terminating a server (in the case where the active server is monitored in the monitor mode)

The following figure shows the flow of processing that HA Monitor performs when terminating a server in the case where the active server is monitored in the monitor mode.

Figure 4‒51: Processing flow for terminating a server (in the case where the active server is monitored in the monitor mode)

Note

In the case where APIs are issued from a UAP (when the monitor-mode program management function is used), the processing flow from the beginning of active server termination processing to the completion of active server normal termination processing differs from the flow shown here. For details, see (2) Flow of server termination processing (when UAPs are monitored).

(3) Processing flow for terminating a server (in the case where monitoring is performed in the standby system in the monitor mode)

The following figure shows the flow of processing that HA Monitor performs when terminating a server in the case where monitoring is performed in the standby system in the monitor mode.

Figure 4‒52: Processing flow for terminating a server (in the case where monitoring is performed in the standby system in the monitor mode)

4.8.3 Hot standby processing in the event of a server failure

This subsection explains the flow of HA Monitor processing in the event of a server failure. The details of the processing flow depend on whether the servers are used in the server mode or the monitor mode.

The processing flow explained here assumes that shared resources are disconnected in the same order that they were connected. For details about the HA Monitor processing that occurs when shared resources are disconnected in the reverse order from which they were connected, see 4.8.7 Processing flow for disconnecting shared resources in the reverse order from which they were connected.

Note

If you want to access shared resources by using a user command, you must disconnect them in the reverse order from which they were connected. For details about the specification method, see 3.5.5 Specifying the shared resource separation order.

(1) Processing flow for hot-standby switchover in the event of a server failure (in the server mode)

The following figure shows the flow of hot-standby switchover processing performed by HA Monitor in the event of a server failure in the server mode.

Figure 4‒53: Processing flow for hot-standby switchover in the event of a server failure (in the server mode)

(2) Processing flow for hot-standby switchover if a server failure occurs (in the case where the active server is monitored in the monitor mode)

The following figure shows the flow of hot-standby switchover processing that HA Monitor performs if a server failure occurs in the case where the active server is monitored in the monitor mode.

Figure 4‒54: Processing flow for hot-standby switchover if a server failure occurs (in the case where the active server is monitored in the monitor mode)

Note

In the case where APIs are issued from a UAP (when the monitor-mode program management function is used), the processing flow from the occurrence of a server failure to the completion of active server hot standby processing differs from the flow shown here. For details, see (3) Flow of hot standby processing in the event of a server failure (when UAPs are monitored).

(3) Processing flow for hot-standby switchover if a server failure occurs (in the case where monitoring is performed in the standby system in the monitor mode)

The following figure shows the flow of hot-standby switchover processing that HA Monitor performs if a server failure occurs in the case where monitoring is performed in the standby system in the monitor mode.

Figure 4‒55: Processing flow for hot-standby switchover if a server failure occurs (in the case where monitoring is performed in the standby system in the monitor mode)

4.8.4 Hot standby processing in the event of a host failure

This subsection explains the flow of HA Monitor processing in the event of a host failure. The details of the processing flow depend on the following conditions:

Whether servers are used in the server mode or the monitor mode

The data protection method and failure type

Host failures that result in host reset or host failures other than monitoring path failures when SCSI reservation for shared disk is used or when the function for controlling hot standby based on the availability of LAN communications is used

Monitoring path failures when SCSI reservation for shared disk is used or when the function for controlling hot standby based on the availability of LAN communications is used

(1) Hot standby processing in the event of a host failure (server mode)

(a) In the event of a host failure that results in host reset or a host failure other than monitoring path failures when SCSI reservation for shared disk is used or when the function for controlling hot standby based on the availability of LAN communications is used

The following figure shows the hot standby processing performed by HA Monitor in the server mode in the event of a host failure that results in host reset or a host failure other than monitoring path failures when SCSI reservation for shared disk is used or when the function for controlling hot standby based on the availability of LAN communications is used.

Figure 4‒56: Hot standby processing in the event of a host failure (server mode)

(b) In the event of a monitoring path failure when SCSI reservation for shared disk is used or when the function for controlling hot standby based on the availability of LAN communications is used

When SCSI reservation for shared disk is used or the function for controlling hot standby based on the availability of LAN communications is used, the processing flow for handling a monitoring path failure is different. The following figure shows the hot standby operation performed by HA Monitor in the event of a monitoring path failure.

Figure 4‒57: Hot standby processing in the event of a monitoring path failure (server mode: SCSI reservation for shared disk is used or the function for controlling hot standby based on the availability of LAN communications is used)

(2) Hot standby processing in the event of a host failure (monitor mode)

(a) In the event of a host failure that results in host reset or a host failure other than monitoring path failures when SCSI reservation for shared disk is used or when the function for controlling hot standby based on the availability of LAN communications is used

The following figure shows the hot standby processing performed by HA Monitor in the monitor mode in the event of a host failure that results in host reset or a host failure other than monitoring path failures when SCSI reservation for shared disk is used or when the function for controlling hot standby based on the availability of LAN communications is used.

Figure 4‒58: Hot standby processing in the event of a host failure (monitor mode)

(b) In the event of a monitoring path failure when SCSI reservation for shared disk is used or the function for controlling hot standby based on the availability of LAN communications is used

When SCSI reservation for shared disk is used or the function for controlling hot standby based on the availability of LAN communications is used, the processing flow for handling a monitoring path failure is different. The following figure shows the hot standby operation performed by HA Monitor in the event of a monitoring path failure.

Figure 4‒59: Hot standby processing in the event of a monitoring path failure (monitor mode: SCSI reservation for shared disk is used or the function for controlling hot standby based on the availability of LAN communications is used)

4.8.5 Processing flow for handling a hot standby error in the event of a server failure

This subsection explains the flow of processing performed by HA Monitor when a server failure has occurred and an attempt to switch over to the standby system has failed. The details of the processing flow depend on whether the servers are used in the server mode or the monitor mode.

The processing flow explained here assumes that the shared resources are disconnected in the same order that they were connected. For details about the HA Monitor processing that occurs when shared resources are disconnected in the reverse order from which they were connected, see 4.8.7 Processing flow for disconnecting shared resources in the reverse order from which they were connected.

(1) Processing flow for handling a hot standby error in the event of a server failure (server mode)

The following figure shows the flow of processing performed by HA Monitor in the event of a hot standby error in the server mode.

Figure 4‒60: Processing flow for handling a hot standby error in the event of a server failure (server mode)

(2) Processing flow for handling a hot standby error in the event of a server failure (monitor mode)

The following figure shows the flow of processing performed by HA Monitor in the event of a hot standby error in the monitor mode.

Figure 4‒61: Processing flow for handling a hot standby error in the event of a server failure (monitor mode)

4.8.6 Processing flow for handling a hot standby error in the event of a host failure

This subsection explains the flow of processing performed by HA Monitor when a host failure has occurred and an attempt to switch over to the standby system has failed. The details of the processing flow depend on whether the servers are used in the server mode or monitor mode.

The processing flow explained here assumes that the shared resources are disconnected in the same order that they were connected. For details about the HA Monitor processing that occurs when shared resources are disconnected in the reverse order from which they were connected, see 4.8.7 Processing flow for disconnecting shared resources in the reverse order from which they were connected.

(1) Processing flow for handling a hot standby error in the event of a host failure (server mode)

The following figure shows the flow of processing performed by HA Monitor in the event of a hot standby error in the server mode.

Figure 4‒62: Processing flow for handling a hot standby error in the event of a host failure (server mode)

(2) Processing flow for handling a hot standby error in the event of a host failure (monitor mode)

The following figure shows the flow of processing performed by HA Monitor in the event of a hot standby error in the monitor mode.

Figure 4‒63: Processing flow for handling a hot standby error in the event of a host failure (monitor mode)

4.8.7 Processing flow for disconnecting shared resources in the reverse order from which they were connected

When an active server is terminated, HA Monitor disconnects the shared resources. You can disconnect the shared resource in the reverse order from which they were connected. Disconnecting shared resources in reverse order enables you to execute user commands before you disconnect the shared disk and LAN, thereby implementing processing that depends on the shared disk and LAN in user-specific termination processing.

(1) Server termination processing

This subsection explains the flow of server termination processing performed by HA Monitor when you disconnect the shared resources in the reverse order from which they were connected. The details of the processing flow depend on whether the servers are used in the server mode or the monitor mode.

(a) Terminating servers (server mode)

In the server mode, there are two types of server termination processing:

Terminating standby servers when active server termination processing is completed

Terminating standby servers when active server termination processing is started

This subsection explains the flow of HA Monitor processing when servers in the server mode are terminated and shared resources are to be disconnected in the reverse order from which they were connected.

Terminating standby servers when active server termination processing is completed

The following figure shows the flow of HA Monitor processing when standby servers are terminated at the time the active server termination processing is completed and the shared resources are to be disconnected in the reverse order from which they were connected.

Figure 4‒64: Processing flow for server termination when the shared resources are to be disconnected in the reverse order from which they were connected (server mode: terminating standby servers when active server termination processing is completed)

Terminating standby servers when active server termination processing is started

The following figure shows the flow of HA Monitor processing when standby servers are terminated at the time the active server termination processing is started and the shared resources are to be disconnected in the reverse order from which they were connected.

Figure 4‒65: Processing flow for server termination when the shared resources are to be disconnected in the reverse order from which they were connected (server mode: terminating standby servers when active server termination processing is started)

(b) Terminating servers (monitor mode)

This subsection explains the flow of HA Monitor processing when servers in the monitor mode are terminated and shared resources are to be disconnected in the reverse order from which they were connected.

Figure 4‒66: Processing flow for server termination when the shared resources are to be disconnected in the reverse order from which they were connected (monitor mode)

(2) Hot standby processing in the event of a server failure

This subsection explains the flow of hot standby processing performed by HA Monitor in the event of a server failure when you disconnect the shared resources in the reverse order from which they were connected. The details of the processing flow depend on whether the servers are used in the server mode or the monitor mode.

(a) Hot standby processing in the event of a server failure (server mode)

The following figure shows the hot standby processing performed by HA Monitor in the event of a server failure in the server mode when you disconnect the shared resources in the reverse order from which they were connected.

Figure 4‒67: Hot standby processing in the event of a server failure when the shared resources are to be disconnected in the reverse order from which they were connected (server mode)

(b) Hot standby processing in the event of a server failure (monitor mode)

The following figure shows the hot standby processing performed by HA Monitor in the event of a server failure in the monitor mode when you disconnect the shared resources in the reverse order from which they were connected.

Figure 4‒68: Hot standby processing in the event of a server failure when the shared resources are to be disconnected in the reverse order from which they were connected (monitor mode)

(3) Hot standby error processing in the event of a server failure

This subsection explains the flow of processing performed by HA Monitor when a server failure has occurred and an attempt to switch over to the standby system has failed when the shared resources are to be disconnected in the reverse order from which they were connected. The details of the processing flow depend on whether the servers are used in the server mode or the monitor mode.

(a) Hot standby error in the event of a server failure (server mode)

The following figure shows the flow of processing performed by HA Monitor when a server failure has occurred in the server mode and an attempt to switch over to the standby system has failed when the shared resources are to be disconnected in the reverse order from which they were connected.

Figure 4‒69: Processing flow for handling a hot standby error in the event of a server failure when the shared resources are to be disconnected in the reverse order from which they were connected (server mode)

(b) Hot standby error in the event of a server failure (monitor mode)

The following figure shows the flow of processing performed by HA Monitor when a server failure has occurred in the monitor mode and an attempt to switch over to the standby system has failed when the shared resources are to be disconnected in the reverse order from which they were connected.

Figure 4‒70: Processing flow for handling a hot standby error in the event of a server failure when the shared resources are to be disconnected in the reverse order from which they were connected (monitor mode)

(4) Hot standby error processing in the event of a host failure

This subsection explains the flow of processing performed by HA Monitor when a host failure has occurred and an attempt to switch over to the standby system has failed when the shared resources are to be disconnected in the reverse order from which they were connected. The details of the processing flow depend on whether the servers are used in the server mode or the monitor mode.

(a) Hot standby error in the event of a host failure (server mode)

The following figure shows the flow of processing performed by HA Monitor when a host failure has occurred and an attempt to switch over to the standby system has failed when the shared resources are to be disconnected in the reverse order from which they were connected. This processing flow is for the server mode.

Figure 4‒71: Processing flow for handling a hot standby error in the event of a host failure when the shared resources are to be disconnected in the reverse order from which they were connected (server mode)

(b) Hot standby error in the event of a host failure (monitor mode)

The following figure shows the flow of processing performed by HA Monitor when a host failure has occurred and an attempt to switch over to the standby system has failed when the shared resources are to be disconnected in the reverse order from which they were connected. This processing flow is for the monitor mode.

Figure 4‒72: Processing flow for handling a hot standby error in the event of a host failure when the shared resources are to be disconnected in the reverse order from which they were connected (monitor mode)

4.8.8 Processing flow for start retries in the event of a failure during hot standby processing

This subsection explains the processing flow for using the start retry function when a server or host failure has occurred and an attempt to switch over to the standby system has failed. The details of the processing flow depend on whether the servers are used in the server mode or the monitor mode.

(1) Processing flow for start retries in the event of a server failure (server mode)

The following figure shows the processing flow for start retries when a hot standby error has occurred in the server mode.

Figure 4‒73: Processing flow for start retries in the event of a server failure (server mode)

(2) Processing flow for start retries in the event of a server failure (monitor mode)

The following figure shows the processing flow for start retries when a hot standby error has occurred in the monitor mode.

Figure 4‒74: Processing flow for start retries in the event of a server failure (monitor mode)

(3) Processing flow for start retries in the event of a host failure (server mode)

The following figure shows the processing flow for start retries when a hot standby error has occurred in the server mode.

Figure 4‒75: Processing flow for start retries in the event of a host failure (server mode)

(4) Processing flow for start retries in the event of a host failure (monitor mode)

The following figure shows the processing flow for start retries when a hot standby error has occurred in the monitor mode.

Figure 4‒76: Processing flow for start retries in the event of a host failure (monitor mode)

(5) Processing flow for start retries in the event of a failure on the active server after hot standby processing (monitor mode)

The figure below shows the flow of HA Monitor processing in the event of a failure on the active server after hot standby processing. This example explains hot standby processing in the event of a server failure.

Figure 4‒77: Processing flow in the event of a failure on the active server after hot standby processing (monitor mode)

(6) Processing flow when the server cannot be started by start retries

The figure below shows the processing flow when hot standby processing has failed and start retries were attempted, but the server cannot be started within the specified number of start retries.

This example shows hot standby processing for a server failure in the monitor mode. The start retry count is one.

Figure 4‒78: Processing flow when the server cannot be started by start retries

4.8.9 Processing flow in the case where hybrid fencing is used

This subsection describes the processing flow in the case where hybrid fencing is used. The processing flow differs depending on whether SCSI reservation for shared disk succeeds and whether the server is operating in the server mode or monitor mode.

(1) In the case where SCSI reservation for shared disk succeeds with hybrid fencing (in the server mode)

The following figure shows the processing flow in the case where SCSI reservation for shared disk succeeds with hybrid fencing (in the server mode).

Figure 4‒79: Processing flow in the case where SCSI reservation for shared disk succeeds with hybrid fencing (in the server mode)

(2) In the case where SCSI reservation for shared disk succeeds with hybrid fencing (in the monitor mode)

The following figure shows the processing flow in the case where SCSI reservation for shared disk succeeds with hybrid fencing (in the monitor mode).

Figure 4‒80: Processing flow in the case where SCSI reservation for shared disk succeeds with hybrid fencing (in the monitor mode)

(3) In the case where SCSI reservation for shared disk fails with hybrid fencing (in the server mode)

The following figure shows the processing flow in the case where SCSI reservation for shared disk fails with hybrid fencing (in the server mode).

Figure 4‒81: Processing flow in the case where SCSI reservation for shared disk fails with hybrid fencing (in the server mode)

(4) In the case where SCSI reservation for shared disk fails with hybrid fencing (in the monitor mode)

The following figure shows the processing flow in the case where SCSI reservation for shared disk fails with hybrid fencing (in the monitor mode).

Figure 4‒82: Processing flow in the case where SCSI reservation for shared disk fails with hybrid fencing (in the monitor mode)

Part 3: Installation and Operation

5 Installation of HA Monitor and System Design

This chapter describes the considerations to be evaluated before you install HA Monitor in a job processing system.

This manual assumes that you have finished designing a job processing system. It discusses only the considerations required in order to design a hot-standby configuration by using HA Monitor.

5.1 Installation and design procedures

This section describes the decisions that you must make according to the jobs your system will process before you evaluate the considerations on the installation of HA Monitor.

Determine the following beforehand:

Jobs for which a hot-standby configuration is to be applied

Your hot-standby configuration will be determined by the number of jobs that are to be handled in the configuration.

Level of reliability and performance needed

Programs used for the jobs

The monitoring method used by HA Monitor depends on the types of programs.

Type of resources needed by your programs

The following describes the considerations and the decisions to be made based on this information before you install HA Monitor.

Evaluation of hot-standby configuration

You must determine the items listed below based on such information as the number of jobs and the level of reliability and performance required. For details, see 5.2 Evaluation of hot-standby configuration.

Hot-standby configuration

Number of machines

Number of hosts

Whether LPARs are used

Whether virtual machines are used

Evaluation of server configuration

You must determine the items listed below based on the programs used for the jobs. For details, see 5.3 Evaluation of server configuration.

Unit of hot standby operation (server)

Server operation method (server mode or monitor mode)

Placement of servers

Number of servers

Number of programs (when using the program management function)

Evaluation of resources configuration

You must determine the items listed below based on the resources used by the programs. For details, see 5.4 Evaluation of resources configuration.

Required resources

Number of resources

How to share resources

Evaluation of hardware configuration

After determining the servers and resources, you must determine a hardware configuration. For details, see 5.5 Evaluation of hardware configuration.

5.2 Evaluation of hot-standby configuration

Installation of HA Monitor enables you to choose from among several hot-standby configurations. This section explains the differences in the hot-standby configurations and the considerations to be evaluated in selecting and designing a configuration.

Evaluation of a hot-standby configuration involves determining the following:

Hot-standby configuration to be used

Number of machines

Number of hosts (when using HA Monitor Extension)

Whether LPARs are used

Whether virtual machines are used

5.2.1 Differences in hot-standby configurations

This subsection explains the differences in the hot-standby configurations.

You must be familiar with the characteristics of each hot-standby configuration and choose the one that is best-suited for your job processing system, taking into account the cost factors and the processing performance after hot standby processing. The following table describes the types and characteristics of the hot-standby configurations.

Table 5‒1: Types and characteristics of the hot-standby configurations

No.

Type of hot-standby configuration

Characteristics

1

1-to-1 switchover configuration

Provides a dedicated secondary system.

Cost is high, but processing performance remains at roughly the same level after hot standby switching.

2

Mutual hot-standby configuration

In this hot-standby configuration, operations are performed by multiple active servers running on two different hosts, with corresponding standby servers placed on the hosts. It can use resources effectively, but after hot standby switching, there is increased processing load since both operations are performed on a single host.

3

2-to-1 switchover configuration

Because one secondary system is provided for two primary systems, this configuration can use resources more effectively than when you use multiple 1-to-1 switchover configurations.

If only one system fails, there is no degradation in processing performance after hot standby switching, but if both of the primary systems fail, there is a processing load because operations are concentrated on a single host.

4

Multi-standby configuration

Because multiple secondary systems are provided for one primary system, the cost is high, but processing performance remains at roughly the same level after hot standby switching. In addition, even if a secondary system fails after hot standby switching, you can switch to another secondary system and continue operations.

5

Cluster hot-standby configuration

This is a hot-standby configuration for distributing the processing load. It represents a combination of configurations 1 through 4 above.

By having multiple hosts serve as active and standby systems, a single job can run in parallel on multiple servers on different hosts, distributing the processing load.

For examples of these hot-standby configurations, see 2.1 Hot-standby configurations.

5.2.2 Considerations for designing a configuration

When you design a hot-standby configuration, you must ensure the reliability of the monitoring path and reset path.

The reliability of the monitoring and reset paths is important for detecting host failures. If all of the monitoring and reset paths fail at the same time, HA Monitor will not be able to detect the failure on the reset path using a health check, and instances of the same server might end up running on both the active system and the standby system. Therefore, to ensure the reliability of the monitoring path and reset path, do the following:

Always use multiple monitoring paths, and dedicate at least one of these to monitoring only.

Always perform health checks on the monitoring and reset paths.

In the event of failure on a monitoring or reset path, promptly eliminate the cause of the failure.

5.2.3 Considerations for changing the maximum number of hosts (when using HA Monitor Extension)

When you use HA Monitor Extension, you can change the maximum number of hosts per hot-standby configuration to more than 32. If you will be setting the maximum number of hosts to more than 32, you must provide a machine environment that satisfies the following conditions:

Gigabit Ethernet is used for monitoring paths.

LANs that support multicasting are used for monitoring paths.

Monitoring path throughput is at least 100 megabits/second.

All packets can be transmitted completely via monitoring paths when hot standby processing can be performed on servers on all hosts within the hot-standby configuration.

For details about how to check whether all packets are transmitted completely via monitoring paths, see 6.24 Checking the operation when the maximum number of hosts is changed (when using HA Monitor Extension).

5.2.4 Considerations for using a hot-standby configuration in a virtualization environment (when using Hitachi server virtualization (Virtage))

In BladeSymphony with Hitachi server virtualization installed, you can divide the processor into logical partitions (LPARs) and perform hot standby processing for each LPAR (LPAR mode). For details about hot standby processing in units of LPARs when host reset is performed, see 3.3.6 Physical partition reset function in a virtualization environment. For details about the setup procedure for running LPARs using Hitachi server virtualization, see the BladeSymphony documentation.

This subsection discusses considerations on using LPARs as the hot standby units.

(1) Evaluating the LPARs and processor to be used in a hot-standby configuration

Between the LPAR and Basic modes, there is no difference in HA Monitor operations other than whether the reset target is an LPAR or a processor. You can perform hot standby processing either between LPARS or between an LPAR and a processor.

There is no problem when the hosts constituting a hot-standby configuration are in the same chassis. However, in terms of reliability, we recommend that you use separate processors for the hosts that constitute a hot-standby configuration.

(2) Handling hardware limitations

In BladeSymphony, there is a limit to the number of peripheral devices that can be added to a single chassis. If you increase the number of hosts per chassis in the LPAR mode, the normal hardware configuration might no longer be achieved.

For details about the hardware limitations, see the hardware documentation.

5.2.5 Considerations for using a hot-standby configuration in a virtualization environment (when using VMware ESXi)

If you use VMware ESXi-based virtualization, you can divide a single physical partition into multiple logical partitions (virtual machines) and perform hot standby processing for each virtual machine. For details about hot standby processing in units of LPARs when host reset is performed, see 3.3.6 Physical partition reset function in a virtualization environment. For details about the setup procedure for running virtual machines using VMware ESXi, see the VMware ESXi documentation.

This subsection discusses considerations on using virtual machines as the hot standby units.

Important

If you reset hosts in a VMware ESXi-based virtualization environment, do not suspend the virtual machines while HA Monitor is running. For details, see Release Notes.

(1) Evaluating the virtual machines to be used in a hot-standby configuration

Between a VMware ESXi-based virtualization environment and the normal environment, there is no difference in HA Monitor operations other than whether the reset target is a virtual machine or a processor. If host reset is performed, you can perform hot standby processing between virtual machines. If SCSI reservation or the function for controlling hot standby based on the availability of LAN communications is used, you can perform hot standby processing between virtual machines or between a virtual machine and a processor.

However, in terms of reliability, we recommend that you use separate processors for the hosts that constitute a hot-standby configuration.

(2) Checking the version of HA Monitor

If host reset is performed in a hot-standby configuration that includes a VMware ESXi-based virtualization environment, the versions of all HA Monitors in the hot-standby configuration must be 01-50 or later. If the configuration contains an HA Monitor whose version is earlier than 01-50, it will not be possible to reset hosts in the virtualization environment.

If you use the same monitoring paths as in the existing system, use HA Monitor version 01-50 or later in the existing system, or set the monitoring paths so that the HA Monitor in the existing system is not connected to a host in the hot-standby switchover configuration.

(3) Checking the machine environment

If you use VMware ESXi-based virtualization, make sure that the machine environment satisfies the following conditions:

The /usr/bin/ssh command has been installed (when host reset is performed).

To reset a host on a virtual machine, HA Monitor uses the /usr/bin/ssh command to connect to VMware ESXi. If this command has not been installed, install the openssh-clients package provided with Linux.

SSH connection can be established with VMware ESXi (when host reset is performed).

The root user must be able to connect to VMware ESXi by using SSH.

We also recommend that at least one physical CPU core is allocated for each virtual machine running on a physical partition.

If this condition is not satisfied, HA Monitor might become unavailable temporarily because no core is allocated to the virtual machines during certain periods of time. This might result in a delay of host failure detection and alive message notification.

(4) Checking the functions to be used

If host reset is performed in a VMware ESXi-based virtualization environment, the hot standby processing time cannot be reduced by using a mutual hot-standby configuration (system cannot be specified in the cpudown operand in the HA Monitor environment settings).

If all of the following conditions are met, before starting HA Monitor, start the virtual machine on which HA Monitor Connector operates.

The server model is HA8000xN or later.

A management server is not used as a failure management processor.

HA Monitor and HA Monitor Connector operate on different virtual machines.

5.3 Evaluation of server configuration

After you have evaluated the hot-standby configuration, you must evaluate the server configuration. Because the server configuration determines the number of resources required, you must determine the server configuration before evaluating the resources configuration.

The server is the unit of hot standby processing. You can allocate one program to one server or multiple programs to one server.

The evaluation of your server configuration involves determining the following:

Unit of hot standby processing (server)

Server operating method (server mode or monitor mode)

Placement of servers

Number of servers

Number of programs (when using the program management function)

5.3.1 Determining the unit of hot standby processing (server)

HA Monitor performs hot standby processing in units of servers. This subsection explains servers.

A server is a program that performs a job. Whether a single program or a group of multiple programs is to be handled as one server depends on the jobs performed in the hot-standby configuration.

(1) Relationship between servers and programs

Two methods are available. In one, a single program corresponds to a single server. In the other, a group of multiple programs correspond to a single server.

If you run servers in the server mode, you must use the one program per server method.

If you run servers in the monitor mode, you can choose either method. If a program that performs a job uses a shared resource, that shared resource is connected and disconnected for each server. Therefore, you must provide shared resources for each server. Furthermore, if you run servers in the monitor mode, you must create a server start command for each server. Consider these points before you choose the relationship between servers and programs.

(2) Number of servers

HA Monitor allows a maximum of 128 servers to be placed on a single host. The default setting is 16. If you place more than 16 servers, you must specify a maximum number of servers with the servmax operand in the HA Monitor environment settings.

If you use HA Monitor Extension, you can change the maximum number of servers. For details, see 3.7.1 Changing the maximum number of servers.

(3) Number of programs (when using the program management function)

If you run servers in the monitor mode and you use the program management function, you can specify the number of programs that will use the program management function and that can run concurrently on a single host. HA Monitor supports a maximum of 256 programs that use the program management function and that can run concurrently on a single host.

Specify the maximum number of programs in the pgmmax operand in the HA Monitor environment settings. Make sure that you specify the pgmmax operand value because the default value for this operand is 0.

For details about the program management function, see 3.6 Function for controlling programs (monitor mode).

(4) Considerations when the server operating methods (server and monitor modes) differ

When you place a server on a host, there is no need to determine whether the server is to be run in the server mode or monitor mode. You can place servers in the server mode intermixed with servers in the monitor mode on the same host. You can also group multiple servers together.

5.3.2 Considerations on using multiple servers

You must consider the points discussed in this subsection when you use multiple servers in a hot-standby configuration.

(1) Whether multiple servers are used to perform a single job

If multiple servers are used in tandem to perform a single job, evaluate the use of the grouped-system switchover function that groups multiple servers together and performs hot standby processing.

When multiple servers are grouped together, they are treated as a group of servers. If a failure occurs on one of the servers in the group, HA Monitor performs hot standby processing for the entire group. You can avoid unneeded hot standby processing because you can base whether hot standby processing is to be performed on the importance of each server.

For details about grouping servers together and the required environment settings, see 3.2.3 Grouped-system switchover achieved by grouping servers.

(2) Whether it is necessary to determine the server start order

If you need to determine the start order of multiple servers, evaluate the use of the server switchover order control function.

You can group multiple servers and determine the order in which the servers in the group are to be started and terminated during hot standby processing in the event of a failure. For example, if a group of servers to be placed includes a required server, you can specify settings in such a manner that the required server is started first and then the remaining servers are started.

The server switchover order control function does not apply to the order in which servers are started during server startup. Therefore, during server startup, the operator must manually start the servers in the predefined start order. If you want to automatically control the start order during server startup, evaluate running servers in the monitor mode and treating multiple programs as a single server.

For details about server switchover order control and the required environment settings, see 3.2.4 Controlling the server switchover order.

(3) Whether a specific host has too much load after hot standby processing

In a configuration in which a secondary system is shared, such as the 2-to-1 switchover configuration, a large amount of job processing might be concentrated on a single host after hot standby processing. HA Monitor provides a lock function that terminates the standby servers in a secondary system whenever the active server in the secondary system is started. If you want to improve system performance by preventing a heavy load from being placed on a specific host, consider using this lock control function. This function enables you to prevent a concentration of load because it can limit the servers to be run on a single host.

We recommend that you use this lock function in the n-to-1 switchover configuration (n: 2 or greater) in which loads tend to be concentrated in the secondary system. For details about the lock function, see 3.2.6 Locking servers to prevent concentration of the processing load after the hot standby operation.

5.4 Evaluation of resources configuration

Evaluation of the resources configuration is affected by the hot-standby configuration, number of servers, and types of shared resources required for servers. Before you evaluate the resources configuration, you must evaluate the hot-standby configuration and server configuration.

In the evaluation of the resources configuration, you determine the following:

Required resources

Number of resources

Resource sharing method

5.4.1 Required resources and number of resources

This subsection discusses the types and number of resources that are required in a hot-standby configuration. The required resources include those required for HA Monitor and those required for servers. After you have determined the types of resources required, you must determine the number of resources of each type that you will require, based on the number of hosts and servers.

(1) Resources required for HA Monitor

HA Monitor requires the following resources:

Monitoring paths

Reset paths

(2) Resources required for servers

You must determine the resources required for servers according to the nature of your jobs. HA Monitor automatically performs the hot standby operation on the following types of resources:

Shared disks

File systems

LANs

If you use other types of shared resources on servers, you must create appropriate shared resource connection and disconnection processing as user commands beforehand.

Important

Check the specifications of the server that is subject to hot standby processing for the information required with respect to deploying shared disks or file systems.

(3) Maximum and minimum resource configurations

The table below describes the maximum and minimum hardware and software configurations that are supported by HA Monitor.

Note that this table provides only logical values. When you configure an actual system, you must determine a configuration that is appropriate to the actual processor performance and operating environment. The maximum configuration provided in this table is based on HA Monitor's logical values. Therefore, these values might be restricted by the OS and the system configuration that are actually being used. For the actual values, check the appropriate product documentation.

Table 5‒2: Maximum and minimum resource configurations supported by HA Monitor

Classification

Item

Maximum configuration

Minimum configuration

Hardware

Number of hosts per hot-standby configuration

32#1

2

Number of monitoring paths per host

6

2

Number of reset paths per host#6

1#7

1#7

Number of volume groups on a shared disk per server

3,000

0

Number of file systems on a shared disk per server

3,000

0

Number of shared disk physical devices per server#2

3,000

0

Software

Number of HA Monitors per host

1

1

Number of servers per host (not including resource servers)

128#1, #3

1

Number of programs per server that use the program management function#4

256

1

Number of alias IP addresses per host

--

0

Number of resource servers per host

64#5

0

Legend: --: There is no limitation.

#1: The value differs when HA Monitor Extension is used. For details, see (1) Maximum number of hosts that can be specified.

#2: Applicable only when SCSI reservation for shared disk is used

#3: This value applies if the maximum number of servers that can operate is set. In other cases, the value is 16.

#4: Applicable only when the monitor-mode program management function is used.

#5: This value applies if the maximum number of servers that can operate is set. In other cases, the value is 8.

#6: Reset paths are not necessary when SCSI reservation or the function for controlling hot standby based on the availability of LAN communications is used.

#7: The value is 0 when you are using SCSI reservation or the function for controlling hot standby based on the availability of LAN communication.

5.4.2 Necessary IP addresses and port numbers

In order to employ a hot-standby configuration, you must acquire IP addresses. This subsection explains the targets to which IP addresses are to be assigned and the number of IP addresses. The following figure shows the targets to which IP addresses are to be assigned.

Figure 5‒1: Targets to which IP addresses are to be assigned (for BladeSymphony, or model HA8000xM or earlier)

Figure 5‒2: Targets to which IP addresses are to be assigned (for model HA8000xN or later in the case where a management server is used as a failure management processor)

Figure 5‒3: Targets to which IP addresses are to be assigned (for model HA8000xN or later in the case where a management server is not used as a failure management processor)

The following table describes the targets to which IP addresses are to be assigned, as shown in the figure, and the number of IP addresses to be specified.

Table 5‒3: IP-address and port-number assignment targets and the number of IP addresses to be assigned

No. in figure

Target

Number of IP addresses

Unit

Description

Location of setting

1

Monitoring path

6

Per host

A monitoring path must be multiplexed. A maximum of six monitoring paths can be used.

A unique IP address and port number must be assigned to each monitoring path.

For details about the IP addresses, see (3) IP addresses to be assigned to multiple monitoring paths.

For a port number, specify 5001 or higher that is not used by the system. For a monitoring path that pairs the active and standby systems, specify the same port number.

For details about the IP addresses and host names when HA Monitor Extension is used to change the maximum number of hosts, also see 3.7.2 Changing the maximum number of hosts.

lan operand (host name) in the HA Monitor environment settings

lanport operand (service name) in the HA Monitor environment settings

netmask operand (netmask) in the HA Monitor environment settings

multicast_lan operand (host name) in the HA Monitor environment settings

/etc/hosts file (IP address, host name)

/etc/services file (port number, service name)

HA Monitor connection configuration file

2

Reset path#1

1

Per host

You must have a dedicated IP address and port number.

For the port number, specify the same number on each host in the range from 5001 to 65535 that is not in use.

If the system contains multiple hot-standby configurations that do not monitor each other and share the same reset path, you must specify settings carefully. For details, see (2) Notes about sharing a reset path among hot-standby configurations.

Reset path settings

(using the HA Monitor environment setup command (monsetup command))

3

Failure management processor#1

1

Per host

You need a dedicated IP address and port number.

For the IP address, specify the one having the same network address as for the reset path.

Specify the same IP address that is used for the reset path in a configuration that satisfies the following conditions: 1) the server model is HA8000xN or later, 2) a management server is not used as a failure management processor, and 3) HA Monitor and HA Monitor Option exist in the same OS.

For the port number, specify the same number on each host in the range from 5001 to 65535 that is not in use. Make sure that you specify a port number that is not used for the reset path.

If the system contains multiple hot-standby configurations that do not monitor each other and share the same reset path, you must specify settings carefully. For details, see (2) Notes about sharing a reset path among hot-standby configurations.

Reset path settings

(using the HA Monitor environment setup command (monsetup command))

4

LAN

0 or more

Per host

If servers use a LAN, at least one IP address is required. There is no limit. If no LAN is used, no IP address is needed.

OS settings

For details about how to specify OS settings, see the OS documentation.

5

Server

0 or more

Per server

If servers use a LAN, at least one IP address is required.

Specify an alias IP address. This value must be the same in both active and standby systems. For details about alias IP addresses, see (2) IP addresses to be assigned to the servers.

LAN status settings file

6

VMware ESXi#1, #2

1

Processor or housing

You need an IP address that can be used by HA Monitor and a port for SSH connection.

Set the IP address to be used by HA Monitor statically without using DHCP.

For the port number for SSH connection, 22 is assigned by default, but you can assign a different port number. If you assign a different port number, specify the port number for SSH connection in VMware ESXi for HA Monitor.

Reset path settings

(using the HA Monitor environment setup command (monsetup command))

#1: This setting is required in order to reset hosts. It cannot be specified when SCSI reservation for shared disk is used or the function for controlling hot standby based on the availability of LAN communications is used.

#2: This setting is required in order to reset hosts, and it is also required in a VMware ESXi-based virtualization configuration.

5.4.3 Approach to evaluating the resource sharing method

This subsection describe an approach to evaluating how resources are to be shared.

You must provide each shared resource for each server. However, if you use a resource server, you can share one shared resource among multiple servers. For a LAN, an alternative is to specify multiple IP addresses for a LAN adapter.

If you use multiple shared resources, you can localize the effects of shared resource failures. However, the time required for hot standby processing increases as the number of resources increases. Evaluate the resource configuration taking into account the pros and cons.

The following table shows the units of the shared resources that must be provided for each server.

Table 5‒4: Units of shared resources to be provided

Type of shared resource

Unit of shared resource required for each server

Shared disk

If an LVM is used for connecting a shared disk:

Volume group

File system

Logical volume

LAN

IP address

5.4.4 Configurations using resource servers

This subsection explains a configuration in which shared resources are shared among multiple servers by using resource servers.

(1) Number of resource servers

You can specify only one resource server for a group. The maximum number of resource servers that can be placed on a single host depends on the maximum number of servers that are run on a single host, which is specified in the servmax operand in the HA Monitor environment settings.

The following shows the maximum number of resource servers that can be placed for each maximum number of servers that is specified in the servmax operand.

When the maximum number of servers is 16: 8

When the maximum number of servers is 64: 32

When the maximum number of servers is 128 or greater: 64

(2) Concept of a resource server configuration

Resources include shared disks, file systems, and LANs. You can share one resource among multiple servers and dedicate the other resources to individual servers, or you can share multiple resources via a single resource server.

(a) Possible configurations

You can use a resource server to achieve the following configurations:

Use the resource server to share multiple shared resources

Use the resource server to share one of the shared resources and allocate the remaining shared resources to individual servers

The figure below shows a configuration in which multiple shared resources are shared by using a single resource server. In this example, servers A and B share the shared disk and alias IP address.

Figure 5‒4: Configuration in which multiple shared resources are shared by using a single resource server

The figure below shows a configuration in which one of the shared resources is shared by using a resource server and the remaining shared resources are allocated to individual servers. In this example, servers A and B share the shared disk and use separate alias IP addresses.

Figure 5‒5: Configuration in which one of the shared resources is shared by using a resource server and the remaining shared resources are allocated to individual servers

(b) Impossible configuration

A single shared resource cannot be shared among multiple servers or resource servers. All servers that share a shared resource must belong to the same group as the resource server.

For example, in the configuration shown below, a single shared disk is used by a resource server and by server C, which does not belong to the server group. This configuration cannot use the resource server to share the shared resource.

Figure 5‒6: Configuration that cannot use a resource server to share a shared resource

5.5 Evaluation of hardware configuration

This section explains the hardware configuration required for HA Monitor.

It explains only the considerations required when you design a hot-standby configuration using HA Monitor. For the settings that are not discussed in this manual, see your hardware documentation.

5.5.1 Shared disk configuration

A shared disk is controlled for each volume group.

To control a shared disk for each volume group, you must connect the shared disk from all the hosts that use the shared disk and control it by using LVM. Place the data that is used by each server in a separate group and provide an independent volume group to each server. If you use a resource server, you can share a single shared disk among multiple servers.

Preventing a single point of failure

HA Monitor does not directly handle disk failures or failures on I/O paths such as disk adapters. Use both of the following methods to create a configuration that prevents a shared disk from becoming a single point of failure:

Data redundancy using RAID

Achieve data redundancy by using RAID devices, such as Hitachi disk array system.

I/O path redundancy

Achieve I/O path redundancy by using disk path switching software, such as HDLM.

If you use SCSI reservation for shared disk, you need a shared disk. LVM mirroring and Linux standard software RAID (md) are not supported.

Note

In the event that SCSI reservation for shared disk is used, HA Monitor supports HDLM, DMMP, EMPD, and HFC-PCM as multipath software. For details about setting up multipath software, see 6.4.1 Setting up multipath software.

5.5.2 LAN configuration

A LAN is used to connect active and standby systems, as well as to connect workstations and PCs that are used as clients. Active and standby systems are connected by using a LAN adapter on the same LAN.

(1) Preventing a single point of failure

HA Monitor does not directly handle LAN failures, such as failures in HUB and LAN adapters. To handle these failures, the following methods can be employed to create a configuration that prevents a LAN from becoming a single point of failure:

HUB redundancy

LAN adapter redundancy using a kernel extension package in Linux, such as a bonding package

(2) IP addresses to be assigned to the servers

You specify alias IP addresses for servers. You can specify multiple alias IP addresses for the same server. Note the following about specifying alias IP addresses for servers:

The source address of a program that is subject to the hot standby operation must be an alias IP address.

A host name that has an alias IP address must not be used as a local host name that is set by using the hostname(1) command. If the host is switched over during the hot standby operation, a program that obtains IP addresses from host names can no longer function on the source host.

(3) IP address configuration

HA Monitor switches LANs for each alias IP address assigned to a server. Therefore, if a hot-standby configuration consists of multiple servers, you must configure network addresses in such a manner that each server uses a different alias IP address. If you use a resource server, you can share a single alias IP address among multiple servers.

(4) VMware ESXi-based virtualization configuration

If you use a VMware ESXi-based virtualization configuration, a LAN adapter and a host bus adapter (HBA) can be shared among or used exclusively by the virtual machines.

To ensure reliability, we recommend that you use a redundant configuration as described below by using functions provided in VMware ESXi.

Provide two LAN adapters each for redundancy when configuring the reset path or the VMware ESXi-connection reset path.

Provide two host bus adapters for redundancy.

5.5.3 Monitoring path configuration

Monitoring paths are used to mutually monitor hosts and to exchange information needed for the hot standby operation between active and standby systems.

The conditions for monitoring paths differ when you use HA Monitor Extension to set the maximum number of hosts per hot-standby configuration to 33 or more. For details, see 5.2.3 Considerations for changing the maximum number of hosts (when using HA Monitor Extension).

(1) How to connect monitoring paths

You must provide a dedicated TCP/IP LAN for a monitoring path. A TCP/IP LAN can be used in any hot-standby configuration, but it requires a dedicated IP address and port number.

A monitoring path connects two processors on the hosts (in the active and standby systems) that monitor each other. Configure the active and standby systems and the monitoring path that connects them in the same network. If one active system has multiple standby systems, connect the active system and all the standby systems with a monitoring path and configure them in the same network.

(2) Using multiple monitoring paths

If there is only one monitoring path and a failure occurs on the monitoring path, HA Monitor might treat the failure as a host failure and perform the hot standby operation. To prevent hot-standby switchover from occurring at a single point of failure, always multiplex the monitoring path. Note that at least one of the multiple monitoring paths must be used for only monitoring purposes. If there is only one monitoring path, HA Monitor might not be able to determine whether the hot standby operation can be continued.

If the system being used is BladeSymphony, the number of supported LAN adapters is limited. Therefore, before you provide multiple monitoring paths, you must consider sharing a monitoring path with a business application LAN or with the reset path.

If you use SCSI reservation for shared disk, we recommend that you share a monitoring path with a business application LAN. If multiple failures occur on multiple monitoring paths (disabling communication), and the hot standby operation is performed, alias IP addresses might remain assigned to both hosts before the source HA Monitor detects a loss of reservation. If you share a monitoring path with a business application LAN, the source host uses its alias IP address for communication less frequently.

You can use a maximum of six monitoring paths. If you provide multiple monitoring paths and a failure occurs on one of them, HA Monitor can continue monitoring hosts by using another path. Also, you can choose either of the following operation modes:

Reducing the amount of data load per monitoring path

The amount of data load per monitoring path can be reduced because HA Monitor uses a different monitoring path for each instance of communication between hosts. If you do not specify a preferred monitoring path, this operation mode is used.

Specifying a preferred monitoring path

If you specify a preferred monitoring path in the lan operand in the HA Monitor environment settings, HA Monitor uses the specified monitoring path first for communication between hosts. If a failure occurs on that monitoring path, HA Monitor uses another monitoring path to continue communication.

Note that if you specify an alive message transmission interval in the alive_interval operand in the HA Monitor environment settings, HA Monitor sends alive messages to all monitoring paths. For this reason, a preferred monitoring path cannot be specified.

If you use redundant LAN adapters by using a kernel extension package in Linux, such as a bonding package and VMware ESXi functions, you can prevent the hot standby operation resulting from a single point of failure without having to provide multiple monitoring paths. In such a case, you must specify a value that is larger than the LAN adapter switching time in the patrol operand in the HA Monitor environment settings because it takes time to switch LAN adapters.

(3) IP addresses to be assigned to multiple monitoring paths

A monitoring path requires a dedicated IP address and port number. In order to provide multiple monitoring paths, you must note the following:

Use a different LAN segment for each monitoring path.

Use a different network address for each monitoring path.

A network address consists of a network number and a subnet number.

When assigning a different network number to each monitoring path

There is no limitation to the supported IP addresses. The following table shows examples of IP address combinations.

Table 5‒5: Examples of IP address combinations (when assigning a different network number to each monitoring path)

LAN

IP address

Netmask

Network number

Subnet number

Host number

LAN1

100.2.1.130

255.255.255.0

100

2.1

130

LAN2

101.2.1.130

255.0.0.0

101

None

2.1.130

When assigning the same network number to all monitoring paths and a different subnet to each monitoring path

You must specify the same netmask.

In a network configuration where the same network number is assigned to all monitoring paths and a different subnet is assigned to each monitoring path, make sure that you do not specify different netmasks. The example shown below cannot use multiple monitoring paths because its IP address combination has the same network number but different netmasks.

Table 5‒6: Example of IP address combinations that cannot use multiple monitoring paths

LAN

IP address

Netmask

Network number

Subnet number

Host number

LAN1

100.2.1.130

255.255.255.0

100

2.1

130

LAN2

100.2.2.130

255.255.0.0

100

2

2.130

The IP address combination shown below can use multiple monitoring paths because it uses the same network number and netmask.

Table 5‒7: Example of IP address combinations that can use multiple monitoring paths

LAN

IP address

Netmask

Network number

Subnet number

Host number

LAN1

100.2.1.130

255.255.255.0

100

2.1

130

LAN2

100.2.2.130

255.255.255.0

100

2.2

130

The following figure shows a network configuration using multiple monitoring paths in which the same network number is assigned to all monitoring paths and a different subnet is assigned to each monitoring path.

Figure 5‒7: Network configuration using multiple monitoring paths

5.5.4 Reset path configuration

A reset path is used for the connection when you need to issue a host input/output reset instruction because of a failure in the active system. A reset path is not used when SCSI reservation for shared disk is used or when the function for controlling hot standby based on the availability of LAN communications is used.

For a reset path, use a TCP/IP LAN that is for only resetting purposes (reset-only LAN). Make sure that the LAN allows communication via broadcast.

If the server is BladeSymphony, or model HA8000xM or earlier, connect the failure management processor of each host to the reset-only LAN. If the server is model HA8000xN or later, connect the BMC and the management server that will be used as a failure management processor on each host to the reset-only LAN.

Also, to issue reset instructions from HA Monitor to the failure management processor, connect each host's LAN adapter to the LAN dedicated to resetting.

Make sure that you provide a LAN dedicated to resetting separate from LANs used for other jobs in order to avoid the delay caused by host reset or host reset errors that might result from an increase in traffic. You can also protect the failure management processors from unauthorized access.

Important

Do not share a reset path as a monitoring path. If a reset path is shared in this manner, both functions might be disabled by a single point of failure.

Note that if you use BladeSymphony, there is a limitation to the supported number of LAN adapters. Share the reset path only for the purpose of providing multiple monitoring paths.

This subsection discusses the considerations for the reset path configuration and provides notes about sharing a reset path among multiple hot-standby configurations.

(1) Considerations for the reset path configuration

The reset path configuration depends on the system being used and the hot-standby configuration. This subsection explains the reset path configuration. For details about the system being used, see (1) Processors. For details about types of hot-standby configurations, see 2.1 Hot-standby configurations.

The following table describes the considerations for the reset path configuration.

Table 5‒8: Considerations for the reset path configuration

Consideration

Description

How to connect failure management processors and the reset path

A failure management processor has a separate LAN port from the system LAN port. You connect the failure management processor to the reset path by connecting that LAN port to the reset path.

The location of the LAN port depends on the system. For details, see your hardware documentation.

Providing dual reset paths

The reset path itself cannot be made dual. You must use a kernel extension package in Linux, such as a bonding package, to provide dual LAN adapters that are connected to the reset path.

If you use dual LAN adapters and a failure occurs on one of the LAN adapters, HA Monitor can use the other LAN adapter to perform hot standby processing.

If you use VMware ESXi-based virtualization, we recommend that you make LAN adapters redundant by using the VMware ESXi functions.

Whether Hitachi server virtualization is used (BladeSymphony)

If you use Hitachi server virtualization to perform hot standby processing between LPARs, one LPAR is treated as a host. Therefore, you must connect the reset path from each LPAR.

Note:

If the model of the server in use is HA8000xN or later, interpret the failure management processor as BMC.

Examples of reset path connection configurations are shown below.

To create a 1-to-1 switchover configuration (with model HA8000xM or earlier)

Figure 5‒8: Reset path connection configuration (1-to-1 switchover configuration (with model HA8000xM or earlier))

To create a 1-to-1 switchover configuration (with model HA8000xN or later)

Figure 5‒9: Reset path connection configuration (1-to-1 switchover configuration in which a management server is used as a failure management processor (with model HA8000xN or later))

Figure 5‒10: Reset path connection configuration (1-to-1 switchover configuration in which a management server is not used as a failure management processor (with model HA8000xN or later))

1-to-1 switchover configuration in the same chassis (BladeSymphony)

Figure 5‒11: Reset path connection configuration (1-to-1 switchover configuration in the same chassis) (BladeSymphony)

If the system being used is BladeSymphony, you can employ a hot-standby configuration within the same chassis. If you use Hitachi server virtualization to perform hot standby processing between LPAR, you can also employ a hot-standby configuration within the same chassis.

(2) Notes about sharing a reset path among hot-standby configurations

If you use BladeSymphony or HA8000 and create multiple hot-standby configurations that do not monitor each other in the system, the reset path might be shared among the hot-standby configurations due to hardware limitations.

If the environment is set up incorrectly in such a network configuration, HA Monitor will not be able to identify hosts correctly. As a result, a host that has nothing to do with a host failure that has occurred might be reset or the HA Monitor might terminate abnormally. This subsection describes the settings required for HA Monitor to identify hosts correctly in the following cases:

When sharing failure management processors (BladeSymphony)

When sharing a failure management processor (model HA8000xN or later)

When not sharing failure management processors (BladeSymphony or HA8000)

When sharing standby server modules in an N+M cold standby configuration (BladeSymphony)

When you configure your system, you must check the IP addresses specified here for any duplications. For details about how to perform an IP address duplication check, see 6.21.1 Checking the settings and definitions between multiple hot-standby configurations.

Note

This applies to hot-standby configurations that use the following OSs and systems supported by HA Monitor:

AIX (EP8000)

HP-UX (IPF) (BladeSymphony)

(a) When sharing failure management processors (BladeSymphony)

Depending on the model of BladeSymphony, it might or might not be possible to set a separate port number for each hot-standby switchover configuration by using the HA Monitor linkage settings of SVP. The following describes how to specify the settings in both cases.

For models that allow you to use a separate port number for each hot-standby switchover configuration

We recommend that you use a separate port number for each hot-standby switchover configuration to prevent different hot-standby switchover configurations from communicating with each other. You can also use the same port number for multiple configurations. In this case, if you set the same host address for multiple configurations by mistake, a problem might occur in the event of a host failure. For example, a host that is not related to the failure might be reset, or HA Monitor might terminate abnormally.

Figure 5‒12: Network configuration that shares a reset path among multiple hot-standby configurations (when sharing SVPs)

As shown in the figure, if two hot-standby configurations are configured with one pair of hosts 1 and 3 and another pair of hosts 2 and 4, an SVP in each chassis is shared between hot-standby configurations because it is connected to two hosts in different hot-standby configurations. In this case, you must specify the following settings:

Specify a local host address that is unique in all hosts. This host address must also be unique among the hosts in different chassis so that each host can be identified.

Use the address operand in the HA Monitor environment settings to specify a host address. Note that if the specified host address is duplicated, the KAMN414-E message is issued. In such a case, see 7.5.14 Handling a duplicated address operand value and take appropriate action.

Make sure that the port number of the reset path differs from the port numbers of the SVPs.

Make sure that the port number of the reset path is different for each hot-standby switchover configuration. However, to use virtualization, set the same value for the hosts on the same device.

For SVP port numbers, set a different value for each hot-standby switchover configuration. However, to use virtualization, you must set the same value for the hosts on the same device.

For an example of the settings, see (3) Configuration consisting of two chassis and four hosts (Basic mode).

For an N+M cold standby configuration, also note that an assigned partition name must be unique within the entire host. For specification examples, see (6) N+M cold standby configuration.

For models that do not allow you to use a separate port number for each hot-standby switchover configuration

The following shows an example of a network configuration that shares a reset path among multiple hot-standby configurations and in which the SVPs that serve as failure management processors are shared by the hot-standby configurations.

Figure 5‒13: Network configuration that shares a reset path among multiple hot-standby configurations (when sharing SVPs)

As shown in the figure, if two hot-standby configurations are configured with one pair of hosts 1 and 3 and another pair of hosts 2 and 4, an SVP in each chassis is shared between hot-standby configurations because it is connected to two hosts in different hot-standby configurations. In this case, you must specify the following settings:

Specify a local host address that is unique in all hosts. This host address must also be unique among the hosts in different chassis so that each host can be identified.

Use the address operand in the HA Monitor environment settings to specify a host address. Note that if the specified host address is duplicated, the KAMN414-E message is issued. In such a case, see 7.5.14 Handling a duplicated address operand value and take appropriate action.

Specify the same port number for the reset path in all hosts. Even in a separate chassis, specify the same value for all the hosts that share the SVP.

For the port number of an SVP, specify the same value.

Make sure that the port number of the reset path differs from the port numbers of the SVPs.

For an example of the settings, see (3) Configuration consisting of two chassis and four hosts (Basic mode).

For an N+M cold standby configuration, also note that an assigned partition name must be unique within the entire host. For specification examples, see (6) N+M cold standby configuration.

(b) When sharing a failure management processor (model HA8000xN or later)

The following shows an example of network connections in which multiple hot-standby switchover configurations share a reset path and failure management processor.

Figure 5‒14: Network connections in which a reset path is shared by hot-standby switchover configurations (in the case where a failure management processor is shared)

To create two hot-standby switchover configurations with two host pairs (hosts 1 and 3, and hosts 2 and 4) in a network shown in the preceding figure, the hot-standby switchover configurations need to share a failure management processor. In this case, you must specify the following settings:

Specify a local host address that is unique in all hosts. Use the address operand in the HA Monitor environment settings to specify a host address. Note that if the specified host address is duplicated, the KAMN414-E message is issued. In such a case, see 7.5.14 Handling a duplicated address operand value and take appropriate action.

Specify the port number of the reset path and the port number of the failure management processor as follows:

Set the same value for all hosts in one hot-standby switchover configuration.

Set a different value for each hot-standby switchover configuration. Note that if you use VMware to perform virtualization, the same value must be set for the hosts on the same device.

Specify different values for the port number of the reset path and the port number of the failure management processor.

For an example of the settings, see (2) When there are two pairs of hot-standby configurations.

(c) When not sharing failure management processors (BladeSymphony or HA8000)

The following shows an example of a network configuration that shares a reset path among multiple hot-standby configurations but in which the SVPs that serve as failure management processors are not shared by the hot-standby configurations.

Figure 5‒15: Network configuration that shares a reset path among multiple hot-standby configurations (when not sharing failure management processors)

As shown in the figure, if two hot-standby configurations are configured with one pair of hosts 1 and 3 and another pair of hosts 2 and 4, SVPs are not shared between different hot-standby configurations. In this case, you must specify one of the following (specifying both will not cause a problem):

Specify a local host address that is unique in all hosts. This host address must also be unique among the hosts in different chassis so that each host can be identified.

Use the address operand in the HA Monitor environment settings to specify a host address.

Specify a different port number for the reset path in each hot-standby configuration. In a pair of hot-standby configurations, specify the same value for all hosts. Also specify a different port number for the SVP in each hot-standby configuration.

Make sure that the port number of the reset path differs from the port numbers of the SVPs.

For an example of the settings, see (4) Configuration consisting of four chassis and four hosts (Basic mode) for BladeSymphony and (2) When there are two pairs of hot-standby configurations for HA8000.

For an N+M cold standby configuration, also note that an assigned partition name must be unique within the entire host. For specification examples, see (6) N+M cold standby configuration.

(d) When sharing standby server modules in an N+M cold standby configuration (BladeSymphony)

This subsection explains a network configuration that satisfies the following conditions:

A set path is shared among multiple hot-standby configurations.

A standby server module is shared among multiple N+M cold standby configurations.

The following figure shows an example of such a network configuration.

Figure 5‒16: Network configuration that shares a reset path among multiple hot-standby configurations (a standby server module is shared among multiple N+M cold standby configurations)

To configure a total of two sets of hot-standby configurations, where Hosts 1 and 3 make up one set and Hosts 2 and 4 make up another set, specify the following settings:

Set the local host's host address to a value that is unique among all hosts so that a host in a separate chassis can also be identified by its host address.

Specify a host address in the address operand in the HA Monitor environment settings. If the specified host address is duplicated, the KAMN414-E message will be issued; in such a case, see 7.5.14 Handling a duplicated address operand value and take appropriate action.

Specify the same port number for the reset path on all hosts. Also specify the same SVP port number on all hosts.

Specify a different partition name for each host.

For a specification example, see (6) N+M cold standby configuration.

5.5.5 Examples of hardware configurations (BladeSymphony (Basic mode))

This subsection shows examples of hardware configurations with different hot-standby configurations for BladeSymphony when host reset is performed.

Because BladeSymphony allows multiple hosts in a single chassis, there is no need to provide as many chassis as there are hosts. Examples of the 1-to-1 switchover configuration include one using a single chassis and one using two chassis. The other examples of hardware configurations presented here use two chassis.

For examples of hardware configurations in which BladeSymphony is run in the LPAR mode, see 5.5.6 Examples of hardware configurations (BladeSymphony (LPAR mode)).

(1) 1-to-1 switchover configuration

The following figure shows an example of a hardware configuration that uses one chassis and a 1-to-1 switchover configuration for two processes in the same chassis.

Figure 5‒17: Example of a hardware configuration that uses a 1-to-1 switchover configuration in the same chassis (BladeSymphony (Basic mode))

The following figure shows an example of a hardware configuration that uses two chassis and a 1-to-1 switchover configuration.

Figure 5‒18: Example of a hardware configuration that uses a 1-to-1 switchover configuration between two chassis (BladeSymphony (Basic mode))

(2) 2-to-1 switchover configuration

The following figure shows an example of a hardware configuration that uses a 2-to-1 switchover configuration.

Figure 5‒19: Example of a hardware configuration that uses a 2-to-1 switchover configuration (BladeSymphony (Basic mode))

(3) Multi-standby configuration

A hardware configuration using a multi-standby configuration is the same as a hardware configuration using a 2-to-1 switchover configuration. For an example of a hardware configuration using a 2-to-1 switchover configuration, see (2) 2-to-1 switchover configuration.

(4) Cluster hot-standby configuration

The following figure shows an example of a hardware configuration that uses a cluster hot-standby configuration.

Figure 5‒20: Example of a hardware configuration using a cluster hot-standby configuration (BladeSymphony (Basic mode))

(5) VMware ESXi-based virtualization configuration

The following figure shows an example of a hardware configuration that uses a VMware ESXi-based virtualization configuration.

Figure 5‒21: Example of a hardware configuration that uses a VMware ESXi-based virtualization configuration (BladeSymphony (Basic mode))

5.5.6 Examples of hardware configurations (BladeSymphony (LPAR mode))

This subsection shows examples of hardware configurations with different hot-standby configurations for performing host resets on BladeSymphony with Hitachi server virtualization running in LPAR mode.

For examples of hardware configurations in which BladeSymphony is run in the Basic mode, see 5.5.5 Examples of hardware configurations (BladeSymphony (Basic mode)).

(1) 1-to-1 switchover configuration

The following figure shows an example of a hardware configuration that uses one chassis and a 1-to-1 switchover configuration for two LPARs in the same chassis. In this example, hot standby processing occurs between LPAR2 and LPAR3.

Figure 5‒22: Example of a hardware configuration that uses a 1-to-1 switchover configuration in the same chassis (BladeSymphony (LPAR mode))

The following figure shows an example of a hardware configuration that uses two chassis and a 1-to-1 switchover configuration. In this example, hot standby processing occurs between LPAR1 and LPAR3.

Figure 5‒23: Example of a hardware configuration that uses a 1-to-1 switchover configuration between two chassis (BladeSymphony (LPAR mode))

(2) 2-to-1 switchover configuration

The figure below shows an example of a hardware configuration that uses a 2-to-1 switchover configuration. In this example, hot standby processing occurs among LPAR2, LPAR3, and LPAR6.

Figure 5‒24: Example of a hardware configuration that uses a 2-to-1 switchover configuration (BladeSymphony (LPAR mode))

(3) Multi-standby configuration

A hardware configuration using a multi-standby configuration is the same as a hardware configuration using a 2-to-1 switchover configuration. For an example of a hardware configuration using a 2-to-1 switchover configuration, see (2) 2-to-1 switchover configuration.

(4) Cluster hot-standby configuration

The figure below shows an example of a hardware configuration that uses a cluster hot-standby configuration. In this example, hot standby processing occurs among LPAR2, LPAR3, LPAR6, and LPAR7.

Figure 5‒25: Example of a hardware configuration using a cluster hot-standby configuration (BladeSymphony (LPAR mode))

5.5.7 Examples of hardware configurations (model HA8000xM or earlier)

This subsection shows examples of hardware configurations with different hot-standby switchover configurations that use machines whose model is HA8000xM or earlier and in which host reset is performed.

(1) 1-to-1 switchover configuration

The following figure shows an example of a hardware configuration that uses a 1-to-1 switchover configuration.

Figure 5‒26: Hardware configuration example: 1-to-1 switchover configuration (model HA8000xM or earlier)

(2) 2-to-1 switchover configuration

The following figure shows an example of a hardware configuration that uses a 2-to-1 switchover configuration.

Figure 5‒27: Hardware configuration example: 2-to-1 switchover configuration (model HA8000xM or earlier)

(3) Multi-standby configuration

The following figure shows an example of a hardware configuration that uses a multi-standby configuration.

Figure 5‒28: Hardware configuration example: Multi-standby configuration (model HA8000xM or earlier)

(4) Cluster hot-standby configuration

The following figure shows an example of a hardware configuration that uses a cluster hot-standby configuration.

Figure 5‒29: Hardware configuration example: Cluster hot-standby switchover configuration (model HA8000xM or earlier)

(5) VMware ESXi-based virtualization configuration

The following figure shows an example of a hardware configuration that uses a VMware ESXi-based virtualization configuration.

Figure 5‒30: Example of a hardware configuration that uses a VMware ESXi-based virtualization configuration (model HA8000xM or earlier)

5.5.8 Example of a hardware configuration (model HA8000xN or later)

This subsection shows examples of hardware configurations with different hot-standby switchover configurations that use machines whose model is HA8000xN or later and in which host reset is performed.

(1) 1-to-1 switchover configuration

The following figure shows hardware configuration examples for a 1-to-1 switchover configuration in which a management server is used as a failure management processor.

Figure 5‒31: Hardware configuration example: 1-to-1 switchover configuration in which a management server is used as a failure management processor (model HA8000xN or later)

The following figure shows a hardware configuration example for a 1-to-1 switchover configuration in which a management server is not used as a failure management processor.

Figure 5‒32: Hardware configuration example: 1-to-1 switchover configuration in which a management server is not used as a failure management processor (model HA8000xN or later)

(2) 2-to-1 switchover configuration

The following figure shows a hardware configuration example for a hot-standby switchover configuration in which a management server is used as a failure management processor.

Figure 5‒33: Hardware configuration example: 2-to-1 switchover configuration in which a management server is used as a failure management processor (model HA8000xN or later)

The following figure shows a hardware configuration example for a hot-standby switchover configuration in which a management server is not used as a failure management processor.

Figure 5‒34: Hardware configuration example: 2-to-1 switchover configuration in which a management server is not used as a failure management processor (model HA8000xN or later)

(3) Multi-standby configuration

The following figure shows hardware configuration examples for a multi-standby configuration in which a management server is used as a failure management processor.

Figure 5‒35: Hardware configuration example: Multi-standby configuration in which a management server is used as a failure management processor (model HA8000xN or later)

The following figure shows a hardware configuration example for a multi-standby configuration in which a management server is not used as a failure management processor.

Figure 5‒36: Hardware configuration example: Multi-standby configuration in which a management server is not used as a failure management processor (model HA8000xN or later)

(4) Cluster hot-standby configuration

The following figure shows hardware configuration examples for a cluster hot-standby switchover configuration in which a management server is used as a failure management processor.

Figure 5‒37: Hardware configuration example: Cluster hot-standby switchover configuration in which a management server is used as a failure management processor (model HA8000xN or later)

The following figure shows hardware configuration examples for a cluster hot-standby switchover configuration in which a management server is not used as a failure management processor.

Figure 5‒38: Hardware configuration example: Cluster hot-standby switchover configuration in which a management server is not used as a failure management processor (model HA8000xN or later)

(5) VMware ESXi-based virtualization configuration

The following figure shows hardware configuration examples for a VMware ESXi-based virtualization configuration in which a management server is used as a failure management processor.

Figure 5‒39: Hardware configuration example: VMware ESXi-based virtualization configuration in which a management server is used as a failure management processor (model HA8000xN or later)

The following figure shows hardware configuration examples for a VMware ESXi-based virtualization configuration in which a management server is not used as a failure management processor.

Figure 5‒40: Hardware configuration example: VMware ESXi-based virtualization configuration in which a management server is not used as a failure management processor (model HA8000xN or later)

5.5.9 Examples of hardware configurations (using SCSI reservation for shared disk and PC/AT-compatible systems from another company)

This subsection shows examples of hardware configurations with different hot-standby configurations when SCSI reservation for shared disk is used with PC/AT-compatible systems from another company. These examples assume the use of PC/AT-compatible systems from another company.

(1) 1-to-1 switchover configuration

The following figure shows an example of a hardware configuration that uses a 1-to-1 switchover configuration.

Figure 5‒41: Example of a hardware configuration that uses a 1-to-1 switchover configuration (PC/AT-compatible system from another company)

(2) 2-to-1 switchover configuration

The following figure shows an example of a hardware configuration that uses a 2-to-1 switchover configuration.

Figure 5‒42: Example of a hardware configuration that uses a 2-to-1 switchover configuration (PC/AT-compatible system from another company)

(3) Multi-standby configuration

The following figure shows an example of a hardware configuration that uses a multi-standby configuration.

Figure 5‒43: Example of a hardware configuration that uses a multi-standby configuration (PC/AT-compatible system from another company)

(4) Cluster hot-standby configuration

The following figure shows an example of a hardware configuration that uses a cluster hot-standby configuration.

Figure 5‒44: Example of a hardware configuration that uses a cluster hot-standby configuration (PC/AT-compatible system from another company)

(5) VMware ESXi-based virtualization configuration

The following figure shows an example of a hardware configuration that uses a VMware ESXi-based virtualization configuration.

Figure 5‒45: Example of a hardware configuration that uses a VMware ESXi-based virtualization configuration (PC/AT-compatible system from another company)

5.5.10 Examples of hardware configurations (using the function for controlling hot standby based on the availability of LAN communications)

This subsection shows examples of hardware configurations with a different hot-standby configuration when the function for controlling hot standby based on the availability of LAN communications is used. These examples assume the use of PC/AT-compatible systems from another company.

(1) 1-to-1 switchover configuration

The following figure shows an example of a hardware configuration that uses a 1-to-1 switchover configuration.

Figure 5‒46: Example of a hardware configuration that uses a 1-to-1 switchover configuration (using the function for controlling hot standby based on the availability of LAN communications)

(2) 2-to-1 switchover configuration

The following figure shows an example of a hardware configuration that uses a 2-to-1 switchover configuration.

Figure 5‒47: Example of a hardware configuration that uses a 2-to-1 switchover configuration (using the function for controlling hot standby based on the availability of LAN communications)

(3) Multi-standby configuration

The following figure shows an example of a hardware configuration that uses a multi-standby configuration.

Figure 5‒48: Example of a hardware configuration that uses a multi-standby configuration (using the function for controlling hot standby based on the availability of LAN communications)

(4) Cluster hot-standby configuration

The following figure shows an example of a hardware configuration that uses a cluster hot-standby configuration.

Figure 5‒49: Example of a hardware configuration that uses a cluster hot-standby configuration (using the function for controlling hot standby based on the availability of LAN communications)

(5) VMware ESXi-based virtualization configuration

The following figure shows an example of a hardware configuration that uses a VMware ESXi-based virtualization configuration.

Figure 5‒50: Example of a hardware configuration that uses a VMware ESXi-based virtualization configuration (function for controlling hot standby based on the availability of LAN communications)

6 System Configuration

This chapter explains the procedure for configuring a system in which HA Monitor has been installed, the HA Monitor directory structure, and the settings required to configure a system in which HA Monitor is installed. The user must have the superuser permissions to configure a system.

This chapter assumes that all the hardware required for the system configuration has already been configured.

6.1 Configuration procedure

After completing the installation of HA Monitor, a system administrator with the superuser permissions must configure the system.

This section explains the system configuration procedures from installation of HA Monitor, through specification of settings, to verification of system operation. Explanations are provided for three different cases, one that involves resetting of hosts, one that involves SCSI reservation, and one that involves using the function for controlling hot standby based on the availability of LAN communications.

Important

Before starting system operation in the production environment, always perform the procedure described in 6.23 Checking the operation of the configured system. By performing this procedure, you can check whether the system has been configured correctly. If you start system operation in the production environment without performing this procedure, the system might fail to perform hot-standby switchover in the event of a failure. Never omit this procedure, or your job processing system might stop.

6.1.1 When resetting hosts (BladeSymphony or HA8000)

This subsection explains a system configuration for BladeSymphony or HA8000 when hosts are reset.

(1) System configuration procedure

The figure below shows the system configuration procedure. For the system configuration procedure when HA Monitor Extension is used, see (2) System configuration procedure (when using HA Monitor Extension).

Figure 6‒1: System configuration procedure (when resetting hosts)

The following explains the correspondence between the purpose for configuring the system and the steps in the system configuration procedure that must be performed.

If you are installing HA Monitor and configuring a system for the first time

Perform all the configuration steps shown in the figure.

If you are adding hosts

If you are adding hosts to an existing hot-standby configuration, perform all the configuration steps shown in the figure. You need one HA Monitor per host.

Note that the step Configure VMware ESXi for use in HA Monitor operation is required only if VMware ESXi-based virtualization is used.

If you are adding servers

If you are adding servers to an existing hot-standby configuration, perform the configuration steps shown under Configuration steps when a server is added. The steps shown under Configuration steps when a host is added are not needed.

The step Create server start, termination, and monitoring commands is required only for added servers that will run in the monitor mode. In this case, the creation of a server start command is required, but creation of server termination and monitoring commands is optional.

The steps Create program restart commands, Create and configure UAPs, and Create definition files (monitor-mode program) are required only for added servers that will run in the monitor mode and use the program management function.

For details about how to install HA Monitor and notes about installation, see Release Notes.

For details about how to add hosts and servers to an existing system, see 7.10.1 Adding hosts and 7.10.2 Adding servers.

(2) System configuration procedure (when using HA Monitor Extension)

The following figure shows the system configuration procedure when HA Monitor Extension is used.

Figure 6‒2: System configuration procedure (when using HA Monitor Extension) (when resetting hosts)

The following explains which steps in the system configuration procedure must be performed depending on the purpose for configuring the system.

If you are installing HA Monitor and HA Monitor Extension and configuring a system for the first time

Perform all the configuration steps shown in the figure.

If you are adding hosts

If you are installing HA Monitor Extension and adding hosts to an existing hot-standby configuration, perform all the configuration steps shown in the figure as needed on each host in the hot-standby configuration.

You need one each of HA Monitor and HA Monitor Extension per host.

Note that the step Configure VMware ESXi for use in HA Monitor operation is required only if VMware ESXi-based virtualization is used.

If you are adding servers

If you are installing HA Monitor Extension and adding servers to an existing hot-standby configuration, perform the configuration steps shown under Configuration steps when a server is added. The steps shown under Configuration steps when a host is added are not needed.

The step Create server start, termination, and monitoring commands is required only for added servers that will run in the monitor mode. In this case, creation of a server start command is required, but creation of server termination and monitoring commands is optional.

The steps Create program restart commands, Create and configure UAPs, and Create definition files (monitor-mode program) are required only for added servers that will run in the monitor mode and use the program management function.

For details about how to install HA Monitor and HA Monitor Extension and notes about installation, see Release Notes.

For details about how to add hosts and servers to an existing system, see 7.10.1 Adding hosts and 7.10.2 Adding servers.

6.1.2 When using SCSI reservation for shared disk

The following figure shows the system configuration procedure.

Figure 6‒3: System configuration procedure (when using SCSI reservation for shared disk)

The following explains the correspondence between the purpose for configuring the system and the steps in the system configuration procedure that must be performed.

If you are installing HA Monitor and configuring a system for the first time

Perform all the configuration steps shown in the figure.

If you are adding hosts

If you are adding hosts to an existing hot-standby configuration, perform all the configuration steps shown in the figure. You need one HA Monitor per host.

Note that the step Configure VMware ESXi for use with HA Monitor is required only if VMware ESXi-based virtualization is used.

If you are adding servers

If you are adding servers to an existing hot-standby configuration, perform the configuration steps shown under Configuration steps when a server is added. The steps shown under Configuration steps when a host is added are not needed.

The step Create server start, termination, and monitoring commands is required only for added servers that will run in the monitor mode. In this case, creation of a server start command is required, but creation of server termination and monitoring commands is optional.

The steps Create program restart commands, Create and configure UAPs, and Create definition files (monitor-mode program) are required only for added servers that will run in the monitor mode and use the program management function.

For details about how to install HA Monitor and notes about installation, see Release Notes.

For details about how to add hosts and servers to an existing system, see 7.10.1 Adding hosts and 7.10.2 Adding servers.

6.1.3 When using hybrid fencing

If you use hybrid fencing, configure the system by referring to 6.1.1 When resetting hosts (BladeSymphony or HA8000). Note that when you configure the system, you must configure the OS by referring to 6.3 Configuring the OS, and then specify the settings for SCSI reservation for shared disk by referring to 6.4 Specifying settings for using SCSI reservation for shared disk. If the settings to use host reset and the settings to use SCSI reservation for shared disk are different, specify both settings.

6.1.4 When using the function for controlling hot standby based on the availability of LAN communications

The following figure shows the system configuration procedure.

Figure 6‒4: System configuration procedure (when using the function for controlling hot standby based on the availability of LAN communications)

The following explains which steps in the system configuration procedure must be performed depending on the purpose for configuring the system and.

If you are installing HA Monitor and configuring a system for the first time

Perform all the configuration steps shown in the figure.

If you are adding hosts

If you are adding hosts to an existing hot-standby configuration, perform all the configuration steps shown in the figure. You need one HA Monitor per host.

If you are adding servers

If you are installing HA Monitor Extension and adding servers to an existing hot-standby configuration, perform the configuration steps shown under Configuration steps when a server is added. The steps shown under Configuration steps when a host is added are not needed.

The step Create server start, termination, and monitoring commands is required only for added servers that will run in the monitor mode. In this case, creation of a server start command is required, but creation of server termination and monitoring commands is optional.

The steps Create program restart commands, Create and configure UAPs, and Create definition files (monitor-mode program) are required only for added servers that will run in the monitor mode and use the program management function.

For details about how to install HA Monitor and notes about installation, see Release Notes.

For details about how to add hosts and servers to an existing system, see 7.10.1 Adding hosts and 7.10.2 Adding servers.

6.2 Directory structure

HA Monitor's HAmon directory is created under /opt/hitachi when you install HA Monitor.

The following figure shows the directory structure that is created once HA Monitor has been installed.

Figure 6‒5: Directory structure

Of these HA Monitor directories, only the directory for environment settings can be created and edited by the user. Users must not edit files in any of the other directories.

6.3 Configuring the OS

After completing installation of HA Monitor, the system administrator must configure the OS so that HA Monitor can be used.

6.3.1 Specifying environment variables

The system administrator must specify the following environment variable:

PATH environment variable

In order to use HA monitor commands, you must specify the PATH environment variable. HA Monitor commands are stored under the /opt/hitachi/HAmon/bin/ directory. Add the following path:

/opt/hitachi/HAmon/bin

6.3.2 Specifying system files

Set HA Monitor in system files. Use the monsetup command to set the HA Monitor to automatically start and stop.

(1) If the OS is RHEL7 or a later version

HA Monitor provides the Unit settings file (/usr/lib/systemd/system/HAmon-powercontrol.service). Execute the systemctl command of the OS to enable the service.

Use the following method to enable the service:

# systemctl enable HAmon-powercontrol.service

The following shows the contents of the Unit settings file:

[Unit]
Description=HA Monitor(power control)
After= sysinit.target network.target sshd.service jp1_base.service HABooster.service ltd.service haconnd.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c '/opt/hitachi/HAmon/bin/HApwon >/dev/null 2>&1'
ExecStop=/bin/sh -c '/opt/hitachi/HAmon/bin/HApwoff >/dev/null 2>&1'

[Install]
WantedBy=multi-user.target

To change the automatic start and stop sequence of HA Monitor, modify the values specified for the After entry under the [Unit] section in the Unit settings file. For details about the Unit settings file, see the documentation for systemd of the OS.

Notes

You can change only the values for the After entry from the default settings. Do not change or delete other default settings provided since installation.

Although you can add services to the After entry, you must not change or delete the other default settings provided since installation.

If you add options, adequately make prior consideration and verification. If you specify incorrect settings, such as those causing service dependencies to loop, the OS might be unable to start.

The Unit settings file under /usr/lib/systemd/system is overwritten when an overwrite installation of HA Monitor is performed. Therefore, if you want to edit the Unit settings file, copy HAmon-powercontrol.service under /usr/lib/systemd/system to /etc/systemd/system, and then edit the file at the copy destination.

(2) If the OS is RHEL6

HA Monitor provides the /etc/rc.d/init.d/HAmon script. Register the script by executing the chkconfig command of the OS.

Use the following method to register the script:

# /sbin/chkconfig --add HAmon

To change the automatic start and termination sequence of HA Monitor, modify the chkconfig control statement at the beginning of the /etc/rc.d/init.d/HAmon file. For details about the control statement, see Help for the chkconfig command of the OS.

Note that the /etc/rc.d/init.d/HAmon file is overwritten when an overwrite installation of HA Monitor is performed. Therefore, if you modified the /etc/rc.d/init.d/HAmon file, back up the /etc/rc.d/init.d/HAmon file before you perform an overwrite installation of HA Monitor. After an overwrite installation of HA Monitor is performed, modify the /etc/rc.d/init.d/HAmon file again based on the backup file.

6.3.3 Setting the system clock

You manage the time at all computers by setting the system clock. Make sure that you do not set the system clock to a time that has already passed (at the level of seconds). HA Monitor uses the system clock to check the integrity of start notification packets for communication between hosts.

You must set the system clock before you start using HA Monitor. If you change the system clock while HA Monitor is running, HA Monitor might erroneously detect a failure, resulting in connection errors between HA Monitors.

6.3.4 Setting the DNS

If you use DNS and the DNS has a higher priority in host name settings than the /etc/hosts file, there might be a delay before HA Monitor starts. We recommend that you set the host name settings in such a manner that the /etc/hosts file has a higher priority than DNS.

6.3.5 Setting system log files

When you use HA Monitor, set the syslog file to be output so that the causes of failures can be determined.

6.3.6 Specifying the kernel parameters

If you change the default value (which is 16) for the maximum number of servers that can run concurrently or you change the default value for the maximum number of hosts when using HA Monitor Extension to a value greater than 32, you must change the kernel parameters. You can display the kernel parameters by using the sysctl command of the OS. You can change the kernel parameters by modifying the /etc/sysctl.conf file (or the /etc/sysctl.d/99-sysctl.conf file if the OS is RHEL7 or a later version). The table below lists the two kernel parameters that must be changed and their values.

Table 6‒1: Kernel parameter settings (when changing the default values for the maximum numbers of servers and hosts)

Kernel parameter to be changed

Value to be specified

Specifying 64 as the maximum number of servers

Specify 128 or greater as the maximum number of servers

Specifying 33 or greater as the maximum number of hosts

net.core.rmem_max

303200 or greater

1048576 or greater

1048576 or greater

net.core.wmem_max

303200 or greater

1048576 or greater

1048576 or greater

Notes

If the default value for a kernel parameter is greater than the value shown in the table, there is no need to change the default value.

Specify the same value for both kernel parameters.

If the maximum number of servers that can be run concurrently is set to the default value (16), specify the default value or a greater value for the kernel parameters.

6.3.7 Specifying the output language

You must specify the output language for messages (English or Japanese).

(1) Specifying the LANG environment variable

The value specified in the kernel's LANG environment variable determines the message output language for HA Monitor.

To change from English to Japanese or vice versa, specify the LANG environment variable as follows:

To set the output language to English: LANG=C

To set the output language to Japanese (EUC): LANG=ja_JP.eucJP

To set the output language to Japanese (UTF-8); LANG=ja_JP.UTF-8

For details about specifying the output language when all the following conditions are satisfied, see (a) If the OS is RHEL7 or a later version:

The HA Monitor version is 01-65 or later.

If the OS is RHEL7 or a later version

The HA Monitor start command (monstart command) is used to start HA Monitor.

(2) Specifying output of Japanese messages when HA Monitor is automatically started at system startup

For notes on editing the Unit settings file and startup script /etc/rc.d/init.d/HAmon, see 6.3.2 Specifying system files.

(a) If the OS is RHEL7 or a later version

The language of messages that are output to syslog by HA Monitor that is automatically started at system startup depends on the value of the Environment entry under the [Service] section in the Unit settings file of HA Monitor.

To output messages in Japanese, add the LANG environment variable that specifies Japanese as the output language. The following shows an example. Add the underlined entry.

[Unit]
Description=HA Monitor(power control)
...
...
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c '/opt/hitachi/HAmon/bin/HApwon >/dev/null 2>&1'
ExecStop=/bin/sh -c '/opt/hitachi/HAmon/bin/HApwoff >/dev/null 2>&1'
Environment="LANG=ja_JP.UTF-8"
...
...
(omitted)

(b) If the OS is RHEL6

The language of messages that are output to syslog by HA Monitor that is automatically started at system startup depends on the language environment in which the kernel executes the startup script (/etc/rc.d/init.d/HAmon).

To output messages in Japanese, add the LANG environment variable that specifies Japanese as the output language to the HA Monitor startup script. The following shows an example. Add the underlined entry.

#!/bin/sh
#
# chkconfig: 2345 60 40

NOLOCALE=1
export LANG=ja_JP.UTF-8
PATH=/sbin:/usr/sbin:/usr/bin:/bin
export PATH

(omitted)

6.3.8 Setting the security level

HA Monitor's monitoring and reset paths use UDP communication. You must set the security level so that HA Monitor's UDP communication is not blocked by a firewall.

6.3.9 Setting the file system

(1) File system in which HA Monitor is installed

Do not target the file system (/opt/hitachi/HAmon) in which HA Monitor is installed with specification of the nosuid option in the /etc/fstab file or with the mount command.

HA Monitor does not monitor file systems. Consider configuring HA Monitor in such a manner that an OS panic is issued when a file system error that might adversely affect job processing occurs so that hot standby processing can be performed quickly.

The example below shows a setting for /etc/fstab that issues an OS panic if an error occurs on the root disk (/). For details, see the OS documentation.

LABEL=/    /    ext4    errors=panic    1    1

(2) File system on a shared disk that is used by servers

If you use a file system on a shared disk, you must specify the file system settings described below for the OS.

(a) Setting that prohibits file system mounting during system startup

Because HA Monitor controls the mounting and unmounting of the file system, specify the setting in such a manner that the file system will not be mounted automatically during system startup. For details about how to specify this setting, see the OS documentation.

(b) File system type setting

When you specify the file system to be controlled by HA Monitor in the /etc/fstab file, make sure that the ignore option is not specified in the third field (file system type). If the ignore option is specified, the fsck command executed by HA Monitor might fail, making the file system unavailable.

(c) Settings related to the file system check time

If the fsck command needs time to check the file system, this will cause an increase in the time required for hot standby processing. To reduce the amount of time required for checking by the fsck command, evaluate implementing the following:

Perform hot standby processing only on files that are actually needed. Do not include any unneeded files.

Do not execute the fsck command automatically with the following settings:

Disabling the fsck command based on the mount count (maximum mount count)

Disabling the fsck command based on the interval (check interval)

(d) Settings for enabling external access by using the NFS

To externally access directories by using the NFS, specify the fsid option in the /etc/exports file. For details, see the documentation for the OS.

6.4 Specifying settings for using SCSI reservation for shared disk

This section explains the settings for using SCSI reservation for shared disk in HA Monitor.

In a VMware ESXi-based virtualization environment, a single path configuration is normally assumed because paths are switched by VMware ESXi even in a redundant configuration. The following figure shows an example of a VMware ESXi-based virtualization environment in which Fibre Channel connections are employed in a redundant configuration.

Figure 6‒6: Example of a redundant configuration in a VMware ESXi-based virtualization environment (using Fibre Channel)

If you configure the OS on a virtual machine to directly recognize the shared disk with iSCSI connection, you can also use VMware ESXi's optional functions, such as vMotion. In this case, set up VMware ESXi to recognize only the LANs, not the shared disk. In a redundant configuration, use the multipath software (DMMP) with the OS on the virtual machine. In this configuration, DMMP switches paths.

The following figure shows an example of a redundant configuration that uses DMMP in a VMware ESXi-based virtualization environment.

Figure 6‒7: Example of a redundant configuration in a VMware ESXi-based virtualization environment (using DMMP)

6.4.1 Setting up multipath software

You might need to specify settings according to the type of the multipath software. The following describes the multipath software for which you must specify settings.

(1) Using HDLM multipath software

If you use HDLM multipath software, execute -prsup on of the HDLM driver option configuration utility (dlmsetopt) and restart the hosts. The HDLM driver option configuration utility must be run in all hot-standby configurations. Otherwise, the shared disk might not be protected. For details about the setup and verification procedures, see the Hitachi Dynamic Link Manager Software User's Guide (for Linux(R) systems).

(2) Using DMMP multipath software

If you use DMMP multipath software, specify the following settings.

devices section

For the path checker (path_checker), specify a value other than tur (which issues TEST UNIT READY to a device). The queue_if_no_path function is not used (specify 0 for features). For failback (failback), specify the settings for using manual failback.

An example is shown below. For vendor and product, specify values appropriate to the storage being used.

devices {
       device {
               vendor                  "HITACHI"
               product                 "DF.*"
               path_checker            directio
               features                "0"
               failback                manual
       }
}

Settings when the blacklist section is specified

If you specify the blacklist section, specify it for each WWID or device type. Do not specify it for each device name.

The following example specifies the blacklist section for each WWID:

blacklist {
   wwid 360060e8010462fe004f2b6ae00000065
}

For details about the name of the settings file and how to specify the settings, see the DMMP documentation.

6.4.2 Determining the operand value needed for SCSI reservation for shared disk

This subsection explains how to determine the value for the scsi_device or dmmp_device operand in the server environment definition.

Specify the scsi_device operand in the following cases:

A single-path configuration is used.

A VMware ESXi-based virtualization environment is used (except when DMMP is used).

A redundant configuration is used with multipath software (HDLM).

If you use a redundant configuration with multipath software (DMMP), specify the dmmp_device operand.

(1) scsi_device operand value

(a) If HFC-PCM is used with a single-path configuration in a VMware ESXi-based virtualization environment

Specify in the scsi_device operand the symbolic link for the shared disk. You can obtain the symbolic link from the path name for the shared disk that is obtained on the basis of the LVM's physical volume name.

To obtain the scsi_device operand value:

Check the physical volume name.

The checking method is as follows:

#/sbin/vgdisplay -v volume-group-name

The following example specifies vg01 for the volume group name.

Example:

#/sbin/vgdisplay -v vg01

Execution results:

 --- Volume group ---
 VG Name               vg01
...
...
 --- Physical volumes ---
 PV Name               /dev/sda1
...
...
(omitted)

The underlined portion is the physical volume name. The numeric digit at the end is the partition number.

Obtain the symbolic link.

Use the command below to obtain the symbolic link that is located under /dev/disk/by-id and that is created by the OS. For the physical volume name, specify the physical volume name checked in step 1 without /dev/ or the partition number.

# ls -l /dev/disk/by-id | grep physical-volume-name$

The following example specifies sda for the physical volume name.

Example:

# ls -l /dev/disk/by-id | grep sda$

Execution results:

lrwxrwxrwx 1 root  9 Jun  7 17:08
scsi-360060e8010462fe004f2b6ae00000065 -> ../../sda

The underlined portion is a symbolic link. In the scsi_device operand, specify an absolute path.

Notes

Do not specify the following symbolic links for the scsi_device operand:

• A symbolic link that ends with part

• A symbolic link that begins with wwn-

(b) Redundant configuration using multipath software (HDLM)

If you use HDLM to make the shared disk path redundant, specify the absolute path of the HDLM's logical device in the scsi_device operand. You can obtain the absolute path from the LVM's physical volume name, which is the same as the HDLM's logical device name with a partition number added.

Execute the following command to check the physical volume name:

#/sbin/vgdisplay -v volume-group-name

The following shows an example that specifies vg01 for volume group.

Example:

#/sbin/vgdisplay -v vg01

Execution results:

 --- Volume group ---
 VG Name               vg01
.
.
 --- Physical volumes ---
 PV Name               /dev/sddlmaa1
.
.
(omitted)

The underlined portion is the LVM's physical volume name. The partition number would be p1 at the end. Specify in the scsi_device operand the LVM's physical volume name without the partition number.

Important

HDLM's logical device name might not be the same among all hosts. Perform this procedure on all hosts in the hot-standby configuration and check the scsi_device operand value.

(2) dmmp_device operand value (redundant configuration using multipath software (DMMP))

Specify in the dmmp_device operand the absolute path of the DMMP's logical device. You can obtain the absolute path from the LVM's physical volume name, which is the same as the DMMP's logical device.

Execute the following command to check the physical volume name:

#/sbin/vgdisplay -v volume-group-name

The following example specifies vg01 for the volume group.

Example:

#/sbin/vgdisplay -v vg01

Execution results:

 --- Volume group ---
 VG Name               vg01
. . --- Physical volumes ---
 PV Name               /dev/mapper/mpath1p1
. .
(omitted)

The underlined portion is the LVM's physical volume name, which is the DMMP's logical volume. p1 at the end is the partition number. Specify in the dmmp_device operand the DMMP's logical volume without the partition number.

6.4.3 Specifying the udev rules

Use a udev rules file to specify the owner, group, and permissions of the device file on the shared disk that was specified in the scsi_device or dmmp_device operand in the server environment definition.

If the device file on the shared disk is opened in the write mode, that event is sent to the udev rules when the device file is closed and the file is placed in the status specified in the udev rules (if udev rules are not specified, the default status). HA Monitor opens a device file on the shared disk in the write mode in order to reserve the shared disk specified in the scsi_device or dmmp_device operand in the server environment definition. If you change the permissions by using chmod or chown, the device file is reset to the status specified in the udev rules. Therefore, you must use the udev rules to change permissions.

An example of setting the owner, group, and permissions (specifying a symbolic link in the scsi_device operand) is shown below. If you want to use HDLM multipath software, contact your Hitachi sales representative. If you want to use multipath software (DMMP or HFC-PCM), see the documentation for the respective product.

Under /etc/udev/rules.d, create a rule file for setting the owner, group, and permissions.

The following shows how to create a rule file:

# touch /etc/udev/rules.d/numeric-value-rule-name.rule

Legend:

numeric-value: Any 2-digit number

rule-name: Any character string

The example shown below specifies 99 for numeric-value and my-rules for rule-name.

Example:

# touch /etc/udev/rules.d/99-my-rules.rules

Note the following:

Make sure that the specified rule name does not duplicate the name of any other rule file under /etc/udev/rules.d.

If you specify multiple rule files for the same device, the rule file with the largest number will be used.

We recommend that you specify a number that is larger than any disk device-related udev rule. By default, 60 or a smaller number is specified for a disk device-related udev rule.

Edit the rule file.

Add the following line by editing the rule file (use a program such as vi editor):

ACTION=="add|change", SYMLINK=="symbolic-link", OWNER="owner-name", GROUP="group-name", MODE="permission", OPTIONS+="nowatch"

For symbolic-link, specify the scsi_device operand value without /dev/. The following shows an example:

ACTION=="add|change", SYMLINK=="disk/by-id/scsi-360060e8010462fe004f2b6ae00000065", OWNER="user1", GROUP="group1", MODE="0666", OPTIONS+="nowatch"

Restart the OS to apply the rule file.

Copy the file edited in step 2 to the remote host, and then perform step 3.

There is no need to edit a symbolic link under /dev/disk/by-id because it is always the same value regardless of the referencing host.

6.4.4 Notes about SCSI reservation for shared disk

It is not possible to create multiple partitions on a single physical device so that each server can use a different partition.

SCSI reservation is performed for each physical device. If separate partitions are used by the different servers, the disk cannot be protected because SCSI reservations cannot be obtained correctly.

The same names must be specified in the same order on each host for the device names specified in the scsi_device and dmmp_device operands in the server environment definition. If the specified names or their specification order differ between hosts, SCSI reservations might not be obtained correctly.

In the case of a redundant configuration using HDLM, the order in which the device names are specified in the scsi_device operand must be the same on each host. If the same disk is used for all hosts in a hot-standby configuration, different names can be specified.

In the scsi_device and dmmp_device operands, you must specify all device names that correspond to the device file names of the physical volumes that make up each volume group specified in the disk operand.

For the shared disks corresponding to the logical devices specified in the scsi_device and dmmp_device operands, do not use an LVM mirror or Linux standard software RAID (md).

6.5 Configuring VMware ESXi for use in HA Monitor operation

You must configure VMware ESXi to use it with HA Monitor. The required settings are for resetting hosts and for using SCSI reservation for shared disk.

6.5.1 Settings required for resetting hosts and using SCSI reservation for shared disk

The settings listed below are required when hosts are reset as well as when SCSI reservation for shared disk is used. These settings are related to the shared disk that is subject to hot standby processing by HA Monitor.

These settings are not required when both the following conditions are satisfied:

SCSI reservation for shared disk is used.

DMMP is used as multipath software.

(1) Specifying the device type of the shared disk to be allocated to the virtual machine

In VMware vSphere Client, you must specify Raw device mapping for the shared disk that will be allocated to the virtual machine. To specify the shared disk:

For a version earlier than VMware ESXi6

Connect to VMware ESXi with VMware vSphere Client.

Open the window used for editing the virtual machine settings, and then click Add.

Select Hard disk as the type of device to be added, and then click Next.

Select Raw device mapping as the type of disk to be used.

For VMware ESXi6 or a later version

Connect to VMware ESXi with VMware vSphere Client.

Open the window used for editing the virtual machine settings, and then click Edit.

Click Virtual Hardware, and then select Add hard disk and then New raw disk.

Select Physical as Disk Compatibility.

For a virtual device node that is selected during this setup, we recommend that you select a new virtual device node for the shared disk (such as [SCSI (1:0)]).

If you use SCSI reservation for shared disk, also specify the settings described in 6.5.3 Settings required for using SCSI reservation for shared disk.

(2) Specifying the sharing of the SCSI bus

You must set the sharing of the SCSI bus to Physical so that the SCSI bus can be shared between virtual machines and between chassis.

To specify the sharing of the SCSI bus:

Connect to VMware ESXi with VMware vSphere Client.

Open the window used for editing the virtual machine settings.

Select SCSI controller, and then select Physical for sharing the SCSI bus.

6.5.2 Settings required for resetting hosts

If you will be resetting hosts, you must specify the settings described below.

(1) Specifying a password for VMware ESXi

Specify the root user's password that matches the password for VMware ESXi on the remote host in the hot-standby configuration.

(2) Specifying the settings needed for establishing SSH connection with VMware ESXi

You must specify the settings needed for establishing SSH connection with VMware ESXi. To specify the settings:

Example of specifying the settings for VMware ESXi (VMware ESXi 4.1):

Start the console window of VMware ESXi, and then log in.

Choose Troubleshooting Options, and then press the Enter key.

From the Troubleshooting Mode Options menu, choose Enable Remote Tech Support (SSH), and then press the Enter key.

Log out from VMware ESXi.

Example of specifying the settings for VMware ESXi (VMware ESXi 5.0 or later):

Start the console window of VMware ESXi, and then log in.

Choose Troubleshooting Options, and then press the Enter key.

From the Troubleshooting Mode Options menu, select Enable ESXi Shell, and then press the Enter key.

The status changes from ESXi Shell is Disabled to ESXi Shell is Enabled.

From the Troubleshooting Mode Options menu, select Enable SSH, and then press the Enter key.

The status changes from SSH is Disabled to SSH is Enabled.

Log out from VMware ESXi.

(3) Specifying the settings needed for vCenter management

If you use vCenter management, we recommend that you disable the suspend operation by using vCenter's role settings.

Important

If you use HA Monitor in VMware ESXi's virtualization environment, do not suspend virtual machines while HA Monitor is running. For details, see Release Notes.

6.5.3 Settings required for using SCSI reservation for shared disk

If you will be using SCSI reservation for shared disk, you must specify the following settings:

Hard disk (virtual disk) settings

Compatible mode: Physical

SCSI controller settings

SCSI controller type: LSI Logic SAS or VMware Paravirtual

Path management

Path selection: Fixed, Most recent, or Round-robin

These settings are related to the shared disk that is subject to hot standby processing by HA Monitor.

6.6 Configuring the reset path (BladeSymphony)

If the system being used is BladeSymphony, you must configure a reset path so that in the event of a host failure HA Monitor can reset the host.

Important

When a reset path is configured, HA Monitor checks whether it is possible to communicate with a failure management processor and VMware ESXi. Therefore, before you configure a reset path, make sure that the settings for a failure management processor and VMware ESXi have been specified completely.

You use an HA Monitor environment setup command (monsetup command) to configure the reset path. For details about the HA Monitor environment setup command (monsetup command), see 9.22 monsetup (specifies HA Monitor environment settings) in 9. Commands.

6.6.1 Information required for configuration (BladeSymphony)

You need the following information about the target hosts and SVP.

Figure 6‒8: Information required for configuring the reset path (BladeSymphony: not using VMware ESXi-based virtualization)

Figure 6‒9: Information required for configuring the reset path (BladeSymphony: using VMware ESXi-based virtualization)

The following table explains the specification items in the figures.

Table 6‒2: Information required for configuring the reset path (BladeSymphony)

No. in figure

Information required for configuration

Description

Remarks

1

Partition name

Specifies the management system name of HA Monitor.

This is information specified when you configured partitions.

Set the same processor's system name that was set for SVP#1.

If you use an N+M cold standby configuration, specify a value that is unique among all hosts that share the reset path.

2

LPAR name#2

Specifies the name of the LPAR in which HA Monitor is run.

This is information specified when you configured LPARs.

Specify the LPAR name that was specified on the HVM control screen.

3

IP address of reset path

Specifies the IP address of the reset path allocated to the target host.

For details about how to allocate IP addresses and port numbers, see 5.4.2 Necessary IP addresses and port numbers.

If auto is specified for the IP address of the SVP, set Enable for the SVP. This setting is required for HA Monitor to operate in a system configuration that supports the N+M cold-standby function#1.

4

Port number of reset path

Specifies the port number of the reset path allocated to the target host.

5

IP address of SVP

Specifies the IP address of the reset path allocated to SVP.

For an N+M cold standby configuration, specify auto.

6

Port number of SVP

Specifies the port number of the reset path allocated to SVP.

7

Machine address#3

Specifies any number that indicates the physical partition (processor) in which HA Monitor is run.

This must be the same value specified for the host in the same physical partition but different from one specified for a host in a different physical partition.

Make sure that this value is unique among all physical partitions that share the reset path.

8

IP address of VMware ESXi#3

Specifies the management IP address of VMware ESXi.

For details about how to allocate IP addresses and port numbers, see 5.4.2 Necessary IP addresses and port numbers.

9

Port number of VMware ESXi#3

Specifies the port number assigned to VMware ESXi for SSH connection.

10

Password#3, #4

Specifies the password required for authorization when SSH connection is established with VMware ESXi on the local host.

Specify the same password throughout the same hot-standby configuration.

11

Virtual machine name#3

Specifies the virtual machine name that is managed by VMware ESXi.

This is the name specified when you created virtual machines with VMware ESXi. Specify the name that was checked by VMware vSphere Client or vCenter.

This virtual machine name must be unique among all VMware ESXi systems that can be connected by HA Monitor.

The following describes how to check a virtual machine name:

Execute the ifconfig command on the guest OS of the virtual machine and make note of the information specific to the virtual machine (such as the MAC address).

Use VMware vSphere Client to connect to VMware ESXi.

Open the window for editing the virtual machine settings and find the virtual machine with the same virtual machine-specific information that was obtained in step 1.

#1

Specify this in the HA Monitor linkage settings for the SVP. For model BS2000 or earlier, specify this setting by using the HA command in the SVP command mode. For details, see SVP-related documentation.

#2

This information is required only if you use Hitachi server virtualization to employ a hot-standby configuration in the LPAR mode.

#3

This information is required only if you use VMware ESXi-based virtualization.

#4

HA Monitor stores in a unique format on the local disk the password needed for establishing SSH connection with VMware ESXi on the local host. Although this password is stored in such a manner that only the root user can reference it, you must evaluate what security measures might be needed, such as physically disconnecting the local disk from external networks.

Depending on the configuration, the specification of a host address (address operand in the HA Monitor environment settings) requires special care. For details, see (2) Notes about sharing a reset path among hot-standby configurations.

If you employ a mutual hot-standby switchover configuration and do not specify a host that has reset priority, you must configure the cluster management function in the HA Monitor linkage settings for the SVP. For details, see SVP-related documentation.

6.6.2 Examples of settings (BladeSymphony)

This subsection explains examples of reset path settings for BladeSymphony. In the tables in this subsection, Setting item corresponds to Information required for configuration in Table 6-2 Information required for configuring the reset path (BladeSymphony).

(1) Configuration consisting of two chassis and two hosts (Basic mode)

This subsection presents an example of settings specified with the monsetup command for the following configuration (performing the hot standby operation between hosts 1 and 2).

Figure 6‒10: Example 1 of reset path settings (BladeSymphony)

Setting item

Settings on host

Host 1

Host 2

Partition name

MS0001

MS0002

IP address of reset path

x.x.x.a

x.x.x.b

Port number of reset path

ppppp

ppppp

IP address of SVP

x.x.x.c

x.x.x.d

Port number of SVP

qqqqq

qqqqq

(2) Configuration consisting of one chassis and two hosts (LPAR mode)

This subsection presents an example of settings specified with the monsetup command for the configuration shown in the following figure (performing the hot standby operation between hosts 1 and 2).

Figure 6‒11: Example 2 of reset path settings (BladeSymphony)

Setting item

Settings on host

Host 1

Host 2

Partition name

MS0001

MS0001

LPAR name

LPAR1

LPAR2

IP address of reset path

x.x.x.a

x.x.x.b

Port number of reset path

ppppp

ppppp

IP address of SVP

x.x.x.c

x.x.x.c

Port number of SVP

qqqqq

qqqqq

(3) Configuration consisting of two chassis and four hosts (Basic mode)

This subsection explains an example of reset path settings in a system that runs under the following conditions:

The system contains two pairs of 1-to-1 switchover configurations that do not monitor each other.

Two pairs of hot-standby configurations share the reset path.

Two pairs of hot-standby configurations share the SVP.

Important

In the hot-standby configuration explained here, the specification of host address (address operand in the HA Monitor environment settings) requires special care. For details, see (2) Notes about sharing a reset path among hot-standby configurations.

The following shows an example of settings specified with the monsetup command for the configuration shown in the following figure (performing the hot standby operation between hosts 1 and 3 and between hosts 2 and 4) that satisfies all the above conditions. Some models of BladeSymphony allow you to set a different port number for each hot-standby switchover configuration by using the HA Monitor linkage settings of SVPs. The following shows setting examples in the cases where different port numbers are set and the same port number is set for hot-standby switchover configurations.

(a) Case where different port numbers are set for two switchover configurations

Figure 6‒12: Example 3 of reset path settings (BladeSymphony)

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Partition name

MS0001

MS0002

MS0003

MS0004

IP address of reset path

x.x.x.a

x.x.x.b

x.x.x.d

x.x.x.e

Port number of reset path

ppppp

rrrrr

ppppp

rrrrr

IP address of SVP

x.x.x.c

x.x.x.c

x.x.x.f

x.x.x.f

Port number of SVP

qqqqq

sssss

qqqqq

sssss

For this hot-standby configuration, the following shows an example of the address operand values in the HA Monitor environment settings:

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Host addresses of the local hosts

11

21

12

22

(b) Case where the same port number is set for two switchover configurations

The following shows an example of values to be specified in the monsetup command under the conditions shown earlier. In this example, hosts 1 and 3 make up one hot-standby switchover pair, and hosts 2 and 4 make up another hot-standby switchover pair.

Figure 6‒13: Example 4 of reset path settings (BladeSymphony)

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Partition name

MS0001

MS0002

MS0003

MS0004

IP address of reset path

x.x.x.a

x.x.x.b

x.x.x.d

x.x.x.e

Port number of reset path

ppppp

ppppp

ppppp

ppppp

IP address of SVP

x.x.x.c

x.x.x.c

x.x.x.f

x.x.x.f

Port number of SVP

qqqqq

qqqqq

qqqqq

qqqqq

For this hot-standby configuration, the following shows an example of the address operand values in the HA Monitor environment settings:

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Host addresses of the local hosts

11

21

12

22

(4) Configuration consisting of four chassis and four hosts (Basic mode)

This subsection explains an example of reset path settings in a system that runs under the following conditions:

The system contains two pairs of 1-to-1 switchover configurations that do not monitor each other.

Two pairs of hot-standby configurations share the reset path.

Two pairs of hot-standby configurations do not share the SVP.

Important

In the hot-standby configuration explained here, the specification of host address (address operand in the HA Monitor environment settings) requires special care. For details, see (2) Notes about sharing a reset path among hot-standby configurations.

The following shows an example of settings specified with the monsetup command for the configuration shown in the following figure (performing the hot standby operation between hosts 1 and 3 and between hosts 2 and 4) that satisfies all the above conditions.

Figure 6‒14: Example 5 of reset path settings (BladeSymphony)

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Partition name

MS0001

MS0002

MS0003

MS0004

IP address of reset path

x.x.x.a

x.x.x.c

x.x.x.e

x.x.x.g

Port number of reset path

ppppp

rrrrr

ppppp

rrrrrr

IP address of SVP

x.x.x.b

x.x.x.d

x.x.x.f

x.x.x.h

Port number of SVP

qqqqq

sssss

qqqqq

sssss

For this hot-standby configuration, the following shows an example of the address operand values in the HA Monitor environment settings:

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Host addresses of the local hosts

11

21

12

22

(5) VMware ESXi-based virtualization configuration

This subsection explains an example of reset path settings in a system that runs under the following conditions:

A VMware ESXi-based virtualization configuration is used.

The system contains two pairs of 1-to-1 switchover configurations that do not monitor each other.

Two pairs of hot-standby configurations share the reset path.

Two pairs of hot-standby configurations share the SVP.

The following shows an example of settings specified with the monsetup command for the configuration shown in the following figure (performing the hot standby operation between hosts 1 and 3 and between hosts 2 and 4, and managing hosts 1, 2, and 3 by VMware ESXi) that satisfies all the above conditions.

Figure 6‒15: Example 6 of reset path settings (BladeSymphony)

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Partition name

MS0001

MS0001

MS0002

MS0003

IP address of reset path

x.x.x.a

x.x.x.b

x.x.x.d

x.x.x.e

Port number of reset path

ppppp

ppppp

ppppp

ppppp

IP address of SVP

x.x.x.c

x.x.x.c

x.x.x.f

x.x.x.f

Port number of SVP

qqqqq

qqqqq

qqqqq

qqqqq

Machine address

11

11

12

--

IP address of VMware ESXi

y.y.y.a

y.y.y.a

y.y.y.b

--

Port number of VMware ESXi

default#

default#

rrrrr

--

Virtual machine name

VM0001

VM0002

VM0003

--

Legend: --: Specification is not required.

#: If you use the port number for SSH connection (22), specify default.

In this configuration, specify use in the vmware_env operand in the HA Monitor environment settings (sysdef) on all hosts in the hot-standby configuration.

(6) N+M cold standby configuration

This subsection explains an example of reset path settings in a system that runs under the following conditions:

The system employs an N+M cold standby configuration.

The system employs a 1-to-1 switchover configuration.

Host 3 is a standby server module.

Important

In the hot-standby configuration explained here, the specification of host address (address operand in the HA Monitor environment settings) requires special care. For details, see (2) Notes about sharing a reset path among hot-standby configurations.

The following shows an example of settings specified in the system for the configuration shown in the following figure that satisfies all the above conditions (performing the hot standby operation between hosts 1 and 2; host 3 is a standby server module).

Figure 6‒16: Example 7 of reset path settings (BladeSymphony)

The following shows an example of the values that are set with the monsetup command to achieve the above settings:

Setting item

Settings on host

Host 1

Host 2

Host 3

Partition name

MS0001

MS0002

--

IP address of reset path

x.x.x.a

x.x.x.b

--

Port number of reset path

ppppp

ppppp

--

IP address of SVP

auto

auto

--

Port number of SVP

qqqqq

qqqqq

--

Legend: --: Specification is not required.

When using BladeSymphony BS500 or BS2500

For this hot standby configuration, the following shows an example of the values specified for linking HA Monitor:

Setting item

Settings on SVP

SVP on host 1

SVP on host 2

SVP on host 3

HA Monitor:

Enabled

Enabled

--

Port(5001-65535):#

qqqqq

qqqqq

--

System name:

MS0001

MS0002

--

N+M Cold standby:

Enable

Enable

--

Legend: --: Specification is not required.

#: For Blade Symphony BS2500, this setting item appears as "Port number (5001-65535)".

(7) Mutual hot-standby configuration (when a host with reset priority is not specified)

This subsection explains an example of reset path settings in a system that runs under the following conditions:

A mutual hot-standby configuration is employed.

A host with reset priority is not specified (system is specified in the cpudown operand in the HA Monitor environment settings).

The following shows an example of settings specified with the monsetup command for the configuration shown in the following figure (performing hot standby operation between hosts 1 and 2) that satisfies all the above conditions.

Figure 6‒17: Example 7 of reset path settings (BladeSymphony)

The following shows an example of values specified with the monsetup command to obtain the above settings:

Setting item

Settings on host

Host 1

Host 2

Partition name

MS0001

MS0002

IP address of reset path

x.x.x.a

x.x.x.b

Port number of reset path

ppppp

ppppp

IP address of SVP

x.x.x.c

x.x.x.d

Port number of SVP

qqqqq

qqqqq

When using BladeSymphony BS2500

The following shows an example of the values to be specified in the HA Monitor linkage settings of the SVPs in this hot-standby switchover configuration:

Setting item

Settings for SVP

SVP on host 1

SVP on host 2

Port (5001-65535):

ppppp

ppppp

Fast reset assistance:

Disabled

Disabled

Cluster management:

Enabled

Enabled

Cluster manager config:

Enable#

Disable

#: Set Enable for only one of the SVPs in the system.

6.7 Configuring the reset path (model HA8000xM or earlier)

If the model of the system being used is HA8000xM or earlier, you must configure the reset path so that in the event of a host failure HA Monitor can reset the host.

Important

When a reset path is configured, HA Monitor checks whether it is possible to communicate with a failure management processor and VMware ESXi. Therefore, before you configure a reset path, make sure that the settings for a failure management processor and VMware ESXi have been specified completely.

You use an HA Monitor environment setup command (monsetup command) to configure the reset path. For details about the HA Monitor environment setup command (monsetup command), see 9.22 monsetup (specifies HA Monitor environment settings) in 9. Commands.

6.7.1 Information required for configuration (model HA8000xM or earlier)

You need the following information about the target hosts and SVP.

Figure 6‒18: Information required for configuring the reset path (model HA8000xM or earlier: not using VMware ESXi-based virtualization)

Figure 6‒19: Information required for configuring the reset path (model HA8000xM or earlier: using VMware ESXi-based virtualization)

The following table explains the specification items in the figures.

Table 6‒3: Information required for configuring the reset path (model HA8000xM or earlier)

No. in figure

Information required for configuration

Description

Remarks

1

Partition name

Specifies a management system name for the partition in which HA Monitor is run.

Specify any name consisting of from 1 to 31 alphanumeric characters.

--

2

IP address of reset path

Specifies the IP address of the reset path allocated to the target host.

For details about how to allocate IP addresses and port numbers, see 5.4.2 Necessary IP addresses and port numbers.

3

Port number of reset path

Specifies the port number of the reset path allocated to the target host.

4

IP address of SVP

Specifies the IP address of the reset path allocated to SVP.

5

Port number of SVP

Specifies the port number of the reset path allocated to SVP.

6

Machine address#1

Specifies any number that indicates the physical partition (processor) in which HA Monitor is run.

This must be the same value specified for the host in the same physical partition but different from one specified for a host in a different physical partition.

Make sure that this value is unique among all physical partitions that share the reset path.

7

IP address of VMware ESXi#1

Specifies the management IP address of VMware ESXi.

For details about how to allocate IP addresses and port numbers, see 5.4.2 Necessary IP addresses and port numbers.

8

Port number of VMware ESXi#1, #2

Specifies the port number assigned to VMware ESXi for SSH connection.

9

Password#1

Specifies the password required for authorization when SSH connection is established with VMware ESXi on the local host.

Specify the same password throughout the same hot-standby configuration.

10

Virtual machine name#1

Specifies the virtual machine name that is managed by VMware ESXi.

This is the name specified when you created virtual machines with VMware ESXi. Specify the name that was checked by VMware vSphere Client or vCenter.

This virtual machine name must be unique among all VMware ESXi systems that can be connected by HA Monitor.

The following describes how to check a virtual machine name:

Execute the ifconfig command on the guest OS of the virtual machine and make note of the information specific to the virtual machine (such as the MAC address).

Use VMware vSphere Client to connect to VMware ESXi.

Open the window for editing virtual machine settings and find the virtual machine with the same virtual machine-specific information that was obtained in step 1.

Legend:

--: Not applicable

#1

This information is required only if you use VMware ESXi-based virtualization.

#2

Although the password is stored in such a manner that only the root user can reference it, you must evaluate what security measures might be needed, such as physically disconnecting the local disk from external networks.

6.7.2 Examples of settings (model HA8000xM or earlier)

This subsection shows examples of reset path settings for model HA8000xM or earlier. In the tables in this subsection, Setting item corresponds to Information required for configuration in Table 6-3 Information required for configuring the reset path (model HA8000xM or earlier).

(1) When there is one pair of hot-standby configurations

This subsection presents an example of settings specified with the monsetup command for the following configuration (performing the hot standby operation between hosts 1 and 2).

Figure 6‒20: Example 1 of reset path settings (model HA8000xM or earlier)

Setting item

Settings on host

Host 1

Host 2

Partition name

MS0001

MS0002

IP address of reset path

x.x.x.a

x.x.x.b

Port number of reset path

ppppp

ppppp

IP address of SVP

x.x.x.c

x.x.x.d

Port number of SVP

qqqqq

qqqqq

(2) When there are two pairs of hot-standby configurations

This subsection explains an example of reset path settings in a system that runs under the following conditions:

The system contains two pairs of 1-to-1 switchover configurations that do not monitor each other.

Two pairs of hot-standby configurations share the reset path.

Two pairs of hot-standby configurations do not share the SVP.

Important

In the hot-standby configuration explained here, the specification of host address (address operand in the HA Monitor environment settings) requires special care. For details, see (2) Notes about sharing a reset path among hot-standby configurations.

The following shows an example of settings specified with the monsetup command for the configuration shown in the following figure (performing the hot standby operation between hosts 1 and 3 and between hosts 2 and 4) that satisfies all the above conditions.

Figure 6‒21: Example 2 of reset path settings (model HA8000xM or earlier)

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Partition name

MS0001

MS0002

MS0003

MS0004

IP address of reset path

x.x.x.a

x.x.x.c

x.x.x.e

x.x.x.g

Port number of reset path

ppppp

rrrrr

ppppp

rrrrrr

IP address of SVP

x.x.x.b

x.x.x.d

x.x.x.f

x.x.x.h

Port number of SVP

qqqqq

sssss

qqqqq

sssss

For this hot-standby configuration, the following shows an example of the address operand value in the HA Monitor environment settings:

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Host addresses of the local hosts

11

21

12

22

(3) VMware ESXi-based virtualization configuration

This subsection explains an example of reset path settings in a system that runs under the following conditions:

A VMware ESXi-based virtualization configuration is used.

The system contains two pairs of 1-to-1 switchover configurations that do not monitor each other.

Two pairs of hot-standby configurations share the reset path.

One pair of hot-standby configurations shares the SVP.

The following shows an example of settings specified with the monsetup command for the configuration shown in the following figure (performing the hot standby operation between hosts 1 and 3 and between hosts 2 and 4, and managing hosts 1, 2, and 3 by VMware ESXi) that satisfies all the above conditions.

Figure 6‒22: Example 3 of reset path settings (model HA8000xM or earlier)

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Partition name

MS0001

MS0001

MS0002

MS0003

IP address of reset path

x.x.x.a

x.x.x.b

x.x.x.d

x.x.x.f

Port number of reset path

ppppp

ppppp

ppppp

ppppp

IP address of SVP

x.x.x.c

x.x.x.c

x.x.x.e

x.x.x.g

Port number of SVP

qqqqq

qqqqq

qqqqq

qqqqq

Machine address

11

11

12

--

IP address of VMware ESXi

y.y.y.a

y.y.y.a

y.y.y.b

--

Port number of VMware ESXi

default#

default#

rrrrr

--

Virtual machine name

VM0001

VM0002

VM0003

--

Legend: --: Specification is not required.

#: If you use the port number for SSH connection (22), specify default.

In this configuration, specify use in the vmware_env operand in the HA Monitor environment settings (sysdef) on all hosts in the hot-standby configuration.

6.8 Configuring the reset path (model HA8000xN or later)

If the model of the system being used is HA8000xN or later, you must configure the reset path so that in the event of a host failure HA Monitor can reset the host.

Important

When a reset path is configured, HA Monitor checks whether it is possible to communicate with a failure management processor and VMware ESXi. Therefore, before you configure a reset path, make sure that:

The settings for a failure management processor and VMware ESXi have been specified completely.

HA Monitor Connector is running.

You use an HA Monitor environment setup command (monsetup command) to configure the reset path. For details about the HA Monitor environment setup command (monsetup command), see 9.22 monsetup (specifies HA Monitor environment settings).

6.8.1 Information required for configuration (model HA8000xN or later)

You need the following information about the target hosts and failure management processor.

Figure 6‒23: Information required for configuring the reset path (model: HA8000xN or later) (management server used as failure management processor, VMware ESXi-based virtualization not used)

Figure 6‒24: Information required for configuring the reset path (model: HA8000xN or later) (management server not used as failure management processor, VMware ESXi-based virtualization not used)

Figure 6‒25: Information required for configuring the reset path (model: HA8000xN or later) (management server used as failure management processor, VMware ESXi-based virtualization used)

Figure 6‒26: Information required for configuring the reset path (model: HA8000xN or later) (management server not used as failure management processor, VMware ESXi-based virtualization used)

The following table explains the specification items in the figures.

Table 6‒4: Information required for configuring the reset path (model HA8000xN or later)

No. in figure

Information required for configuration

Description

Remarks

1

Partition name

Specify the same value that is specified for System name (management system name of the partition in which HA Monitor operates) set for HA Monitor Option (HA Monitor Connector) on the management server.

Specify any name consisting of from 1 to 31 alphanumeric characters.

--

2

IP address of reset path

Specifies the IP address of the reset path allocated to the target host.

For details about how to allocate IP addresses and port numbers, see 5.4.2 Necessary IP addresses and port numbers.

3

Port number of reset path

Specifies the port number of the reset path allocated to the target host.

4

IP address of the failure management processor

If you use a management server as the failure management processor, specify the IP address of the management server. If you do not use a management server as the failure management processor, specify the IP address of the reset path. However, in a VMware-based virtualization environment in which HA Monitor and HA Monitor Option exist on separate virtual machines, do not specify the IP address of the reset path. In this case, specify the IP address of the reset path over the network to the virtual machine on which HA Monitor Option operates.

5

Port number of the failure management processor

Specify the same value that is specified for Port (port number of the management server for communication with HA Monitor) in the HA Monitor Connector configuration window.

6

Machine address#1

Specifies any number that indicates the physical partition (processor) in which HA Monitor is run.

This must be the same value specified for the host in the same physical partition but different from one specified for a host in a different physical partition.

Make sure that this value is unique among all physical partitions that share the reset path.

7

IP address of VMware ESXi#1

Specifies the management IP address of VMware ESXi.

For details about how to allocate IP addresses and port numbers, see 5.4.2 Necessary IP addresses and port numbers.

8

Port number of VMware ESXi#1, #2

Specifies the port number assigned to VMware ESXi for SSH connection.

9

Password#1

Specifies the password required for authorization when SSH connection is established with VMware ESXi on the local host.

Specify the same password throughout the same hot-standby configuration.

10

Virtual machine name#1

Specifies the virtual machine name that is managed by VMware ESXi.

This is the name specified when you created virtual machines with VMware ESXi. Specify the name that was checked by VMware vSphere Client or vCenter.

This virtual machine name must be unique among all VMware ESXi systems that can be connected by HA Monitor.

The following describes how to check a virtual machine name:

Execute the ifconfig command on the guest OS of the virtual machine and make note of the information specific to the virtual machine (such as the MAC address).

Use VMware vSphere Client to connect to VMware ESXi.

Open the window for editing the virtual machine settings, and then find the virtual machine that matches the information that you made note of in step 1.

Legend:

--: Not applicable

#1

This information is required only if you use VMware ESXi-based virtualization.

#2

Although the password is stored in such a manner that only the root user can reference it, you must evaluate what security measures might be needed, such as physically disconnecting the local disk from external networks.

6.8.2 Examples of settings (model HA8000xN or later)

This subsection shows examples of reset path settings for model HA8000xN or later. In the tables in this subsection, Setting item corresponds to Information required for configuration in Table 6-4 Information required for configuring the reset path (model HA8000xN or later).

(1) When there is one pair of hot-standby configurations

This subsection presents an example of settings specified with the monsetup command for the following configuration (performing the hot standby operation between hosts 1 and 2).

Figure 6‒27: Example 1 of configuring the reset path (model: HA8000xN or later) (management server used as the failure management processor)

Setting item

Settings on host

Host 1

Host 2

Partition name

MS0001

MS0002

IP address of reset path

x.x.x.a

x.x.x.b

Port number of reset path

ppppp

ppppp

IP address of the management server

x.x.x.c

x.x.x.c

Port number of the management server

qqqqq

qqqqq

Figure 6‒28: Example 2 of configuring the reset path (model: HA8000xN or later) (management server not used as the failure management processor)

Setting item

Settings on host

Host 1

Host 2

Partition name

MS0001

MS0002

IP address of reset path

x.x.x.a

x.x.x.b

Port number of reset path

ppppp

ppppp

IP address of the failure management processor

x.x.x.a

x.x.x.b

Port number of the failure management processor

qqqqq

qqqqq

(2) When there are two pairs of hot-standby configurations

This subsection explains an example of reset path settings in a system that runs under the following conditions:

The system contains two pairs of 1-to-1 switchover configurations that do not monitor each other.

Two pairs of hot-standby configurations share the reset path.

Important

In the hot-standby switchover configuration explained here, the specification of host address (address operand in the HA Monitor environment settings) requires special care. For details, see (2) Notes about sharing a reset path among hot-standby configurations.

The following shows an example of settings specified with the monsetup command for the configuration shown in the following figure (performing the hot standby operation between hosts 1 and 3 and between hosts 2 and 4) that satisfies all the above conditions.

Figure 6‒29: Example 3 of configuring the reset path (model: HA8000xN or later) (management server used as the failure management processor)

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Partition name

MS0001

MS0002

MS0003

MS0004

IP address of reset path

x.x.x.a

x.x.x.b

x.x.x.c

x.x.x.d

Port number of the reset path

ppppp

qqqqq

ppppp

qqqqq

IP address of the management server

x.x.x.e

x.x.x.e

x.x.x.e

x.x.x.e

Port number of the management server

rrrrr

sssss

rrrrrr

sssss

For this hot-standby configuration, the following shows an example of the address operand values in the HA Monitor environment settings:

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Host addresses of the local hosts

11

21

12

22

Figure 6‒30: Example 4 of configuring the reset path (model: HA8000xN or later) (management server not used as the failure management processor)

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Partition name

MS0001

MS0002

MS0003

MS0004

IP address of reset path

x.x.x.a

x.x.x.b

x.x.x.c

x.x.x.d

Port number of the reset path

ppppp

qqqqq

ppppp

qqqqq

IP address of the failure management processor

x.x.x.a

x.x.x.b

x.x.x.c

x.x.x.d

Port number of the failure management processor

rrrrr

sssss

rrrrr

sssss

(3) VMware ESXi-based virtualization configuration

This subsection explains an example of reset path settings in a system that runs under the following conditions:

A VMware ESXi-based virtualization configuration is used.

The system contains two pairs of 1-to-1 switchover configurations that do not monitor each other.

Two pairs of hot-standby configurations share the reset path.

The following shows an example of values to be specified in the monsetup command under the preceding conditions in a configuration. In this example, hosts 1 and 3 make up one hot-standby switchover pair, hosts 2 and 4 make up another hot-standby switchover pair, and hosts 1, 2, and 3 are managed by VMware ESXi.

Figure 6‒31: Example 5 of configuring the reset path (model: HA8000xN or later) (management server used as the failure management processor)

Setting item

Settings on host

Host 1

Host 2

Host 3

Host 4

Partition name

MS0001

MS0001

MS0002

MS0003

IP address of reset path

x.x.x.a

x.x.x.b

x.x.x.c

x.x.x.d

Port number of the reset path

ppppp

ppppp

ppppp

ppppp

IP address of the management server

x.x.x.e

x.x.x.e

x.x.x.e

x.x.x.e

Port number of the management server

rrrrr

rrrrr

rrrrr

rrrrr

Machine address

11

11

12

--

IP address of VMware ESXi

y.y.y.a

y.y.y.a

y.y.y.b

--

Port number of VMware ESXi

default#

default#

rrrrr

--

Virtual machine name

VM0001

VM0002

VM0003

--

Legend: --: Specification is not required.

#: If you use the port number for SSH connection (22), specify default.

In this configuration, specify use for the vmware_env operand in the HA Monitor environment settings (sysdef) on all hosts in the hot-standby switchover configuration.

6.8.3 Notes on configuration if a management server is used as the failure management processor (model: HA8000xN or later)

If you reconfigure HA Monitor Connector when, for example, adding a host or changing the configuration, you must stop the HA Monitor programs of all HA8000 servers set on the management server.

The following shows the configuration procedure:

Stop the HA Monitor programs of all HA8000 servers that are set in HA Monitor Connector.

The HA8000 system units do not need to be active, but power must be supplied to them.

Change the settings of HA Monitor Connector.

For details, see the documentation for HA Monitor Connector.

Use the HA Monitor environment setup command (monsetup command) to configure the reset path.

For details about the HA Monitor environment setup command (monsetup command), see 9.22 monsetup (specifies HA Monitor environment settings).

6.9 Configuring the monitoring paths

You must configure monitoring paths so that HA Monitor can monitor hosts and exchange information needed for hot standby processing.

6.9.1 Adding host names and service names

You must add the host names and service names of the TCP/IP LANs that are used as monitoring paths. This information must be added to the following files:

/etc/hosts file

/etc/services file

You must also specify these host and service names in the HA Monitor environment settings. This subsection explains examples of the information to be added and the settings in the files. For details about how to add information to the /etc/hosts and /etc/services files, see the OS documentation.

(1) Adding host names

Specify in the /etc/hosts file the host names of the TCP/IP LANs that are used for all monitoring paths. A unique host name must be specified for each host. For the IP addresses, do not specify alias IP addresses.

Also specify in the /etc/hosts file the IP addresses and host names of the HA Monitors on the remote hosts that are connected via the monitoring paths. If this information is omitted and the monitoring path status display command (monpath -i command) is executed, the host names corresponding to the IP addresses will not be displayed.

If you use HA Monitor Extension to change the maximum number of hosts and a desired multicast group ID for multicasting alive messages, specify in the /etc/hosts file the multicast group ID and the host name corresponding to the multicast group ID.

(2) Adding the service names

Specify in the /etc/services file the service names of the TCP/IP LANs that are used for all monitoring paths. The same names must be specified on all hosts in the hot-standby configuration.

For a port number, specify 5001 or a greater value that is not already being used in the system. The same port numbers must be specified on all hosts. For the protocol name, specify udp.

(3) Example of settings in the files

This subsection explains an example of the settings in the following files:

/etc/hosts file

/etc/services file

HA Monitor environment settings (sysdef)

For an example of the file settings that are required for using HA Monitor Extension to change the maximum number of hosts and multicasting alive messages, see (4) Example of file settings (for using HA Monitor Extension and multicasting alive messages).

(a) Monitoring path configuration used in the example

This example assumes that the monitoring paths are configured as shown in the following figure.

Figure 6‒32: Example of monitoring path configuration

(b) Example of settings in the /etc/hosts file

An example of the settings in the /etc/hosts file is shown below. The same settings are specified on hosts 1 and 2.

# IP address       Host name
192.168.0.11        path11
192.168.1.11        path12
192.168.0.12        path21
192.168.1.12        path22

This example specifies path11 and path12 as the monitoring paths used on host 1 and path21 and path21 as the monitoring paths used on host 2.

(c) Example of settings in the /etc/services file

An example of the settings in the /etc/services file is shown below. The same settings are specified on hosts 1 and 2.

# Service name     Port number/protocol name
HAmon1           7777/udp
HAmon2           7778/udp

This example uses the two services HAmon1 and HAmon2 to employ dual monitoring paths.

(d) Example of HA Monitor environment settings

An example of the HA Monitor environment settings (/opt/hitachi/HAmon/etc/sysdef) is shown below. The information specified for host 1 differs from that specified for host 2. Note that this example shows only the settings applicable to addition of host and service names.

Example of settings on host 1

environment
   lan     path11:path12,
   lanport HAmon1:HAmon2
;

Example of settings on host 2

environment
   lan     path21:path22,
   lanport HAmon1:HAmon2
;

Specify in the lan operand the monitoring path names specified in the /etc/hosts file. Specify in the lanport operand the service names specified in the /etc/services file. Make sure that the values specified in the lan and lanport operands correspond correctly to each other. In this example, path11 and path21 are specified first because these monitoring paths correspond to the service HAmon1.

For details about the lan and lanport operands, see 8.3.1 HA Monitor environment settings (sysdef).

(4) Example of file settings (for using HA Monitor Extension and multicasting alive messages)

This subsection explains examples of the settings in the following files when HA Monitor Extension is used to change the maximum number of hosts and to multicast alive messages:

/etc/hosts file

/etc/services file

HA Monitor environment settings (sysdef)

(a) Monitoring path configuration used in the example

This example assumes that the monitoring paths are configured as shown in the following figure.

Figure 6‒33: Example of monitoring path configuration

(b) Example of settings in the /etc/hosts file

An example of the settings in the /etc/hosts file is shown below. The same settings are specified on hosts 1 and 2.

# IP address       Host name
192.168.0.11        path11
192.168.1.11        path12
192.168.0.12        path21
192.168.1.12        path22
239.1.1.1           alive_id

This example specifies path11 and path12 as the monitoring paths used on host 1 and path21 and path21 as the monitoring paths used on host 2.

The example also specifies 239.1.1.1 as the multicast group ID and alive_id as the host name corresponding to this multicast group ID.

(c) Example of settings in the /etc/services file

An example of the settings in the /etc/services file is shown below. The settings specified on hosts 1 and 2 are the same.

# Service name     Port number/protocol name
HAmon1           7777/udp
HAmon2           7778/udp

This example uses the two services HAmon1 and HAmon2 to employ dual monitoring paths.

(d) Example of HA Monitor environment settings

An example of the HA Monitor environment settings (/opt/hitachi/HAmon/etc/sysdef) is shown below. The information specified for host 1 differs from that specified for host 2. Note that this example shows only the settings applicable to addition of host and service names when HA Monitor Extension is used to multicast alive messages.

Example of settings on host 1

environment
   lan               path11:path12,
   lanport           HAmon1:HAmon2,
   hostmax           33,
   alive_multicast   use,
   multicast_lan     alive_id
;

Example of settings on host 2

environment
   lan               path21:path22,
   lanport           HAmon1:HAmon2,
   hostmax           33,
   alive_multicast   use,
   multicast_lan     alive_id
;

Specify in the lan operand the monitoring path names specified in the /etc/hosts file. Specify in the lanport operand the service names specified in the /etc/services file. Make sure that the values specified in the lan and lanport operands correspond correctly to each other. In this example, path11 and path21 are specified first because these monitoring paths correspond to the HAmon1 service.

Specify in the hostmax operand the maximum number of hosts in one hot-standby configuration. Because this example sets the maximum number of hosts to more than 32, alive messages are multicast without having to specify use in the alive_multicast operand.

Specify in the multicast_lan operand alive_id as the host name corresponding to the multicast group ID.

For details about these operands, see 8.3.1 HA Monitor environment settings (sysdef).

6.9.2 Creating an HA Monitor connection configuration file

There is one HA Monitor per host and multiple HA Monitors communicate with each other between hosts. When you configure a system, you must define an HA Monitor connection configuration in a definition file. This definition file is called the HA Monitor connection configuration file.

The HA Monitor connection configuration file is referenced when HA Monitors are started, and the HA Monitors are connected with each other according to the contents of the HA Monitor connection configuration file. The contents of HA Monitor connection configuration file are checked when HA Monitors are started and when the definition check command (moncheck command) is executed.

(1) How to create an HA Monitor connection configuration file

You can create an HA Monitor connection configuration file automatically or manually. Normally, we recommend that you create this file automatically. However, if either of the following is applicable, create the file manually:

Monitoring paths are to be configured in such a manner that the local host and a specific host are monitored mutually

In this case, you must create the HA Monitor connection configuration file manually because you must specify the pair option in this file.

Changing or deleting an HA Monitor connection configuration

However, if you are simply adding an HA Monitor, you can create the HA Monitor connection configuration file automatically.

This subsection explains how to create an HA Monitor connection configuration file manually. The automatic creation method is not explained here because you can create the HA Monitor connection configuration file when you check the system operation after completing the system configuration. For details about how to create an HA Monitor connection configuration file automatically, see (3) Checking termination of HA Monitors and checking the connection configuration file (that is created automatically).

(2) Manual creation procedure

You create an HA Monitor connection configuration file under the directory for HA Monitor environment settings using connection for the file name. We recommend that you create one connection file, and then copy it to all the hosts.

The following shows the format of the parameters that are specified in the HA Monitor connection configuration file:

Δ₀host-nameΔ₁LANΔ₁IP-addressΔ₁pair;↓

Legend:

Δ₀: Indicates 0 or more spaces or tabs.

Δ₁: Indicates 1 or more spaces or tabs.

↓: Indicates an end-of-line code.

Make sure that each line ends with a semicolon (;). You specify the settings for one host per line. If there are four hosts, you must specify four lines.

Note that a hash mark (#) indicates that the text from the hash mark through the end-of-line code constitutes a comment.

Important

If there is an error in a connection configuration file in which pair is specified, correct hot standby operation cannot be performed because the correct connection configuration between hosts cannot be recognized.

When you use a connection configuration file in which pair is specified, you cannot move a server to another host because the pair relationship between servers cannot be changed. If you will be moving a server to another host, you must re-create the connection configuration file, and then restart the HA Monitors.

The following explains the items that are specified in the connection configuration file.

# HA Monitor connection configuration settings
host-name LAN IP-address [pair];
         :

host-name

Specifies the host name of a host. All hosts, including the local host, in the hot-standby configuration must be specified. You can specify the host names in any order, but they must be host names specified in the name operand in the HA Monitor environment settings (not the host names specified in the /etc/hosts file).

LAN IP-address

Specifies the reserved word LAN and the IP address of the TCP/IP LAN used as the monitoring path. You must specify the IP addresses of all LANs if you use multiple monitoring paths.

If you specify pair, specify all the monitoring paths that are actually used for monitoring.

pair

When pair is specified, it works as follows:

The hot-standby configuration connects only to hosts for which pair is specified.

When the active server is starting, if you cannot check the status of a host for which pair is specified, it enters the active server start wait state so that the active server does not start on more than one host.

In the following case, pair must be specified because there is a possibility that the active server might start on both the primary and secondary systems:

You restart HA Monitor on the primary system in a state of planned hot standby to the secondary system, and the active server starts before the monitoring path connects, or no connection is possible due to monitoring path failure.

For example, this applies in cases where the server starts automatically when the OS and HA Monitor start.

Note that even if a host for which pair is specified is not yet running, it will be in the active server start wait state. To continue the startup of the active server, first confirm that the host for which pair is specified is not yet running, or that no active server exists, and then execute the monact command.

If you specify pair, you must specify it for all monitoring paths on all hosts in the hot-standby configuration. Note that the pair specification cannot be generated automatically. You must specify it manually.

When you specify pair, you can perform health checking on the monitoring paths and execute HA monitor commands only between hosts for which pair is specified. Therefore, less time is required to perform health checking and execute HA Monitor commands compared to when pair is omitted. The following HA Monitor commands are executed only between hosts for which pair is specified:

Server and host status display command (monshow command)

Reset path status display command (monrp command)

This command is used when the system being used is BladeSymphony or HA8000.

Monitoring path status display command (monpath command

HA Monitors manual connection command (monlink command)

(3) Examples

This subsection presents examples of creating HA Monitor connection configuration files.

(a) When all hosts monitor mutually

The following figure shows an example of creating an HA Monitor connection configuration file when all hosts monitor mutually.

Figure 6‒34: Example of creating an HA Monitor connection configuration file when all hosts monitor mutually

(b) When some hosts do not monitor mutually

The figure below shows an example of creating an HA Monitor connection configuration file when there are hosts that do not perform mutual monitoring on a monitoring path. In this example, pair is specified for the hosts that perform mutual monitoring.

Figure 6‒35: Example of creating an HA Monitor connection configuration file when there are hosts that do not perform mutual monitoring

Because pair is specified for the hosts that perform mutual monitoring, less time is required for performing health checking and for executing HA Monitor commands.

In this example, hosts 1 and 3 do not monitor mutually.

6.10 Creating definition files (HA Monitor)

You must create definition files for specifying the environment settings for HA Monitor.

For details, see 8.3 Configuring an environment for HA Monitor.

When you use HA Monitor Extension, you must set up an environment for the functions of HA Monitor Extension on all hosts in the hot-standby configuration. For details about setting up an environment for the functions of HA Monitor Extension, see 3.7 Functions of HA Monitor Extension.

6.11 Configuring the shared resources to be used by servers

You must configure shared resources so that the servers can use them. You need to specify these settings only if servers use shared resources.

6.11.1 Configuring a shared disk

The configuration of a shared disk depends on LVM is used for switching. The following table provides an overview of the shared disk configurations. Note that, in the following table, connecting volume groups for referencing and updating has the same meaning as activating volume groups.

Table 6‒5: Overview of shared disk configurations

Required setting

Shared disk switching method

Remarks

LVM

HA Booster

Configuring volume groups

Creating volume groups

Create the volume groups to be switched.

Create the volume groups on the device that is to be registered in the control group.#1

See (1) Configuring volume groups.

Editing the /etc/rc.sysinit file (or the /etc/lvm/lvm.conf file#2).

Set the volume groups that are not to be connected automatically for referencing and updating when the system starts.

Set the volume groups that are to be connected automatically for referencing and updating when the system starts.#1

Specifying file systems

Specify settings.

Specify settings.

Required if file systems are to be switched.

See (2) Specifying file systems.

Specifying character special files

Create an HA Monitor's user command to bind a character special file at the time of system startup and unbind at the time of server shutdown.

Bound at the time of system startup.

Required if character special files are to be switched.

See (3) Specifying character special files.

#1

Applicable when LVM is also used.

#2

If the OS is RHEL7 or a later version, edit the /etc/lvm/lvm.conf file.

(1) Configuring volume groups

(a) Creating volume groups

Creating volume groups that are to be switched by LVM

The system administrator uses an OS command to create volume groups because switchover occurs on the shared disk in units of volume groups.

Note the following about creating volume groups:

To create a hot-standby switchover configuration with different servers, configure shared disks so that the data used by each server belongs to a different volume group.

If a resource server is used for a server group, you can place the data to be used by all the servers in the server group in the same volume group.

For a volume group created on a shared disk, assign the same name on both hosts.

HA Monitor controls the volume groups created on the shared disk. Therefore, the volume groups must be set to be not connected automatically for referencing and updating when the system starts. If the OS is RHEL7 or a later version, edit the /etc/lvm/lvm.conf file. If the OS is RHEL6, edit the /etc/rc.sysinit file.

(b) Editing the /etc/rc.sysinit file

This subsection explains editing of the /etc/rc.sysinit file.

When using LVM for switchover processing

Search the file for the lines that contain "/sbin/lvm vgchange -a y" or "/sbin/lvm vgchange -a ay", and then delete "/sbin/lvm.static vgchange -a y --ignorelockingfailure".

If a command to be executed by action is deleted as a result of the above deletion, either delete action itself or comment it out. If deletion of action results in deletion of a command that is executed by the then clause of the if command, insert a null line by using a colon (:) to avoid shell syntax errors.

If the volume group contains a resource other than the shared disk to which you want to connect automatically for referencing and updating at system startup, add the following information immediately after the deleted line: "/sbin/lvm.static vgchange -a y name-of-volume-group-to-be-connected --sysinit" or "/sbin/lvm vgchange -a ay name-of-volume-group-to-be-connected --sysinit".

Important

Use special caution when editing the /etc/rc.sysinit file. If you edit it incorrectly, the system might be unable to start. For example, the system becomes unable to start in the following cases:

The /etc/rc.sysinit file is incorrectly edited and a shell syntax error occurs.

LVM is allocated in /root, /home, or another area that is required for startup and connection by the vgchange -a y command is not specified in the /etc/rc.sysinit file.

After editing the /etc/rc.sysinit file, if you update the initscripts package of Red Hat Enterprise Linux, the /etc/rc.sysinit file might be overwritten. If the file is overwritten, the changes to the file are lost. If the changes to the /etc/rc.sysinit file are lost, you must edit it again.

The following subsection presents examples of editing the /etc/rc.sysinit file.

(c) Examples of editing the /etc/rc.sysinit file (when using LVM)

This subsection presents examples of editing the /etc/rc.sysinit file when using LVM.

The following show an example when the OS being used is Red Hat Enterprise Linux 6.1.

Before editing (when all volume groups in the system are to be connected automatically for referencing and updating)

...
...
if [ -x /sbin/lvm ]; then
    action $"Setting up Logical Volume Management:" /sbin/lvm vgchange -a y --sysinit
fi
...
...

After editing (when the volume group is not to be connected automatically for referencing and updating)

.
.
  if [ -x /sbin/lvm ]; then
#     action $"Setting up Logical Volume Management:" /sbin/lvm vgchange -a y --sysinit
    :
  fi
.
.

After editing (when vg01 and vg02 are to be connected automatically for referencing and updating)

.
.
if [ -x /sbin/lvm ]; then
#    action $"Setting up Logical Volume Management:" /sbin/lvm vgchange -a y --sysinit
     /sbin/lvm vgchange -a y vg01 --sysinit
     /sbin/lvm vgchange -a y vg02 --sysinit
    :
fi
.
.

The following shows an example when the OS being used is Red Hat Enterprise Linux 6.4.

Before editing (when all volume groups in the system are to be connected automatically for referencing and updating)

...
...
if [ -x /sbin/lvm ]; then
    action $"Setting up Logical Volume Management:" /sbin/lvm vgchange -a ay --sysinit
fi
...
...

After editing (when not connecting the volume groups automatically for referencing and updating)

.
.
if [ -x /sbin/lvm ]; then
#     action $"Setting up Logical Volume Management:" /sbin/lvm vgchange -a ay --sysinit
     :
fi
.
.

After editing (when vg01 and vg02 are to be connected automatically for referencing and updating)

...
...
if [ -x /sbin/lvm ]; then
#    action $"Setting up Logical Volume Management:" /sbin/lvm vgchange -a ay --sysinit
     /sbin/lvm vgchange -a ay vg01 --sysinit
     /sbin/lvm vgchange -a ay vg02 --sysinit
fi
...
...

The following shows an example of when the OS being used is Red Hat Enterprise Linux 6.6, 6.7, 6.8, 6.9, or 6.10.

Before editing (when all volume groups in the system are to be connected automatically for referencing and updating)

...
...
if [ -x /sbin/lvm ]; then
    if [ ! -f /.nolvm ] && ! strstr "$cmdline" nolvm ; then
        action $"Setting up Logical Volume Management:" /sbin/lvm vgchange -a ay --sysinit --ignoreskippedcluster
    else
        echo $"Logical Volume Management disabled at boot."
    fi
fi
...
...

After editing (when not connecting the volume groups automatically for referencing and updating)

...
...
if [ -x /sbin/lvm ]; then
    if [ ! -f /.nolvm ] && ! strstr "$cmdline" nolvm ; then
#        action $"Setting up Logical Volume Management:" /sbin/lvm vgchange -a ay --sysinit --ignoreskippedcluster
   ...
    else
        echo $"Logical Volume Management disabled at boot."
    fi
fi
...
...

After editing (when vg01 and vg02 are to be connected automatically for referencing and updating)

...
...
if [ -x /sbin/lvm ]; then
    if [ ! -f /.nolvm ] && ! strstr "$cmdline" nolvm ; then
#        action $"Setting up Logical Volume Management:" /sbin/lvm vgchange -a ay --sysinit --ignoreskippedcluster
         /sbin/lvm vgchange -a ay vg01 --sysinit --ignoreskippedcluster
         /sbin/lvm vgchange -a ay vg02 --sysinit --ignoreskippedcluster
    else
        echo $"Logical Volume Management disabled at boot."
    fi
fi
...
...

(d) Editing the /etc/lvm/lvm.conf file

Edit the "auto_activation_volume_list" entry in the /etc/lvm/lvm.conf file so that the shared disks controlled by HA Monitor are controlled as follows:

Volume groups controlled by HA Booster (volume groups on a device registered in a control group):

The volume groups are automatically connected for referencing and updating (that is, activated) when the system starts.

Volume groups not controlled by HA Booster:

The volume groups are not automatically connected for referencing and updating (that is, activated) when the system starts.

(e) Examples of editing the /etc/lvm/lvm.conf file

This subsection presents examples of editing the /etc/lvm/lvm.conf file when using LVM.

The following shows an example when the OS being used is Red Hat Enterprise Linux Server 7.1.

Before editing (when all volume groups in the system are to be connected automatically for referencing and updating)

activation {
...
...
# If auto_activation_volume_list is defined but empty, no volumes will be
# activated automatically and --activate ay/-a ay will do nothing.
#
# auto_activation_volume_list = []

# If auto_activation_volume_list is defined and it's not empty, only matching
# volumes will be activated either automatically or by using --activate ay/-a ay.
#
# "vgname" and "vgname/lvname" are matched exactly.
# "@tag" matches any tag set in the LV or VG.
# "@*" matches if any tag defined on the host is also set in the LV or VG
#
# auto_activation_volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
...
...
}

After editing (when not connecting the volume groups automatically for referencing and updating)

activation {
...
...
# If auto_activation_volume_list is defined but empty, no volumes will be
# activated automatically and --activate ay/-a ay will do nothing.
#
# auto_activation_volume_list = []

# If auto_activation_volume_list is defined and it's not empty, only matching
# volumes will be activated either automatically or by using --activate ay/-a ay.
#
# "vgname" and "vgname/lvname" are matched exactly.
# "@tag" matches any tag set in the LV or VG.
# "@*" matches if any tag defined on the host is also set in the LV or VG
#
# auto_activation_volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]

auto_activation_volume_list = []
...
...
}

After editing (when vg01 and vg02 are to be connected automatically for referencing and updating)

activation {
...
...
# If auto_activation_volume_list is defined but empty, no volumes will be
# activated automatically and --activate ay/-a ay will do nothing.
#
# auto_activation_volume_list = []

# If auto_activation_volume_list is defined and it's not empty, only matching
# volumes will be activated either automatically or by using --activate ay/-a ay.
#
# "vgname" and "vgname/lvname" are matched exactly.
# "@tag" matches any tag set in the LV or VG.
# "@*" matches if any tag defined on the host is also set in the LV or VG
#
# auto_activation_volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]

auto_activation_volume_list = [ "vg01", "vg02" ]
...
...
}

The following shows an example when the OS being used is Red Hat Enterprise Linux Server 7.2.

Before editing (when all volume groups in the system are to be connected automatically for referencing and updating)

# Configuration section activation.
activation {
...
...
        # Configuration option activation/auto_activation_volume_list.
        # Only LVs selected by this list are auto-activated.
...
...
        # Example
        # volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
        #
        # This configuration option does not have a default value defined.
...
...
}

After editing (when not connecting the volume groups automatically for referencing and updating)

# Configuration section activation.
activation {
...
...
        # Configuration option activation/auto_activation_volume_list.
        # Only LVs selected by this list are auto-activated.
...
...
        # Example
        # volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
        #
        # This configuration option does not have a default value defined.

    auto_activation_volume_list = []
...
...
}

After editing (when vg01 and vg02 are to be connected automatically for referencing and updating)

# Configuration section activation.
activation {
...
...
        # Configuration option activation/auto_activation_volume_list.
        # Only LVs selected by this list are auto-activated.
...
...
        # Example
        # volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
        #
        # This configuration option does not have a default value defined.

    auto_activation_volume_list = [ "vg01", "vg02" ]
...
...
}

The following shows an example of when the OS being used is Red Hat Enterprise Linux Server 7.3 to 7.7, 8.1, or 8.2.

Before editing (when all volume groups in the system are to be connected automatically for referencing and updating)

# Configuration section activation.
activation {
...
...
# Configuration option activation/auto_activation_volume_list.
# Only LVs selected by this list are auto-activated.
...
...
# Example
# auto_activation_volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
#
# This configuration option does not have a default value defined.
...
...
}

After editing (when not connecting the volume groups automatically for referencing and updating)

# Configuration section activation.
activation {
...
...
# Configuration option activation/auto_activation_volume_list.
# Only LVs selected by this list are auto-activated.
...
...
# Example
# auto_activation_volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
#
# This configuration option does not have a default value defined.
auto_activation_volume_list = []
...
...
}

After editing (when vg01 and vg02 are to be connected automatically for referencing and updating)

# Configuration section activation.
activation {
...
...
# Configuration option activation/auto_activation_volume_list.
# Only LVs selected by this list are auto-activated.
...
...
# Example
# auto_activation_volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
#
# This configuration option does not have a default value defined.
auto_activation_volume_list = [ "vg01", "vg02" ]
...
...
}

(2) Specifying file systems

If you use file systems on the shared disk, you must specify the file systems for the OS.

HA Monitor controls switching of file systems by mounting and unmounting them. Therefore, you must set the file systems to not be mounted automatically at the time of system startup. For details about the specification method, see the OS documentation.

Note the following points when you specify a file system to be controlled by HA Monitor in the /etc/fstab file:

In the third field (file system type), do not specify ignore. If ignore is specified, the fsck command executed by HA Monitor might fail, disabling the file system.

Omit the sixth field (fsck sequence at OS startup) or specify 0 in the field. If 1 or a larger value is specified and the OS of the standby system starts while the file system is being mounted in the primary system, fsck is performed. This might corrupt the data on the shared disk.

(3) Specifying character special files

For character special files, the necessity and timing of binding and unbinding depend on the condition of using them.

The following table describes the conditions for using character special files and the necessity of binding and unbinding.

Table 6‒6: Conditions for using character special files and necessity of binding and unbinding

Condition for using character special file

Necessity of binding and unbinding

On a shared disk that is not in a control group managed by HA Booster

A character special file is bound not at system startup, but when a volume group is connected for referencing and updating. It is unbound when the volume group is disconnected. An HA Monitor user command is used to specify the settings.

Because a character special file must be unbound before the volume group is disconnected, you must specify reverse in the deviceoff_order operand# in the HA Monitor environment settings and the server environment definition.

Specify the settings for all servers that use the character special file.

#

For details about the deviceoff_order operand in the HA Monitor environment settings, see 8.3.1 HA Monitor environment settings (sysdef). For details about the deviceoff_order operand in the server environment definition, see 8.4.1 Server environment definition (servers).

The subsections below explain the user commands related to specification of character special files. For details about how to create user commands, see 6.18 Creating user commands.

(a) Timing of issuance of a user command

Specify a user command in such a manner that the raw OS command is issued at the following times:

Binding a character special file

When the active server starts (-s start) (server type: online)

When hot standby processing starts on a standby server

When a server failure occurs on the active server (-a start) (server type: standby)

When a host failure occurs in the active system (-h start) (server type: standby)

When planned hot standby processing is executed from the active system to a standby system (-w start) (server type: standby)

Unbinding a character special file

When the active server terminates

When a server failure occurs on the active server (-a start) (server type: online)#

When planned hot standby processing is executed from the active system to a standby system (-w start) (server type: online)

When a server failure occurs on the active server (hot standby is disabled) (-o start) (server type: online)

When the active server terminates normally (-e start) (server type: online)

When the active server is terminated by planned termination (-p start) (server type: online)#

When the standby server starts (-s start) (server type: standby)

When hot standby processing fails on a standby server (-f start) (server type: standby)

#

If the server is in the monitor mode or it is a resource server, no parameters are passed. Therefore, unbinding is not necessary at this timing.

(b) Coding example for a user command

This coding example for a user command binds a character special file (/dev/raw/raw1) to a logical volume (lvol0) on a shared disk (/dev/vg10) for a server (LinuxTP1).

This code is stored under the file name usrcommand_raw_bind in HA Monitor's sample file directory.

#! /bin/sh

# The definition of server
SERV="LinuxTP1"                  # server name:LinuxTP1
# user name
USER_NAME="TP1user"              # user name, TPI administrator, or HiRDB administrator
# The definition of device
RAW_NAME="/dev/raw/raw1"         # raw device name
DEV_NAME="/dev/vg10/lvol0"       # the name of logical volume on shared disk

# Unbinding the raw device
func_unbind(){
# Wait to unbind the raw device.
COUNT=0
while [ \( ${COUNT} -lt 10 \) -a \( -e "$1" \) ]
do
   # Kill processes accessing the raw device.
   fuser -k "$1" >/dev/null 2>&1
   # Wait for 1 second until processing finishes.
   sleep 1
   # Unbind the raw device.
   raw "$1" 0 0
   COUNT=`expr ${COUNT} + 1`
done
}

# Binding the raw device
func_bind(){
# Bind the raw device.
raw "$1" "$2"
# Wait to bind the raw device.
COUNT=0
while [ ${COUNT} -lt 10 ]
do
   # If the raw device can be read, finish binding.
   dd of=/dev/null if="$1" bs=1 count=0 >/dev/null 2>&1
   if [ "$?" = 0 ]
   then
     break
   fi
   # Wait for 1 second until the raw device sets.
   sleep 1
   COUNT=`expr ${COUNT} + 1`
done
# Change the owner of the device
chown $USER_NAME "$1"
}
# Main
# Bind the raw device, when the online server starts or Hotstandby is started.
# Unbind the raw device, when the online server ends, standby server starts
# or Hotstandby is failed.
PATH=/bin:/usr/bin:/sbin
export PATH
if [ "$2" = "${SERV}" ]
then
   if [ "$4" = "online" ]
   then
     case "$5" in
     "-s" )                      # online server startup
       if [ "$6" = "start" ]
       then
         func_bind ${RAW_NAME} ${DEV_NAME}
       fi
     ;;
     "-a" | "-w" | "-o" | "-e" | "-p" )  # online server end
                                 # (server error | planned system switch
                                 # | server error (no standby server)
                                 # | normal end | planned stop)
       if [ "$6" = "start" ]
       then
         func_unbind ${RAW_NAME} ${DEV_NAME}
       fi
     ;;
     esac
   else
     case "$5" in
     "-a" | "-h" | "-w" )        # Standby server is done into Hotstandby.
                                 # (server error | host down | planned system switch)
       if [ "$6" = "start" ]
       then
         func_bind ${RAW_NAME} ${DEV_NAME}
       fi
     ;;
     "-s" | "-f" )               # standby server startup or Hotstandby fail
       if [ "$6" = "start" ]
       then
         func_unbind ${RAW_NAME} ${DEV_NAME}
       fi
     ;;
     esac
   fi
fi

This coding example binds a character special file by executing func_bind in the active system when the active server starts or on the target host when hot standby processing is performed. It also unbinds the character special file by executing func_unbind in the active system when the active server terminates, in a standby system when the standby server starts, or in a standby system resulting in a hot standby error in the event that hot standby processing fails.

func_bind uses the raw command to bind a character special file. After that, func_bind retries the following processing once a second (for a maximum of 10 seconds) until the character special file becomes accessible:

dd command

Checks whether the character special file has become accessible.

sleep command

Waits until the character special file is bound.

chown command

Changes the owner of the character special file. Note that if the OS is RHEL6 or later, you do not need to use a user command to change the owner. You can change the owner by using the udev function of the OS. For details about the udev settings, see the documentation for the OS.

In addition, the owner of the bound character special file is set to the specified user.

func_unbind retries the following processing once a second (for a maximum of 10 seconds) until the character special file is unbound:

fuser command

If a process is using the character special file during hot standby processing, disconnection of the shared disk fails because the character special file cannot be unbound. Therefore, the fuser command is used to forcibly terminate all processes that are using the character special file.

sleep command

Waits until the character special file is unbound.

raw command

Unbinds the character special file.

Note that in this coding example, if bind processing takes 10 seconds or longer for any reason, the chown command, which is executed last, fails to change the privilege. If the settings are specified so that server startup processing continues#, the user cannot notice the error until the server accesses a raw device.

#

uoc_neck is omitted or nouse is specified in the server environment settings.

To prevent this, consider the following measures:

In the server environment settings, specify use for uoc_neck so that server startup processing stops when the chown command fails to change the privilege.

In the sample user command, the result of the chown command, which is executed last, is the result of the user command. However, if you change the sample processing (for example, processing is added after the chown command), make sure that the results of the chown command and other processing are reflected to the final result of the user command.

If you do not set use for uoc_neck in the server environment settings, add processing that outputs a message to syslog or another location to the user command as a means for error notification.

If necessary, change the interval at which processing is retried until the character special file becomes accessible.

6.11.2 Specifying LAN status settings files

In HA Monitor, a LAN is connected and disconnected for each server. You must create LAN status settings files and then specify the files.

There are two types of LAN status settings files:

server-alias-name.up file

This file is used when a LAN is connected. It specifies the information needed for assigning to the LAN adapter the IP address used by the server.

server-alias-name.down file

This file is used when a LAN is disconnected. It specifies the information needed for deleting the IP address used by the server from the LAN adapter.

You must create these files under the directory for HA Monitor environment settings for each server. For server-alias-name in the file names, specify the value that was specified in the alias operand in the server environment definition.

HA Monitor provides sample LAN status settings files named server.up, server.down, server_ip.up, and server_ip.down in HA Monitor's sample file directory. If you copy these files to the directory for HA Monitor environment settings, you can save time in creating your LAN status settings files. Assign execution permissions to the files.

HA Monitor uses alias IP addresses to switch LANs. In the LAN status settings files, you specify these alias IP addresses, which are to be added to or deleted from LAN adapters, as arguments of the ifconfig, ip, and arping OS commands. For details about the OS commands, see the OS documentation.

If the OS is RHEL7 or a later version

Use either of the following combinations of sample LAN status settings files according to the command to be used:

ip command: server_ip.up and server_ip.down

ifconfig command: server.up and server.down

If the OS is RHEL6

Use the ifconfig command. Use the following combination of sample LAN status settings files:

server.up and server.down

This subsection explains how to specify LAN status settings files by using alias IP addresses.

(1) Specifying the server-alias-name.up file

In the case where the ip command is used in RHEL7 or a later version

In the server-alias-name.up file, specify the interface name, alias IP address, and prefix (length of bits indicating the network) of the LAN adapter allocated to each host. If necessary, also specify the broadcast address and number of retries. After you have added an alias IP address, execute the echo and arping commands. The echo command clears the routine cache, and the arping command broadcasts an ARP request and updates the mapping of IP addresses and MAC addresses. Pay particular attention to the broadcast timings of ARP requests. In such a case, see (c) ARP request broadcast timings and take appropriate action.

The settings to be specified in the server-alias-name.up file differ depending on whether hbonding is used in RHEL7 or a later version.

If hbonding is not used in RHEL7 or a later version:

The following shows the contents of a server-alias-name.up file.

#!/bin/sh
set -x

INTERFACE=eXXX                     # LAN interface name
IPADDR=n.n.n.n                       # alias IP address
PREFIX=ee                           # address prefix
BROADCAST=nnn.nnn.nnn.nnn            # broadcast address
RETRYCNT=m                         # retry count

# Retry for the alias IP address adding.
exitcode=1
loopcnt=0
while [ $loopcnt -lt $RETRYCNT ]
do
    # The alias IP address is added to the LAN interface.
    /usr/sbin/ip addr add $IPADDR/$PREFIX broadcast $BROADCAST dev $INTERFACE

    # The information on old routing cache is deleted.
    /bin/echo 0 > /proc/sys/net/ipv4/route/flush

    # The ARP caches are updated.
    /sbin/arping -U -c 2 -I $INTERFACE $IPADDR
    sleep 1
    /sbin/arping -U -c 2 -I $INTERFACE $IPADDR

    # Was the alias IP address registered to the system ?
    RCD=`/usr/sbin/ip addr show dev $INTERFACE | grep " $IPADDR/$PREFIX "`
    if [ "$RCD" = "" ]
    then
        # When it is not registered, "1" is returned as the termination code.
        exitcode=1

        loopcnt=`/usr/bin/expr $loopcnt + 1`
    else

        # When it is registered, "0" is returned as the termination code.
        exitcode=0

        break

    fi
done

exit $exitcode

Legend:

eXXX: Interface name

n.n.n.n: Alias IP address

ee: Prefix

nnn.nnn.nnn.nnn: Broadcast address

m: Retry count

Taking into account occurrence of a temporary failure, we recommend that you specify 2 or a larger value for the number of retries (m).

If hbonding is used in RHEL7 or a later version:

The following shows the contents of a server-alias-name.up file. Note that, in this case, you must use the label specification to assign an IP address to the alias device. Also note that you must specify a physical device when executing the arping command.

#!/bin/sh
set -x

INTERFACE=hbondX                    # LAN interface name
IF_LABEL=hbondX:Y                   # alias device name
IPADDR=n.n.n.n                      # alias IP address
PREFIX=ee                           # address prefix
BROADCAST=nnn.nnn.nnn.nnn           # broadcast address
RETRYCNT=m                          # retry count

# Retry for the alias IP address adding.
exitcode=1
loopcnt=0
while [ $loopcnt -lt $RETRYCNT ]
do
    # The alias IP address is added to the LAN interface.
    /usr/sbin/ip addr add $IPADDR/$PREFIX broadcast $BROADCAST dev $INTERFACE label $IF_LABEL

    # The information on old routing cache is deleted.
    /bin/echo 0 > /proc/sys/net/ipv4/route/flush

    # The ARP caches are updated.
    /sbin/arping -U -c 2 -I $INTERFACE $IPADDR
    sleep 1
    /sbin/arping -U -c 2 -I $INTERFACE $IPADDR

    # Was the alias IP address registered to the system ?
    RCD=`/usr/sbin/ip addr show dev $INTERFACE | grep " $IPADDR/$PREFIX "`
    if [ "$RCD" = "" ]
    then
        # When it is not registered, "1" is returned as the termination code.
        exitcode=1

        loopcnt=`/usr/bin/expr $loopcnt + 1`
    else
        # When it is registered, "0" is returned as the termination code.
        exitcode=0

        break

    fi
done

exit $exitcode

Legend:

hbondX: Interface name

hbondX:Y: Alias device name

n.n.n.n: Alias IP address

ee: Prefix

nnn.nnn.nnn.nnn: Broadcast address

m: Retry count

Taking into account occurrence of a temporary failure, we recommend that you specify 2 or a larger value for the number of retries (m).

If the ifconfig command is used:

In the server-alias-name.up file, specify the interface name, IP index number, and alias IP address of the LAN adapter allocated to each host. If necessary, also specify a netmask, broadcast address, and number of retries. After you have added an alias IP address, execute the echo and arping commands. The echo command clears the routine cache, and the arping command broadcasts an ARP request and updates the mapping of IP addresses and MAC addresses. Pay particular attention to the broadcast timings of ARP requests. In such a case, see (c) ARP request broadcast timings and take appropriate action.

The following shows the contents of a server-alias-name.up file:

#!/bin/sh
set -x

INTERFACE=xxx:y                       # LAN interface name
IPADDR=n.n.n.n                        # alias IP address
NETMASK=nn.nn.nn.nn                  # netmask address
BROADCAST=nnn.nnn.nnn.nnn            # broadcast address
RETRYCNT=m                            # retry count

# Retry for the alias IP address adding.
exitcode=1
loopcnt=0

while [ $loopcnt -lt $RETRYCNT ]
do
    # The alias IP address is added to the LAN interface.
    /sbin/ifconfig $INTERFACE inet $IPADDR netmask $NETMASK broadcast $BROADCAST

    # The information on old routing cache is deleted.
    /bin/echo 0 > /proc/sys/net/ipv4/route/flush

    # The ARP caches are updated.
    IFNAME=`echo $INTERFACE | /bin/sed -e 's/:[0-9]*$//'`
    /sbin/arping -U -c 2 -I $IFNAME $IPADDR
    sleep 1
    /sbin/arping -U -c 2 -I $IFNAME $IPADDR

    # Was the alias IP address registered to the system ?
    RCD=`/sbin/ifconfig -a | /bin/grep -E "[:| ]$IPADDR "`
    if [ "$RCD" = "" ]
    then
        # When it is not registered, "1" is returned as the termination code.
        exitcode=1

        loopcnt=`/usr/bin/expr $loopcnt + 1`
    else
        # When it is registered, "0" is returned as the termination code.
        exitcode=0

        break
    fi
done

exit $exitcode

Legend:

xxx: Interface name

y: IP index number

n.n.n.n: Alias IP address

nn.nn.nn.nn: Netmask

nnn.nnn.nnn.nnn: Broadcast address

m: Retry count

If you use multiple alias IP addresses for a single interface, you must specify a different IP index number for each alias IP address.

If there is only one LAN status settings file or alias IP addresses will not be assigned to the same interface by multiple LAN status settings files, there is no need for retries. In this case, a server-alias-name.up file that was used with version 01-52 or earlier can also be used. However, we recommend that you specify 2 or a larger value for the number of retries (m), taking into account occurrence of a temporary failure.

(a) Overview of retry processing

This subsection provides an overview of retry processing by using the ifconfig command. Note that the information in this subsection does not apply if the ip command is used.

Be careful if there are multiple LAN status settings files that assign alias IP addresses to the same interface and these LAN status settings files are processed at the same time#. In such a case, one of the ifconfig commands might fail to assign alias IP addresses. For example, this problem might occur if server1.up is set to assign an alias IP address to eth1:1 and server2.up is set to assign an alias IP address to eth1:2.

This problem can be handled by retrying the processing. Specify the number of LAN status settings files that assign alias IP addresses to the same interface as the retry count (m) in the server-alias-name.up file. For example, in the case of server1.up and server2.up above, specify 2.

#

Multiple LAN status settings files might execute at the same time if server1 and server2 are started as active servers concurrently at the destination during hot standby processing resulting from a host failure.

(b) Assigning multiple alias IP addresses by one LAN status settings file

This subsection shows an example that assigns alias IP addresses to eth1 and eth2 by using one LAN status settings file.

This example assumes the following conditions:

In addition to this LAN status settings file, another LAN status settings file assigns a different alias IP address to eth1 (retry count for eth1 is 2).

Two LAN status settings files assign different alias IP addresses to eth2 (retry count for eth2 is 3).

If assignment of an alias IP address to eth1 fails, no IP address is assigned to eth2.

#!/bin/sh
set -x

INTERFACE=eth1:1                    # LAN interface name
IPADDR=192.168.100.100              # alias IP address
NETMASK=255.255.255.0               # netmask address
BROADCAST=192.168.100.255           # broadcast address
RETRYCNT=2                          # retry count

# Retry for the alias IP address adding.
exitcode=1
loopcnt=0
while [ $loopcnt -lt $RETRYCNT ]
do
    # The alias IP address is added to the LAN interface.
    /sbin/ifconfig $INTERFACE inet $IPADDR netmask $NETMASK broadcast $BROADCAST

    # The information on old routing cache is deleted.
    /bin/echo 0 > /proc/sys/net/ipv4/route/flush

    # The ARP caches are updated.
    IFNAME=`echo $INTERFACE | /bin/sed -e 's/:[0-9]*$//'`
    /sbin/arping -U -c 2 -I $IFNAME $IPADDR
    sleep 1
    /sbin/arping -U -c 2 -I $IFNAME $IPADDR

    # Was the alias IP address registered to the system ?
    RCD=`/sbin/ifconfig -a | /bin/grep ":$IPADDR "`
    if [ "$RCD" = "" ]
    then
        # When it is not registered, "1" is returned as the termination code.
        exitcode=1

        loopcnt=`/usr/bin/expr $loopcnt + 1`
    else
        # When it is registered, "0" is returned as the termination code.
        exitcode=0

        break
    fi
done

if [ "$exitcode" = "1" ]
then
    exit $exitcode
fi

INTERFACE=eth2:1                    # LAN interface name
IPADDR=192.168.200.100              # alias IP address
NETMASK=255.255.255.0               # netmask address
BROADCAST=192.168.200.255           # broadcast address
RETRYCNT=3                          # retry count

# Retry for the alias IP address adding.
exitcode=1
loopcnt=0
while [ $loopcnt -lt $RETRYCNT ]
do
    # The alias IP address is added to the LAN interface.
    /sbin/ifconfig $INTERFACE inet $IPADDR netmask $NETMASK broadcast $BROADCAST

    # The information on old routing cache is deleted.
    /bin/echo 0 > /proc/sys/net/ipv4/route/flush
    # The ARP caches are updated.
    IFNAME=`echo $INTERFACE | /bin/sed -e 's/:[0-9]*$//'`
    /sbin/arping -U -c 2 -I $IFNAME $IPADDR
    sleep 1
    /sbin/arping -U -c 2 -I $IFNAME $IPADDR

    # Was the alias IP address registered to the system ?
    RCD=`/sbin/ifconfig -a | /bin/grep ":$IPADDR "`
    if [ "$RCD" = "" ]
    then
        # When it is not registered, "1" is returned as the termination code.
        exitcode=1

        loopcnt=`/usr/bin/expr $loopcnt + 1`
    else
        # When it is registered, "0" is returned as the termination code.
        exitcode=0

        break
    fi
done

exit $exitcode

(c) ARP request broadcast timings

The ARP entry lock duration# might be set for clients and other machines (nodes) in the same network. When hot-standby switchover occurs, if a broadcast of an ARP request is performed in the lock duration of those nodes, the ARP entries on the nodes that received the ARP request are not updated. In this case, these nodes might fail to communicate with the switchover-destination host.

Therefore, an ARP request generated when hot-standby switchover occurs needs to be broadcast after the lock duration of those nodes. To achieve this, make sure that after the arping command is executed, sleep is used to wait for the lock duration, and then the arping command is executed again.

Note that in a system where the duration between stoppage of the hot-standby switchover source program that uses TCP communication and broadcasting of ARP request by the hot-standby switchover destination exceeds the lock duration, the sleep and second arping commands are unnecessary.

#

The ARP entry lock duration is the time during which ARP entries are not updated after the last time TCP communication is closed.

In RHEL, the default is 1 second.

In RHEL, the ARP entry lock duration that is set for each host can be checked with /proc/sys/net/ipv4/neigh/interface-name/locktime (unit: jiffy).

(2) Specifying the server-alias-name.down file

In the case where the ip command is used in RHEL7 or a later version

In the server-alias-name.down file, specify the interface name, alias IP address, and prefix (length of bits indicating the network) of the LAN adapter allocated to each host. If necessary, also specify the number of retries.

The settings to be specified in the server-alias-name.down file differ depending on whether hbonding is used in RHEL7 or a later version.

If hbonding is not used in RHEL7 or a later version:

The following shows the contents of a server-alias-name.down file.

#!/bin/sh
set -x

INTERFACE=eXXX                      # LAN interface name
IPADDR=n.n.n.n                        # alias IP address
PREFIX=ee                            # address prefix
RETRYCNT=m                          # retry count

# Retry for the alias IP address delete.
exitcode=1
loopcnt=0
while [ $loopcnt -lt $RETRYCNT ]
do
    # The alias IP address is deleted from the LAN interface.
    /usr/sbin/ip addr del $IPADDR/$PREFIX dev $INTERFACE

    # Was the alias IP address deleted from the system ?
    RCD=`/usr/sbin/ip addr show dev $INTERFACE | grep " $IPADDR/$PREFIX "`
    if [ "$RCD" = "" ]
    then
        # When it is deleted, "0" is returned as the termination code.
        exitcode=0
        break
    else
        # When it is not deleted, "1" is returned as the termination code.
        exitcode=1
        loopcnt=`/usr/bin/expr $loopcnt + 1`
        sleep 1
    fi
done

# The information on old routing cache is deleted.
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

exit $exitcode

Legend:

eXXX: Interface name

n.n.n.n: Alias IP address

ee: Prefix

m: Number of retries

Taking into account occurrence of a temporary failure, we recommend that you specify 2 or a larger value for the number of retries (m).

If hbonding is used in RHEL7 or a later version:

The following shows the contents of a server-alias-name.down file. Note that, in this case, you must use the label specification to assign an IP address to the alias device. Also note that you must specify a physical device when executing the arping command.

#!/bin/sh
set -x

INTERFACE=hbondX                    # LAN interface name
IF_LABEL=hbondX:Y                   # alias device name
IPADDR=n.n.n.n                      # alias IP address
PREFIX=ee                           # address prefix
RETRYCNT=m                          # retry count

# Retry for the alias IP address delete.
exitcode=1
loopcnt=0
while [ $loopcnt -lt $RETRYCNT ]
do
    # The alias IP address is deleted from the LAN interface.
    /usr/sbin/ip addr del $IPADDR/$PREFIX dev $INTERFACE label $IF_LABEL

    # Was the alias IP address deleted from the system ?
    RCD=`/usr/sbin/ip addr show dev $INTERFACE | grep " $IPADDR/$PREFIX "`
    if [ "$RCD" = "" ]
    then
        # When it is deleted, "0" is returned as the termination code.
        exitcode=0
        break
    else
        # When it is not deleted, "1" is returned as the termination code.
        exitcode=1
        loopcnt=`/usr/bin/expr $loopcnt + 1`
        sleep 1
    fi
done

# The information on old routing cache is deleted.
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

exit $exitcode

Legend:

hbondX: Interface name

hbondX:Y: Alias device name

n.n.n.n: Alias IP address

ee: Prefix

m: Number of retries

Taking into account occurrence of a temporary failure, we recommend that you specify 2 or a larger value for the number of retries (m).

If the ifconfig command is used:

In the server-alias-name.down file, specify the interface name and IP index number of the LAN adapter allocated to each host.

The following shows the contents of a server-alias-name.down file:

#!/bin/sh
set -x

INTERFACE=xxx:y                      # LAN interface name

# The alias IP address is deleted from the LAN interface.
/sbin/ifconfig   $INTERFACE down

# The information on old routing cache is deleted.
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

Legend:

xxx: Interface name

y: IP index number

If use is specified in the fence_lan operand in the HA Monitor environment settings, the results of deleting alias IP addresses is checked. Create a server-alias-name.down file with the contents shown below.

Note that these contents can still be used even when use is not specified in the fence_lan operand.

#!/bin/sh
set -x

INTERFACE=xxx:y                      # LAN interface name
RETRYCNT=m                           # retry count

# Retry for the alias IP address delete.
exitcode=1
loopcnt=0
while [ $loopcnt -lt $RETRYCNT ]
do
    # The alias IP address is deleted from the LAN interface.
    /sbin/ifconfig $INTERFACE down

    # Was the alias IP address deleted from the system ?
    RCD=`/sbin/ifconfig -a | /bin/grep -E "$INTERFACE( |: )"`
    if [ "$RCD" = "" ]
    then
    # When it is deleted, "0" is returned as the termination code.
        exitcode=0
        break
    else
        # When it is not deleted, "1" is returned as the termination code.
        exitcode=1
    loopcnt=`/usr/bin/expr $loopcnt + 1`
        sleep 1
    fi
done

# The information on old routing cache is deleted.
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

exit $exitcode

Legend:

xxx: Interface name

y: IP index number

m: Retry count

Taking into account occurrence of a temporary failure, we recommend that you specify 2 or a larger value for the number of retries (m).

6.12 Creating server start, termination, and monitoring commands (for the active server)

If a server runs in the monitor mode, you create for it a server start command, a server termination command, and a server monitoring command. Creation of a server start command is mandatory. Creation of server termination and monitoring commands is optional. Note that the server monitoring command does not need to target any UAPs that are monitored by using the monitor-mode program management function.

If you code the processing for multiple programs in the server start, termination, and monitoring commands, you can use those programs as a single server.

Create each command for each server by using a language, such as C language and shell language.

Important

We recommend that you specify the processing in a created command in such a manner that errors can be analyzed (by, for example, outputting logs as needed).

6.12.1 Creating a server start command

A server start command is used to start a program. If the server runs in the monitor mode, you must create a server start command so that HA Monitor can verify completion of server startup.

This subsection explains the timing of calling a server start command and how to create a server start command.

(1) Timing of calling a server start command

A created server start command is executed by HA Monitor with the superuser permissions.

The server start command is called after shared resources are connected in the following cases:

The active server is to be started by executing the monitor-mode server start command (monbegin command).

The active server is to be restarted when the active server is set to be restarted in the event of a server failure.

(2) How to create a server start command

For a server start command, you can specify a program or a shell that codes program startup processing.

Specify an operating environment needed to start the program within the shell. In RHEL7 or a later version, you can specify environment variables in the Unit settings file. However, the Unit settings file is overwritten when an overwrite installation of HA Monitor is performed. Therefore, we recommend that you specify an operating environment (environment variables) within the shell.

The following shows the environment variables that are inherited when the server startup command is executed.

When the monitor-mode server start command (monbegin command) is executed

The environment variables of the user who executed the monbegin command are inherited.

When the server is restarted#

The environment variables specified in the Unit settings file of HA Monitor are inherited in the following cases:

The OS is RHEL7 or a later version and the HA Monitor version is 01-65 or later.

The OS is RHEL7 or a later version, the HA Monitor version is earlier than 01-65, and HA Monitor automatically started when the system started.

The environment variables of the user who executed the HA Monitor start command (monstart command) are inherited in the following cases:

The OS is RHEL7 or a later version, the HA Monitor version is earlier than 01-65, and HA Monitor was started by using the monstart command.

The OS is RHEL6 and HA Monitor was started by using the monstart command.

The environment variables specified in the startup script /etc/rc.d/init.d/HAmon are inherited in the following cases:

The OS is RHEL6 and HA Monitor automatically started when the system started.

#

If the server started automatically by the function of the monbegin_restart operand in the HA Monitor environment settings, or if the server restarted by the function of the servexec_retry operand in the server environment definition.

To use a server start command to start a UAP that issues APIs when using the monitor-mode program management function, specify the program name in a UAP environment variable. In the server start command, specify all target UAP start processes. For details about the UAP environment variables, see 6.15.2 Specifying UAP environment variables. To acquire core files for investigation in the event of a UAP failure, add ulimit -c unlimited before the process for starting the UAP. The kernel parameter's default setting does not collect core files.

Specify the created server start command in the name operand in the server environment definition. To specify arguments in the server start command, also specify the arguments in the actcommand operand in the server environment definition.

Specify the following settings for the waitserv_exec operand in the server environment definition according to the type of the server start command.

Table 6‒7: Value to be specified for the waitserv_exec operand according to the type of the server start command

Type of the server start command

Setting when servers are monitored

Setting when servers are not monitored

A server start command that does not return the result when the start processing of the target programs is completed.

For example, a server start command in which programs are directly specified.

no

no

A server start command that returns the result when the start processing of the target programs is completed.

yes

Either value

For a program that directly operates as a server process

Specify no in the waitserv_exec operand in the server environment definition. This is the default. For a program that directly functions as a server process, make sure that you specify no because it does not wait for completion of startup processing.

For a program that does not directly operate as a server process

If you monitor servers in the monitor mode, make sure that you specify yes in the waitserv_exec operand in the server environment definition.

If you do not specify yes when servers in the monitor mode are to be monitored, an active server's startup processing might be completed on HA Monitor and monitoring of servers might begin before the program completes its startup processing. If monitoring of servers begins before the program's startup processing is completed, the incomplete program start status might be treated as a server failure, resulting in a restart of the active server and hot standby processing.

If servers in the monitor mode are not to be monitored, you can specify either yes or no. However, you must specify yes in the following cases:

The server switchover order is to be controlled between servers in the monitor mode.

The processing to be performed will be determined on the basis of the EXIT code from the server start command.

If yes is specified in the waitserv_exec operand in the server environment definition, HA Monitor evaluates the return value of the server start command that is specified in the name or actcommand operand in the server environment definition. The following describes the HA Monitor processing that depends on the return value:

Return value: 0

HA Monitor assumes that the program has successfully started and performs start completion processing on the active server.

Return value: Nonzero value

HA Monitor assumes that the program's startup processing failed and terminates the active server startup processing. Server restart processing will fail, if attempted. When restart processing fails, HA Monitor does not execute the server termination command specified in the termcommand operand in the server environment definition. For details about the handling of restart errors, see 7.5.2 Handling server restart errors.

If you specify yes in the waitserv_exec operand in the server environment definition, you can specify a timeout value from start to termination of the server start command in the start_timeout operand in the server environment definition.

Note that hot-standby switchover is not performed in the following cases:

The server could not be started or restarted because the server start command returned a value other than 0.

The server could not be started because a start timeout was detected.

If you want hot-standby switchover to be performed in the preceding cases, create a start command that always returns a return value of 0. In addition, make sure that the monitoring command can detect a server start failure.

If you specify no in the waitserv_exec operand in the server environment definition, HA Monitor performs start completion processing on the active server regardless of the server start command's return value.

6.12.2 Creating a server termination command

A server termination command is used to terminate a program automatically. Creation of a server termination command is optional. You create it when you want to terminate a program automatically for a purpose such as grouped-system switchover.

This subsection explains the timing of calling a server termination command and how to create a server termination command.

(1) Timing of calling a server termination command

A created server termination command is executed by HA Monitor with the superuser permissions.

The server termination command is called before shared resources are disconnected in the following cases:

Planned termination is to be performed on the active server by executing the server hot-standby switchover command (monswap command).

The active server is to be terminated normally by executing the monitor-mode server termination command (monend command).

The server is set to be restarted in the event of a server failure and a failure has been detected on the server (before restarting the active server).

The active server is to be terminated due to hot standby processing in the event of a server failure.

Planned termination is to be performed on the active server due to a grouped-system switchover.

HA Monitor passes an argument that is determined by the event to the server termination command. The arguments include -e, -w, and -c. You must code processing that corresponds to the arguments as shown below.

Table 6‒8: Arguments for a server termination command and the processing to be coded

Argument

Processing to be coded when the argument is passed

-e

Normal termination of the server

-w

Planned termination when planned hot standby processing is performed on the server

-c

Processing required in order to restart the abnormally terminated server

The following table shows the relationships between server termination command start conditions and the arguments.

Table 6‒9: Server termination command start conditions and the arguments to be passed

Server termination command start condition

Argument to be passed#

When termcmd_at_abort=nouse is specified

When termcmd_at_abort=use is used

The server is to be terminated by the monitor-mode server termination command (monend command).

-e

-e

In the event of a server failure, an active server that has no standby server is to be terminated by the monitor-mode server termination command (monend command).

-e

-c

Hot standby processing is to be performed on the server due to a server failure.

-w

-c

The server hot-standby switchover command (monswap or monswap -g command) is to be executed (as planned hot-standby switchover).

-w

-w

In the event of a failure on an active server that has no standby server, a standby server is to be started and then the server hot-standby switchover command (monswap or monswap -g command) is to be executed (as planned hot-standby switchover).

-w

-c

The active server is to be forcibly terminated and then hot standby processing is to be performed for grouped-system switchover.

-w

-w

In the event of a server failure, an active server that has no standby server is to be forcibly terminated for grouped-system switchover.

-w

-c

In the event of a failure on an active server that has no standby server, a standby server is to be started and then hot standby processing is to be performed for grouped-system switchover.

-w

-c

In the event of a server failure, preprocessing is to be performed prior to restarting the active server.

-c

-c

The server startup processing is to be cancelled due to a server start command error.

--

-c

Legend:

--: The server termination command is not issued.

#

The argument to be passed depends on the value of the termcmd_at_abort operand specified in the HA Monitor environment settings. For details about the termcmd_at_abort operand, see 8.3.1 HA Monitor environment settings (sysdef).

(2) How to create a server termination command

For a server termination command, you specify a script such as a shell that codes program termination processing.

If necessary, specify an operating environment needed to terminate the program within the shell. In RHEL7 or a later version, you can specify environment variables in the Unit settings file. However, the Unit settings file is overwritten when an overwrite installation of HA Monitor is performed. Therefore, we recommend that you specify an operating environment (environment variables) within the shell.

The following shows the environment variables that are inherited when the server termination command is executed.

The environment variables specified in the Unit settings file of HA Monitor are inherited in the following cases:

The OS is RHEL7 or a later version and the HA Monitor version is 01-65 or later.

The OS is RHEL7 or a later version, the HA Monitor version is earlier than 01-65, and HA Monitor automatically started when the system started.

The environment variables of the user who executed the HA Monitor start command (monstart command) are inherited in the following cases:

The OS is RHEL7 or a later version, the HA Monitor version is earlier than 01-65, and HA Monitor was started by using monstart command.

The OS is RHEL6 and HA Monitor was started by using the monstart command.

The environment variables specified in the startup script /etc/rc.d/init.d/HAmon are inherited in the following cases:

The OS is RHEL6 and HA Monitor automatically started when the system started.

To use a server termination command to terminate a UAP that issues APIs when using the monitor-mode program management function, specify the program name in a UAP environment variable. In the server termination command, specify all target UAP termination processes. If a UAP creates processes, also specify the processing to terminate those processes. For details about the UAP environment variables, see 6.15.2 Specifying UAP environment variables.

Specify the created server termination command in the termcommand operand in the server environment definition. You can also use the stop_timeout or stop_timeout_sw operand in the server environment definition to specify the timeout value for the server termination command.

HA Monitor processing is not affected by the return value of a server termination command that is called during normal or planned termination of the active server. When you create a termination command, note the following points:

The host pair shutdown function is used to perform hot-standby switchover if the following conditions are met: use is specified for the term_pairdown operand in the server environment definition, and the server termination command called when planned termination is performed terminates with a return value other than 0.

If the server termination command called when the server is restarted terminates with a value other than 0, HA Monitor stops the subsequent processing (restarting the server, starting the monitoring command, and monitoring the server). Therefore, server restart fails and no hot-standby switchover occurs. For details about the handling of restart errors, see 7.5.2 Handling server restart errors. If you want hot standby processing to be performed even when termination of the server fails, code the server termination command in such a manner that it always returns 0.

If you code termination processing for multiple programs in the server termination command, you can use those programs as a server.

6.12.3 Creating a server monitoring command

A server monitoring command is used to monitor program status. If you create a server monitoring command, you can use HA Monitor to restart the server and perform automatic hot standby processing in the event of a server failure. If the server consists of only those UAPs that are monitored by the monitor-mode program management function, there is no need to create a server monitoring command.

This subsection explains the timing of calling a server monitoring command and how to create a server monitoring command. For details about how to monitor servers, see 3.2.1 Monitoring a server in the monitor mode (for the active server).

(1) Timing of calling a server monitoring command

A created server monitoring command is executed by HA Monitor with the superuser permissions.

The server monitoring command is started after startup of the active server is completed in the active system.

(2) How to create a server monitoring command (if the command is specified for the ptrlcmd_ex operand in the server environment definition)

Create a server monitoring command so that the following conditions are met:

If a particular operating environment is required for monitoring programs, set it. In RHEL7 or a later version, you can specify environment variables in the Unit settings file. However, the Unit settings file is overwritten when an overwrite installation of HA Monitor is performed. Therefore, we recommend that you specify an operating environment (environment variables) within the shell.

For details about the environment variables that are inherited when the server monitoring command is executed, see (2) How to create a server termination command in 6.12.2 Creating a server termination command.

The command must be able to receive the SIGTERM signal.

HA Monitor stops the monitoring command by sending the SIGTERM signal if a timeout occurs when the server is terminated normally, planned hot-standby switchover is performed, or the monitoring command is executed. Therefore, the command must be able to receive the SIGTERM signal.

Create a server monitoring command so that its execution does not loop indefinitely.

If processing of the monitoring command loops indefinitely for some reason (for example, periodic monitoring is performed within the monitoring command), HA Monitor detects the loop as a slowdown or hang of the monitoring command. Therefore, HA Monitor assumes that a server failure occurred on a monitor-mode server.

The monitoring command must return the monitoring result (EXIT code) as the termination result.

The following table describes the termination results (EXIT codes) that can be used. Note that EXIT codes 30 to 49 cannot be used because they have been reserved by HA Monitor.

Table 6‒10: Termination results (EXIT codes) that can be used according to the status of the monitoring-target job processing

Status of the monitoring-target job processing

Termination results (EXIT codes) that can be used

Description

Normal

0

If this EXIT code is returned, HA Monitor assumes that the monitoring-target job processing is in the normal status or has been recovered from a failure.

Make sure that the command returns this code if the monitoring-target job processing is operating normally.

Status check failed

Job processing error

Indeterminable

1 to 9

If this EXIT code is returned, HA Monitor re-executes the monitoring command. HA Monitor operates as specified by the ptrlcmd_ex_retry operand in the server environment definition. Note that the re-execution interval is the same value that is specified for the ptrlcmd_ex_inter operand in the server environment definition.

Make sure that the command returns these codes if a problem that seems to be temporary has occurred for the target job processing (for example, a status check failed, a job processing error occurred, or the status is indeterminable).

Failure

10 to 19

If this EXIT code is returned, HA Monitor does not re-execute the monitoring command and assumes that a server failure occurred on the active server. HA Monitor operates as specified by the servexec_retry operand in the server environment definition.

Because HA Monitor does not re-execute the monitoring command, make sure that the command returns these codes in cases such as when it is possible to determine that the job processing has failed.

20 to 29

If this EXIT code is returned, HA Monitor does not re-execute the monitoring command and performs hot-standby switchover, assuming that a server failure occurred on the active server. HA Monitor immediately performs hot-standby switchover, ignoring the specification of the servexec_retry operand in the server environment definition.

Make sure that the command returns these codes in cases such as when HA Monitor can judge that job processing cannot continue even by restarting the server due to a disk failure or another permanent failure.

The following provides an overview of processing for each EXIT code.

If the EXIT code is 0 (normal), HA Monitor starts the monitoring command at regular intervals (the interval is specified for the ptrlcmd_ex_inter operand in the server environment definition). The following figure shows an overview of failure monitoring for a monitor-mode server (in the case where the EXIT code 0 is returned).

Figure 6‒36: Failure monitoring for a monitor-mode server (in the case where the EXIT code 0 is returned)

The command returns an EXIT code of 1 to 9 in such a case when a status check failed, a job processing error occurred, or the status is indeterminable. If one of these codes is returned a certain number of times, HA Monitor determines that the server has failed. If the number of times a status check fails exceeds the value specified for the ptrlcmd_ex_inter operand in the server environment definition, HA Monitor assumes that a server has failed. The following figure shows an overview of failure monitoring for a monitor-mode server (in the case where the EXIT code 1 to 9 is returned). Note that this figure is an example when 2 is specified for the ptrlcmd_ex_retry operand in the server environment definition.

Figure 6‒37: Failure monitoring for a monitor-mode server (in the case where the EXIT code 1 to 9 is returned)

If the command returns the EXIT code 10 to 19 or 20 to 29 (failure), HA Monitor does not re-execute the monitoring command and determines that a server failure occurred on the active server. The following figure shows an overview of failure monitoring for a server in the monitor mode (if the command returns the EXIT code 10 to 19 or 20 to 29).

Figure 6‒38: Failure monitoring for a monitor-mode server (in the case where the EXIT code 10 to 19 or 20 to 29 is returned)

If the command does not return the result even after the time specified for the ptrlcmd_ex_tmout operand in the server environment definition passes, HA Monitor assumes that a slowdown or hang of job processing occurred. In this case, HA Monitor determines that a server failure occurred on the active server. The following figure shows an overview of monitoring a failure on a server in the monitor mode (in the case no result is returned even after the time specified for the ptrlcmd_ex_tmout operand in the server environment definition passes).

Figure 6‒39: Failure monitoring for a monitor-mode server (in the case where no result is returned even after the time specified for the ptrlcmd_ex_tmout operand passes)

Specify the server monitoring command that you created for the ptrlcmd_ex operand in the server environment definition. In addition, specify the following operands in the server environment definition:

ptrlcmd_ex_inter (cannot be omitted)

ptrlcmd_ex_retry (can be omitted)

ptrlcmd_ex_tmout (can be omitted)

For details about how to check the server logs if a server failure is detected by the server monitoring command, see 7.3 Checking the server logs.

The patrol_ex.sh file is provided as a sample file for a server monitoring command in the HA Monitor sample file directory. Please use this file if necessary.

#!/bin/sh
#******************************************************************************
#*                                                                            *
#*    Linux(x86) HA Monitor                                                   *
#*    This is a sample of the patrol command.                                 *
#*    This is specified in the ptrlcmd_ex operand.                            *
#*    (For monitor mode server)                                               *
#*    Attention: This can not be specified in the patrolcommand operand.      *
#*                                                                            *
#*    All Rights Reserved. Copyright (C) 2017, Hitachi, Ltd.                  *
#*                                                                            *
#******************************************************************************
set -x
# The object program to monitor
PROGRAM=/home/xxxx/yyyy

# The definition of command
PS=/bin/ps
GREP=/bin/grep

# Is the object program to monitor operating ?
EXIST=`$PS -efl | $GREP $PROGRAM | $GREP -v grep`

# When the object program to monitor is not operating,
# the variable EXIST is empty.
if [ "$EXIST" = "" ]
         then
         # This patrol command terminates, because the object program
         # to monitor is not operating.
         exit 10
fi

# Please describe commands and processes that can confirm that applications and tasks are running.
# For example, a command that throws a request to an application and receives a response.
# If it is a database, describe SQL to access the database.
# The end result (EXIT code) is returned according to the command execution result.
# For details, refer to the manual.
#
# ex)
#        Describe the SQL to access the database.
#        if [ "$?" -ne "0" ]
#        then
#                  exit 1
#        fi

exit 0

The EXIST=`$PS -efl | $GREP $PROGRAM | $GREP -v grep` entry checks whether there is a program process in the server monitoring command. You can check the process, for example, from the output results of the ps OS command.

In the section indicated by Describe the SQL to access the database, specify a command or another process that allows you to confirm that the application or job processing is operating. For example, you can specify a command that issues a request to an application and receives a reply or an SQL statement that accesses a database. The termination result (EXIT code) is returned as the execution result of the command. For details about the termination result (EXIT code), see Table 6-11 Termination results (EXIT codes) that can be used according to the status of the monitoring-target job processing.

(3) How to create a server monitoring command (if the command is specified for the patrolcommand operand in the server environment definition)

Create a server monitoring command so that the following conditions are met:

If a particular operating environment is required for monitoring programs, set it. In RHEL7 or a later version, you can specify environment variables in the Unit settings file. However, the Unit settings file is overwritten when an overwrite installation of HA Monitor is performed. Therefore, we recommend that you specify an operating environment (environment variables) within the shell.

For details about the environment variables that are inherited when the server monitoring command is executed, see (2) How to create a server termination command in 6.12.2 Creating a server termination command.

The monitoring-target programs are monitored periodically.

If a program failure is detected during monitoring, the monitoring command is terminated (by exit).

Any termination code can be used.

The command must be able to receive the SIGTERM signal.

HA Monitor stops the server monitoring command by sending the SIGTERM signal to the monitoring command. Therefore, the command must be able to receive the SIGTERM signal.

Specify the server monitoring command that you created for the patrolcommand operand in the server environment definition.

The patrol.sh file is provided as a sample file for a server monitoring command in the HA Monitor sample file directory. Please use this file if necessary.

The following shows an example of creating a B-shell-based monitoring command.

#!/bin/sh
# The object program to monitor
PROGRAM=/home/xxxx/yyyy

# The definition of command
PS=/bin/ps
GREP=/bin/grep

# Main loop
while true
do
    # Is the object program to monitor operating ?
    EXIST=`$PS -efl | $GREP $PROGRAM | $GREP -v grep`

    # When the object program to monitor is not operating,
    # the variable EXIST is empty.
    if [ "$EXIST" = "" ]
    then
        # This patrol command terminates, because the object program
        # to monitor is not operating.
        exit
    fi

    # The monitoring is continued, because the object program
    # to monitor is operating.
    sleep 5
done

In the preceding coding example, the server monitoring command has an internal loop and checks whether a program process exists inside the loop. You can check the process, for example, from the output results of the ps OS command. If a program process exists, the command continues processing of the loop to continue checking whether the process is alive.

6.13 Creating commands that will be used to start, stop, and monitor the standby server (in the case where monitoring is performed in the standby system)

To perform monitoring in the standby system, create commands that will be used to start, stop, and monitor the standby server.

6.13.1 Creating a command that is executed when the standby server starts

By creating a command that is to be executed when the standby server starts, you can execute process startup and other desired processing when the standby server starts.

This subsection describes how to create a command that is to be executed when the standby server starts and when the created command is called.

(1) Timing at which the command to be executed when the standby server starts is called

If you create the command, HA Monitor executes it with the superuser permissions.

The command is called after the shared resources are disconnected in the following cases:

When the standby server is started by executing the monitor-mode server start command (monbegin command)

When the standby server is restarted by the settings that restart the standby server if a failure occurs on the standby server

(2) How to create a command to be executed when the standby server starts

Create a command that is to be executed when the standby server starts so that startup of monitoring-target programs and other processing are executed by that command.

Specify an operating environment needed to start the program within the shell. In RHEL7 or a later version, you can specify environment variables in the Unit settings file. However, the Unit settings file is overwritten when an overwrite installation of HA Monitor is performed. Therefore, we recommend that you specify an operating environment (environment variables) within the shell.

For details about the environment variables that are inherited when the standby server start command is executed, see (2) How to create a server start command in 6.12.1 Creating a server start command.

If you have created a command, specify it for the sby_actcommand operand in the server environment definition. HA Monitor evaluates the return value from the command specified for the sby_actcommand operand in the server environment definition. The following describes the HA Monitor processing that depends on the return value.

If the return value is 0:

HA Monitor assumes that the command processing succeeded and completes startup of the standby server.

If the return value is other than 0:

HA Monitor assumes that the command processing failed and stops startup of the standby server. If an attempt was made to restart the standby server, the attempt has failed. If restart fails, HA Monitor executes the command specified for the sby_termcommand operand in the server environment definition. For details about the handling of a restart failure, see 7.5.2 Handling server restart errors.

You can use the sby_start_timeout operand in the server environment definition to specify the timeout value for the command.

6.13.2 Creating a command to be executed when the standby server stops

By creating a command that is to be executed when the standby server stops, you can execute process stop and other desired processing when the standby server stops.

This subsection describes how to create a command that is to be executed when the standby server stops and when the created command is called.

(1) Timing at which the command to be executed when the standby server stops is called

If you create the command, HA Monitor executes it with the superuser permissions.

The command is called in the following cases:

The monitor-mode server termination command (monend command) is executed in the active system, and the standby server terminates normally when the active server terminates normally.

The standby server termination command (monsbystp command) is executed, and the standby server terminates normally.

A failure on the standby server is detected.

The wait-state server termination command (mondeact command) is executed to forcibly terminate a server that is in the hot-standby switchover wait state.

A failure on a server in the hot-standby switchover wait state is detected.

HA Monitor passes an appropriate argument to the command according to the case in which the command is called. The arguments that can be passed are -e and -c. The following table describes the relationships among the condition in which the command is started, the argument to be passed, and the processing performed when the argument is passed. When you create a command, code the appropriate processing according to the argument.

Table 6‒11: Relationships among the condition in which the command is started, the argument to be passed, and the processing performed when the argument is passed

Condition in which the command is started

Argument to be passed

Processing performed when the argument is passed

The wait-state server termination command (monsbystp command) is used to terminate the server.

-e

Normally terminate the program.

The standby server is terminated due to a failure on the standby server.

-c

Perform processing to make it possible to restart the abnormally terminated program.

Pre-processing is performed for restarting the standby server in the event of a failure on the standby server.

Startup of the standby server is canceled because the standby-server start command failed.

The wait-state server termination command (mondeact command) is used to forcibly terminate a server in the hot-standby switchover wait state.

The server is forcibly terminated because a failure occurred on a server in the hot-standby switchover wait state.

Notes

The termination command is not called in the standby system when a hot-standby switchover occurs.

(2) How to create a command to be executed when the standby server stops

Create a command that is to be executed when the standby server stops, and code necessary processing. For example, code processing that terminates a command that was started when the standby server started (command specified for the sby_actcommand operand in the server environment definition).

If necessary, specify an operating environment needed to terminate the program within the shell. In RHEL7 or a later version, you can specify environment variables in the Unit settings file. However, the Unit settings file is overwritten when an overwrite installation of HA Monitor is performed. Therefore, we recommend that you specify an operating environment (environment variables) within the shell.

For details about the environment variables that are inherited when the standby server terminating command is executed, see (2) How to create a server termination command in 6.12.2 Creating a server termination command.

If you have created a command, specify it for the sby_termcommand operand in the server environment definition.

You can use the sby_stop_timeout operand in the server environment definition to specify the timeout value for the command.

HA Monitor processing is not affected by the return value of the server termination command that is called when the standby server terminates normally. If the command called when the standby server is restarted terminates with a value other than 0, HA Monitor stops the subsequent processing (restarting the standby server, starting the monitoring command, and monitoring the server). Therefore, restart of the standby server fails. For details about the handling of restart errors, see 7.5.2 Handling server restart errors.

6.13.3 Creating a command to be executed when monitoring of the standby server is performed

HA Monitor can execute a monitoring command at regular intervals during monitoring of the standby server by creating a command that is to be executed when monitoring of the standby server is performed.

This subsection describes how to create a command that is to be executed when monitoring of the standby server is performed and when the created command is called.

(1) Timing at which the command to be executed when monitoring of the standby server is performed is called

If you create the command, HA Monitor executes it with the superuser permissions.

The command is called in the following cases:

Startup of the standby server is completed in the standby system.

The standby server in the standby system is ready for hot-standby switchover.

The standby server is in the hot-standby switchover wait state.

(2) How to create a command to be executed when monitoring of the standby server is performed

Create a command so that the following conditions are met:

If a particular operating environment is required for monitoring programs, set it. In RHEL7 or a later version, you can specify environment variables in the Unit settings file. However, the Unit settings file is overwritten when an overwrite installation of HA Monitor is performed. Therefore, we recommend that you specify an operating environment (environment variables) within the shell.

For details about the environment variables that are inherited when the standby server monitoring command is executed, see (2) How to create a server termination command in 6.12.2 Creating a server termination command.

The command must be able to receive the SIGTERM signal.

HA Monitor stops the monitoring command by sending the SIGTERM signal if a timeout occurs when the standby server is terminated normally or the monitoring command is executed. Therefore, the command must be able to receive SIGTERM.

Create a monitoring command so that its execution does not loop indefinitely.

If processing of the monitoring command loops indefinitely for some reason (for example, periodic monitoring is performed within the monitoring command), HA Monitor detects the loop as a slowdown or hang of the monitoring command. Therefore, HA Monitor assumes that a server failure occurred on the monitor-mode standby server.

The monitoring command you create must terminate with one of the EXIT codes listed in the following table according to the status of the monitoring-target job processing.

Table 6‒12: Correspondence between the statuses of monitoring-target job processing and EXIT codes

No.

Status of the monitoring-target job processing

EXIT code

Description

1

Normal

0

If the monitoring target is operating normally, the command must return the EXIT code 0. HA Monitor assumes that the monitoring target is in the normal status or has been recovered from a failure.

2

Status check failed

Monitoring error

Indeterminable

1 to 9

HA Monitor re-executes the monitoring command.

HA Monitor operates as specified by the sby_ptrlcmd_ex_retry operand in the server environment definition.

The re-execution interval is the same value that is specified for the sby_ptrlcmd_ex_inter operand in the server environment definition.

Make sure that the command returns these codes when a temporary failure has occurred (for example, it is impossible to judge whether monitoring of the monitoring target failed or whether the problem that occurred pertains to the monitoring target).

3

Failure

10 to 19

HA Monitor assumes that a failure has occurred without re-executing the monitoring command.

HA Monitor operates as specified by the sby_servexec_retry operand in the server environment definition.

Make sure that the command returns these codes in such a case when HA Monitor can judge that a failure has occurred without re-executing the monitoring command.

4

20 to 29

HA Monitor judges that a server failure has occurred on the standby server without re-executing the monitoring command, and terminates the standby server.

HA Monitor immediately terminates the standby server so that hot-standby switchover does not occur, ignoring the specification of the sby_servexec_retry operand in the server environment definition.

Make sure that the command returns these codes in cases such as when HA Monitor can judge that processing cannot continue even by restarting the standby server due to a disk failure or another permanent failure.

5

--

30 to 49

Do not use these codes because they are reserved by HA Monitor.

6

Other codes

HA Monitor re-executes the monitoring command.

HA Monitor operates as specified by the sby_ptrlcmd_ex_retry operand in the server environment definition.

The re-execution interval is the same value that is specified for the sby_ptrlcmd_ex_inter operand in the server environment definition.

Make sure that the command returns these codes when a temporary failure has occurred (for example, it is impossible to judge whether monitoring of the monitoring target failed or whether the problem that occurred pertains to the monitoring target).

6.14 Creating a program restart command (when the program management function is used)

A program restart command is used to restart a UAP that is monitored by the monitor-mode program management function in the event of a UAP failure.

This section explains the timing of calling a program restart command and how to create a program restart command.

6.14.1 Timing of calling a program restart command

A created program restart command is executed by HA Monitor with the superuser permissions.

The program restart command is called in the following cases:

A failure is detected in the UAP that is monitored by the program management function.

The server start command terminated, but an API for monitoring the UAP (hamon_patrolstart function) is not issued.

6.14.2 How to create a program restart command

For a program restart command, you must specify a shell that codes processing for starting the UAP after the UAP has terminated.

Specify in the shell the operating environment required for restarting the program. In RHEL7 or a later version, you can specify environment variables in the Unit settings file. However, the Unit settings file is overwritten when an overwrite installation of HA Monitor is performed. Therefore, we recommend that you specify an operating environment (environment variables) within the shell.

For details about the environment variables that are inherited when the program restart command is executed, see (2) How to create a server termination command in 6.12.2 Creating a server termination command.

To acquire core files for investigation in the event of a UAP failure, add ulimit -c unlimited before the process for starting the UAP. The kernel parameter's default setting does not collect core files.

For the processing for terminating the UAP, if the UAP creates processes, also specify the processing to terminate those processes. Because HA Monitor waits for completion of the shell specified in the program restart command, make sure that the shell returns control after the UAP startup processing has been performed.

To use a program restart command to restart a UAP that issues APIs, specify the program name in a UAP environment variable. For details about the UAP environment variables, see 6.15.2 Specifying UAP environment variables.

Specify the created program restart command in the restartcommand operand in the monitor-mode program environment definition. You can also specify in the restart_timeout operand in the monitor-mode program environment definition a timeout value before the program restart command is terminated since it was started and before the hamon_patrolstart function is issued.

HA Monitor checks the return value of the program restart command specified in the restartcommand operand. The following explains HA Monitor processing depending on the return value.

When the return value is 0

HA Monitor assumes that the program has been restarted successfully and detects the completion of program restart. For a UAP, HA Monitor waits for an API to be issued.

When the return value is not 0

HA Monitor assumes that the program restart has failed and terminates the program restart processing.

6.15 Creating and configuring UAPs

If you use the monitor-mode program management function to monitor a UAP, you must include APIs in the UAP and specify UAP environment variables.

For details about the program management function, see 3.6 Function for controlling programs (monitor mode).

6.15.1 Including APIs in UAPs

If you use the monitor-mode program management function to monitor a UAP, you must edit the UAP's source code to issue the APIs provided by HA Monitor from the UAP. This subsection explains how in include APIs .

(1) UAPs in which APIs can be included

A UAP that satisfies all the following conditions can use APIs:

Programmed in C (ANSI C)

None of the following function and area names that are used in APIs provided by HA Monitor is used:

hamon_patrolstart

hamon_patrolstop

hamon_upapi_env

hamon_upapi_inf

Neither of the following environment variables that are used by APIs provided by HA Monitor is used:

HAMON_UAPNAME

HAMON_API_PATROL

The UAP does not contain processing for releasing all file descriptors of a process.

When an API is issued, the API connects to HA Monitor via UNIX domain communication and holds one file descriptor. Therefore, APIs cannot be used if the UAP contains processing that releases all file descriptors of a process.

The UAP is not monitored by any program product other than HA Monitor.

For example, APIs cannot be issued from an SPP of OpenTP1.

Because an API generates one thread and uses that thread to transmit signals specified in the arguments of the API, the following conditions are all satisfied:

Threads can be generated.

Signals can be used.

There is one or more signals that can be used by APIs. For a corresponding signal, the signal handler is undefined and unblocked.

The UAP has been designed to handle return of a system call due to signal interruption (error number: EINTR).

For example, if control is returned due to a signal interruption (error number: EINTR), system calls (sleep, usleep, and nanosleep) might be retried for the remainder of the processing time or system calls (read and write) might retry processing after control is returned from the interruption.

(2) Files provided by HA Monitor

HA Monitor provides the files listed below so that UAPs can issue APIs.

Table 6‒13: Files provided by HA Monitor to issue APIs

Name

File storage path

Description

Header file

/opt/hitachi/HAmon/api/include/HA_mon_uap.h

Header file required for using APIs

Object file for API

/opt/hitachi/HAmon/api/lib/hamon_api32.a

Object file that requires a static link to the UAP to use APIs. This is for UAPs that run in a 32-bit environment.

/opt/hitachi/HAmon/api/lib/hamon_api64.a

Object file that requires a static link to the UAP to use APIs. This is for UAPs that run in a 64-bit environment.

Shared library file for API#

/opt/hitachi/HAmon/api/lib/hamon_api32.so

API library file that is used dynamically during UAP execution. This is for UAPs that run in a 32-bit environment.

/opt/hitachi/HAmon/api/lib/hamon_api64.so

API library file that is used dynamically during UAP execution. This is for UAPs that run in a 64-bit environment.

#

This is a library that is loaded dynamically from the object file of API during UAP execution. It cannot be linked to UAPs.

(3) Creating UAPs

Create a UAP in C. Specify APIs provided by HA Monitor in the UAP.

To use a shell that is run in the event of a UAP slowdown, code the processing for collecting trace information before and after a system call or a function such as an SQL function is issued. An effective method is to collect trace information in memory and then analyze it from core and output messages.

(4) Compiling and linking

Use the gcc command to compile the UAP source file that contains APIs.

The following table lists and describes the options required during compilation.

Table 6‒14: Options required during compilation

Option

Meaning

-o

Specifies a name for the binary file to be created.

-I/opt/hitachi/HAmon/api/include

Specifies the path of the header file.

-L/opt/hitachi/HAmon/api/lib

Specifies the path of the shared library.

-Wl,-Bdynamic

Specifies to the linker that the dynamic library is to be linked (because this is the default value, this option can be omitted).

-ldl

Uses a library such as for the dlopen function.

Shown below are examples of compilation in a 64-bit environment and a 32-bit environment.

Example for a 64-bit environment

Name of the UAP source program created in C

sample.c (main function)

Example of command line during UAP compilation

gccΔ-oΔsampleΔ-I/opt/hitachi/HAmon/api/includeΔ-L/opt/hitachi/HAmon/api/libΔsample.cΔ/opt/hitachi/HAmon/api/lib/hamon_api64.aΔ-Wl,-BdynamicΔ-ldl

Legend:

Δ: Single-byte space

Example for a 32-bit environment

Name of the UAP source program created in C

sample.c (main function)

Example of command line during UAP compilation

gccΔ-oΔsampleΔ-I/opt/hitachi/HAmon/api/includeΔ-L/opt/hitachi/HAmon/api/libΔsample.cΔ/opt/hitachi/HAmon/api/lib/hamon_api32.aΔ-Wl,-BdynamicΔ-ldl

Legend:

Δ: Single-byte space

6.15.2 Specifying UAP environment variables

You must specify UAP environment variables to start a UAP that contains APIs. Specify the following environment variables by means of a shell used to execute the server start command:

HAMON_UAPNAME environment variable

Specify the name of a program that was specified in the name operand in the monitor-mode program environment definition. This environment variable is required.

When HA Monitor receives a monitoring start request or a monitoring end request from an API, it uses the program name specified in this environment variable to identify the UAP that issued the API.

HAMON_API_PATROL environment variable

Specify nouse in this environment variable only if the UAP is to run in an environment in which HA Monitor has not been installed or in a test environment in which HA Monitor has not started. When nouse is specified, the hamon_patrolstart and hamon_patrolstop functions return with return code 0.

6.16 Creating definition files (server)

You must create two types of definition files in order to configure an environment for the active and standby servers that run on a host.

For details, see 8.4 Configuring a server environment.

6.17 Creating definition files (monitor-mode program)

Create the definition files needed to set up an environment for a program that will use the program management function on a server in the monitor mode.

For details, see 8.5 Configuring an environment for programs in the monitor mode.

6.18 Creating user commands

The user commands are created by the user and executed automatically by HA Monitor. If you create user commands, HA Monitor can issue them automatically when the status of a server or of HA Monitor changes. This enables the user to perform desired processing as appropriate to a change in the server's or HA Monitor's status.

Creation of user commands is optional. You create them as needed.

Important

We recommend that you specify the processing in created commands in such a manner that errors can be analyzed (by, for example, outputting logs as needed).

6.18.1 Timing of user command issuance: when the server status has changed

HA Monitor issues a user command automatically in response to a change in the server's or HA Monitor's status. There are multiple timings for status changes. HA Monitor specifies parameters corresponding to the timing in arguments and then issues the user command. If the server whose status has changed is in the server mode, you can specify whether HA Monitor is to issue a user command when processing begins or when it ends. For servers and resource servers in the monitor mode, HA Monitor can issue a user command only when processing begins. Note that HA Monitor continues its processing of the relevant server after the user command is completed, and depending on the parameters, this processing might be executed asynchronously with MA Monitor processing.

This subsection explains the parameters that are passed from HA Monitor when a user command is issued based on a change in the server's status and explains the timing of user command issuance.

(1) Parameters that are passed based on a change in the server's status

The following table lists and describes the parameters that are passed to user commands based on a change in the server's status.

Table 6‒15: List of parameters that are passed to user commands based on a change in the server's status

Server status

Server status parameter

Start and end parameters on the active server#1

Start and end parameters on the standby server#1

When a server starts

-s

start (start startup processing)

start (start startup processing)

end (terminate startup processing)#7

end (terminate startup processing)

When a server terminates normally

-e

start (start normal termination processing)#2

start (start normal termination processing)

--

end (terminate normal termination processing)

When a server is terminated according to the schedule

-p

start (start planned termination processing)#2

start (start planned termination processing)

--

end (terminate normal termination processing)

When the standby server terminates

-e

--

sbyend (start command processing)

--

--

Active server failure

When hot-standby switchover is possible

-a

start (start error handling processing)

start (start hot-standby switchover processing)

end (terminate error handling processing)

end (terminate hot-standby switchover processing)#7

When hot-standby switchover is impossible

-o#8

start (start error handling processing)

--

end (terminate error handling processing)

--

When HA Monitor waits for the active server to be restarted

-r#7

start (start waiting for restart)#2

--

--

--

When the restart limit of the active server is detected

-n#7

start (detects a restart limit)#3

start (detects a restart limit)#4

--

--

When the standby server fails

-a

--

sbyend (start error handling processing)

--

--

When a host fails

-h

--

start (start hot-standby switchover processing)#9

--

end (terminate hot-standby switchover processing)#7, #9

Planned hot standby

-w#5

start (start command processing)

start (start hot standby processing)

end (terminate command processing)

end (terminate hot-standby switchover processing)#7

When hot-standby switchover fails

-f

--

start (hot standby error)

--

--

Start retry limit error during hot standby processing#6

-t#7

start (start retry limit error notification)

--

--

--

Legend:

--: A user command is not issued.

Bold type: If the server whose status changed is a monitor-mode server or resource server, the parameter is not passed. Therefore, a user command is not issued in this case.

#1

The top item on a row is the start parameter, and the bottom item on a row is the termination parameter.

#2

end is not issued because the time is the same as for start.

#3

-a is issued after this parameter.

#4

If restart is specified in the switchtype operand in the server environment definition, -a is issued after this parameter. If manual is specified in the switchtype operand, nothing is issued after this.

#5

The user command is also started with -w in the following cases:

A failure occurs on the active server in the monitor mode.

When servers are grouped and hot standby processing is performed due to a failure on another active server (including the resource server).

#6

The parameter is passed if retry is specified in the switch_error operand in the server environment definition.

#7

This processing is executed asynchronously with HA Monitor processing.

#8

The user command is also started with the -o parameter if the server fails to start for any of the following reasons:

Server startup is canceled by the ip_neck, vg_neck, fs_neck, uoc_neck, or start_timeout operand specified in the server environment definition.

The wait-state server termination command (mondeact) is run while the server that failed to start is in the start retry state (>ONL).

In a case where yes is specified for waitserv_exec in the server environment definition, the start command specified for the name or actcommand operand in the server environment definition returns a result other than 0.

#9

When the active server is started by using the monact command in the online serverless mode, hot-standby switchover processing is assumed and this parameter is passed.

The following table lists and describes the parameters that are passed to user commands based on a change in the HA Monitor status.

Table 6‒16: List of parameters that are passed to user commands based on a change in the HA Monitor status

HA Monitor status

Start and end parameters on the local host

Detailed information parameter for a remote host

When HA Monitor starts on the local host

-m

Start startup processing

start

--

When HA Monitor terminates on the local host

-m

Terminate startup processing

end

--

When HA Monitor fails on a remote host

-m

--

-d host-name (HA monitor failure detected on the remote host indicated by host-name)#

Legend:

--: A user command is not issued.

#

When a standby server in the online serverless mode is started as the active server by using the monact command, the user command for this parameter is not executed.

(2) Timing of user command issuance (when the server status has changed)

This subsection explains the timing of user command issuance.

The following figure shows the parameters that are passed when the server is started and their issuance timing.

Figure 6‒40: Parameters that are passed when the server is started and their issuance timing

The following figure shows the parameters that are passed when the server is terminated normally and their issuance timing.

Figure 6‒41: Parameters that are passed when the server is terminated normally and their issuance timing

The following figure shows the parameters that are passed when planned termination is performed on the server and their issuance timing.

Figure 6‒42: Parameters that are passed when planned termination is performed on the server and their issuance timing

The following figure shows the parameters that are passed when the standby server is terminated and their issuance timing.

Figure 6‒43: Parameters that are passed when the standby server is terminated and their issuance timing

The following figure shows the parameters that are passed in the event of an active server failure and their issuance timing.

Figure 6‒44: Parameters that are passed in the event of an active server failure and their issuance timing

The following figure shows the parameters that are passed when hot standby processing cannot be performed and their issuance timing.

Figure 6‒45: Parameters that are passed when hot standby processing cannot be performed and their issuance timing

The following figure shows the parameters that are passed when the system is to wait until the active server is restarted and their issuance timing.

Figure 6‒46: Parameters that are passed when the system is to wait until the active server is restarted and their issuance timing

The following figure shows the parameters that are passed when the active server's restart wait limit time has elapsed and their issuance timing.

Figure 6‒47: Parameters that are passed when the active server's restart wait limit time has elapsed and their issuance timing

The following figure shows the parameters that are passed in the event of a standby server failure and their issuance timing.

Figure 6‒48: Parameters that are passed in the event of a standby server failure and their issuance timing

The following figure shows the parameters that are passed in the event of a host failure and their issuance timing.

Figure 6‒49: Parameters that are passed in the event of a host failure and their issuance timing

The following figure shows the parameters that are passed when planned hot standby processing is performed and their issuance timing.

Figure 6‒50: Parameters that are passed when planned hot standby processing is performed and their issuance timing

The following figure shows the parameters that are passed when an active server failure has occurred and hot standby processing has failed and their issuance timing.

Figure 6‒51: Parameters that are passed when an active server failure has occurred and hot standby processing has failed and their issuance timing

The following figure shows the parameters that are passed when an active server failure has occurred in a multi-standby configuration, but hot standby processing has failed and the server is to be switched over to another standby server and their issuance timing.

Figure 6‒52: Parameters that are passed when an active server failure has occurred in a multi-standby configuration, but hot standby processing has failed and the server is to be switched over to another standby server and their issuance timing

The following figure shows the parameters that are passed when a host failure has occurred and hot standby processing has failed and their issuance timing.

Figure 6‒53: Parameters that are passed when a host failure has occurred and hot standby processing has failed and their issuance timing

The following figure shows the parameters that are passed when a host failure has occurred in a multi-standby configuration, but hot standby processing has failed and the server is to be switched over to another standby server and their issuance timing.

Figure 6‒54: Parameters that are passed when a host failure has occurred in a multi-standby configuration, but hot standby processing has failed and the server is to be switched over to another standby server and their issuance timing

The following figure shows the parameters that are passed when planned hot standby processing has failed and their issuance timing.

Figure 6‒55: Parameters that are passed when planned hot standby processing has failed and their issuance timing

The following figure shows the parameters that are passed when planned hot standby processing has failed and the server is to be switched over to another standby server and their issuance timing.

Figure 6‒56: Parameters that are passed when planned hot standby processing has failed and the server is to be switched over to another standby server and their issuance timing

Figure 6‒57: Parameters that are passed in the event of a start retry limit error and their issuance timing

6.18.2 Timing of user command issuance: when the HA Monitor status has changed

HA Monitor issues a user command automatically in response to a change in the server's or HA Monitor's status. There are multiple timings for status changes. HA Monitor issues a user command with the correct parameters corresponding to the timing of the status change specified in an argument. Note that HA Monitor continues its processing of the relevant host after the user command is completed.

This subsection explains the parameters that are passed from HA Monitor when a user command is to be issued based on a change in HA Monitor's status and explains the timing of user command issuance.

(1) Parameters that are passed based on a change in HA Monitor's status

The following table lists and describes the parameters that are passed to user commands based on a change in HA Monitor's status.

Table 6‒17: List of parameters that are passed to user commands based on a change in HA Monitor's status

HA Monitor status

HA Monitor status parameter

Start and end parameters on the local host

Remote host details parameter

Start of HA Monitor on the local host

-m

start (start startup processing)

--

Termination of HA Monitor on the local host

end (terminate termination processing)

--

Detection of an error by HA Monitor on the remote host

--

-d host-name (HA Monitor on the remote host indicated as host-name has detected a failure)

Legend:

--: No parameter is passed.

(2) Timing of user command issuance (when the HA Monitor status has changed)

This subsection explains the timing of user command issuance.

The following figure shows the parameters that are passed when HA Monitor is started and terminated and their issuance timing.

Figure 6‒58: Parameters that are passed when HA Monitor is started and terminated and their issuance timing.

The following figure shows the parameters that are passed when HA Monitor on the remote host has detected a failure and their issuance timing.

Figure 6‒59: Parameters that are passed when HA Monitor on the remote host has detected a failure and their issuance timing

When HA Monitor on the remote host has detected a failure (-m -d host-name), a user command is issued only on the host that is paired with the host resulting in the failure and on which hot standby processing is to be performed.

6.18.3 User command issuance format: when the server status has changed

This subsection explains the format of user commands that are issued by HA Monitor when the server status has changed.

(1) Format of user commands issued by HA Monitor

When the server status has changed, HA Monitor issues a user command created by the user in the following format:

command-name -n server-alias-name -k server-type server-status-parameter start/end-parameter

(2) Parameters

Command name

Indicates the command name of a user command created by the user and specified in the HA Monitor environment settings.

-n server-alias-name

Indicates the server alias name of the server whose status change results in issuance of the user command.

-k server-type

Indicates the type of server indicated by server-alias-name. One of the following values is passed:

online: Active server

standby: Standby server

At the start and end of hot standby processing, and during a failure, the server type of the target server is standby.

Server status parameter

Indicates the status of the server indicated by server-alias-name. One of the following values is passed:

-s: Server active status

-e: Server normal termination status

-p: Server planned termination status

-a: Server failure status (hot standby processing could be performed after error processing terminated)

-o: Server failure status (hot standby processing could not be performed after error processing terminated)

-f: Server failure status (hot standby processing failed after error processing terminated)

-r: Server failure status (HA Monitor is to wait for the active server to be restarted after error processing terminated)

-n: Server failure status (the active server restart monitoring interval has elapsed after error processing terminated)

-h: Host failure status

-w: Planned hot standby status

-t: Start retry limit error during hot standby processing

start/end-parameter

Indicates whether this is the start or the end point of the status change (HA Monitor processing) on the server indicated by server-alias-name. One of the following values is passed:

start: Start point of the server status change (HA Monitor processing)

end: End point of the server status change (HA Monitor processing)

sbyend: Start point of the status change only on the standby server (HA Monitor processing)

For a server in the monitor mode or a resource server, only the following server-status-parameter and start/end-parameter combinations can be passed:

Server start status: -s start

Server normal termination status: -e start, -e sbyend

Server failure status (impossible to perform hot-standby switchover): -o start or -o end

Server failure status (standby server failure): -a sbyend

Server failure status (hot standby error): -f start

Host failure status: -h start

Planned hot standby status: -w start

Start retry limit error during hot-standby switchover: -t start

(3) Examples of user commands issued by HA Monitor

This example issues a user command (/usr/bin/usrcmd) when the active server (server) starts:

/usr/bin/usrcmd -n server -k online -s start

This example issues a user command (/usr/bin/usrcmd) in the event of a server failure on the active server (server) (when error processing begins on the active server):

/usr/bin/usrcmd -n server -k online -a start

This example issues a user command (/usr/bin/usrcmd) in the event of a server failure on the active server (server) (when error processing ends on the active server):

/usr/bin/usrcmd -n server -k online -a end

This example issues a user command (/usr/bin/usrcmd) in the event of a server failure on the active server (server) (when hot standby processing begins on the standby server):

/usr/bin/usrcmd -n server -k standby -a start

This example issues a user command (/usr/bin/usrcmd) in the event of a server failure on the active server (server) (when hot standby processing ends on the standby server):

/usr/bin/usrcmd -n server -k standby -a end

6.18.4 User command issuance format: when the HA Monitor status has changed

This subsection explains the format of user commands that are issued by HA Monitor when the HA Monitor status has changed.

(1) Format of user commands issued by HA Monitor

When the HA Monitor status has changed, HA Monitor issues a user command created by the user in the following format:

command-name HA-Monitor-status-parameter [start/end-parameter] [details-parameter]

(2) Parameters

command-name

Indicates the command name of a user command created by the user and specified in the HA Monitor environment settings.

HA-Monitor-status-parameter

Indicates the status of HA Monitor. The following parameter is passed:

-m: HA Monitor start or termination status or HA Monitor error status on the remote host

start/end-parameter

Indicates whether this is the start or the end point of the HA Monitor status change (HA Monitor processing). One of the following values is passed:

start: Start point of the HA Monitor status change (HA Monitor processing)

end: End point of the HA Monitor status change (HA Monitor processing)

details-parameter

If HA-Monitor-status-parameter indicates an HA Monitor failure status on the remote host, this parameter indicates the corresponding remote host. The following parameter is passed:

-d host-name: Name of the host on which an HA Monitor failure has occurred

If an HA Monitor failure is detected on the remote host (-m -d host-name), the user command is issued only on the host that is paired with the host resulting in the failure and that is subject to hot standby processing.

(3) Examples of user commands issued by HA Monitor

This example issues a user command (/usr/bin/usrcmd) when HA Monitor starts on the local host:

/usr/bin/usrcmd -m start

This example issues a user command (/usr/bin/usrcmd) when an HA Monitor failure is detected on the host named host:

/usr/bin/usrcmd -m -d host

6.18.5 How to create user commands

This subsection explains how to create user commands. You must specify a created user command in the usrcommand command in the HA Monitor environment settings.

You create one user command per host. You use a programming language, such as C language or shell language, to create user commands.

When a user command is issued, HA Monitor passes to the user command arguments that provide HA Monitor information and information about the server whose status has changed. The user can code desired processing for all types of status changes in a single file by using the arguments passed from HA Monitor to determine the status change on the server or on HA Monitor.

Specify the necessary operating environment in the user command. In RHEL7 or a later version, you can specify environment variables in the Unit settings file. However, the Unit settings file is overwritten when an overwrite installation of HA Monitor is performed. Therefore, we recommend that you specify an operating environment (environment variables) within the user command.

For details about the environment variables that are inherited when the user command is executed, see (2) How to create a server termination command in 6.12.2 Creating a server termination command.

Note the following about issuing a user command when the server status has changed:

If the processing to be performed on the active server differs from the processing to be performed on the standby server, you must specify processing for identifying the type of server.

If you run multiple servers on a single host and execute different processing on each server, you must specify processing for identifying the server alias name.

The server type and server alias name are passed as arguments from HA Monitor when a user command is executed. For details about the arguments that are passed from HA Monitor, see 6.18.3 User command issuance format: when the server status has changed.

Important

Because a user command is issued within HA Monitor processing, if looping occurs within the user command, HA Monitor processing cannot be guaranteed. You must create user commands carefully so that looping will not occur. If looping does occur in a user command, the server might remain indefinitely under start or termination processing.

Commands that can be specified in user commands:

You can execute HA Monitor commands from within a user command.

Note that HA Monitor commands cannot be executed from within a user command that is issued while HA Monitor is starting or terminating (-m start or -m end).

There is a command for inheriting information between active and standby servers.

If you use the command for specifying and displaying server inheritance information (moninfo command), you can inherit any information from the active server to a standby server. This command also enables you to view the information to be inherited.

For details about these commands, see 9. Commands.

Checking a user command's return value:

HA Monitor does not check a user command's execution results. User command execution results have no effect on the HA Monitor processing.

However, there is a function for checking user command execution results and then canceling server startup processing or hot standby processing. For a user command executed at the start of server startup (-s start) or hot-standby switchover (-w start, -a start, or -h start), you can enable the function by specifying the uoc_neck operand in the server environment definition. In this case, if the execution result (termination code) of the user command is not 0, the server startup or hot-standby switchover is canceled.

Synchronizing or not synchronizing HA Monitor processing with user command processing:

HA Monitor processing is performed synchronously with user command processing. However, termination of HA Monitor processing is not synchronized with termination of a user command in the following cases:

Termination of active server startup processing (-k online -s end)

Termination of hot standby processing that was performed due to a failure at the active server (-k standby -a end)

Termination of hot standby processing that was performed due to planned hot standby processing (-k standby -w end)

Termination of hot standby processing that was performed due to a host failure (-h end)

If HA Monitor is to wait for the active server to be restarted after a failure at the active server (-r)

If the restart limit is detected in the event of a failure at the active server (-n)

A start retry limit error was detected during hot standby processing (-t)

Creating user commands that control shared resources:

Note the following about creating user commands for controlling shared resources:

Connect the shared resources before the active server is started.

Shared resources must be connected before the active server is started so that the active server can use the shared resources.

Disconnect a resource that needs locking between hosts after the active server has terminated.

The resource must be disconnected after the active server has terminated in order to prevent the resource from being connected from both hosts at the same time.

Inherit information about shared resources from the active server to the standby server, if necessary.

To inherit information about shared resources from the active server to the standby server, you can use the command for specifying and displaying server inheritance information (moninfo command).

6.18.6 Coding example for a user command

This coding example for a user command changes the IP addresses of servers. Part of the coding of the user command is shown and explained in (1). The entire coding is shown in (2).

You can also change the IP addresses of servers by specifying a LAN status settings file. If you specify a LAN status settings file, there is no need to create a user command.

(1) Explanation of the user command shown in the coding example

This example adds and deletes IP addresses that are used by servers at the following times:

An IP address is added on the active server when the server starts (when the server startup processing begins) and nothing is performed on the standby server.

An IP address is deleted from the active server in the event of an active server failure (when error processing begins) and an IP address is added on the standby server.

The figure below shows the timing of these operations. The circled numbers in the figure correspond to the numbers in the partial coding example.

Figure 6‒60: Issuance timing shown in the coding example for a user command

The following figure shows the correspondence between the coding example for a user command and the issuance timing.

Figure 6‒61: Correspondence between the coding example for a user command and the issuance timing

Specify in (1) the processing that you want to perform on the active server when the server starts (when server startup processing begins). This example specifies ipaddr_add as the command used to add IP addresses.

Specify in (2) the processing that you want to perform on the standby server when the server starts (when server startup processing begins). This example specifies nothing because no action is to be performed.

Specify in (3) the processing that you want to perform on the active server in the event of a failure on the active server (when error processing begins). This example specifies ipaddr_delete as the command for deleting IP addresses.

Specify in (4) the processing that you want to perform on the standby server in the event of a failure on the active server (when error processing begins). This example specifies ipaddr_add as the command for adding IP addresses.

The following table shows the correspondence between the variables passed from HA Monitor and the user commands issued by HA Monitor for each of the issuance timings (1) through (4).

Table 6‒18: Correspondence between the variables passed from HA Monitor and the user commands issued by HA Monitor

Issuance timing

$4

$5

$6

User command issued by HA Monitor#

(1)

online

-s

start

/usr/bin/usrcmd -n server -k online -s start

(2)

standby

-s

start

/usr/bin/usrcmd -n server -k standby -s start

(3)

online

-a

start

/usr/bin/usrcmd -n server -k online -a start

(4)

standby

-a

start

/usr/bin/usrcmd -n server -k standby -a start

#: This example uses the following user command name and server alias name:

User command name: /usr/bin/usrcmd

Server alias name: server

(2) Coding example

This code is stored as a sample file under HA Monitor's sample file directory.

#!/bin/sh

# The definition of the 1st server
SERV1="server1"
IPADDR_SERV1="a.b.c.d"
BROADCAST_SERV1="e.f.g.h"
IFNAME_SERV1="ethX:Y1"

# The definition of the 2nd server
SERV2="server2"
IPADDR_SERV2="i.j.k.l"
BROADCAST_SERV2="m.n.o.p"
IFNAME_SERV2="ethX:Y2"

# The alias IP address is added to the LAN interface.
# The ARP caches are updated.
ipaddr_add()
{
        /sbin/ifconfig $WIFNAME inet $WIPADDR netmask 255.255.255.0 broadcast $WBROADCAST
        ARPIFNAME=`echo $WIFNAME | /bin/sed -e 's/:[0-9]*$//'`
        /sbin/arping -U -c 2 -I $ARPIFNAME $WIPADDR
        /bin/echo 0 > /proc/sys/net/ipv4/route/flush
        return 0
}

# The alias IP address is deleted from the LAN interface.
ipaddr_delete()
{
        /sbin/ifconfig $WIFNAME down
        /bin/echo 0 > /proc/sys/net/ipv4/route/flush
        return 0
}
#  Main
KIND_ONLINE="online"
KIND_STANDBY="standby"
SERV_START="-s"
SERV_END="-e"
SERV_PLANEND="-p"
SERV_ABORT="-a"
SERV_ABORT_NS="-o"
SERV_FAULT="-f"
SERV_HOSTDOWN="-h"
SERV_PLANSWAP="-w"
STATUS_START="start"
STATUS_END="end"
STATUS_SBYEND="sbyend"

# This user command terminates, when the state of HAmonitor has changed.
if [ "$1" = "-m" ]
then
        exit  1
fi

# Processing of the server
for serv in $SERV1 $SERV2
do
        if [ "$2" = "$serv" ]
        then
                case "$serv" in
                "$SERV1" )
                        WIPADDR="$IPADDR_SERV1"
                        WBROADCAST="$BROADCAST_SERV1"
                        WIFNAME="$IFNAME_SERV1"
                        ;;
                "$SERV2" )
                        WIPADDR="$IPADDR_SERV2"
                        WBROADCAST="$BROADCAST_SERV2"
                        WIFNAME="$IFNAME_SERV2"
                        ;;
                esac
                if [ "$4" = "$KIND_ONLINE" ]
                then
                        case "$5" in
                        "$SERV_START" )
                                [ "$6" = "$STATUS_START" ] && ipaddr_add
                                ;;
                        "$SERV_END" )
                                ipaddr_delete
                                ;;
                        "$SERV_PLANEND" )
                                ipaddr_delete
                                ;;
                        "$SERV_ABORT" )
                                [ "$6" = "$STATUS_START" ] && ipaddr_delete
                                ;;
                        "$SERV_ABORT_NS" )
                                [ "$6" = "$STATUS_START" ] && ipaddr_delete
                                ;;
                        "$SERV_PLANSWAP" )
                                [ "$6" = "$STATUS_START" ] && ipaddr_delete
                                ;;
                        esac
                else
                        case "$5" in
                        "$SERV_START" )
                                ;;
                        "$SERV_END" )
                                ;;
                        "$SERV_PLANEND" )
                                ;;
                        "$SERV_ABORT" )
                                [ "$6" = "$STATUS_START" ] && ipaddr_add
                                ;;
                        "$SERV_FAULT" )
                                ipaddr_delete
                                ;;
                        "$SERV_HOSTDOWN" )
                                [ "$6" = "$STATUS_START" ] && ipaddr_add
                                ;;
                        "$SERV_PLANSWAP" )
                                [ "$6" = "$STATUS_START" ] && ipaddr_add
                                ;;
                        esac
                fi
        fi
done

exit 0

6.19 Settings for resource monitoring

This section describes the settings for resource monitoring. Resources include LANs and disks.

6.19.1 Settings in the files required for monitoring a LAN

If you will be monitoring LANs and performing hot standby processing automatically in the event of failure, you must create LAN monitor definition files and specify the files. There are two types of LAN monitor definition files:

LAN monitor definition file (LAN-interface-name.conf)

You need to specify this file for each monitored target. Specify in the file the IP addresses of the check packet transmission targets for monitoring the LAN.

LAN monitoring script (lanpatrol.sh)

Prepare this file if HA Monitor monitors a LAN by monitoring ping responses or the number of received packets. If the monitored target is hbonding, and if only the hbonding status is to be monitored, there is no need to specify this file.

Create these files under the directory for HA Monitor environment settings for each server. This subsection explains how to specify the LAN monitor definition files.

For details about monitoring LANs, see 3.4.1 LAN monitoring and automatic hot standby in the event of a failure.

(1) Specifying a LAN monitor definition file

You specify in a LAN monitor definition file the IP addresses of the check packet transmission targets for monitoring a LAN. If you monitor only the hbonding status, the LAN monitor definition file must be empty (0 bytes).

You must specify this file for each monitored target before you start HA Monitor. A LAN monitor definition file cannot be added, changed, or deleted while HA Monitor is running.

The file name to be specified is LAN-interface-name.conf. For LAN-interface-name, specify the switchbyfail operand value that was specified in the server environment definition. Specifying a period (.) at the beginning disables the file.

The following rules apply to creating LAN monitor definition files:

One IP address or IP label must be specified per line.

Make sure that each IP address you specify is the IP address of a permanently activated remote host (such as a router) connected to the network to which the monitoring-target LAN interface is connected. The IP address used in the local node cannot be specified. If you specify an IP label, it might take time to resolve the host name, so we recommend that you specify an IP address.

If HA Monitor monitors the number of received packets, in addition to specifying the IP address of a remote host, always specify the IP address of the host on which the paired instance of HA Monitor runs.

Comments cannot be specified in these files.

The same IP address or IP label cannot be specified more than once.

A tab, space, or end-of-line code that immediately precedes or follows an IP address or IP label is ignored.

If HA Monitor monitors ping responses, you can specify 1 to 1,024 IP addresses or IP labels. If HA Monitor monitors the number of received packets, you can specify 2 to 1,024 IP addresses or IP labels.

If you specify IP labels in a LAN monitor definition file, specify each IP address and the corresponding IP label in the /etc/hosts file.

You must not create in the directory for HA Monitor environment settings any file with the extension .conf other than a LAN monitor definition file.

An example of a LAN monitor definition file is shown below. In this example, the LAN interface name is eth0, the physical IP address is 192.168.0.1, and the network is 192.168.0.

File name: eth0.conf

192.168.0.2
192.168.0.3
192.168.0.4

(2) Specifying a LAN monitoring script

Prepare this file if HA Monitor monitors a LAN by monitoring ping responses or the number of received packets. The name of the file to be set is lanpatrol.sh. If the monitored target is hbonding and only the hbonding status is to be monitored, there is no need to specify this file.

A sample is provided for this file under the directory for HA Monitor's sample files. Copy the sample file to the directory for HA Monitor environment settings and then use the copy. The following shows the contents of the sample file:

#!/bin/sh
#******************************************************************************
#*                                                                            *
#*    Linux(x86) HA Monitor                                                   *
#*    This is a sample of the LAN patrol command.                             *
#*                                                                            *
#*    [SYNOPSIS]                                                              *
#*        lanpatrol.sh interface [-r]                                         *
#*    [OPTIONS]                                                               *
#*        interface                                                           *
#*            The name of the LAN interface.                                  *
#*        -r                                                                  *
#*            Route monitoring method                                         *
#*    [DIAGNOSTICS]                                                           *
#*        0 : normal                                                          *
#*        1 : abnormal                                                        *
#*        2 : error                                                           *
#*                                                                            *
#*    All Rights Reserved. Copyright (C) 2011, 2018, Hitachi, Ltd.            *
#*                                                                            *
#******************************************************************************
#set -x

# retry interval(seconds) for receive packet number method. or
# return wait(seconds) of ping command for route monitoring method.
WAIT_TIME=3

# retry counter
RETRY=3

# environment variables
PATH=/usr/bin:/usr/sbin:/bin
export PATH

:
(Omitted)

Notes

Be careful if you create a LAN monitoring script with a sample file provided by HA Monitor whose version is earlier than 01-69. In this case, HA Monitor monitors the number of received packets independently of the setting of the lancheck_mode operand in the HA Monitor environment settings.

You can check the version of the sample file as follows:

If the [OPTIONS] section does not include a description of the -r option, the version is earlier than 01-69.

If the [OPTIONS] section includes a description of the -r option, the version is 01-69 or later.

The processing executed by this script differs depending on the monitoring method. The following shows the processing for each method.

If HA Monitor monitors ping responses:

Ping commands are issued in parallel for all IP addresses specified in the LAN monitor definition file by using the wait time specified (in seconds) in the WAIT_TIME environment variable. HA Monitor checks the return values from all ping commands that were issued in parallel, and if at least one of the return values is 0, HA Monitor judges that the LAN is in a normal state. If none of the return values is 0, HA Monitor retries issuing ping commands for all IP addresses specified in the LAN monitor definition file. The retry count can be set by using the RETRY environment variable. After all retries are completed, if none of the return values is 0, HA Monitor judges that the LAN is in an abnormal state.

If HA Monitor monitors the number of received packets:

The script issues the ping command to the IP address specified in the LAN monitor definition file.

When the time specified (in seconds) for the WAIT_TIME environment variable passes, the script determines repetitively the number of packets received by the monitored LAN interface and then compares the current value with the previous value. If the numbers of packets received do not match, the script determines that the LAN is functioning normally.

If they do match, the script retries issuance of the ping command and again determines the number of packets received. The number of retries can be set for the RETRY environment variable. If the number of packets received remains the same over all retries, the script determines that an error has occurred.

The command line to be specified in the LAN monitoring script differs depending on the monitoring method. The following shows the command line for each method.

If HA Monitor monitors ping responses:

lanpatrol.sh∆LAN-interface-name∆-r

If HA Monitor monitors the number of received packets:

lanpatrol.shΔLAN-interface-name

Legend:

Δ: Indicates a single-byte space.

You can edit the sample file, if necessary. The following subsections provide notes about the sample file when it is used as is and when it is edited.

(a) Using the sample file as is

If you use the sample file as is and specify a value in the lancheck_patrol operand that is smaller than the default (15) in the HA Monitor environment settings, you must reduce the LAN monitoring script execution time. You must specify a LAN monitoring script execution time that satisfies the following formula:

lancheck_patrol operand value > LAN monitoring script execution time + 2

The execution time of the LAN monitoring script differs depending on the monitoring method. The following shows the calculation expressions for each method.

If HA Monitor monitors ping responses:

WAIT_TIME variable value × RETRY variable value

For example, if a sample file for the LAN monitoring script is used, the execution time is 3 × 3 = 9 seconds.

If HA Monitor monitors the number of received packets:

(WAIT_TIME variable value + 1) × RETRY variable value

In the sample file, the LAN monitoring script execution time is (3 + 1) × 3 = 12 seconds.

To reduce the LAN monitoring script execution time, reduce the value of the WAIT_TIME or the RETRY variable. The minimum values of WAIT_TIME and RETRY variables are 0 and 1, respectively.

Note

If HA Monitor monitors the number of received packets, it takes 1 second to execute the ping command in the LAN monitoring script. The time required for monitoring received packets equals this value plus the time specified in the WAIT_TIME variable. Monitoring of received packets is repeated as many times as specified in the RETRY variable. Therefore, the LAN monitoring script execution time equals the received packet monitoring time times the count specified in the RETRY variable.

(b) Editing the sample file

The following notes apply to editing the sample file:

A LAN monitoring script is executed with the root permissions.

Grant execution permissions for a LAN monitoring script.

In a LAN monitoring script, specify any environment variable that is required.

Create a LAN monitoring script in such manner that it terminates with the EXIT code shown in the table below according to the status of the monitored LAN.

Table 6‒19: EXIT code returned according to the LAN status

Status of monitored LAN

EXIT code

Description

Normal

0

If the script determines that the LAN can be used successfully, it returns 0 for the EXIT code. HA Monitor determines that the LAN is normal or has recovered.

Failure

1

If the script determines that the LAN cannot be used, it returns 1 for the EXIT code. HA Monitor determines that a failure has occurred on the LAN. If the corresponding LAN interface is specified in the switchbyfail operand in the server environment definition, HA Monitor switches the active server.

Not monitorable

Value other than 0 or 1

If the script cannot identify the LAN status, it returns a value other than 0 or 1 for the EXIT code. Because the LAN cannot be monitored, HA Monitor will not switch the active server even if the corresponding LAN interface name is specified in the switchbyfail operand in the server environment definition.

When HA Monitor terminates normally or when the function for controlling hot standby based on the availability of LAN communications is used and a host failure is detected, SIGTERM is sent to the executing LAN monitoring script.

6.19.2 Settings in the files required for disk monitoring

You must specify settings in the necessary files to perform disk monitoring. This subsection describes the files required for monitoring system disks and disks for business use.

(1) Settings in the files required for monitoring the system disk

To monitor the system disk, you must perform the following operations:

Specifying the settings in the monitoring definition file for the system disk

Creating monitoring-target files

Specifying the settings in the monitoring definition file for the system disk

The monitoring definition file for the system disk is /opt/hitachi/HAmon/etc/system_patrol.disk. In this monitoring definition file, specify a file that exists on the system disk.

A sample file of the monitoring definition file is provided at /opt/hitachi/HAmon/lib/system_patrol.disk. Copy this file to /opt/hitachi/HAmon/etc/system_patrol.disk, and use a copied file.

The contents of the sample monitoring definition file are as follows:

/opt/hitachi/HAmon/spool/hamonitor_disk_patrol

In the sample file, /opt/hitachi/HAmon/spool/hamonitor_disk_patrol is specified as the monitoring-target file. If the /opt/hitachi/HAmon file system exists on the system disk and the system disk uses only one mount point, you can use the sample file without changing the file contents.

If the /opt/hitachi/HAmon file system is not placed on the system disk:

On the system disk, create a monitoring-target file in a desired file system, and specify the file in the monitoring definition file.

The following shows a specification example (in the case where /home exists on the system disk and the hamonitor_disk_patrol file is monitored).

/home/hamonitor_disk_patrol

If the system disk can use multiple mount points (such as /home, /var, and /opt):

Create a monitoring-target file in a desired file system for each mount point, and specify the files in the monitoring definition file.

The following shows a specification example (in the case where additional mount point /home can also be used and the /home/hamonitor_disk_patrol file is also monitored).

/opt/hitachi/HAmon/spool/hamonitor_disk_patrol
/home/hamonitor_disk_patrol

Do not create a monitoring-target file in virtual file systems such as /boot, /proc, and /sys.

Observe the following rules when specifying settings in the monitoring definition file:

Specify the absolute paths of the monitoring-target files on separate lines (one path on one line).

The path of each monitoring-target file can have a maximum of 247 bytes.

The path of each monitoring-target file must not include tabs or spaces.

Tabs or spaces before or after the path of a monitoring-target file are ignored.

The number of monitoring-target file paths that can be specified is 1 to 128.

Creating monitoring-target files

The monitoring-target files specified in the monitoring definition file must be created. You do not need to create the /opt/hitachi/HAmon/spool/hamonitor_disk_patrol file, which is automatically created when HA Monitor is installed.

A sample monitoring-target file is provided at /opt/hitachi/HAmon/lib/hamonitor_disk_patrol. Create the necessary number of copies of the sample file with the names specified in the monitoring definition file. For example, assume that the monitoring definition file contains the following settings:

/opt/hitachi/HAmon/spool/hamonitor_disk_patrol
/home/hamonitor_disk_patrol

In this case, copy the file as follows:

cp /opt/hitachi/HAmon/lib/hamonitor_disk_patrol /home/hamonitor_disk_patrol

(2) Settings in the files required for monitoring disks for business use

To monitor disks for business use, you must perform the following operations:

Specifying the settings in the monitoring definition file for the disk for business use

Creating monitoring-target files

Specifying the settings in the monitoring definition file for the disk for business use

The monitoring definition file for the disk for business use must be created at a specific location with a name in the following format: /opt/hitachi/HAmon/etc/server-alias-name.disk. You must set this monitoring definition file on each server. In this monitoring definition file, specify the following files:

Files at the mount points for the monitoring-target disks for business use

Observe the following rules when specifying settings in the monitoring definition file:

Specify the absolute paths of the monitoring-target files on separate lines (one path on one line).

The path of each monitoring-target file can have a maximum of 247 bytes.

The path of each monitoring-target file must not include tabs or spaces.

Tabs or spaces before or after the path of a monitoring-target file are ignored.

The number of monitoring-target file paths that can be specified is 1 to 128.

Creating monitoring-target files

The monitoring-target files specified in the monitoring definition file must be created.

A sample monitoring-target file is provided at /opt/hitachi/HAmon/lib/hamonitor_disk_patrol. Create the necessary number of copies of the sample file with the names specified in the monitoring definition file. For example, assume that the monitoring definition file contains the following settings:

/data1/hamonitor_disk_patrol
/data2/hamonitor_disk_patrol

In this case, copy the file as follows:

cp /opt/hitachi/HAmon/lib/hamonitor_disk_patrol /data1/hamonitor_disk_patrol
cp /opt/hitachi/HAmon/lib/hamonitor_disk_patrol /data2/hamonitor_disk_patrol

6.20 Configuring HA Monitor Extension

To use HA Monitor Extension, you must execute the HA Monitor environment setup command (monsetup -ex command) to specify settings needed for using the functions of HA Monitor Extension. For details about the HA Monitor environment setup command (monsetup -ex command), see 9.22 monsetup (specifies HA Monitor environment settings) in 9. Commands.

6.21 Checking the settings and definitions for the configured system

After you have configured the reset path and created the definition files, you must check the settings for errors. This section explains how to check the settings and definitions and the definition files between multiple hot-standby configurations.

6.21.1 Checking the settings and definitions between multiple hot-standby configurations

Regardless of the OS being used, if a system contains multiple hot-standby configurations in a network in which the same reset path is used and there is any duplication of address operand values in the HA Monitor environment settings or duplication of port numbers specified for HA Monitor's reset path, the following events might occur between the hot-standby configurations:

In the event of a host failure, a host containing a hot-standby configuration that has nothing to do with that host failure might also result in a failure.

When a host is rebooted, another host containing a hot-standby configuration that has nothing to do with the rebooted host might result in a failure.

To prevent these events in a system that contains multiple hot-standby configurations, you must check for duplications in the address operand values and the port numbers specified for HA Monitor's reset path. You can use the IP address detection command (monlistip command) to list the devices in the network that use the same reset path. For details about the IP address detection command (monlistip command), see 9.13 monlistip (detects IP addresses) in 9. Commands.

The following figure shows the procedure for checking the settings and definitions between multiple hot-standby configurations.

Figure 6‒62: Procedure for checking the settings and definitions between multiple hot-standby configurations

The following provides the details of the checking procedure, where the item numbers correspond to the step numbers in the figure.

To check settings and definitions:

List the devices in the reset path's network.

You can use either of the methods described below. Make sure that all devices are listed.

Execute the IP address detection command (monlistip command).

Execute this command from one of the hosts. The execution results are output to a CSV file. If you load this file on a computer running a Windows OS, you can edit it with a spreadsheet program. You can use the obtained results as the input in the steps that follow.

Contact the network administrator.

Check if HA Monitor can be applied to the devices.

Check the system and OS of each device listed in step 1 to determine whether the devices might be using HA Monitor. For the systems that support HA Monitor, see Release Notes. The OSs that support HA Monitor include AIX, HP-UX, and Linux. The following example shows how to check the OS:

Determine from the prompt that is displayed when you log in with telnet.

Execute the uname command in remote shell (rsh), and then determine from the result.

Check whether the devices are using HA Monitor.

For the devices that were determined in step 2 to have the possibility of using HA Monitor, check if they have an HA Monitor directory to determine whether they are actually using HA Monitor. Log in to each computer checked in step 2 and execute the following command:

ls /opt/hitachi/HAmon

If the four directories bin, etc, lib, and spool are found, you can conclude that the device is using HA Monitor.

Check the address operand values in the HA Monitor environment settings for all devices that are using HA Monitor.

Check the address operand values for any duplications in the HA Monitor that is used by multiple devices.

If an address operand value is duplicated, go to step 5. If none of the address operand values is duplicated, the check is now complete.

Execute the HA Monitor environment setup command (monsetup command) on all devices that are using HA Monitor to check the reset path settings.

In the HA Monitor environment setup command (monsetup command), specify the -tty option (if the OS is AIX) or -resetpath option (if the OS is HP-UX or Linux) to check the reset path settings.

The check items are as follows:

Port number of reset path

Port number of SVP or hot-standby mechanism at the connection destination

Port number of alternate reset path

Port number of alternate switchover mechanism at the connection destination

Check the port numbers for any duplications.

For each of the hosts that contains a duplicated address operand value (as checked in step 4), if there are duplications in any of the items checked in step 5, go to step 7. If there are no duplications, checking is now completed.

Correct the duplicated port numbers.

Separate the networks so that hosts using duplicated port numbers cannot communicate with each other.

6.21.2 Checking the definitions

You must check the definitions specified in the HA Monitor environment settings, server environment settings, and monitor-mode program environment settings to make sure that there are no errors. To perform a definition check, you use the definition check command (moncheck command).

When HA Monitor and servers start, a definition check is performed automatically. An HA Monitor with invalid definitions cannot be started correctly. Therefore, before you start HA Monitor, perform a definition check to make sure that there are no errors in the definition files. You can perform a definition check not only during the configuration stage but after you have edited the definition files. For details about the definition check command (moncheck command), see 9.7 moncheck (checks definitions) in 9. Commands.

The following items are checked by definition checking:

Basic rules

Whether the basic rules for definitions are observed

Specification formats

Whether the specification formats are correct for definitions

Definition statements

Whether the definition statements are correct

Whether the formats of the definition statements are correct

Whether all the required definition statements are specified

Whether more than one set of HA Monitor environment settings has been specified

Operands

Whether the operands are correct

Whether the operand formats are correct

Whether all the required operands are specified

Values

Whether all the required values are specified

Whether the character specification methods are correct

Whether the values are within valid value ranges

Whether the specified number of values is correct when multiple values are specified

Whether the same value is specified in an operand for shared resources between multiple servers

Characters

Whether all specified characters and symbols are valid

6.22 Settings for collecting error information

HA Monitor enables you to run a shell to collect information that can be used to determine the cause of a server slowdown. You must create the shell before you start HA Monitor.

For details, see 7.6.4 Determining the cause of a server slowdown.

6.23 Checking the operation of the configured system

After you have completed the system configuration and have checked the settings and definitions, you must check the operation of the configured system. In the operation check, you actually start and terminate HA Monitor and the servers and perform the hot standby operation to determine whether HA Monitor runs successfully.

Note the following check items when you check the system operation:

Whether HA Monitor and the servers can be started and terminated normally

Whether there are any error messages

Whether servers and clients can communicate with each other

Whether job processing using servers can be performed successfully

Whether shared resources can be used from target hosts after hot standby processing

The following figure shows the procedure for checking system operation.

Figure 6‒63: Procedure for checking system operation

#1: When HA Monitor is terminated, the HA Monitor connection configuration file is also checked.

#2: Performed only when the monitor-mode program management function is used.

6.23.1 Prior confirmation

Before you start checking the operation of the system that you have configured, confirm the following so that this system will not affect other systems or will not become corrupted:

The address operand values differ from those at any other host or system.

The volume groups set as shared resources will not connect automatically with a file system when the OS starts.

If all the following conditions are satisfied, a host in an unrelated hot-standby configuration might result in a failure:

The applicable model is BladeSymphony, HA8000, or RV3000.

Hot-standby switchover based on host reset is selected as the switchover method.

An address operand value in the HA Monitor environment settings is the same as that in any other hot-standby configuration.

Confirm that the address operand values in the HA Monitor environment settings are unique from any other hot-standby configuration by following the procedure described in 6.21.1 Checking the settings and definitions between multiple hot-standby configurations.

If a volume group set as a shared resource will connect automatically to a file system when the OS starts, the shared disk might be accessed from both hosts at the same time, resulting in corruption of data on the shared disk and in the file system.

Specify the settings so that a volume group set as a shared resource will not be connected automatically to a file system when the OS starts by following the procedures described in 6.3.9 Setting the file system and 6.11.1 Configuring a shared disk.

6.23.2 Checking the HA Monitor operation

After you have finished checking the settings and definitions, you must check the start, status, and termination of HA Monitor to verify its operation. When you check the termination of HA Monitor, you can automatically create an HA Monitor connection configuration file. The procedure for checking the HA Monitor status described below assumes that you automatically create the connection configuration file.

(1) Checking startup of HA Monitors

Start all HA Monitors that are connected in the primary and secondary systems.

If you set HA Monitors to be started manually, execute the HA Monitor start command (monstart command) and manually start all HA Monitors. If the KAMN002-I message is issued, HA Monitor has started.

If you have set HA Monitors to be started automatically, restart the OS to confirm that HA Monitors start automatically.

(2) Checking the HA Monitor status

Check if all the HA Monitors that have started are connected correctly. In this step, check the HA Monitor status for the following:

Hosts

Execute the server and host status display command (monshow -c command).

The example shown below is the command execution results when HA Monitor recognized the hosts successfully. If all the hosts connected to HA Monitor are displayed exactly as specified, HA Monitor has successfully recognized the hosts.

KAMN335-I  Connected host information
       Host name     Host address Patrol time
       host1              100          10
       host2              200          10

Monitoring paths

Execute the monitoring path status display command (monpath command).

The example shown below is the command execution results when HA Monitor recognized the monitoring paths successfully. If the monitoring path statuses shown under status are all shown as OK, HA Monitor has successfully recognized the monitoring paths.

KAMN390-I Path status Display
device name host name status
path11      host2     OK  *
path12      host2     OK

Reset path (when a host reset is performed)

Execute the reset path status display command (monrp command).

The example shown below is the command execution results when HA Monitor recognized the reset path successfully. If the reset path status shown under status is shown as OK, HA Monitor has successfully recognized the reset path.

KAMN395-I Reset Path status Display
host name                         status
host2                             OK

(3) Checking termination of HA Monitors and checking the connection configuration file (that is created automatically)

After you have verified that the HA Monitors that were started were connected correctly, you must check if the HA Monitors can be terminated. When HA Monitor is terminated, an HA Monitor connection configuration file is created automatically. Therefore, you must also check the contents of the HA Monitor connection configuration file.

To check the stoppage of HA Monitors and check the connection configuration file:

Terminate all HA Monitors that are to be connected.

Manually terminate the HA Monitors that are to be connected. You can terminate them by executing the HA Monitor termination command (monstop command). If the KAMN050-I message is issued, an HA Monitor has been terminated.

An HA Monitor connection configuration file is created on each host where an HA Monitor was terminated.

Check the contents of the connection configuration file that was created automatically.

Check the contents of the connection configuration file that was created automatically for any errors. The contents of the automatically created connection configuration file are the same on all hosts.

If the connection configuration file is created automatically, the configuration cannot be changed or deleted. If you want to set a monitoring path directly on the local host and a remote host that will monitor each other or you will be changing or deleting the configuration, you must create the connection configuration file manually.

If you created a connection configuration file manually, confirm that connection information for HA Monitor is not added when HA Monitor is terminated. If connection information for HA Monitor is added, check and, if necessary, revise the connection configuration file and the monitoring path settings.

If you want to increase the number of HA Monitors to be connected, perform the procedure from start to termination of HA Monitor again. Connection information for the new HA Monitor will be added automatically to the connection configuration file. If you have added an HA Monitor automatically, all the items in the existing connection configuration file, except for comment lines, will be changed automatically.

If you have set HA Monitors to be terminated automatically, also confirm that HA Monitors are terminated automatically. When you shut down the OS, the KAMN056-I message is output to syslog.

6.23.3 Checking the server operation

Check the server operation after you have started HA Monitor.

(1) Checking the start of servers

You must check if the servers can start successfully. If the KAMN252-I message is issued, a server has started and is ready for hot standby processing.

The following describes how to start servers.

For servers in the server mode:

Start the servers by using the corresponding program's start method. HA Monitor determines whether a server is to be started as the active server or a standby server. This determination is made based on the specified server environment definition and the status of the remote host.

For servers in the monitor mode:

Start the servers by executing the monitor-mode server start command (monbegin command). HA Monitor determines whether a server is to be started as the active server or a standby server. This determination is made based on the specified server environment definition and the status of the remote host.

If you have specified a server start timeout value (start_timeout) in the server environment definition, confirm that the specified server start time is appropriate. If the start time is affected by recovery processing during server startup or by the number of preceding processes, also take these factors into consideration when you confirm the server start time.

Confirm by using a method such as a server-provided command that all processes making up a server have started successfully.

Also confirm that the shared resources specified in the server environment definition are connected successfully. For details about how to check the connection status of shared resources, see the OS documentation.

(2) Checking the server status

After you have confirmed that the servers start successfully, you must check the server status to determine whether HA Monitor recognized that the servers started. To check the server status, you execute the server and host status display command (monshow command).

The example of command execution results shown below indicates that HA Monitor was able to recognize server startup. If the active server's status is ONL and the standby server's status is SBY, HA Monitor has recognized the servers correctly.

#/opt/hitachi/HAmon/bin/monshow
KAMN213-I Own host name : host1
    Own servers               Pair servers
    Alias     Status          Status     Host name
    server1   ONL             SBY        host2

If the server is in the monitor mode and the program management function is used, also check the program status. To check the program status, execute the server and host status display command (monshow command) with the -u option specified.

Shown below is an example of command execution results when HA Monitor was able to recognize startup of a program. If the program status is ACT, HA Monitor has recognized the program startup correctly.

#/opt/hitachi/HAmon/bin/monshow -u
KAMN213-I Own host name : host1
     Alias     Program Alias  Status   Patrol time
     server1   UAP1            ACT        3600

(3)  Checking the restart of servers

If you have set the local host to be restarted in the event of a server failure, generate a server failure to confirm that the server will restart successfully.

A server in the server mode has been restarted successfully if the KAMN255-D and KAMN254-I messages are issued in this order.

A server in the monitor mode has been restarted successfully if the KAMN275-E and KAMN274-I messages are issued in this order.

(4) Checking the restart of programs

If you have used the monitor-mode program management function to create a program restart command, confirm that the program can be restarted successfully.

After you have confirmed that the servers start successfully, generate a program error to check the status of the program. You can determine whether HA Monitor restarted the program. To check the program status, execute the server and host status display command (monshow command) with the -u option specified.

Shown below is an example of command execution results when HA Monitor has successfully recognized program restart. If the program status is ACT, HA Monitor successfully recognized program restart.

#/opt/hitachi/HAmon/bin/monshow -u
KAMN213-I Own host name : host1
     Alias     Program Alias  Status   Patrol time
     server1   UAP1            ACT        3600

(5) Checking the termination of servers

You must check if the servers can be terminated successfully. If no servers are displayed when you execute the server and host status display command (monshow command), the server termination processing has been completed.

The following describes how to terminate servers.

If the servers were started in the server mode:

Terminate the servers by using the corresponding program's termination method. If the corresponding standby server is running, HA Monitor will terminate it.

If the servers were stated in the monitor mode:

The user must terminate all servers manually by executing the monitor-mode server termination command (monend command).

If you have specified a server termination timeout value (stop_timeout or stop_timeout_sw) in the server environment definition, confirm that the specified server termination time is appropriate. If the server termination time exceeds the specified value, HA Monitor forcibly terminates the server termination command and outputs the KAMN293-E message to syslog. If the termination time is affected by other processing, such as server termination processing during job processing, also take such factors into consideration when you confirm the server termination time.

Use a method such as a server-provided command to confirm that all processes making up a server have terminated successfully.

Also confirm that the shared resources specified in the server environment definition have disconnected successfully. For details about how to check the connection status of shared resources, see the OS documentation.

6.23.4 Testing hot standby processing

After you have finished checking the operation of HA Monitors and servers, you must test hot standby processing. Perform this test after you have started HA Monitors and servers.

(1) Checking planned hot standby

Execute the planned hot-standby switchover command (monswap command) to test planned hot-standby switchover. After that, execute the server and host status display command (monshow command).

Shown below is an example of the execution results of the server and host status display command (monshow command) when planned hot standby processing performed from host 1 to host 2 was successful. If the status of the target server is ONL, the planned hot standby processing was successful.

#/opt/hitachi/HAmon/bin/monshow
KAMN213-I Own host name : host2
    Own servers               Pair servers
    Alias     Status          Status     Host name
    server2   ONL

If you are using the multi-standby function, confirm that servers have started successfully in all secondary systems.

(2) Checking automatic hot standby in the event of a server failure

The user must generate a server failure to check whether HA Monitor executes automatic hot standby processing successfully. After the failure has been recovered, execute the server and host status display command (monshow command).

Shown below is an example of the execution results of the server and host status display command (monshow command) when automatic hot standby processing from host 1 to host 2 was successful. If the status of the target server is ONL, the automatic hot standby processing was successful.

#/opt/hitachi/HAmon/bin/monshow
KAMN213-I Own host name : host2
    Own servers               Pair servers
    Alias     Status          Status     Host name
    server2   ONL

(3) Checking automatic hot standby in the event of a program error

If a server is in the monitor mode and the program management function is used, the user must generate a program error to check whether HA Monitor performs hot standby processing automatically. After the error has been recovered, execute the server and host status display command (monshow command) with the -u option specified.

Shown below is an example of the execution results of the server and host status display command (monshow command) when automatic hot standby processing was successful after a program error. If the program status is ACT, the automatic hot standby processing was successful.

#/opt/hitachi/HAmon/bin/monshow -u
KAMN213-I Own host name : host2
     Alias     Program Alias  Status   Patrol time
     server2   UAP1            ACT        3600

(4) Checking automatic hot standby in the event of a host failure

The user must generate a host failure to check whether HA Monitor executes automatic hot standby processing successfully. If you have specified VMware ESXi-based virtualization in the reset path configuration, you must verify that the reset path is set correctly by generating a host failure on the local host and then checking whether the virtual machine is reset.

After the failure has been recovered, execute the server and host status display command (monshow command).

You can generate a host failure by forcibly terminating the host.

Shown below is an example of the execution results of the server and host status display command (monshow command) when automatic hot standby processing from host 1 to host 2 was successful. If the status of the target server is ONL, the automatic hot standby processing was successful.

#/opt/hitachi/HAmon/bin/monshow
KAMN213-I Own host name : host2
    Own servers               Pair servers
    Alias     Status          Status     Host name
    server2   ONL

If a file system is created on a shared disk, it might take time to perform a file system integrity check when the server is started. Note that a timeout might occur if the dev_timelimit, dev_offlimit, or dev_onlimit operand (shared-resource switchover timeout value) is specified in the server environment definition for a server that will use the file system. Therefore, perform hot-standby switchover resulting from a host failure to confirm that a timeout does not occur.

After you have performed hot standby processing resulting from a host failure, confirm that OS dump files are output successfully and that neither setting errors nor capacity shortages occur. For details about how to verify that dump file output processing was successful, see the Linux Tough Dump documentation.

If hosts are to be reset to protect data on shared disks, but a specified reset path is invalid, a wrong host might be reset. To check the validity of the reset path settings, generate a host failure on each host and confirm that hot standby processing is performed correctly. If you use the physical partition reset function in a virtualization environment created by either Hitachi server virtualization (Virtage) or VMware ESXi, check hot standby processing resulting from a failure in the guest OS as well as hot standby processing resulting from a host failure caused by a failure in the host OS and hardware.

If you use SCSI reservation for shared disk, confirm that all shared disk reservations were obtained in the active system after hot standby processing. For details about how to check the reservation status of shared disks, see 4. Obtain the reserved status for the disk in 7.5.7 Handling device failures on a shared disk (while the active server is running) (using SCSI reservation for shared disk).

6.23.5 Testing the system load

After you have tested hot standby processing, test the system load. Run the system in the same manner as for actual operations to check for any problem with the host failure monitoring time or the server failure monitoring time specified in the definition files.

To test the system load, use monitoring history. For details about the monitoring history, see 7.6 Operation for preventing failures caused by heavy load.

If you perform operations in unbalanced or reduced mode after hot standby processing in a mutual hot-standby configuration or a multi-standby configuration, confirm that no slowdown occurs during operation with as much load as during actual operations and that the system throughput and response performance requirements are satisfied.

Once you finish testing the system load, checking of system operation is complete.

6.24 Checking the operation when the maximum number of hosts is changed (when using HA Monitor Extension)

If you use HA Monitor Extension to change the maximum number of hosts, you must check whether all packets are transmitted completely via monitoring paths. Perform this checking on all hosts in the hot-standby configuration.

Before performing the procedure shown here, if necessary, install the net-tools package so that the netstat command can be used.

The following figure shows the checking procedure.

Figure 6‒64: Procedure for checking the operation when the maximum number of hosts is changed

To check the operation shown in the figure:

Start HA Monitors on all hosts in the hot-standby configuration.

Use the server and host status display command (monshow -c command) to check whether HA Monitors started successfully.

For details about starting HA Monitors, see (1) Starting HA Monitor.

Start servers on all hosts in the hot-standby configuration to enable hot standby processing.

Use the server and host status display command (monshow command) to check whether the systems are ready for hot standby processing, or check messages to determine whether the systems are ready for hot standby processing.

For details about starting servers, see (3) Starting servers.

Check UDP statistical information for a change in UDP packet reception errors.

Execute the netstat -s -u command to obtain the value for packet receive errors in the UDP statistical information. Repeat this step several times at the host failure monitoring interval (specified in the patrol operand in the HA Monitor environment settings) to check for any upward trend.

If there is no change in the UDP packet reception errors values, packets are being received completely, in which case the operation check is finished.

If the values for UDP packet reception errors are increasing, go to step 4.

Adjust the alive message transmission interval.

Whether all packets are transmitted completely depends on the machine's performance and environment, while the performance required by HA Monitor depends on the volume of alive messages that are transmitted. Therefore, use the alive_interval operand in the HA Monitor environment settings to adjust the alive message transmission interval and the volume of alive messages to be transmitted.

Adjust the alive_interval operand value as follows:

If the alive_interval operand value can be increased

Increase the alive_interval operand value.

If the alive_interval operand value is already set to the maximum

Delete the alive_interval operand from the HA Monitor environment settings.

Check the operation again.

Repeat steps 1 through 4. If there is no change in the value for UDP packet reception errors in step 3, the operation check when the maximum number of hosts is changed is finished.

Important

Specify in the alive_interval operand the same value on all hosts in the hot-standby configuration. If you have changed the alive_interval operand value on one of the hosts, also change the operand's value on all other hosts in that hot-standby configuration.

7 System Operation

This chapter discusses the operation of the configured system.

It explains how to start and terminate the system, the actions to be taken by the operator when a failure occurs, how to prevent failures, how to automate operations, and how to modify the system.

7.1 Operation flow

The following figure provides an overview of the operation flow in a hot-standby configuration that includes HA Monitor.

Figure 7‒1: Operation flow in a hot-standby configuration

7.1.1 Preparations for operation

You must specify the PATH environment variable before you start system operation because HA Monitor commands will execute during operation. Add /opt/hitachi/HAmon/bin to the operator's PATH environment variable.

The two types of HA Monitor commands are the commands that can be executed by superusers only and the commands that can be executed by both superusers and general users. For details, see 9. Commands.

7.2 Starting and terminating

This section explains how to start and terminate a system that uses HA Monitor. You can also have the system start automatically. For details about performing the start procedures automatically, see 7.7.1 Automating the operation from system start through server start.

7.2.1 Starting

This subsection explains how to start the system, including the procedure from powering up the computer to reaching the system status in which servers are ready for the hot standby operation.

To start the system, you must start both the active system and the standby system. The following figure shows the procedure for starting the system.

Figure 7‒2: System start procedure

When the power has been turned on at both hosts and the servers have started, the two HA Monitors communicate with each other to check that they are ready for the hot standby operation. The KAMN252-I message is issued at this point.

In the case where host reset is performed with model HA8000xN or later:

If a management server is used as a failure management processor:

Before starting HA Monitor, you must start the management server and HA Monitor Connector must be started on the management server.

If HA Monitor is started first, the reset path configuration fails. Therefore, hot-standby switchover is not performed automatically if a host failure occurs. In such a case, use the following procedure to configure the reset path:

Start the management server, HCSM, and HA Monitor Connector.

Execute the reset path status display command (monrp command) on all hosts.

If a management server is not used as a failure management processor:

Before starting HA Monitor, you must start the services of HA Monitor Connector that manages the device on which the HA Monitor will operate.

If HA Monitor is started first, the reset path configuration fails. Therefore, hot-standby switchover is not performed automatically if a host failure occurs. In such a case, use the following procedure to configure the reset path:

Start HA Monitor Connector.

Execute the reset path status display command (monrp command) on all hosts.

Important

Do not terminate the management server and HA Monitor Connector while HA Monitor is operating. If you terminate it, the HA Monitors on all monitoring-target hosts detect a reset path failure. If system operation continues in that status, it is impossible to reset a host on which a host failure occurs. Therefore, hot-standby switchover cannot be performed automatically. In such a case, restart the management server or HA Monitor Connector, and then execute the reset path status display command (monrp command) on all hosts.

The following subsections explain how to start HA Monitor, the resource server, and servers, as shown in Figure 7-2.

(1) Starting HA Monitor

To start HA Monitor manually, you execute the HA Monitor start command (monstart). After you have started HA Monitor, make sure that the KAMN002-I message indicating that the HA Monitor startup processing is completed is displayed.

When HA Monitor is to be started automatically, it is started by the OS after the kernel has started.

When HA Monitor is first installed, it is set to be started automatically. To change the HA Monitor start method, execute the HA Monitor environment setup command (monsetup command). For details about the HA Monitor environment setup command (monsetup command), see 9.22 monsetup (specifies HA Monitor environment settings) in 9. Commands.

(2) Starting a resource server

A resource server is started automatically when a server that has a resource server as its parent server is started. You specify a parent server in the server environment definition.

A resource server is terminated automatically by HA Monitor when all servers using the resource server have terminated.

If you are starting a resource server manually for a reason such as a shared resource connection error, make sure that the KAMN002-I message indicating that the HA Monitor startup processing is completed is displayed and then execute the resource server start command (monresbgn command).

(a) Action to be taken if the system is placed in wait for startup of the active server

When the resource server is started, the system might be placed in wait for startup of the active server. For cases in which the system is placed in wait for startup of the active server, see 4.1.1 HA Monitor's control of server startup and 4.5.3 Controlling the server start and termination (multi-standby).

For details about the action to be taken if the system is placed in wait for startup of the active server, see 7.4.1 Starting a server in the wait state and then restarting jobs.

(3) Starting servers

The servers in both the active system and the standby system must be started. Before you start servers manually, make sure that the KAMN002-I message indicating that the HA Monitor startup processing is completed is displayed.

To start a server in the server mode, you execute the start command provided by the program. When you start a server, it creates an environment for the hot standby operation by communicating with HA Monitor during the server initialization process. A server started in the standby system becomes the standby server and prepares for any failure that might occur in the active system.

To start a server in the monitor mode, you use the monitor-mode server start command (monbegin command). A server started in the standby system becomes the standby server and prepares for any failure that might occur in the active system.

After you have started a server, make sure that the KAMN251-I message indicating that the server startup processing is completed is displayed. The KAMN251-I message is displayed for each server. If multiple servers are run on the same host, this message is displayed for each of them.

(a) Points to be noted when servers are grouped together

When multiple servers are grouped together, all the servers in the group are started. The servers are started in the order specified by the operator. After you have started grouped servers, make sure that the KAMN251-I message is displayed for all of the servers in the group.

(b) Action to be taken if the system is placed in wait for startup of the active server

When servers are started, the system might be placed in wait for startup of the active server. For cases in which the system is placed in wait for startup of the active server, see 4.1.1 HA Monitor's control of server startup and 4.5.3 Controlling the server start and termination (multi-standby).

For details about the action to be taken if the system is placed in wait for startup of the active server, see 7.4.1 Starting a server in the wait state and then restarting jobs.

7.2.2 Terminating

This subsection explains the system termination procedure and methods. When the system is to be terminated, the servers are terminated and then the kernel is shut down. HA Monitor terminates automatically when the kernel is shut down.

This subsection explains how to terminate the servers, the resource server, and HA Monitor.

(1) Terminating the servers

You can terminate both active and standby servers with the following methods:

In the server mode

In the active system, execute the termination command provided by the program.

If a corresponding standby server is running in the standby system, HA Monitor terminates it.

In the monitor mode

In the active system, execute the monitor-mode server termination command (monend command). If a corresponding standby server is running in the standby system, HA Monitor terminates it.

To terminate the standby server only, execute the standby server termination command (monsbystp command).

To verify that the server has terminated, execute the server and host status display command (monshow command).

To terminate a server in the monitor mode without creating a server termination command:

Use the termination command provided by the program to terminate the server.

Execute the monitor-mode server termination command (monend command).

(2) Terminating the resource server

HA Monitor terminates the resource server automatically when all servers using the resource server have terminated.

Occasionally, you might need to terminate the resource server for a reason such as a shared resource disconnection error.

To terminate the resource server manually:

Check that all the servers using the resource server have terminated.

The resource server is connected to the shared resources during operation. Once the resource server is terminated, the servers using the resource server can no longer use its shared resources. To make sure that all the servers that have specified the resource server as their parent server have been terminated, execute the server and host status display command (monshow command).

Terminate the resource server.

How the resource server is terminated depends on whether both the running resource server (in the active system) and the standby resource server (running in the standby system) or only the resource server on standby is to be terminated.

When terminating both the running and the standby resource servers

Execute the active resource server termination command (monresend command).

When terminating only the standby resource server

Execute the standby resource server termination command (monressbystp command).

(3) Terminating HA Monitor

You can terminate HA Monitor manually by executing the HA Monitor termination command (monstop command). After you have terminated HA Monitor, check that the KAMN050-I message indicating that HA Monitor has been terminated is displayed. Note that HA Monitor cannot be terminated while servers are running. You must first terminate the servers and then terminate HA Monitor. To verify that the servers have been terminated, use the server and host status display command (monshow command).

If you have set HA Monitor to terminate automatically, it terminates when the kernel is shut down. If the kernel is shut down while servers are running, HA Monitor forcibly terminates the active server on the active system and normally terminates the standby server on the standby system without performing hot standby processing. It is advisable to terminate the active server normally before you shut down the kernel. You can change the HA Monitor termination method by executing the HA Monitor environment setup command (monsetup command). For details about the HA Monitor environment setup command (monsetup command), see 9.22 monsetup (specifies HA Monitor environment settings) in 9. Commands.

7.2.3 Notes about maintaining shared resources

This subsection provides important information about connecting and disconnecting the shared resources when you perform maintenance on a shared disk after operation. For the specific operating methods, see the OS documentation.

Important

Pay attention to the notes provided here when you manipulate shared resources. If you ignore these notes, the shared resources might become corrupted or cause hot standby errors.

(1) Before you manipulate shared resources

HA Monitor controls the connection and disconnection of shared resources. Therefore, you must note the following:

While the servers are running, do not manipulate the shared resources from the outside such as by using commands.

If contention occurs between a user operation and the HA Monitor processing, the HA Monitor will not be able to control the system correctly.

If you need to manipulate shared resources for purposes such as maintenance, be sure to first terminate the servers.

For details about HA Monitor's control of shared resources, see 4.3.4 List of shared resource statuses.

(2) Handling volume groups

If a volume group created on a shared disk is accessed for referencing and updating from both hosts at the same time, operation is not guaranteed. Therefore, you must note the following:

Before you start the servers, make sure that the volume groups are disconnected from both hosts.

When a server is started, HA Monitor connects the corresponding volume group for referencing and updating. If the volume group is already connected when the server is started, the volume group might be referenced and updated from both hosts, resulting in data corruption.

After a server is terminated, HA Monitor disconnects the corresponding volume group. If another job is to use the volume group, connect it by executing the vgchange command.

In this case, make sure that the volume group is not referenced or updated from both hosts.

The following shows the vgchange command's execution formats:

Connecting for referencing and updating

vgchange -a y path-name-of-volume-group

Disconnecting

vgchange -a n path-name-of-volume-group

(3) Handling file systems

This subsection provides important information about handling file systems.

Before you start a server, check that the file system is unmounted at both hosts.

Mounting of a file system might fail if a process is accessing the mount point (target directory for mounting) before HA Monitor mounts the file system.

Avoid such handling of a file system.

If the operator uses a file system for another job, make sure that the file system is not mounted from both hosts.

If a file system created on the shared disk is mounted from both hosts at the same time, the file system itself becomes corrupted.

For the timing of mounting and unmounting of file systems by HA Monitor, see 4.3.2 Managing file systems.

(4) Handling of a shared disk after OS migration

The handling of a shared disk is not the same in Linux as in HI-UX/WE2 and AIX. If you have migrated your OS from HI-UX/WE2 or AIX to Linux, you must check and, if necessary, revise the handling of a shared disk and use it carefully.

For example, in HI-UX/WE2 and AIX, a shared disk can be accessed after the active server has terminated. In Linux, a shared disk cannot be accessed after the active server has terminated. In order to access a shared disk in Linux for a purpose such as making a backup, you must first place the shared disk in accessible status.

7.3 Checking the server logs

This subsection describes how to check the server logs if a server failure is detected by the server monitoring command. Note that the server logs generically refer to the following logs:

Execution log for the monitoring command

Details log for the monitoring command

7.3.1 Checking the execution log for the monitoring command (when the ptrlcmd_ex or sby_ptrlcmd_ex operand is specified on a monitor-mode server)

The start result, termination result, and execution status of the monitoring command for a server are output to the execution log for the monitoring command. The execution status includes skipped execution when the previously executed monitoring command has not returned the result yet.

You can examine the execution results and statuses of the monitoring command for a server by checking the log for the server. While the system is operating, the contents of the monitoring command execution log are always updated.

If a server failure is detected by the monitoring command for a server, HA Monitor makes a backup file (.bak) of the log. If the .bak file already exists, the extension is changed to .bak2, and the log is backed up to a new .bak file. Three generations of log data are kept in this manner.

Important

If you need to check a server log while the server is operating, only view the contents of the log, and do not save the log. If you save the log, the log data being output while you were viewing the log might be lost.

The following describes the execution log.

(1) Types of files

The following table lists execution log files. Each file name includes a server alias name, which is obtained from the value of the alias operand in the server environment definition (servers).

Table 7‒1: List of execution log files

File name

Description

/opt/hitachi/HAmon/spool/ptrlcmd_ex/server-alias-name_ptrlcmdlog

This is the file of the current execution log for the monitoring command.

/opt/hitachi/HAmon/spool/ptrlcmd_ex/server-alias-name_ptrlcmdlog.bak

This is an execution log backup file created when the most recent server failure occurred.

/opt/hitachi/HAmon/spool/ptrlcmd_ex/server-alias-name_ptrlcmdlog.bak2

This is an execution log backup file created when the second most recent server failure occurred.

(2) File contents

The following describes the contents of each execution log file.

The header section of each file consists of a comment on the file and the offset management information (offset), which is used for wrapping around the file.

File's header section

#    HA monitor ptrlcmd_ex Logging File
#
#      offset:131
# DATE     TIME            [PID] [PPID] [PGID] [UID] DATA

File's data section

● Entry information

date time [process-ID] [parent-process-ID] [process-group-ID] [UID] [server-identification-name] operand-name command-line-information + entry-identifier

date: The date on which the command information was logged is output in YYYY/MM/DD format.

time: The time at which the command information was logged is output in HH:MM:SS.mmmmmm format. mmmmmm is the number of microseconds.

process-ID: The process ID of the monitoring command is output.

parent-process-ID: The parent process ID of the monitoring command is output.

process-group-ID: The process group ID of the monitoring command is output.

UID: The actual UID of the user who executed the monitoring command is output. If the actual UID cannot be obtained, 0 is output.

server-identification-name: The identification name of the server on which the monitoring command is executed is output.

operand-name: The name of the operand for which the monitoring command to be executed is specified is output.

command-line + entry-identifier: The command line information about the monitoring command and start, which indicates an entry point, are output. The command line part is enclosed in double quotation marks.

● Execution skip information

date time [process-ID] [parent-process-ID] [process-group-ID] [UID] [server-identification-name] operand-name command-line-information + entry-identifier

date: Same as date under Entry point information.

time: Same as time under Entry point information.

process-ID: The process ID of the process that requested output of the information is output.

parent-process-ID: The parent process ID of the process that requested output of the information is output.

process-group-ID: The process group ID of the process that requested output of the information is output.

UID: 0 is output.

server-identification-name: Same as server-identification-name under Entry point information.

operand-name: Same as operand-name under Entry point information.

command-line + entry-point-identifier: skip, which indicates that the previously executed monitoring command had not returned the result yet, and the process ID of that monitoring command (in pid: process-ID format) are output.

● Exit information

date time [process-ID] [parent-process-ID] [process-group-ID] [UID] [server-identification-name] operand-name command-line-information + entry-identifier termination-information

date: Same as date under Entry point information.

time: Same as time under Entry point information.

process-ID: Same as process-ID under Entry point information.

parent-process-ID: Same as parent-process-ID under Entry point information.

process-group-ID: 0 is output.

UID: 0 is output.

server-identification-name: Same as server-identification-name under Entry point information.

operand-name: Same as operand-name under Entry point information.

command-line + entry-identifier: The command line information about the user command and end, which indicates an exit point, are output. The command line part is enclosed in double quotation marks.

termination-information: The termination status of the monitoring command is output. Note that in a case when the command terminated abnormally, the termination status is followed by the string no exit.

(3) Examples of information output to a file

The following shows examples of information that is output to an execution log file.

● Example of information output when the server was operating normally

#    HA monitor ptrlcmd_ex Logging File
#
#      offset:53614
# DATE     TIME            [PID] [PPID] [PGID] [UID] DATA
...
...
2017/04/01 19:46:15.794243 [9725] [8461] [9725] [0] sv1 ptrlcmd_ex "/bin/sh -c /opt/hitachi/HAmon/etc/patrol_ex.sh 1>./spool/ptrlcmd_ex/sv1.log 2>&1" : start
2017/04/01 19:46:18.324938 [9725] [8461] [0] [0] sv1 ptrlcmd_ex "/opt/hitachi/HAmon/etc/patrol_ex.sh" : end 0
2017/04/01 19:46:23.752926 [9871] [9866] [9871] [0] sv1 ptrlcmd_ex "/bin/sh -c /opt/hitachi/HAmon/etc/patrol_ex.sh 1>./spool/ptrlcmd_ex/sv1.log 2>&1" : start
2017/03/14 19:46:28.253276 [9871] [9866] [0] [0] sv1 ptrlcmd_ex "/opt/hitachi/HAmon/etc/patrol_ex.sh" : end 0
...
...
#####

Output of the string indicating that the monitoring command started (start) and the string indicating that the command terminated normally (end 0) has been repeated.

● Example of information output when the monitoring command timed out due to slowdown of the command or server processing

#    HA monitor ptrlcmd_ex Logging File
#
#      offset:53614
# DATE     TIME            [PID] [PPID] [PGID] [UID] DATA
...
...
2017/04/01 19:46:15.794243 [9725] [8461] [9725] [0] sv1 ptrlcmd_ex "/bin/sh -c /opt/hitachi/HAmon/etc/patrol_ex.sh 1>./spool/ptrlcmd_ex/sv1.log 2>&1" : start
2017/04/01 19:46:25.811760 [8461] [8385] [8384] [0] sv1 ptrlcmd_ex : skip, pid:9725
2017/04/01 19:46:35.811760 [8461] [8385] [8384] [0] sv1 ptrlcmd_ex : skip, pid:9725
2017/04/01 19:46:45.811760 [8461] [8385] [8384] [0] sv1 ptrlcmd_ex : skip, pid:9725
2017/04/01 19:46:55.811760 [8461] [8385] [8384] [0] sv1 ptrlcmd_ex : skip, pid:9725
...
...
#####

After the string indicating that the monitoring command started (start) was output, the string indicating that the command ended (end) was not output because the command did not terminate. After the monitoring command was started, the string indicating that command execution was skipped (skip) was output several times until the command timed out. In such a case, see also the details log, and determine the reason why the monitoring command did not terminate.

● Example of information output when the monitoring command resulted in an error (with the termination status 10)

#    HA monitor ptrlcmd_ex Logging File
#
#      offset:53614
# DATE     TIME            [PID] [PPID] [PGID] [UID] DATA
...
...
2017/04/01 19:46:15.794243 [9725] [8461] [9725] [0] sv1 ptrlcmd_ex "/bin/sh -c /opt/hitachi/HAmon/etc/patrol_ex.sh 1>./spool/ptrlcmd_ex/sv1.log 2>&1" : start
2017/04/01 19:46:18.324938 [9725] [8461] [0] [0] sv1 ptrlcmd_ex "/opt/hitachi/HAmon/etc/patrol_ex.sh" : end 10
...
...
#####

The string indicating that the monitoring command started (start) and a string indicating that the monitoring command failed (end 10) have been output. In such a case, see also the details log, and determine the reason why the monitoring command failed.

7.3.2 Checking the details log for the monitoring command (when the ptrlcmd_ex or sby_ptrlcmd_ex operand is specified on a monitor-mode server)

The detailed information about each process executed in the server monitoring command (contents of the standard output and standard error output) is output to the details log for the monitoring command.

Important

If you need to check a server log while the server is operating, only view the contents of the log, and do not save the log. If you save the log, the log data being output while you were viewing the log might be lost.

The following describes the details log.

(1) Types of files

The following table lists details log files. Each file name includes a server alias name, which is obtained from the value of the alias operand in the server environment definition (servers).

Table 7‒2: List of details log files

File name

Description

/opt/hitachi/HAmon/spool/ptrlcmd_ex/server-alias-name.log

This is the file of the current details log for the monitoring command.

/opt/hitachi/HAmon/spool/ptrlcmd_ex/server-alias-name.log_err

This is a details log backup file created when the most recent server failure occurred.

(2) Example of information output to a file

The following shows an example of information that is output to a details log file.

+ PROGRAM=/home/xxxx/yyyy
+ PS=/bin/ps
+ GREP=/bin/grep
++ /bin/ps -efl
++ /bin/grep /home/xxxx/yyyy
++ /bin/grep -v grep
+ EXIST=
+ '[' 'zzzz' = '' ']'
+ exit 0

7.4 Hot standby operation when a failure occurs

This section explains the actions to be taken by the operator after HA Monitor has detected a failure.

The procedure from detection of a failure to job recovery is explained below. During operation, you can verify from the messages output to syslog whether the system is running normally.

To take an appropriate action when an error message is issued:

Check the error messages.

Check what the error messages output to syslog say.

Check the status of hosts and servers.

Check if HA Monitor has completed hot standby processing. Execute the server and host status display command (monshow command) and check if the following is true:

The active server has restarted in the active system.

You must check this if you have specified restart or manual in the switchtype operand in the server environment definition.

The active server has been switched over to the standby server in the standby system.

If the active server is in the wait state and not running, the job has terminated. To restart the job, an operator intervention is required. For details about the operator intervention, see 7.4.1 Starting a server in the wait state and then restarting jobs.

Make sure that the job has resumed without problems.

Check that there are no problems with the job. The check items and methods depend on the nature of the job. For example, check the following:

Whether the server and client are able to communicate with each other

Whether the program used to execute the job is running correctly

Resolve the cause of the failure at the host where the failure occurred.

Resolve the cause of the failure on the basis of the messages that are output. If you need to manipulate shared resources, observe the notes in 7.2.3 Notes about maintaining shared resources. Also, collect HA Monitor error information, if necessary. For details, see 7.4.2 Collecting error information.

The required actions are explained for the major causes of failures. For details about the failure handling procedures, see 7.5 Handling errors.

Restart the host where the failure occurred as the standby system.

If the failure occurred in the primary system, job execution continues in the secondary system. If you restart the host where the failure occurred as the standby system, you can be prepared for a subsequent failure at the system that is currently executing jobs. For details, see 7.4.3 Restarting the host where a failure occurred as the standby system.

Check the status of servers and hosts.

Use HA Monitor commands to check that the server or host where the failure occurred has restarted successfully. For details, see 7.4.4 Checking the status of servers and hosts after handling an error.

7.4.1 Starting a server in the wait state and then restarting jobs

The HA Monitor on the local host communicates with the HA Monitor on the remote host. An HA Monitor monitors the server status in both the active system and the standby system and determines the server status on the local host according to the server status on the remote host. Therefore, a server being started might be placed in a wait state before startup is completed. This subsection explains how to start and terminate a server that is in the wait state.

There are several types of server wait state and the start and termination methods differ depending on the type. When a server is placed in the wait state, a message to that effect is output at a specific interval. The following table describes the types of server wait states and the operating methods.

Table 7‒3: Types of server wait states and operating methods

Type

Text displayed by the monshow command

Description

Message

Start and termination methods

Restart wait state

*ONL*

The active server can be restarted after a failure has occurred on it. The status in effect until the restart is completed after a failure has occurred is called the restart wait state.

KAMN258-D

Start method: Start command provided by the program

Termination method: mondeact

Hot-standby wait state

ONL??

If a host reset is performed and hot standby processing occurs, HA Monitor switches the standby server in the standby system to the active server. When HA Monitor performs this switchover, it checks whether the active server in the active system (host where the failure occurred) has been terminated.

If HA Monitor determines that the active server has terminated, it starts the standby server in the standby system as the active server.

If HA Monitor cannot verify that the active server has terminated, the standby server in the standby system is placed in the wait state before it is actually started as the active server. The status in which the server in the standby system is waiting to be started as the active system is called the hot-standby wait state.

The server is also placed in the hot-standby switchover wait state if SCSI reservation for shared disk fails after host reset by hybrid fencing fails.

KAMN368-D

Start method: monact

Termination method: mondeact

Active server start wait state

*SBY*

If the status of the active server cannot be checked for a reason such as a monitoring path failure, the standby server waits until the active server has started. The status in which the standby server is waiting for the active server to start is called the active server start wait state.

KAMN238-D

Start method: monact

Termination method: mondeact

Resource server start wait state

If the resource server has not completed its startup processing, its child servers wait until the resource server has started. The status in which servers are waiting for startup of the resource server is called the resource server start wait state.

None

Start method: Executing monact after the resource server has started

Termination method: Executing mondeact after the resource server has terminated or started

Grouped-system switchover wait state

SBY??

If multiple servers are grouped together, you can specify the settings so that a grouped-system switchover is not performed, even if a failure occurs on one of the servers. If a failure occurs on a server that is set to not perform grouped-system switchover, the corresponding standby server is placed in the grouped-system switchover wait state.

If an active or standby server starts on any other host, the corresponding standby server returns to the status in which hot standby processing can be performed.

None

Start method: monact

Termination method: monsbystp

Start retry state

>ONL

If resource connection and server startup fail on the target host during hot standby processing but there is no available switchover host and the start retry function is used during hot standby processing, HA Monitor issues a restart instruction to the server. The status in effect from when the hot standby error occurs until server startup processing starts is called the start retry state.

KAMN296-I

Start method: Start command provided by the program

Termination method: mondeact

In the table, the Start and termination methods column provides the names of the applicable HA Monitor commands. The Text displayed by monshow command column indicates the server status that is displayed when the server and host status display command (monshow command) is executed. For details about the HA Monitor commands, see 9. Commands. For details about the server status transitions, see 4.1.3 Server status transitions.

7.4.2 Collecting error information

This subsection explains the error information and how to collect it in the event of a failure.

HA Monitor error information

Use the troubleshooting information collection command (monts command) to collect HA Monitor error information.

OS error information and statistical information

Use OS commands to collect OS error information and statistical information.

This subsection explains the HA Monitor error information and the troubleshooting information collection command (monts command). For details about the OS error information and statistical information, see the OS documentation.

(1) HA Monitor error information

HA Monitor provides the error information listed below. Use the troubleshooting information collection command (monts command) to collect all of this information. The data collected by the monts command is not compressed.

Definition information

Definition files (all files under /opt/hitachi/HAmon/etc)

Module trace information

Core files (all files beginning with /opt/hitachi/HAmon/core)

Message log information

System log files (all files beginning with /var/log/messages)

Memory information

If HA Monitor is running, its memory information is collected.

Trace information

Trace files (all files under /opt/hitachi/HAmon/spool)

The trace files store the history of HA Monitor operations and the execution results of the commands that have been issued.

You can use the trace files listed below to determine the causes of errors. Note that the files required for HA Monitor operation are stored under /opt/hitachi/HAmon/spool. Make sure that you do not change or delete these files, and that you do not perform any operations on this directory.

Table 7‒4: Files used to determine the causes of HA Monitor errors

File name

Description

Purpose

sms

oldsms

This file collects information about host and server failures and slowdowns.

When the trace information in the file reaches 100 KB, it is wrapped around to the backup file named oldsms. The contents of the sms file are then cleared and rewritten in the oldsms file.

Determining the causes of host and server failures

server-alias-name.fslog

server-alias-name.fslog_old

These files collect the execution results of the OS commands (fsck (or xfs_repair), mount, fuser, and umount) that are executed when HA Monitor switches file systems.

When the file size during trace collection exceeds the value specified in the fs_log_size operand in the HA Monitor environment settings, a backup file named server-alias-name.fslog_old is created.

Determining the causes of file system switchover errors

volume-group-name.vglog

volume-group-name.vglog_old

This file collects the execution results of the OS command (vgchange command) that is executed when HA Monitor connects or disconnects volume groups.

When the file size during trace collection exceeds 65,535 bytes, a backup file named volume-group-name.vglog_old is created.

Determining the causes of shared disk connection errors

Monitoring history

Monitoring history file (/opt/hitachi/HAmon/history/patrol_history)

This file is used to collect host and server slowdown information. It provides more detailed information about slowdown periods than the SMS trace information file.

Monitoring history is useful when you re-evaluate the definition files and the system configuration to prevent failures that might be caused by heavy load. For details about the operation method using monitoring history to reduce failures, see 7.6 Operation for preventing failures caused by heavy load.

(2) Using the monts HA Monitor command to collect error information

If you execute the troubleshooting information collection command (monts command) after a failure, you can store the collected error information as archive files or onto a portable medium.

You must execute the monts command with the superuser permissions. The following figure shows the procedure for collecting error information by using the monts troubleshooting information collection command.

Figure 7‒3: Using the troubleshooting information collection command (monts command) to collect error information

When you execute the monts troubleshooting information collection command, you can specify whether the error information is to be stored as an archive file or onto a portable medium.

7.4.3 Restarting the host where a failure occurred as the standby system

After hot standby processing has been completed, the host on which a failure occurred is reset and placed in the inactive status. However, if SCSI reservation for shared disk is used, or if the function for controlling hot standby based on the availability of LAN communications is used, the hosts and HA Monitor return to a running state after completion of hot standby processing due to a monitoring path failure or host slowdown.

To prepare for the possibility of a subsequent failure on the secondary system that is currently executing jobs, the operator must resolve the cause of the failure and then restart the host on which the failure occurred. The following is the procedure for enabling hot standby processing for a host on which a failure occurred.

If the host on which the failure occurred terminated after hot standby processing finished

To restart the host where a failure occurred:

Resolve the cause of the failure on the host where the failure occurred.

Implement the recommended action to eliminate the cause of the failure according to the displayed messages.

Restart the host.

To be prepared for the possibility of a subsequent failure on the secondary system that is executing jobs, start the host where the failure occurred. The restarted host becomes the standby system in case a failure occurs on the host currently executing jobs.

Start HA Monitor.

If you have set HA Monitor to be started manually, execute the HA Monitor start command (monstart command).

You can skip this step if you have set HA Monitor to start automatically when the kernel starts.

Verify that HA Monitor startup processing is complete, and that there is a connection to the host currently executing jobs.

For details about how to check this, see 7.4.4 Checking the status of servers and hosts after handling an error.

The KAMN002-I message is issued when the HA Monitor startup processing is completed.

Start the servers.

Start the servers on the host where the failure occurred.

For servers in the server mode:

Execute the start command provided by the program.

For servers in the monitor mode:

Execute the monitor-mode server start command (monbegin command).

You can also set servers in the monitor mode to start automatically. For details about how to specify the settings, see 7.7.3 Automating an operation after hot standby processing.

Verify that the server startup processing has been completed and that the system is ready for hot standby processing.

The KAMN252-I message is issued when the system is ready for hot standby processing.

If the host where the failure occurred did not terminate after hot standby processing finished

Resolve the cause of the failure on the host where the failure occurred.

Implement the recommended action to eliminate the cause of the failure according to the displayed messages.

Execute the HA Monitors manual connection command (monlink command).

Verify that there is a connection to the host currently executing jobs.

For details about how to check this, see 7.4.4 Checking the status of servers and hosts after handling an error.

Start the servers.

Start the servers on the host on which the failure occurred.

For servers in the server mode:

Execute the start command provided by the program.

For servers in the monitor mode:

Execute the monitor-mode server start command (monbegin command).

You can also set servers in the monitor mode to start automatically. For details about how to specify the settings, see 7.7.3 Automating an operation after hot standby processing.

Verify that the server startup processing is complete and that the system is ready for hot standby processing.

The KAMN252-I message is issued when the system is ready for hot standby processing.

7.4.4 Checking the status of servers and hosts after handling an error

This subsection explains how to check the status of servers and shared resources after an error has been resolved. For details about the commands explained here, see 9. Commands.

(1) Checking the system status

Checking the system status involves checking the status of monitoring paths, the reset path, and hosts and verifying that the active and standby systems can communicate with each other.

Checking the status of monitoring paths

You use the monitoring path status display command (monpath command) to check the status of monitoring paths and to locate a monitoring path failure on the basis of the monitoring path's IP address. You use this command when a failure has occurred on a monitoring path and when you recover such a failure.

Checking the status of the reset path (applicable if host reset is performed)

You execute the reset path status display command (monrp command). You use this command when a failure has occurred on a failure management processor or the reset path and when you recover such a failure.

Checking the host status

You use the server and host status display command (monshow -c command) to check the status of the local host (host that executed the command) and a remote host that is connected to the local host.

(2) Checking the server status

You use the server and host status display command (monshow command) to check the status of the servers running on the host that executed the command and their paired servers.

(3) Checking the resource server status

You use the server and host status display command (monshow -r command) to check the status of the resource server. You can also check the status of servers that use the resource server at the same time.

7.5 Handling errors

This section explains how the operator is to handle errors based on the messages output after errors have occurred. Typical examples are explained here. If an error that is not discussed here has occurred, take appropriate action based on the displayed message and the message handling procedure. For details about the message handling procedures, see the manual HA Monitor Cluster Software Messages.

7.5.1 Handling host start errors

This subsection explains how to handle host start errors.

(1) Start errors on the primary system

If the primary system fails to start, the operator must start the secondary system as the active system. In the secondary system, the server starts as a standby server and is placed in the active server start wait state. The operator executes the wait-state server start command (monact command) to start the standby server in the secondary system as the active server. The secondary system becomes the active system because the active server has been started in the secondary system.

After the active system has started and jobs have started, the operator must determine the cause of the start error in the primary system and start the primary system as the standby system.

(2) Start errors on the secondary system

If the secondary system fails to start, there is no effect on the jobs in the active system. However, if a failure occurs in the active system, hot standby processing will not be performed because the standby system is not available. We recommend that you resolve promptly the cause of any failure in the secondary system and start the secondary system as the standby system.

7.5.2 Handling server restart errors

This subsection explains how to handle server restart errors when a server is set to be restarted automatically in the event of a server failure but the server has failed to restart. When you perform your action depends on the server operating mode:

For a server in the server mode

Take action when the restart limit is detected after the server restart has been attempted as many times as specified.

For a server in the monitor mode

Take action when the server restart fails.

Take action when program restart fails.

(1) For a server in the server mode

This subsection explains the handling methods that apply in either of the cases noted below. If switch or restart is specified in the switchtype operand in the server environment definition, there is no need for the operator to take an action because HA Monitor performs hot standby processing.

manual is specified in the switchtype operand in the server environment definition.

The KAMN316-E message has been issued.

When either of these conditions is applicable, the active server in the active system (where the failure occurred) is terminated and the standby server in the standby system is placed in the active server start wait state.

Action

Because the standby server has been placed in the active server start wait state, start the standby server in the start wait state as the active server in order to restart the jobs.

To do this:

Check the status of the active system.

Check the following:

The active server has been terminated.

The shared resources that are used by the server have been disconnected.

Start the standby server in the standby system as the active server.

Execute the wait-state server start command (monact command) to start the standby server in the active server start wait state as the active server.

Verify that the active server has started.

Check that the KAMN251-I message has been issued.

Resolve the cause of the server failure in the system where the failure occurred.

From the active system, start the standby server.

Execute the start command provided by the program.

To check after taking action:

Verify that the system is ready for hot standby processing.

Either of the following verifies this:

The KAMN252-I message has been issued.

The server and host status display command (monshow command) displays ONL as the active server's status and SBY as the standby server's status.

(2) For a server in the monitor mode

This subsection explains how to handle restart errors for a server in the monitor mode.

(a) If restart of the active server fails

If you are running servers in the monitor mode, the following might be the causes of an active server restart error:

The server start command executed by HA Monitor has failed.

The server termination command executed by HA Monitor has failed.

This subsection explains the handling methods that apply to the cases below. If a different message has been issued, the possible cause would be different. Take appropriate action according to the displayed messages.

The server is set to be restarted in the event of a server failure.

The KAMN273-E message has been issued.

If the server start command specified in the name or actcommand operand in the server environment definition returns a nonzero value during server restart, the processing is cancelled (the server termination command specified in the termcommand operand in the server environment definition is not executed).

Action

Correct the server start or termination command or the server environment definition and then restart the server.

Resolve the cause of the error.

Resolve the cause of the error on the basis of the error code displayed in the KAMN273-E message. For details about the error codes, see Table 7-5 List of error codes that can be displayed in the KAMN273-E message (if restart of the active server fails).

From the standby system, start the active server.

You can use either of the following methods:

In the active system (where the failure occurred), execute the monitor-mode server termination command (monend command) to terminate the active server, and then in the standby system, execute the monitor-mode server start command (monbegin command) to start the active server.

In the standby system, execute the monitor-mode server start command (monbegin command) to start the standby server, and then in the active system, execute the server hot-standby switchover command (monswap command) to perform planned hot-standby switchover to the standby system.

From the active system, start the standby server.

Execute the monitor-mode server start command (monbegin command) to start the standby server.

To check after taking action:

Verify that the system is ready for hot standby processing.

Either of the following verifies this:

The KAMN252-I message has been issued.

The server and host status display command (monshow command) displays ONL as the active server's status and SBY as the standby server's status.

The following table lists the error codes that are displayed in the KAMN273-E message and describes the actions to be taken.

Table 7‒5: List of error codes that can be displayed in the KAMN273-E message (if restart of the active server fails)

Cause code

Detail code

Explanation of code

Action

1

errno for the system call

A system error occurred when the server start command was executed.

Resolve the cause of the system call error.

2

Command's return value

The server start command returned a nonzero value.

Check the specified server start command and correct it.

126

Execution permissions for the file specified in the name or actcommand operand are not granted.

Grant execution permissions to the server start command.

127

The file specified in the name or actcommand operand was not found.

Check that the value specified in the name or actcommand operand in the server environment definition matches the storage location of the server start command.

3

errno for the system call

A system error occurred when the server termination command was executed.

Resolve the cause of the system call error.

4

Command's return value

The server termination command returned a nonzero value.

Check the specified server termination command and correct it.

126

The server termination command does not have the execution permissions for the file specified in the termcommand operand.

Grant execution permissions to the server termination command that was specified in the termcommand operand in the server environment definition.

127

The file specified in the termcommand operand was not found.

Check that the value specified in the termcommand operand in the server environment definition matches the storage location of the server termination command.

(b) If restart of the standby server fails

If you are running servers in the monitor mode, restart of the standby server is canceled and the standby server stops in the following cases:

HA Monitor failed to execute any of the following commands:

The command specified for the sby_actcommand operand in the server environment definition

The command specified for the sby_termcommand operand in the server environment definition

HA Monitor executes either of the following commands and the return code is other than 0:

The command specified for the sby_actcommand operand in the server environment definition

The command specified for the sby_termcommand operand in the server environment definition

Action

Use the following procedure to restart the standby server:

Resolve the cause of the error.

Resolve the cause of the error on the basis of the error code displayed in the KAMN273-E message. For details about the error codes, see Table 7-6 List of error codes that can be displayed in the KAMN273-E message (if restart of the standby server fails).

From the standby system, start the standby server.

In the standby system (where the failure occurred), execute the monbegin command to start the standby server.

To check after taking action:

Verify that the system is ready for hot standby processing.

Either of the following verifies this:

The KAMN252-I message has been issued.

The server and host status display command (monshow command) displays ONL as the active server's status and SBY as the standby server's status.

The following table lists the error codes that are displayed in the KAMN273-E message and describes the actions to be taken.

Table 7‒6: List of error codes that can be displayed in the KAMN273-E message (if restart of the standby server fails)

Cause

Code

Detail code

Explanation of code

Action

1

errno for the system call

A system error occurred during execution of the command specified for the sby_actcommand operand.

Resolve the cause of the system call error.

2

Command's return value

The command specified for the sby_actcommand operand returned a value other than 0 as a return value.

Check and correct the specified command.

126

Execution permissions are not granted to the command specified for the sby_actcommand operand.

Grant execution permissions to the command.

127

The command specified for the sby_actcommand operand does not exist.

Check whether the location of the command is correct.

3

errno for the system call

A system error occurred during execution of the command specified for the sby_termcommand operand.

Resolve the cause of the system call error.

4

Command's return value

The command specified for the sby_termcommand operand returned a value other than 0 as a return value.

Check and correct the specified command.

126

Execution permissions are not granted to the command specified for the sby_termcommand operand.

Grant execution permissions to the command.

127

The command specified for the sby_termcommand operand does not exist.

Check whether the location of the command is correct.

(c) When program restart fails

If you create a program restart command when using the monitor-mode program management function, you can restart only the program.

The following might be the causes of a program restart error:

HA Monitor failed to execute the program restart command.

The program restart command timed out before its processing was completed.

The UAP timed out before it issued the hamon_patrolstart function.

This subsection explains the handling methods that apply to the cases below. If a different message has been issued, the possible cause would be different. Take appropriate action according to the displayed messages.

The program has been set to be restarted in the event of a program error.

The KAMN285-E message has been issued.

Action

Correct the monitor-mode program environment definition and then restart the server. Alternatively, correct the program restart command and then restart the program.

Resolve the cause of the error.

Resolve the cause of the error on the basis of the error code displayed in the KAMN285-E message. For details about the error codes, see Table 7-7 List of error codes displayed in the KAMN285-E message.

Restart the program in the active system.

You can use either of the following methods:

Restart the server. Execute the monitor-mode server termination command (monend command) to terminate the active server, and then execute the monitor-mode server start command (monbegin command) to start the active server.

Manually restart the UAP that resulted in the error.

To check after taking action:

Verify that the program has started successfully.

Either of the following verifies this:

The KAMN284-I message has been issued.

The server and host status display command (monshow -u command) displays ACT as the program status.

The following table lists the error codes that are displayed in the KAMN285-E message and describes the actions to be taken.

Table 7‒7: List of error codes displayed in the KAMN285-E message

Cause code

Detail code

Explanation of code

Action

1

1

The program restart command was not completed before a timeout occurred.

Resolve the cause so that the program restart command will be completed.

2

The UAP did not issue the hamon_patrolstart function before a timeout occurred.

Resolve the cause so that the UAP issues the hamon_patrolstart function before a timeout occurs.

3

The program restart command was not completed and the UAP did not issue the hamon_patrolstart function before a timeout occurred.

Resolve the cause so that the program restart command is completed and the UAP issues the hamon_patrolstart function before a timeout occurs.

2

Command's return value

The program restart command returned a nonzero value.

Check the specified program restart command and correct it.

126

The program restart command does not have the execution permissions for the file specified in the restartcommand operand.

Grant execution permissions to the program restart command that was specified in the restartcommand operand in the monitor-mode program environment definition.

127

The file specified in the restartcommand operand was not found.

Check that the value specified in the restartcommand operand in the monitor-mode program environment definition matches the storage location of the program restart command.

256

A system error occurred while the program restart command was executing.

Eliminate the cause of the system call error.

7.5.3 Handling host reset errors

If a host reset fails, the server in the standby system is placed in the hot-standby wait state and the jobs stop. This subsection explains how to handle host reset errors when both of the following conditions apply:

A host failure has occurred and a host reset has failed for some reason.

The KAMN368-D message has been issued.

Action

Start the server that is in the hot-standby wait state as the active server.

To do this:

Determine which server is in the hot-standby wait state.

From the standby system, execute the server and host status display command (monshow command) to obtain the server alias name of the server whose server status is ONL??.

From the host where the failure occurred, check that no servers are running.

On the host where the failure occurred, execute the server and host status display command (monshow command) to check that no active server is running on any of the other hosts in the system.

Start the server that is in the wait state as the active server.

From the standby system, execute the wait-state server start command (monact command). Specify for the option the server alias name of the server that is in the hot-standby wait state.

To check after taking action:

Verify that the active server startup processing has been completed.

Execute the server and host status display command (monshow command) and check that the corresponding server's status is ONL.

Check that the server jobs are running normally.

7.5.4 Handling shared resource connection errors

This subsection explains the handling of shared resource connection errors. Normally, HA Monitor connects shared resources. In the event of a failure, the operator must connect the shared resources manually. This subsection explains how to connect shared resources manually.

For notes about manipulating shared resources, see 7.2.3 Notes about maintaining shared resources. If you do not observe these notes, the shared resources might become corrupted or hot standby processing might fail.

Shared disk

Execute the command shown below. You must make sure that the shared disk is not connected for referencing and updating purposes from both hosts.

vgchange -a y path-name-of-volume-group

File system

Perform the following procedure:

Use the fsck command of the OS (or the xfs_repair command if the file system of the shared disk is XFS in RHEL7 or a later version) to check the file system.

Use the mount OS command to mount the file system.

LAN

To manually connect a LAN, you must correct the LAN status settings file and then execute it.

The file to be corrected is /opt/hitachi/HAmon/etc/server-alias-name.up.

7.5.5 Handling shared resource disconnection errors

This subsection explains the handling of shared resource disconnection errors. Normally, HA Monitor disconnects shared resources. In the event of a failure, the operator must disconnect the shared resources manually. The action to be taken depends on when the shared resource disconnection error occurred.

Important

If abnormal or forced termination of a server or a shared resource inheritance timeout is detected during shared-resource control processing, the status of the shared resources becomes undefined. Therefore, you must check the status of the shared resources before restarting the server. For other notes about manipulating shared resources, see 7.2.3 Notes about maintaining shared resources. If you do not observe these notes, the shared resources might become corrupted or hot standby processing might fail.

(1) During hot standby processing

If a shared resource disconnection error occurs in the active system during hot standby processing, HA Monitor switches over to the standby system. If a host reset is to be performed, HA Monitor resets the host and then performs hot standby processing. If SCSI reservation for shared disk is to be used, the host on which the shared resource disconnection error occurred issues a kernel panic to terminate the OS and then performs hot standby processing. No operator intervention is needed.

If a shared resource disconnection error occurs in the active system during hot standby processing, contention might occur for shared resources between hosts, resulting in corruption of the shared resources and system shutdown. To avoid contention for shared resources between both hosts, if a host reset is to be performed, HA Monitor resets the host, and, if SCSI reservation for shared disk is to be used, the host on which the shared resource disconnection error occurred issues a kernel panic to terminate the OS.

(2) During server termination processing

If a shared resource disconnection error occurs during server termination processing, a message to that effect is issued. The operator must check the contents of the message and perform the disconnection processing that is appropriate to each shared resource.

Shared disk

Execute the following command:

vgchange -a n path-name-of-volume-group

File system

Perform the following procedure:

Use the umount OS command to unmount the file system.

If an attempt to unmount the file system fails, use the fuser OS command to forcibly terminate all processes that are using the file system, and then retry unmounting the file system.

LAN

To manually disconnect a LAN, you must correct the LAN status settings file and then execute it.

The file to be corrected is /opt/hitachi/HAmon/etc/server-alias-name.down.

You can also use the ifconfig OS command to disconnect a LAN.

7.5.6 Handling device failures on a shared disk (while the active server is starting) (using SCSI reservation for shared disk)

This subsection explains how to handle failures while the active server is starting on a device that contains a shared disk to be reserved.

The failures discussed here are listed below. Note that the device referred to here is the device specified in the scsi_device or dmmp_device operand in the server environment definition.

Some or all of the devices have already been reserved.

I/O errors have occurred on all devices.

For details about the handling of I/O errors that have occurred on some of the devices, see 7.5.7 Handling device failures on a shared disk (while the active server is running) (using SCSI reservation for shared disk).

(1) Handling when devices have already been reserved

If some or all of the devices have already been reserved, HA Monitor issues the KAMN725-W and KAMN726-E messages and the server startup processing fails. If this happens, determine how reservation occurred, and then take appropriate action.

The following explains possible causes and the actions to be taken.

Cause 1

When the active server was terminated, a reservation could not be released for a reason such as an I/O error (the KAMN726-E message is issued)

A host failure occurred on the active system while hot standby was not possible, such as when the standby server was inactive

The OS was shut down while the active server was running

When hot standby processing was performed due to a host failure, the active server was terminated before the failed host and the standby server were started

Action 1

Release the reservation by using the reservation release command (monscsiclr command) or by following the procedure described in 7.5.11 Dealing with the server that does not release the reservation (using SCSI reservation for shared disk), and then restart the server.

Cause 2

A server for which the same shared disk is defined is running as the active server due to an HA Monitor definition error

The device has been reserved by a program that is not an HA Monitor

Action 2

If the device was reserved for one of these reasons or the cause is unknown, take appropriate action according to the figure shown below, then restart the server.

The numbers in the figure correspond to the step numbers in the detailed explanation below.

Figure 7‒4: Determining the cause and taking appropriate action when devices are already reserved

Is the host address indicated by the KAMN725-W message the same as the address operand value in the local host's HA Monitor environment settings?

Yes: Go to step 2

No: Go to step 7

Is KAMN728-W output when a server indicated by KAMN725-W is stopped?

Yes: Release the reservation by using the reservation release command (monscsiclr command) or by referencing 7.5.11 Dealing with the server that does not release the reservation (using SCSI reservation for shared disk).

No: Go to step 3

Did any one of the following events occur on the local host when the server indicated by the server alias name in the KAMN725-W message was running as the active server?

OS shutdown

Cancellation of reservation failed when the server indicated by the server alias name in the KAMN725-W message was terminating (the KAMN725-W message was issued).

The KAMN725-W message was issued due to a resource disconnection error on one of the servers in the active system and then HA Monitor was terminated.

Yes: Release the reservation by using the reservation release command (monscsiclr command) or by referencing 7.5.11 Dealing with the server that does not release the reservation (using SCSI reservation for shared disk).

No: Go to step 4

Is there an HA Monitor (including a test environment) on the host that is not connected to the local host?

Yes: Go to step 5

No: Go to step 6

Is the scsi_device or dmmp_device operand value in the server environment definition the same as that for the local host?

Yes: Check the shared disk configuration and then correct the value of the scsi_device or dmmp_device operand.

No: Go to step 6

Have the devices been reserved by a software program other than HA Monitor?

Yes: Terminate the program that reserved the devices.

No: Cancel the reservation by referencing 7.5.12 Canceling SCSI reservation for shared disk.

Is KAMN728-W output when a server indicated by KAMN725-W is stopped in the system indicated by KAMN725-W?

Yes: Release the reservation by using the reservation release command (monscsiclr command) or by referencing 7.5.11 Dealing with the server that does not release the reservation (using SCSI reservation for shared disk).

No: Go to step 8

Did any one of the following events occur on the active system when the server indicated by the server alias name in the KAMN725-W message was running as the active server on the host indicated by the host address in the same message?

OS shutdown

Cancellation of reservation failed when the server indicated by the server alias name in the KAMN725-W message was terminating (the KAMN725-W message was issued).

The KAMN725-W message was issued due to a resource disconnection error on one of the servers in the active system and then HA Monitor was terminated.

Yes: Release the reservation by using the reservation release command (monscsiclr command) or by referencing 7.5.11 Dealing with the server that does not release the reservation (using SCSI reservation for shared disk).

No: Go to step 9

Is the device name specified in the scsi_device or dmmp_device operand for the host containing the server indicated by the server alias name in the KAMN725-W message the same as the scsi_device or dmmp_device operand value for the host indicated by the host address in the same message?

Yes: Go to step 4

No: Correct the scsi_device or dmmp_device operand values so that they are the same for both hosts.

(2) Handling when I/O errors have occurred on all devices

If I/O errors occur on all of the devices, HA Monitor issues the KAMN725-W and KAMN726-E messages and the server startup processing fails. To handle I/O errors:

Determine the causes of the I/O errors.

Determine the causes of the errors by checking the KAMN725-W and KAMN726-E messages and the messages issued by the kernel and by using hardware management tools.

Resolve the causes of the I/O errors.

Resolve the causes of the I/O errors by taking measures such as replacing erroneous devices.

If you are using a multi-path configuration, restore the paths recovered from the I/O errors to online status (failback).

Restore a recovered path to online status (failback) by using the appropriate command provided by the multipath software (HDLM, DMMP, or HFC-PCM). For details about how to restore paths to online status, see the manual Hitachi Dynamic Link Manager Software User's Guide (for Linux(R) systems). Alternatively, see the documentation for DMMP or HFC-PCM.

For a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), this step is not necessary.

Restart the server.

7.5.7 Handling device failures on a shared disk (while the active server is running) (using SCSI reservation for shared disk)

If an I/O error occurs while the active server is running on the device specified in the scsi_device or dmmp_device operand in the server environment definition, HA Monitor issues the KAMN725-W and KAMN726-E messages. In a redundant configuration with multipath software, HA Monitor issues the KAMN726-E message only if a failure occurred on all paths to the same shared disk.

This subsection explains the recovery procedure.

Note that when a device file name (/dev/sdx) is added due to replacement or addition of a host adapter (HBA) or addition of a shared disk (LU), you must terminate the active server in order to recover from the error.

Determine the cause of the device failure.

Determine the cause of the failure by referencing the KAMN725-W and KAMN726-E messages and the message issued by the kernel and by using hardware management tools.

Terminate the standby server by using the standby server termination command (monsbystp command) (for a resource server, the standby resource server termination command (monressbystp command)).

Make sure that you terminate the standby server before you start the recovery processing. Immediately after the recovery processing has been completed, a reservation for the shared disk will not yet have been obtained. Therefore, if alive message transmission stops before a reservation is obtained after recovery processing has been completed, resulting in hot standby processing, the shared disk might not be protected.

Resolve the cause of the device failure.

Resolve the cause of the device failure by taking appropriate action, such as by replacing the erroneous device.

In a redundant configuration with multipath software, at this time, do not restore to online status the path that resulted in the failure (failback).

Obtain the reserved status for the disk.

The following shows the checking method:

# /usr/bin/sg_persist --in --no-inquiry --read-reservation --device=device-name

Legend:

device-name: Value indicated in the KAMN725-W message. In a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), this is a symbolic link. In a redundant configuration with multipath software, this is a physical device. If the device name displayed in the KAMN725-W message is followed by a path name, this is the path name.

The following example specifies /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065/ as the device name.

Example

# /usr/bin/sg_persist --in --no-inquiry --read-reservation --device=/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065/

Result 1:

If the following text is displayed, confirm that 0x1 is the same as the key that has been output to the KAMN725-W message.

PR generation=0xnn, Reservation follows:
  Key=0x1
  scope: LU_SCOPE,  type: Write Exclusive, registrants only

If the value matches the key displayed in the KAMN725-W message, a reservation has been obtained. In a single-path configuration, in a VMware ESXi-based virtualization environment (where DMMP is not used), or in an HFC-PCM environment, go to step 9. In a redundant configuration with multipath software, go to step 5, because even if a reservation has been obtained, it might not have been obtained for the path that is to be recovered from the failure.

If the value matches the key displayed in the KAMN725-W message, hot standby processing might have already been performed on the remote host or a reservation obtained by the remote host before the failure might still be available. The server running on the local host will issue the KAMN726-E message and will be terminated forcibly after a while. In such a case, skip the steps beginning with step 5 and perform the procedure described in (1) Handling when devices have already been reserved.

Result 2:

If the following message is issued, no reservation has been obtained. Go to step 5.

PR generation=0xnn, there is NO reservation held

Prepare for obtaining a reservation.

The following shows how to prepare:

# /usr/bin/sg_persist --out --no-inquiry --register-ignore --param-sark=key  --device=device-name

Legend:

key, device-name: These are the values displayed in the KAMN725-W message. In a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), the device name is a symbolic link, and in a redundant configuration with multipath software, it is a physical device. If the device name displayed in the KAMN725-W message is followed by a path name, this is the path name.

The following example specifies /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065 as the device name.

Example

# /usr/bin/sg_persist --out --no-inquiry --register-ignore --param-sark=0x1 --prout-type=5 --verbose --device=/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065

If nothing is displayed after execution, the preparation has been completed. Go to step 6.

In a redundant configuration with multipath software, if a reservation has already been obtained in step 4, go to step 8.

Note that the message shown below might be displayed. In this case also, go to step 6 because the preparation has been completed. In a redundant configuration with multipath software, if a reservation has already been obtained in step 4, go to step 7.

persistent reserve out: scsi status: Reservation Conflict
PR out: command failed

Obtain a reservation.

How to obtain a reservation is shown below. Make sure that you specify 5 for --prout-type.

# /usr/bin/sg_persist --out --reserve --param-rk=key --prout-type=5 --device=device-name

Legend:

key, device-name: These are the values displayed in the KAMN725-W message. In a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), the device name is a symbolic link, and in a redundant configuration with multipath software, it is a physical device. If the device name displayed in the KAMN725-W message is followed by a path name, this is the path name.

The following example specifies /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065 as the device name.

Example

# /usr/bin/sg_persist --out --reserve --param-rk=0x1 --prout-type=5 --device=/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065

If nothing is displayed after execution, a reservation has been obtained successfully. If any message is displayed, check the specified key and device name for any errors and then re-execute the command.

Check whether a reservation has been obtained.

The following shows how to check:

# /usr/bin/sg_persist --in --no-inquiry --read-reservation --device=device-name

Legend:

device-name: This is the value indicated in the KAMN725-W message. In a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), this is a symbolic link, and in a redundant configuration with multipath software, it is a physical device. If the device name displayed in the KAMN725-W message is followed by a path name, this is the path name.

The following example specifies /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065 as the device name.

Example

# /usr/bin/sg_persist --in --no-inquiry --read-reservation --device=/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065

If nothing is displayed after execution, a reservation has been obtained successfully. If any message is displayed, evaluate steps 5 and 6 and then re-execute the command.

Note that the message shown below might be displayed. In this case also, a reservation has been obtained successfully. In a single-path configuration, in a VMware ESXi-based virtualization environment (where DMMP is not used), or in an HFC-PCM environment, go to step 9. In a redundant configuration with multipath software, go to step 8.

PR generation=0xnn, Reservation follows:
  Key=0x1
  scope: LU_SCOPE,  type: Write Exclusive, registrants only

In a redundant configuration with multipath software, restore the path that has been recovered from the failure to online status (failback).

Restore a recovered path to online status (failback) by using the appropriate command provided by the multipath software (HDLM, DMMP, or HFC-PCM). For details about how to restore paths to online status, see the manual Hitachi Dynamic Link Manager Software User's Guide (for Linux(R) systems). Alternatively, see the documentation for DMMP or HFC-PCM.

For a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), this step is not necessary.

Restart the standby server that was terminated in step 2.

7.5.8 Handling device failures on a shared disk (while the active server is terminating) (using SCSI reservation for shared disk)

If an I/O error occurs while the active server is terminating on the device specified in the scsi_device or dmmp_device operand in the server environment definition, HA Monitor issues the KAMN725-W and KAMN726-E messages and terminates the OS. In a redundant configuration with multipath software, HA Monitor issues the KAMN726-E message only when a failure has occurred on all paths to the same shared disk. If a failure has occurred on some but not all of the paths to the same shared disk, HA Monitor resumes the server termination processing without terminating the OS.

To recover from the device failure after the active server has been terminated:

Determine the cause of the device failure.

Determine the cause of the failure by referencing the KAMN725-W and KAMN726-E messages and the message issued by the kernel and by using hardware management tools.

Resolve the cause of the device failure.

Resolve the cause of the device failure by taking appropriate action, such as by replacing the erroneous device.

In a redundant configuration with multipath software, do not at this time restore the path that resulted in the failure to online status (failback).

In a multi-path configuration, restore the path that has been recovered from the failure to online status (failback).

Restore a recovered path to online status (failback) by using the appropriate command provided by the multipath software (HDLM, DMMP, or HFC-PCM). For details about how to restore paths to online status, see the manual Hitachi Dynamic Link Manager Software User's Guide (for Linux(R) systems). Alternatively, see the documentation for DMMP or HFC-PCM.

For a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), this step is not necessary.

Cancel the reservation.

If the failure occurred while the active server was terminating normally, cancel the reservation by referencing 7.5.12 Canceling SCSI reservation for shared disk.

If the failure occurred while planned termination or abnormal termination was underway on the active server, there is no need to cancel the reservation because the standby server forcibly obtains a reservation.

7.5.9 Handling device failures on a shared disk (while the standby server is starting, on standby, or terminating) (using SCSI reservation for shared disk)

If an I/O error occurs while the standby server is starting, on standby, or terminating on the device specified in the scsi_device or dmmp_device operand in the server environment definition, HA Monitor issues the KAMN725-W and KAMN726-E messages and resumes processing. In a redundant configuration with multipath software, HA Monitor issues the KAMN726-E message only when a failure has occurred on all paths to the same shared disk.

To recover from a device failure when the standby server is starting, on standby, or terminating:

Determine the cause of the device failure.

Determine the cause of the failure by referencing the KAMN725-W and KAMN726-E messages and the message issued by the kernel and by using hardware management tools.

Resolve the cause of the device failure.

Resolve the cause of the device failure by taking an appropriate action, such as by replacing the erroneous device.

In a redundant configuration with multipath software, do not at this time restore the path that resulted in the failure to online status (failback).

In a multi-path configuration, restore the path that has been recovered from the failure to online status (failback).

Restore a recovered path to online status (failback) by using the appropriate command provided by the multipath software (HDLM, DMMP, or HFC-PCM). For details about how to restore paths to online status, see the manual Hitachi Dynamic Link Manager Software User's Guide (for Linux(R) systems). Alternatively, see the documentation for DMMP or HFC-PCM.

For a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), this step is not necessary.

If you restart the OS when there is a server that is running as the active system in the other system, the following message might be output to the syslog file: kernel: sd x:x:x:x: reservation conflict (x: numeric value). Note that the message to be output might differ depending on the OS version.

This means that the active server has obtained a SCSI reservation to protect the disk. This is a normal operation and there is no need to take any action.

7.5.10 Handling device failures on a shared disk (during hot standby processing) (using SCSI reservation for shared disk)

If an I/O error occurs when the standby server switches over to the active system on the device specified in the scsi_device or dmmp_device operand in the server environment definition, HA Monitor issues the KAMN725-W and KAMN726-E messages and hot standby processing fails. Take appropriate action by referencing 7.5.8 Handling device failures on a shared disk (while the active server is terminating) (using SCSI reservation for shared disk).

However, in a redundant configuration with multipath software, HA Monitor issues the KAMN726-E message only when a failure has occurred on all paths to the same shared disk. If the KAMN725-W message was issued but the KAMN726-E message was not issued, take appropriate action by referencing 7.5.7 Handling device failures on a shared disk (while the active server is running) (using SCSI reservation for shared disk).

Note that when hot standby processing is performed, the following message might be output to the syslog file: kernel: sd x:x:x:x: reservation conflict (x is a number; the message might differ depending on the OS version).

This means that the active server has obtained a SCSI reservation to protect the disk. This is a normal operation and there is no need to take any action.

7.5.11 Dealing with the server that does not release the reservation (using SCSI reservation for shared disk)

If any of the following events occurs, the reservation has not been released:

A host failure occurred on the active system while hot standby was not possible, such as when the standby server was inactive

The OS was shut down while the active server was running

The KAMN728-W message is output while stopping a server that has experienced system-switching after a host failure

The reservation must be released manually before the server is restarted. If the server is restarted without releasing the reservation, the KAMN725-W message is output and the server startup fails.

The following steps show the procedure for releasing the reservation manually:

Make sure that the failed active server is stopped.

If it is not stopped, stop the server, shut down the OS, or turn off the power of the machine.

See 7.5.12 Canceling SCSI reservation for shared disk, and release the reservation.

You do not have to make the comparison with the keys that are output in the KAMN725-W message in step 1.

For keys and device names specified for each step, do not specify the values output in the KAMN725-W message. Instead, specify the following values:

Key: 0xn of Key=0xn that is output as the execution result in step 1.

Steps after step 2 do not have to be executed for device names for which PR generation=0xnn, there is NO reservation held is output as an execution result in step 1.

Device name: scsi_device operand and dmmp_device operand specified value of the environment setting for servers

If multiple device names are specified to each operand, steps from 1 to 4 are executed for each specified value.

The following shows an example of a shell that uses the sg_persist command to release reservations.

The following reservations are released:

/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000001

/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000002

/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000003

#!/bin/sh

DEV_LIST="/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000001 /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000002 /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000003"
KEY="0x01"

for DESTDEV in ${DEV_LIST}
do
  echo "It processes ${DESTDEV} device."
  echo "execute register."
  /usr/bin/sg_persist --out --no-inquiry --register --param-sark=${KEY} --device=${DESTDEV}

  echo "execute clear."
  /usr/bin/sg_persist --out --no-inquiry --clear --param-rk=${KEY} --device=${DESTDEV}

  echo "execute read-reservation."
  /usr/bin/sg_persist --in --no-inquiry --read-reservation --device=${DESTDEV}

  echo "execute read-keys."
  /usr/bin/sg_persist --in --no-inquiry --read-keys --device=${DESTDEV}
  echo ""
done

exit 0

7.5.12 Canceling SCSI reservation for shared disk

This subsection explains how to cancel SCSI reservation for shared disk. Do not perform this procedure if not instructed to do so in the error handling procedures.

Only the superuser can cancel reservations. You can perform this procedure on any host. The procedure is the same regardless of whether the configuration is a single-path configuration, a VMware ESXi-based virtualization environment, or a redundant configuration with multipath software.

To cancel SCSI reservation for shared disk:

Verify that the shared disk has been reserved.

The following shows how to do this:

# /usr/bin/sg_persist --in --no-inquiry --read-reservation --device=device-name

Legend:

device-name: This is the value indicated in the KAMN725-W message. In a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), this is a symbolic link, and in a redundant configuration with multipath software, it is a physical device. If the device name displayed in the KAMN725-W message is followed by a path name, this is the path name.

The following example specifies /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065 as the device name.

Example

# /usr/bin/sg_persist --in --no-inquiry --read-reservation --device=/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065

Result:

If the message shown below is displayed, check that 0x1 matches the key displayed in the KAMN725-W message.

PR generation=0xnn, Reservation follows:
  Key=0x1
  scope: LU_SCOPE,  type: Write Exclusive, registrants only

If the key does not match the key displayed in the KAMN725-W message or the following message is displayed, check and, if necessary, revise the specified device name, and then re-execute the command.

PR generation=0xnn, there is NO reservation held

Prepare to cancel reservations.

The following shows how to prepare:

# /usr/bin/sg_persist --out --no-inquiry --register --param-sark=key --device=device-name

Legend:

key, device-name: This is the value indicated in the KAMN725-W message. In a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), this is a symbolic link, and in a redundant configuration with multipath software, it is a physical device. If the device name displayed in the KAMN725-W message is followed by a path name, this is the path name.

The following example specifies /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065 as the device name.

Example

# /usr/bin/sg_persist --out --no-inquiry --register --param-sark=0x1 --device=/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065

If nothing is displayed after execution, the preparation has been completed.

Note that the message shown below might be displayed. In this case also, the preparation has been completed. Continue with cancellation of reservation.

persistent reserve out: scsi status: Reservation Conflict
PR out: command failed

Cancel the reservation.

The following shows how to cancel the reservation:

# /usr/bin/sg_persist --out --no-inquiry --clear --param-rk=key --device=device-name

Legend:

key, device-name: This is the value indicated in the KAMN725-W message. In a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), this is a symbolic link, and in a redundant configuration with multipath software, it is a physical device. If the device name displayed in the KAMN725-W message is followed by a path name, this is the path name.

The following example specifies /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065 as the device name.

Example

# /usr/bin/sg_persist --out --no-inquiry --clear --param-rk=0x1 --device=/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065

If nothing is displayed after execution, the reservation has been cancelled successfully.

Verify that the reservation has been cancelled.

The following shows how to verify reservation cancellation:

# /usr/bin/sg_persist --in --no-inquiry --read-reservation --device=device-name
# /usr/bin/sg_persist --in --no-inquiry --read-keys --device=device-name

Legend:

device-name: This is the value displayed in the KAMN725-W message. In a single-path configuration, or in a VMware ESXi-based virtualization environment (where DMMP is not used), this is a symbolic link, and in a redundant configuration with multipath software, it is a physical device.

The following example specifies /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065 as the device name.

Example

# /usr/bin/sg_persist --in --no-inquiry --read-reservation --device=/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065
# /usr/bin/sg_persist --in --no-inquiry --read-keys --device=/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065

Result:

PR generation=0xnn, there is NO reservation held
PR generation=0xnn, there are NO registered reservation keys

If the above messages are displayed, the reservation has been cancelled. If these messages are not displayed, check the steps beginning with step 1 and then re-execute them.

7.5.13 Handling UAP errors

If process shutdown or a slowdown occurs in a UAP that is being monitored by the monitor-mode program management function, determine the cause of the error by checking the stack trace in the UAP's core file.

If the stack trace contains either of the function names listed below, an internal error has occurred in an API provided by HA Monitor. In this case, contact the system administrator.

hamon_patrolstart

hamon_patrolstop

7.5.14 Handling a duplicated address operand value

If all the following conditions are satisfied, HA Monitor might malfunction:

The applicable model is BladeSymphony, HA8000, or RV3000.

Hot-standby switchover based on host reset is selected as the switchover method.

An address operand value in the HA Monitor environment settings is the same as that in any other hot-standby configuration.

The following explains the corrective action to take if a duplicate address operand value (host address) is specified by mistake, resulting in issuance of the KAMN414-E message.

Terminate HA Monitor on the local host or the HA Monitor indicated by the detail code in the KAMN414-E message.

For details about how to terminate HA Monitor, see (3) Terminating HA Monitor.

Take one of the following actions:

Isolate the network so that no messages are transmitted between the hot-standby configurations.

Change the port numbers for reset path (including SVP) and monitoring path to unique values in each hot-standby configuration so that no messages are transmitted between the hot-standby configurations.

Change the host addresses of all HA Monitors that share the network, not just within the hot-standby configuration, to unique values (change the address operand values in the HA Monitor environment settings).

If you have taken the second or third actions of step 2, restart the HA Monitors whose settings were changed.

Do not restart the OS until the HA Monitor is restarted.

7.5.15 Dealing with server failures that disable restart and hot standby processing (server in the monitor mode)

In the event of a failure on a server in the monitor mode, if automatic restart and hot standby processing cannot be performed, the server status remains under execution processing in the HA Monitor. In such a case, restart the active server or perform hot standby processing with the procedure described here.

If you want to restart the active server:

On the active system, execute the monitor-mode server termination command (monend command) to terminate the active server.

On the active system, execute the monitor-mode server start command (monbegin command) to start the active server.

On the standby system, execute the monitor-mode server start command (monbegin command) to start the standby server.

If you want to perform hot standby processing:

Use one of the following methods to perform hot standby processing:

Method 1

On the standby system, execute the monitor-mode server start command (monbegin command) to start the standby server.

On the active system, execute the server hot-standby switchover command (monswap command) to perform planned hot-standby switchover.

On the same host as in step 2, execute the monitor-mode server start command (monbegin command) to start the standby server.

Method 2

On the active system, execute the monitor-mode server termination command (monend command) to terminate the active server.

On the standby system, execute the monitor-mode server start command (monbegin command) to start the standby server.

On the standby system, execute the wait-state server start command (monact command) to start the server in start wait state as the active server.

On the same host as in step 1, execute the monitor-mode server start command (monbegin command) to start the standby server.

7.6 Operation for preventing failures caused by heavy load

To prevent slowdown of servers and hosts due to heavy load, check whether the settings in definition files and the system configuration are appropriate based on the following information:

Information about the OS

Information about the programs that have an interface with HA Monitor

Slowdown information collected in the monitoring history file

The monitoring history is a record of slowdowns that have occurred in servers and hosts.

HA Monitor collects monitoring history for servers running on the local host in the server mode and the local and remote hosts. This monitoring history is output to monitoring history files. The monitoring history files contain information about slowdowns that exceed the server failure monitoring time and host failure monitoring time. They also contain information about slowdowns that do not reach the server failure monitoring time or host failure monitoring time. In other words, slowdowns that HA Monitor does not detect as failures are also recorded in a monitoring history file. Analyzing the history of slowdowns that are not detected as failures enables you to do the following:

Reevaluate the values set in the definition files for the server failure monitoring time and host failure monitoring time.

Reevaluate the system configuration based on the actual system load status.

If you create a shell beforehand, you can use it to investigate the cause of server slowdowns when slowdowns occur.

This section explains how to collect and analyze monitoring history and how to investigate the causes of server slowdowns.

Important

Server monitoring history cannot be collected in the following cases:

The server is a standby server.

Collection of server monitoring history and detection of server slowdowns are not supported in the following cases:

The server is running in the monitor mode.

7.6.1 Collecting server monitoring history

This subsection explains the information that can be collected as server monitoring history and how to collect server monitoring history.

When HA Monitor collects server monitoring history, it records server slowdowns in the monitoring history file regardless of whether a failure has occurred. If the length of time during which no operation report is received from the server does not exceed the server failure monitoring time but does exceed a threshold value set by the user (time when server monitoring history is to be obtained), HA Monitor records a server slowdown in the monitoring history file.

The user must specify the following operands in the definition files beforehand:

ph_log_size operand in the HA Monitor environment settings

Specifies a size for a monitoring history file. For details about how to estimate and specify the size of a monitoring history file, see 8.3.1 HA Monitor environment settings (sysdef).

ph_threshold operand in the server environment definition

Specifies the time when server monitoring history is to be obtained.

7.6.2 Collecting host monitoring history

This subsection explains the information that can be collected as host monitoring history and how to collect monitoring history information.

When HA Monitor collects host monitoring history, it records host slowdowns in the monitoring history file regardless of whether a failure has occurred. If the duration of a slowdown on the local host or the length of time during which no alive message is received from the remote host does not exceed the host failure monitoring time but does exceed a threshold value set by the user (time when host monitoring history is to be obtained), HA Monitor records a host slowdown in the monitoring history file.

The user must specify the following operands in the HA Monitor environment settings beforehand:

ph_log_size operand in the HA Monitor environment settings

Specifies a size for a monitoring history file. For details about how to estimate and specify the size of a monitoring history file, see 8.3.1 HA Monitor environment settings (sysdef).

ph_threshold operand in the HA Monitor environment settings

Specifies the time when host monitoring history is to be obtained. This value must be the same for the local and remote hosts.

7.6.3 Analyzing the collected monitoring history

Collected monitoring history is output to a monitoring history file:

/opt/hitachi/HAmon/history/patrol_history

When the size of the monitoring history file exceeds the size specified in the ph_log_size operand in the HA Monitor environment settings, the file is renamed to the following backup file:

/opt/hitachi/HAmon/history/patrol_history_old

When you analyze monitoring history, use the current monitoring history file and its backup files. The current monitoring history file and its backup files are always available for analysis, but make sure that you do not directly edit the files or the directory in which the files are stored.

(1) Output format for a monitoring history file

Each monitoring history item is output as one line in the monitoring history file. The following shows the output format for a monitoring history file item:

year/month/day time;ID;type;server-alias-name-or-host-name;seconds;message

year/month/day time

Date and time the monitoring history was collected.

ID

Unique ID assigned to each message that is output as monitoring history information.

type

Type of the target for which monitoring history was collected:

S: Active server running on the local host

H: Remote host

O: Local host

server-alias-name-or-host-name

Server alias name or host name for which the monitoring history was collected.

seconds

Duration of the slowdown in seconds. This information might not be output depending on the nature of the message that is output as the monitoring history.

message

Message text that is output as the monitoring history item. For details about the actual message texts that are output, see B. Messages Output as Monitoring History.

When HA Monitor creates a monitoring history file, it pads the file with spaces to fix the file size. Therefore, blank lines might remain in the file after monitoring history has been added. You can ignore these blank lines during analysis.

(2) Saving the backup files

When the size of the current monitoring history file exceeds a specified value, HA Monitor issues the KAMN740-I message. When this message is issued, save the file as a backup file. You will use the saved backup files when you analyze the monitoring history.

7.6.4 Determining the cause of a server slowdown

If one of the following types of a slowdown occurs, HA Monitor enables you to run a shell to collect information required for determining the cause of the slowdown:

Slowdown of a server-mode server

Slowdown of a server that uses the monitor-mode server monitoring function (enabled by using the ptrlcmd_ex operand in the server environment definition)

Slowdown of a UAP on a monitor-mode server

This subsection explains the name for the shell to be created and its run-time format. The shell explained here is executed asynchronously with HA Monitor processing.

(1) Shell to be run in the event of a server slowdown

This subsection explains the shell used for collecting information needed to determine the cause of a server slowdown. Run the shell if one of the following types of a slowdown occurs:

Slowdown of a server-mode server

Slowdown of a server that uses the monitor-mode server monitoring function (enabled by using the ptrlcmd_ex operand in the server environment definition)

The following shows the name of the shell to be created and the run-time format.

Name of the shell to be created

/opt/hitachi/HAmon/etc/servdown.sh

Code the processing that you want to perform in this shell. Grant execution permissions to the shell.

Run-time format of the shell

servdown.shΔserver-alias-name

Legend:

Δ: Single-byte space

server-alias-name: Alias name of the active server on which slowdown was detected

(2) Shell to be run in the event of a UAP slowdown

This subsection explains the shell used for collecting information needed to determine the cause of a UAP slowdown. UAP slowdowns can be detected only when UAP operation reports are monitored. The following shows the name of the shell to be created and the run-time format.

Name of the shell to be created

/opt/hitachi/HAmon/etc/uapslowdown.sh

Code the processing that you want to perform in this shell. Grant execution permissions to the shell.

Shell run-time format

uapslowdown.shΔprogram-alias-nameΔprocess-ID

Legend:

Δ: Single-byte space

program-alias-name: Program alias name of the UAP whose slowdown was detected on the active server

process-ID: Process ID of the UAP whose slowdown was detected

(3) Notes

You can detect a server slowdown only if TP1/Server Base or HiRDB is used as a server-mode server or if the monitor-mode server monitoring function (enabled by using the ptrlcmd_ex operand in the server environment definition) is used.

If a server slowdown is detected but there is no available standby server, HA Monitor continues to monitor the status of this server because hot standby processing cannot be performed. If the server slowdown is not resolved, this shell is run each time a slowdown is detected.

If there is a shortage of resources throughout the entire system, the shell might not run even when a slowdown is detected. In such a case, the information might not be collected because HA Monitor terminates abnormally the server or UAP whose slowdown was detected and then runs the shell.

7.7 Automating operations

This section explains how to automate operations in a system that uses HA Monitor. If you want to automate other operations that those discussed here, use operations management software such as JP1. HA Monitor issues events at major points in time, such as when HA Monitor and servers start and when hot standby processing begins. For details about such events, see A. Event IDs Used by HA Monitor.

7.7.1 Automating the operation from system start through server start

This subsection explains how to perform automatically the operations from system start through server start after you turn on the power to the computer. Whether server startup processing can be automated depends on the type of servers.

The following subsections explain how to automate the HA Monitor, resource server, and server startup processing.

(1) Automating the HA Monitor startup processing

Normally, HA Monitor starts automatically after the kernel has started. You can also set HA Monitor to not start automatically. You use the HA Monitor environment setup command (monsetup command) to change this setting.

(2) Automating the resource server startup processing

The resource server is started automatically by HA Monitor. There is no need to specify a setting to automate this processing.

(3) Automating server startup processing

Servers in the server mode can be started automatically by using the method provided by the program. For details, see the program documentation.

Servers in the monitor mode cannot be started automatically. You must use the monitor-mode server start command (monbegin command) to start a server manually.

7.7.2 Automating the processing to be performed when the status of a server or HA Monitor changes

You can execute commands automatically on the basis of changes in the server or HA Monitor status.

If a standard operation needs to be performed by the operator when a server is started or when planned hot standby processing is performed, you can automate that operator action by creating a user command. You can also issue HA Monitor commands from within a user command. However, HA Monitor commands cannot be issued when HA Monitor is being started or terminated.

For the timing of issuing user commands by HA Monitor, see 3.5.2 Issuing commands when the server and HA Monitor status changes.

7.7.3 Automating an operation after hot standby processing

(1) Automating an operation after hot standby processing in the event of a server failure

For servers in the server mode, you can automate the performance of an operation after hot standby processing by using functions provided by the program. For details, see the program documentation.

For servers in the monitor mode, performance of an operation after hot standby processing cannot be automated.

(2) Automating an operation after hot standby processing in the event of a host failure

When hot standby processing occurs due to a host failure, the operator must manually start the host and the servers on the host where the host failure occurred. The method for automating this processing depends on whether the servers are running in the server mode or the monitor mode.

For servers in the server mode, you can automate this processing by using functions provided by the program. For details, see the program documentation.

For servers in the monitor mode, you can automate this processing by using functions provided by HA Monitor. To start servers automatically when the servers are to be restarted on a host on which a host failure occurred, specify the monbegin_restart operand in the HA Monitor environment settings. By default, such server startup processing is set to be performed automatically.

In the event of a host failure that disables monitoring paths, the standby server might be placed in the hot-standby wait state on the target host, regardless of whether the server is in the server mode or the monitor mode. Handling of a server in the hot-standby wait state cannot be automated. For details about the handling, see 7.4.1 Starting a server in the wait state and then restarting jobs.

7.8 Planned hot standby

This section explains how to perform hot standby processing intentionally in order to maintain hosts and distribute the processing load.

7.8.1 Performing a planned hot standby operation

Hot standby that is performed at any time regardless of whether a failure has occurred is called planned hot standby. The planned hot standby method depends on whether the servers are running in the server mode or the monitor mode and on whether the servers are grouped.

(1) For servers in the server mode

To perform planned hot-standby switchover, execute the server hot-standby switchover command (monswap command) in the active system. HA Monitor performs planned termination on the server in the active system and inherits job processing to the server in the standby system.

(2) For servers in the monitor mode

To perform planned hot-standby switchover, execute the server hot-standby switchover command (monswap command) in the active system.

To perform planned hot standby when no server termination command has been created:

Perform a planned termination on the active server running in the active system.

Execute the planned termination command provided by the program.

Perform planned hot standby in the active system.

Execute the server hot-standby switchover command (monswap command).

(3) When servers are grouped

To perform a planned hot-standby switchover in an environment where servers have been grouped, execute the server hot-standby switchover command (monswap command) with the -g option specified in the active system. HA Monitor performs planned termination on all the servers in the group running in the active system and transfers job processing to the servers in the standby system.

7.9 Forced hot-standby switchover

This section describes how you can forcibly perform hot-standby switchover in such a case when you cannot log in to the OS in the active system or when attempted hot-standby switchover does not finish.

7.9.1 Forcibly performing hot-standby switchover

The OS operation in the active system might become unstable if the CPU load, memory load, and disk I/O load become high. In such a case, the active server might be unable to execute job processing. Note that if HA Monitor operates normally even when the OS operation is unstable, HA Monitor might not perform hot-standby switchover without detecting a host failure. In such a case, you can perform hot-standby switchover by forcibly terminating the OS on which the active server is running. This operation is called forced hot-standby switchover. You can use forced hot-standby switchover in the following cases:

When you cannot log in to the OS of the active system

If the OS operation of the active system is unstable and you cannot log in to the active system, you might be unable to perform planned hot-standby switchover from the active system. In such a case, you can perform hot-standby switchover by executing the monswap command with the -f or -f -g option in the standby system.

When attempted hot-standby switchover does not finish

Planned hot-standby switchover attempted from the active system or hot-standby switchover attempted when a failure is detected by the server monitoring command might not be completed for some reason. For example, this situation occurs if it takes time to terminate the server or to disconnect shared resources in the active system. If disconnection of shared resources or another process takes time, the KAMN310-I message (hot-standby switchover start message) is not output in the standby system. In such a case, you can perform hot-standby switchover by executing the monswap command with the -f or -f -g option in the standby system.

7.10 Modifying the system

This section explains the settings that are required when you modify the system and provides notes on system modification.

7.10.1 Adding hosts

This subsection explains how to add hosts to a hot-standby configuration depending on the number of hosts after addition of hosts.

There is a limit to the number of hosts that can run concurrently in a single system. Before you add a host, make sure that the added host will not exceed the maximum permitted number of hosts, which is 32 by default. However, if you use HA Monitor Extension, you can increase the maximum number.

(1) When the number of hosts after addition will not exceed 32

This subsection explains how to add hosts when the number of hosts after addition will not exceed 32. You must install HA Monitor on a host that is to be added. For details about the configuration procedure after HA Monitor installation, see 6.1 Configuration procedure. This subsection explains only the differences when hosts are added from when the entire system is newly configured.

Configuring the monitoring paths

When you add a new host, you must change the HA Monitor connection configuration file. The names of all connected hosts are specified in the HA Monitor connection configuration file.

If you create the HA Monitor connection configuration files automatically, start the HA Monitors on all hosts. HA Monitor connection configuration files will be created automatically.

If you create the HA Monitor connection configuration files manually, edit the file and then make sure that the contents of the files on all hosts are the same.

For details about how to create an HA Monitor connection configuration file, see 6.9.2 Creating an HA Monitor connection configuration file.

(2) When the number of hosts after addition will be 33 or more

This subsection explains how to add hosts when the number of hosts after addition will be 33 or more. If you use HA Monitor Extension, you can set the maximum number of hosts to 33 or more. For details about increasing the maximum number of hosts by using HA Monitor Extension, see 3.7.2 Changing the maximum number of hosts.

This subsection explains how to add hosts up to the maximum number (upper limit) specified during configuration and beyond that maximum number.

(a) Adding hosts within the range of the maximum value

To add hosts up to the maximum value specified during configuration, perform all steps shown in the figure in (2) System configuration procedure (when using HA Monitor Extension).

For details about the Configure the monitoring paths step shown in Figure 6-1, see (1) When the number of hosts after addition will not exceed 32.

(b) Changing the maximum value

To change the upper limit specified during configuration:

Terminate all servers running on all hosts in the hot-standby configuration.

To ensure that all servers have terminated, execute the server and host status display command (monshow command) and check that all the servers on that host have terminated.

Terminate all HA Monitors on all hosts in the hot-standby configuration.

To ensure that the HA Monitors have terminated, check syslog for the KAMN050-I messages issued for the HA Monitors on both hosts.

Specify the settings required for adding hosts on all hosts in the hot-standby configuration.

Perform all steps shown in the figure in (2) System configuration procedure (when using HA Monitor Extension).

For details about the Configure the monitoring paths step shown in Figure 6-1, see (1) When the number of hosts after addition will not exceed 32.

To perform the Create definition files (HA Monitor) step shown in Figure 6-1, modify the hostmax operand in the definition file used to set up a HA Monitor operating environment. For details about the hostmax operand, see 8.3.1 HA Monitor environment settings (sysdef).

7.10.2 Adding servers

This subsection explains how to add servers to a hot-standby configuration that has already been configured.

When you add servers, you install the servers in both the active system and the standby system, and then you configure the environment.

(1) Preparations

If you use HA Monitor to inherit a shared disk and LANs that are used by the servers, the preparations described below are required. If you do not inherit a shared disk or LANs by using HA Monitor, these preparations are not needed.

Add volume groups to the shared disk.

In a hot-standby configuration using HA Monitor, one volume group is switched for each server. Therefore, you must create volume groups for the servers that are added as needed.

Obtain the alias IP addresses that are used by the servers.

HA Monitor inherits alias IP addresses. Because the servers use the same alias IP addresses in the primary system and the secondary system, at least one alias IP address is needed per server. The same server can also use multiple alias IP addresses.

(2) Settings required to add servers

This subsection explains the settings required to add servers that differ from when the entire system is newly configured.

There is a limit to the number of servers that can be run concurrently on a single host. Before you add a server, make sure that adding the added server will not exceed the maximum permitted number of servers per host, which is 16 by default. You can change this default value by changing the servmax operand value in the HA Monitor environment settings.

(a) Adding servers within the range of the maximum value

If adding servers does not exceed the maximum permissible number of servers, follow the procedure for Configuration steps when a server is added provided in 6.1 Configuration procedure. There is no need to terminate HA Monitor.

(b) Changing the maximum value

If you change the maximum value, you must follow the procedure for Configuration steps when a server is added provided in 6.1 Configuration procedure and you must also specify HA Monitor environment settings.

To do this:

Terminate all servers running in the active and standby systems.

To ensure that all servers have terminated, execute the server and host status display command (monshow command) and check that all the servers on that host have terminated.

Terminate the HA Monitors in the active and standby systems.

To ensure that the HA Monitors have terminated, check syslog for the KAMN050-I message issued for the HA Monitors on both hosts.

Specify the settings required for adding servers and the HA Monitor environment settings.

Specify the settings by following the procedure for Configuration steps when a server is added provided in 6.1 Configuration procedure.

You must also perform the Create definition files (server) and Create definition files (HA Monitor) steps. In the Create definition files (HA Monitor) step, you must correct the servmax operand in the definition file used to specify the HA Monitor operating environment. For details about the servmax operand, see 8.3.1 HA Monitor environment settings (sysdef).

7.10.3 Changing shared resources

This subsection explains how to change shared resources. Some shared resources can be changed dynamically while hosts and servers are running. For details about changing shared resources dynamically, see 7.10.4 Changing the shared resource configuration while hosts and servers are running.

The procedure for changing shared resources is described below. You must perform this procedure in the active and the standby systems.

Terminate the servers.

You need terminate only servers that use shared resources. HA Monitor can continue running.

Specify the shared resources.

Specify the shared resources to be used. For details, see 6.11 Configuring the shared resources to be used by servers.

Change the definition file containing the environment settings.

Change the server environment definition or other definitions as appropriate for the type of shared resource.

The table below lists the settings to be changed for the various types of shared resources. For details about the server environment definition, see 8.4.1 Server environment definition (servers).

Table 7‒8: Types of shared resources and settings to be changed

Resource type

Operands to be changed (server environment definition)

Environment settings to be changed (other than the server environment definition)

Section

Shared disk

disk

vg_on_opt

vg_neck

--

--

File system

fs_name

fs_mount_dir

fs_mount_opt

fs_umount_retry

fs_neck

--

--

LAN

lan_updown

ip_neck

server-alias-name.up

server-alias-name.down

6.11.2 Specifying LAN status settings files

Other resources

uoc_neck

uoc_pairdown

User command

6.18 Creating user commands

All resources

dev_timelimit

dev_onlimit

dev_offlimit

deviceoff_order

--

--

Legend:

--: Not applicable

Check the definitions.

Execute the definition check command (moncheck command).

Start the servers.

For servers in the server mode, execute the start command provided by the program. For servers in the monitor mode, execute the monitor-mode server start command (monbegin command).

Check the server status.

Execute the server and host status display command (monshow command).

7.10.4 Changing the shared resource configuration while hosts and servers are running

Adding and deleting shared resources while hosts and servers are running is called dynamically changing the shared resources. This subsection explains how to dynamically change the shared resources.

The method depends on the type of shared resource. To add and delete shared disks dynamically, you use the command for dynamically changing a server's shared resources (mondevice command). To add and delete LANs, you edit the LAN status settings file.

You can use the command for dynamically changing a server's shared resources (mondevice command) in either the active or the standby system.

To add and delete shared resources by using the command for dynamically changing a server's shared resources (mondevice command):

Make sure that the active and standby servers have completed their startup processing.

Execute the server and host status display command (monshow command) to verify that ONL is displayed for the active server and SBY is displayed for the standby server.

Connect or disconnect the shared resources.

Execute the command for dynamically changing a server's shared resources (mondevice command) to connect new shared resources or to disconnect existing shared resources. The server environment definitions for the active and standby servers that are sharing the resources are changed according to the results of the change made to the shared resources.

Check the new settings after change.

Use the server and host status display command (monshow -d command) to verify that the shared resources have been changed as specified.

When you execute the command for dynamically changing a server's shared resources (mondevice command), the server environment definitions for the active and standby servers that are sharing the resources are changed automatically as appropriate for the change made to the shared resources. After you have changed the shared resources dynamically, you must also verify that the server environment definitions have been changed correctly.

If the shared resources for the active server cannot be changed, HA Monitor will not change the shared resources for the standby server. If the shared resources for the standby server cannot be changed, HA Monitor changes only the shared resources for the active server. In such a case, the settings for the shared resources will no longer match between the active and standby servers. To synchronize the settings for the shared resources, you will have to terminate the standby server, synchronize the contents of the server environment definitions between the active and standby servers, and then restart the standby server.

If the shared resources cannot be changed by using the command for dynamically changing a server's shared resources (mondevice command), possible reasons include HA Monitor errors, conflicting command specifications, and the following:

When the command for dynamically changing server's shared resources (mondevice command) is executed on the active server:

The active server has not finished its startup processing.

A server failure occurred on the active server.

A communication error occurred when notifying the standby server of the change (the shared resources for the standby server cannot be changed).

When the command for dynamically changing a server's shared resources (mondevice command) is executed on the standby server:

The active or standby server has not finished its startup processing.

A server failure occurred on the active or standby server.

A communication error occurred when notifying the active server of the change to the shared resources or when receiving notification of completion of the change to the shared resources from the active server (the shared resources for the active server cannot be changed).

7.10.5 Changing the environment settings for HA Monitor and servers

This subsection explains how to change the environment settings.

HA Monitor environment settings (sysdef):

Terminate HA Monitor before you change the HA Monitor environment settings. Make sure that HA Monitor has terminated normally. If you change the HA Monitor environment settings on a host that was reset during hot standby processing and then restart HA Monitor, HA Monitor might not function correctly. When you have finished editing the definition file, start the HA Monitor and the servers.

Server environment settings (servers and servers_opt):

Terminate the server before you change the server environment definition. If you change the server environment definition for a grouped server, or if you add servers to or delete servers from a server group, you must terminate all the servers in the group. You can keep HA Monitor running. When you have finished editing the definition file, start the server (or servers).

Monitor-mode program environment settings (programs)

To change the monitor-mode program environment definition, terminate the server specified in the ALIAS operand in the monitor-mode program environment definition. HA Monitor can keep running. After you have finished editing the definition file, start the server specified in the server_alias operand in the monitor-mode program environment definition.

You can also change the settings for HA Monitor, servers, and programs that use the monitor-mode program management function while they are running. For details about the methods, see 7.10.6 Changing the settings and operations while HA Monitor and servers are running.

7.10.6 Changing the settings and operations while HA Monitor and servers are running

HA Monitor provides a command for changing the settings and operations while HA Monitor and servers are running (monchange command). You can use this command to change the settings and operations while HA Monitor and servers are running.

The following table shows the settings and operations that can be changed and the corresponding operands.

Table 7‒9: Settings and operations that can be changed and corresponding operands

Setting and operation

Corresponding operand

Host failure monitoring time

patrol operand in the HA Monitor environment settings

Server failure monitoring time

patrol operand in the server environment definition

Program slowdown monitoring interval

patrol operand in the monitor-mode program environment definition

Temporarily stopping and restarting the monitoring while HA Monitor and servers are running

HA Monitor environment settings:

disk_ptrl operand

Server environment definition:

disk_ptrl operand

ptrlcmd_ex operand

sby_ptrlcmd_ex operand

This subsection explains how to change each setting.

Important

The monchange command, which changes the settings and operations while HA Monitor and servers are running, does not change the HA Monitor and server environment definition settings files. This is a difference from the mondevice command, which reconfigures shared resources while the active server is running. If you restart a host or server after changing its settings, the change reverts to the settings specified in the definition file.

(1) Changing the host failure monitoring time

You can change the host failure monitoring time specified in the patrol operand in the HA Monitor environment settings. The command for changing the settings and operations while HA Monitor and servers are running (monchange command) can be executed from any host. This command executed on a host can change the host failure monitoring time of all hosts that are connected to that host.

Executing this command on one host will change the host failure monitoring time on all hosts that are connected to that host. If host-to-host communication is disabled during command execution due to a failure on the remote host or a monitoring path failure, the command's processing will be cancelled.

Note the following about changing the host failure monitoring time:

Check that all hosts whose host failure monitoring time is to be changed are connected.

Do not execute the command from more than one host at the same time.

Different host failure monitoring times might be set on hosts in cases such as when a host is restarted after the host failure monitoring time is changed and when a communication failure occurs. If inconsistency in the host failure monitoring time occurs among hosts, re-execute the command for changing the setting and operations while HA Monitor and servers are running (monchange command) so that the hosts share the same host failure monitoring time.

If host monitoring history is to be collected and the specified host failure monitoring time is shorter than the time when host monitoring history is to be obtained, no monitoring history is collected.

(2) Changing the server failure monitoring time

You can change the server failure monitoring time specified in the patrol operand in the server environment definition.

When the command for changing the settings and operations while HA Monitor and servers are running (monchange command) is executed, HA Monitor immediately restarts server monitoring with the new server failure monitoring time.

Note the following about changing the server failure monitoring time:

The command for changing the settings and operations while HA Monitor and servers are running (monchange command) changes the server failure monitoring time only for the host on which the command is executed. Therefore, you must also execute the command on the host on which the standby server runs.

If server monitoring history is to be collected and the specified server failure monitoring time is shorter than the time when server monitoring history is to be obtained, no monitoring history is collected.

(3) Changing the program slowdown monitoring interval (when using the program management function)

You can change the program slowdown monitoring interval specified in the patrol operand in the monitor-mode program environment definition.

When the command for changing the settings and operations while HA Monitor and servers are running (monchange command) is executed, HA Monitor immediately restarts program monitoring with the new program slowdown monitoring interval.

When you change the program slowdown monitoring interval, note the following:

The command for changing the settings and operations while HA Monitor and servers are running (monchange command) changes the program slowdown monitoring interval only for the host on which the command is executed. You must also execute the command on the host where the standby server is running.

For a multi-process configuration in which multiple programs with the same program alias name are run, change the program slowdown monitoring interval for all processes.

(4) Temporarily stopping and restarting the monitoring while HA Monitor and servers are running

While HA Monitor and servers are running, you can temporarily stop or restart monitoring of a specific function (operand). When the command for changing the settings and operations while HA Monitor and servers are running (monchange command) is executed, HA Monitor immediately performs temporary stop or restart of monitoring.

Monitoring can be temporarily stopped or restarted for the following operands.

When temporarily stopping or restarting the system disk monitoring:

disk_ptrl in the HA Monitor environment settings

When temporarily stopping or restarting the server monitoring:

When temporarily stopping or restarting the monitoring of disks for business use on servers:

• disk_ptrl in the server environment definition

When temporarily stopping or restarting the monitoring of servers in the monitor mode

• ptrlcmd_ex in the server environment definition

• sby_ptrlcmd_ex in the server environment definition

Note that the monitoring using the patrolcommand operand in the server environment definition cannot be temporarily stopped or restarted.

Part 4: Reference

8 Files Defined in Environment Settings

To use HA Monitor, you must first specify in definition files various settings for HA Monitor and for the server environment.

This chapter explains the definition specification rules and the operands to be specified in the definition files. It also provides examples of environment settings.

8.1 Overview of definition files

This section explains the files that the user defines as environment definitions. The following are the environment definition files that must be defined:

sysdef file

Specifies environment settings required to run HA Monitor. For details, see 8.3.1 HA Monitor environment settings (sysdef).

servers file

Specifies environment settings required to run servers. If you use resource servers, you also use this file to specify environment settings for them. For details, see 8.4.1 Server environment definition (servers).

servers_opt file

Specifies environment settings required to run locked servers. This file is optional. We recommend that you use this file if you do not want more than one active server to run on a single host. If you use locked servers, you must use both the servers and the servers_opt files to specify environment settings. For details, see 8.4.2 Locked server environment settings (servers_opt).

programs file

Specifies the definitions required to run programs that use the monitor-mode program management function. For details, see 8.5.1 Monitor-mode program environment definition (programs).

The following notes apply to specifying definition files on hosts:

You must specify the definition files on all hosts that contain HA Monitor.

Some operands in the definition files must be specified identically in the primary and secondary systems and some must be different.

After you have finished creating the definition files, make sure that you distribute them to all hosts and then execute the definition check command (moncheck command).

8.2 Definition rules

The following basic rules apply to specifying definitions:

The definition statements must be delimited by the semicolon (;).

The operands must be delimited by the comma (,).

Values must be delimited by the colon (:).

A comment must be enclosed between /* and */. If a text enclosed between /* and */ contains within it another text enclosed between /* or */, only the text segment bracketed by the first/* and the first */ is treated as a comment, and the remainder is treated as part of the definition statement.

This section explains the specification format of the environment definitions and the conventions used in the explanations of the definitions.

8.2.1 Specification format

The following shows the format of a definition:

Δ₀definition-statementΔ₁operandΔ₁valueΔ₀:Δ₀...Δ₀;Δ₀

Legend:

Δ₀: Zero or more spaces, tabs, or end-of-line codes

Δ₁: One or more spaces, tabs, or end-of-line codes

Note:

Do not delimit character strings with a space, tab, or end-of-line code at a location other than as indicated above by Δ₀ and Δ₁.

8.2.2 Conventions used in the definitions

This subsection explains the conventions used in the explanations of the definitions.

The syntax symbols, attribute symbols, and component symbols explained here are not actually specified in the definitions.

(1) Syntax symbols

The syntax symbols explain the syntax of the definition statements and operands. The following table explains the syntax symbols.

Table 8‒1: Syntax symbols for definition statements

Syntax symbol

Meaning

[ ]

Square brackets indicate that specification of the enclosed item or items are optional.

Example: [:TCP/IP-LAN-host-name]

In this example, you specify a TCP/IP LAN host name or nothing.

...

An ellipsis (...) indicates that one or more lines of code are not shown for purposes of brevity.

Example: TCP/IP-LAN-host-name...

In this example, you can specify one or more TCP/IP LAN host names.

|

A vertical bar separates multiple items, and has the meaning of OR.

Example: online|standby

In this example, the item online or standby is shown.

{ }

Curly brackets indicate that one of the enclosed items must be selected for specification. The items are separated by a vertical bar (|).

Example: {online|standby}

In this example, you specify either the string online or standby.

underline

An underlined item is assumed if the operand is omitted.

Example: [{online|standby}]

In this example, if the operand is omitted, the system assumes that online is specified.

bold type

Bold type indicates a value that must be the same on all hosts in the hot-standby configuration.

Example:

TCP/IP-LAN-service-name

In this example, the same TCP/IP LAN service name must be specified on all hosts in the hot-standby switchover configuration.

(2) Component symbols

Of the attribute symbols, the component symbols explain the lexical elements for the user values. HA Monitor does not support invalid characters (such as control codes). The following table explains the component symbols.

Table 8‒2: Component symbols

Component symbol

Meaning

<alphanumeric>

Alphabetic characters (A to Z and a to z) and numeric characters (0 to 9)#

<unsigned integer>

Numeric characters (0 to 9)

<path name>

Alphabetic characters (A to Z and a to z), numeric characters (0 to 9), forward slash (/), hyphen (-), underscore (_), and period (.)

<server path name>

Any characters other than a colon (:), semicolon (;), comma (,), space, tab, or end-of-line code

#: The first character must be an alphabetic character.

8.3 Configuring an environment for HA Monitor

The HA Monitor environment settings specify the definitions required for HA Monitor operation.

8.3.1 HA Monitor environment settings (sysdef)

You create a definition file containing the HA Monitor environment settings under the file name sysdef in the directory for HA Monitor environment settings.

A sample sysdef file is provided under the directory for HA Monitor sample files. If you copy this file to the directory for HA Monitor environment settings and then edit it, you can save time over creating the file from scratch.

You must create this definition file for each host. Bold type indicates a value that must be the same on all hosts. Care must be taken in specifying the other items so that there will be no conflicts between the hosts.

The following shows the format of the definition file used to specify the HA Monitor environment settings:

/*    HA Monitor environment settings    */
  environment  name  host-name-of-local-host
              ,address
                     host-address-of-local-host
              ,patrol
                     host-failure-monitoring-time
              ,lan   TCP/IP-host-name[:TCP/IP-host-name...]
              ,lanport
                     TCP/IP-service-name[:TCP/IP-service-name...]
            [,fs_log_size
                   {size-of-file-system-switchover-log-file
                   |65536}]
            [,servmax
                   {16|64|128|129-512#}]
            [,hostmax#
                   {maximum-number-of-hosts-to-be-connected|32}]
            [,pgmmax
                   {maximum-number-of-programs-that-can-run-concurrently|0}];
[function  [ cpudown
                   {online|standby|system}{,|;}]
            [ clearwait
                   {reset-publication-host-wait-time|20}{,|;}]
            [ standbyreset
                   {use|nouse}{,|;}]
            [ pathpatrol
                     monitoring-path-health-check-interval{,|;}]
            [ pathpatrol_retry
                   {re-check-interval|30}:{re-check-count|5}{,|;}]
            [ message_retry
                   {retry-interval|3}{,|;}]
            [ connect_retry
                   {connection-interval|5}:{connection-count|200}{,|;}]
            [ monbegin_restart
                   {use|nouse}{,|;}]
            [ netmask
                   {byte|bit}{,|;}]
            [ usrcommand
                     absolute-path-name-of-user-command{,|;}]
            [ resetpatrol
                   {reset-path-health-check-interval|2}{,|;}]
            [ resetpath_retry
                    {use|nouse}{,|;}]
            [ resetpath_inter
                      reset-path-re-check-interval{,|;}]
            [ multistandby
                   {use|nouse}{,|;}]
            [ deviceoff_order
                   {order|reverse}{,|;}]
            [ reset_type
                   {server|host}{,|;}]
            [ partition_reset
                   {use|nouse}{,|;}]
            [ jp1_event
                   {use|nouse}{,|;}]
            [ ph_log_size
                     monitoring-history-file-size{,|;}]
            [ ph_threshold
                     time-when-host-monitoring-history-is-to-be-obtained{,|;}]
            [ termcmd_at_abort
                   {use|nouse}{,|;}]
            [ alive_interval
                     alive-message-transmission-interval{,|;}]
            [ alive_multicast
                   {use|nouse}{,|;}]
            [ multicast_lan
                     host-name-of-multicast-group{,|;}]
            [ lanfailswitch
                   {use|nouse}{,|;}]
            [ lancheck_patrol
                   {LAN-monitoring-interval|15}{,|;}]
            [ lancheck_mode
                   {packet|route}{,|;}]
            [ disk_ptrl
                   {use|nouse}{,|;}]
            [ disk_ptrl_inter
                   {system-disk-check-interval|120}{,|;}]
            [ disk_ptrl_retry
                   {system-disk-check-retry-interval|60}:
                   {system-disk-check-retry-count|1}{,|;}]
            [ disk_log_size
                   {disk-monitoring-log-size|1048576}{,|;}]
            [ patrol_100ms
                   {use|nouse}{,|;}]
            [ suppress_reset
                     minimum-number-of-active-hosts{,|;}]
            [ exitcode
                   {type1|type2}{,|;}]
            [ vg_off
                   {sequential|parallel}{,|;}]
            [ vmware_env
                    {use|nouse}{,|;}]
            [ patrol_type
                   {server|host}{,|;}]
            [ resetpatrol_mode
                   {mode1|mode2}{,|;}]
            [ fence_reset
                   {use|nouse}{,|;}]
            [ fence_scsi
                   {use|nouse}{,|;}]
            [ fence_lan
                   {use|nouse}{,|;}]
            [ scsi_check
                   {reserve-status-monitoring-interval|5}{,|;}]
            [ scsi_pathcheck
                   {disk-path-status-monitoring-interval|1}{,|;}]
            [ scsi_timeout
                   {SCSI-command-reply-wait-time|30}{,|;}]
            [ scsi_retry
                   {SCSI-command-retry-count|2}{,|;}]
            [ hbond_lacp
                     hbonding-interface-name
                     [:hbonding-interface-name...];]
            [ vg_offskip
                   {use|nouse}{,|;}]
            [ notswitch_notify
                   {use|nouse}{,|;}]
            [ servcomplete_msg
                   {use|nouse}{,|;}]
[options   [ clearcheck
                   {host-reset-end-time|20};]

#: The operand can be specified only when HA Monitor Extension is used.

Note that if you use SCSI reservation for shared disk (use is specified in the fence_scsi operand), the following operands are ignored, if specified:

cpudown

standbyreset

resetpatrol

resetpath_retry

resetpath_inter

reset_type

partition_reset

suppress_reset

vmware_env

resetpatrol_mode

The following operands are ignored when you use the function for controlling hot standby based on the availability of LAN communications used by business applications (use is specified in the fence_lan operand):

cpudown

standbyreset

resetpatrol

resetpath_retry

resetpath_inter

reset_type

partition_reset

suppress_reset

vmware_env

resetpatrol_mode

(1) environment definition statement

The environment definition statement defines HA Monitor's operating environment. The following explains the operands of the environment definition statement.

(a) name

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 32 alphanumeric characters

--

(Units: --)

No

--

Description

Specifies a name for the host. This name is used by HA Monitor to identify the host.

Each host's name must be unique. The name specified in the name operand is displayed as the host name (host name) when HA Monitor commands are executed.

(b) address

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer having 1 to 8 digits

0 to 9999 (or 1 to 9999 if use is specified for the fence_scsi operand in the function definition statement)

(Units: --)

No

--

Description

Specifies a value indicating the local host's host address in order to identify the host when the CPU is reset because host reset has been performed, as well as to identify the host when it reserves a shared disk (when SCSI reservation for shared disk is used).

HA Monitor specifies the hardware settings.

You must specify a unique address for each host connected via monitoring paths and the reset path. If the system contains multiple hot-standby configurations that do not monitor each other but they share the same reset path, each of the hosts must have a unique address.

If two or more hosts have the same address, HA Monitor might malfunction. If you accidentally set the same address and the KAMN414-E message is output, take appropriate action as described in 7.5.14 Handling a duplicated address operand value.

Due to hardware limitations, specify a value in the range from 0 to 9999. If you specify use in the fence_scsi operand in the function definition statement, specify a value in the range from 1 to 9999.

If the multistandby operand is specified in the function definition statement, or if host is specified in the reset_type operand, the host with the lowest host address has the highest reset priority. When deciding on a host address, consider the following points:

If you want to prioritize resetting the active server over the standby server when a host failure is detected simultaneously on both the active system and the standby system, give the standby system a higher host address than the active system. For standby systems in a multi-standby configuration, give lower host addresses to systems with smaller values for the standbypri operand in the server environment definition.

If you want to prioritize hot standby processing over resetting the active server when a host failure is detected simultaneously on both the active system and the standby system, give the standby system a lower host address than the active system. For standby systems in a multi-standby configuration, give lower host addresses to hosts with larger values for the standbypri operand in the server environment definition. This setting is recommended when you need to obtain an accurate dump as soon as possible in order to investigate a problem such as a slowdown of the active system.

In a configuration without a clear separation between active and standby systems, such as a configuration in which all the hosts are active systems, decide on a host address by taking into account the reset priority of the host based on factors such as the importance of the job and the host's amount of resources.

(c) patrol

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

3 to 600

(Units: seconds)

No

--

Description

Specifies the interval for monitoring for receipt of alive messages from remote hosts. This monitoring is used to detect host failures.

You must specify the same value for all hosts in the hot-standby configuration. If no alive message is sent from a remote host within the time specified in the patrol operand, HA Monitor determines that a host failure has occurred. To specify a time shorter than five seconds, specify the alive_interval operand.

(d) lan

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 32 characters

--

(Units: --)

No

--

Description

Specifies the host names of the TCP/IP LANs that are to be used as monitoring paths.

Specify a unique name for each host.

A host name specified in the lan operand must be a host name specified in the /etc/hosts file. For details about the /etc/hosts file, see 6.9.1 Adding host names and service names.

Delimit the TCP/IP LAN host names with the colon (:). You can specify a maximum of six host names. If you use multiple monitoring paths and have a preferred monitoring path, specify a hash mark (#) in front of that path's host name. You can specify only one preferred path. The following shows a specification example:

lan  #path11:path12,

This example uses two monitoring paths, path11 and path12, and path11 is to be used as the preferred path. If the preferred monitoring path fails and transmission of alive messages stops for 70% of the host failure monitoring time, HA Monitor attempts to communicate via all monitoring paths and switches to a normal monitoring path over which a reply is received.

(e) lanport

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 32 alphanumeric characters

--

(Units: --)

No

--

Description

Specifies the service names of the TCP/IP LANs that are to be used as monitoring paths.

Specify the same values for all hosts in the hot-standby configuration.

A service name specified in the lanport operand must be a service name specified in the /etc/services file. For details about the /etc/services file, see 6.9.1 Adding host names and service names.

The service names in the lanport operand must be specified so that they correspond in order and number to the host names specified in the lan operand. You must specify as many service names as there are host names in the lan operand. The following shows a specification example:

      lan      path11 :path12,
                 ↓
      lanport  HAmon1 :HAmon2,

In this example, path11 in the lan operand corresponds to HAmon1 in the lanport operand, and path12 in the lan operand corresponds to HAmon2 in the lanport operand.

(f) fs_log_size

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

0 to 2147483647

(Units: bytes)

Yes

65536

Description

Specifies the maximum size (in bytes) for a log file that is created when file systems are switched.

Specify the same value for all hosts in the hot-standby configuration. When a log file reaches the specified maximum size, it is automatically backed up at the next log acquisition point and then the file is cleared.

(g) servmax

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

16|64|128|129 to 512

--

(Units: --)

Yes

16

Description

Specifies the maximum number of servers that can be run concurrently on a single host.

Specify the same value for all hosts in the hot-standby configuration. You can specify only one of the following numeric values:

16: Sets 16 as the maximum number of servers that can be run concurrently.

64: Sets 64 as the maximum number of servers that can be run concurrently.

128: Sets 128 as the maximum number of servers that can be run concurrently.

129 to 512: Sets a value in the range from 129 to 512 as the maximum number of servers that can be run concurrently. This is applicable only when HA Monitor Extension is used.

If you set the maximum number of servers that can be run concurrently to a value other than 16 (default), you must also change a kernel parameter setting. For details about the kernel parameter setting, see 6.3.6 Specifying the kernel parameters.

(h) hostmax

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

32 to 256

(Units: --)

Yes

32

Description

Specifies the maximum number of hosts, including the local host, that can be connected.

You must specify the same value for all hosts in the hot-standby configuration.

This operand can be specified only when HA Monitor Extension is used.

For details about HA Monitor Extension, see 3.7 Functions of HA Monitor Extension.

If you have changed the maximum number of hosts, you must also change kernel parameter settings. For details about the kernel parameter settings, see 6.3.6 Specifying the kernel parameters.

(i) pgmmax

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

0 to 256

(Units: --)

Yes

0

Description

When the monitor-mode program management function is used, this operand specifies the maximum number of processes that can issue the hamon_patrolstart function per host.

If you will be using the program management function, ensure that this operand value has been changed from its default value. You must specify the same value for all hosts in the hot-standby configuration.

(2) function definition statement

The function definition statement defines operation options for HA Monitor. Specification of this definition statement is optional. If you omit the function definition statement, all the default values are assumed. The following explains the operands of the function definition statement.

(a) cpudown

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

online|standby|system

--

(Units: --)

Yes

online

Description

Specifies whether the active system host or the standby system host is to have reset priority.

Specify the same value for all hosts in the hot-standby configuration. When you specify the host that is to have reset priority and a host failure is detected on both the active system and the standby system at the same time, the host for which reset priority has been specified is the one that resets remote hosts.

online: The active system is the host that has reset priority.

standby: The standby system is the host that has reset priority.

system: Prevents concurrent host resets without having to specify a reset priority. You can specify system only if you employ a mutual hot-standby configuration, BladeSymphony is being used, and the SVPs support the cluster management function.

Note also that system cannot be specified if you use any of the following operand specifications:

reset_type operand for which host is specified

multistandby operand for which use is specified

standbyreset operand for which use is specified

vmware_env operand for which use is specified

(b) clearwait

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 600

(Units: seconds)

Yes

20

Description

If all monitoring paths fail, all hosts might be reset at the same time. To prevent such a problem, this operand is used to specify the time (in seconds) during which the hosts that do not have reset priority wait until a host reset request is issued.

The following shows the points that you must note when specifying this operand. If all monitoring paths fail when an incorrect setting has been specified, all hosts might be reset at the same time, causing job processing to terminate.

Specify the same value that is set for the clearcheck operand. For details, see the description of the clearcheck operand.

Specify the same value for all hosts in the hot-standby configuration. Even if the models of servers in the hot-standby switchover configuration include ones that do not require setting of this operand, make sure that the operand is specified with the same value for all servers.

(c) standbyreset

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether the standby system is reset if the active system detects a host failure in the standby system. Specify the same value for all hosts in the hot-standby configuration. Do not specify use in a configuration other than the 1-to-1 switchover configuration.

use: Resets the standby system.

nouse: Does not reset the standby system.

(d) pathpatrol

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 240

(Units: minutes)

Yes

--#

#: If you omit this operand, HA Monitor does not perform health checks on the monitoring paths.

Description

Specifies the interval at which monitoring path health checks are performed.

Specify the same value for all hosts in the hot-standby configuration.

When you specify this operand, HA Monitor performs health checks of monitoring paths, which facilitates early detection of failures that might occur on any of the multiple monitoring paths. If you omit this operand, HA Monitor does not perform health checks on the monitoring paths. We recommend that you specify this operand.

(e) pathpatrol_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

re-check-interval: Unsigned integer

3 to 600

(Units: seconds)

Yes

30

re-check-count: Unsigned integer

0 to 20

(Units: number of times)

Yes

5

Description

Specifies rechecking of a monitoring path's status after a monitoring path failure. You specify here the interval between the status checks, a colon (:), and the maximum number of times rechecking is performed.

Specify the same values for all hosts in the hot-standby configuration. If you omit the pathpatrol operand, HA Monitor does not recheck a failed monitoring path.

re-check-interval

Specifies the interval at which a failed monitoring path is re-checked.

re-check-count

Specifies the maximum number of times a failed monitoring path is re-checked. If you specify 0, HA Monitor does not recheck monitoring paths.

Specify the recheck interval value and the recheck count value so that their product does not exceed the monitoring path health check interval specified in the pathpatrol operand. Use the following formula to check if the specified values satisfy this condition:

monitoring path health check interval × 60 ≥ recheck interval × (re-check count + 1) + a

Legend:a: 30 to 60 seconds

For example, if the value of the pathpatrol operand is 1 minute and the re-check interval is 5 seconds, the formula is satisfied if the re-check count is 5 or smaller.

(f) message_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

3 to 600

(Units: seconds)

Yes

3

Description

Specifies the interval at which message transmission is retried when transmission of a query response message that uses a monitoring path fails.

Specify the same value for all hosts in the hot-standby configuration.

If you omit the message_retry operand, HA Monitor retries the message transmission every three seconds.

HA Monitor continues resending the message until the transmission is successful. If the hosts are not monitoring each other (alive message transmission is not being monitored) and transmission of a query response message does not succeed after 60 seconds, HA Monitor determines that a host failure has occurred. If a value greater than 60 seconds is specified in the message_retry operand, HA Monitor waits for message reception until the specified amount of time elapses. If a message has still not been received, HA Monitor then determines that a host failure has occurred.

For details about retrying transmission of query response messages, see (4) Retrying transmission of query response messages.

(g) connect_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

connection-interval: Unsigned integer

5 to 60

(Units: seconds)

Yes

5

connection-count: Unsigned integer

1 to 9999

(Units: number of times)

Yes

200

Description

Specifies how connection processing is performed between HA Monitors. You specify here the interval between attempts to establish connection, a colon (:), and the maximum number of times connection establishment is attempted.

Specify the same values for all hosts in the hot-standby configuration.

It is important that you specify this operand, because if many HA Monitors are connected, connection establishment between HA Monitors might fail. This operand becomes effective when an HA Monitor connection configuration file has been created.

connection-interval

Specifies the interval between attempts to establish connection between HA Monitors. A short interval might adversely affect HA Monitor performance, while a long interval might result in delays in establishing connections between HA Monitors. As a guideline, if 10 or more HA Monitors are connected, specify at least 10 seconds.

connection-count

Specifies the number of times connection processing between HA Monitors is retried. If 9999 is specified, connection processing will be retried until all HA Monitors are connected.

If this value is too small, connection between HA Monitor might fail. A large value might adversely affect HA Monitor performance. As a guideline, specify the default values and consider increasing the values only if connection between HA Monitors fails.

If connection has not been established after the specified number of connection attempts, the KAMN176-E message is issued.

(h) monbegin_restart

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

use

Description

This operand specifies whether to start a monitor-mode server by executing the monitor-mode server start command (monbegin command) automatically in the following cases:

After hot-standby switchover occurs due to a host failure, HA Monitor for the host on which the failure occurred is restarted.

HA Monitor for the host on which the standby server is running is restarted.

Specify the same value for all hosts in the hot-standby configuration.

use: Automatically starts the servers in the monitor mode. If you do not use an operations management software program (such as JP1) to start servers automatically, we recommend that you specify use.

nouse: Does not start the servers in the monitor mode automatically. If you use an operations management software program (such as JP1) to start servers automatically, we recommend that you specify nouse.

(i) netmask

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

byte|bit

--

(Units: --)

Yes

byte

Description

Specifies the netmask that was specified for the LAN interfaces used for the monitoring paths.

Specify the same value for all hosts in the hot-standby configuration. If a LAN is configured as multiple subnets with the same network number, specify the same netmask. For details about the netmask settings, see 5.4.2 Necessary IP addresses and port numbers.

If the netmask operand and netmask settings are invalid, HA Monitor will malfunction, disabling communication with the HA Monitors on the remote hosts.

byte: Specify this value only if the netmask consists only of 255 and 0 (in decimal). The following shows an example:

255.255.0.0

bit: Specify this value if the netmask does not consist only of 255 and 0 (in decimal). The following shows an example:

255.255.255.192

When bit is specified in the netmask operand, the following limitations apply to the netmask settings:

The network address part (network number and subnet number) that is recognized by the combination of IP address and netmask must include the entire network number part defined for each class. For example, 255.254.0.0 cannot be specified as the netmask for a network in class B.

You can use the ifconfig OS command to check the netmask value. For details about the netmask and the ifconfig command, see the OS documentation.

(j) usrcommand

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 1000 characters

--

(Units: --)

Yes

--

Description

Specifies the absolute path name of the user commands that are issued automatically by HA Monitor.

Specify the same value for all hosts in the hot-standby configuration.

For details about how to create user commands and examples of user commands, see 6.18 Creating user commands.

(k) resetpatrol

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 60

(Units: minutes)

Yes

2

Description

Specifies the interval at which reset path health checking is performed.

Specify the same value for all hosts in the hot-standby configuration.

An appropriate reset path health check interval depends on the number of HA Monitors to be connected. Specify a value that satisfies the following condition:

Reset path health check interval × 60 > number of hosts × 10

(l) resetpath_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

use

Description

This operand specifies whether to re-check the reset path status so that the reset path can be restored after an error is detected by a reset path health check.

Set the same value for all hosts in the hot-standby configuration. If a temporary reset path failure occurs (for example, due to restart of the failure management processor) when this function is used, the status where host reset is possible is automatically restored after recovery from the failure. When the reset path is restored, the KAMN646-I message is output.

use: HA Monitor automatically restores the status of the reset path.

nouse: HA Monitor does not automatically restore the status of the reset path.

(m) resetpath_inter

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

10 to 1440

(Units: minutes)

Yes

15

Description

This operand specifies the interval at which the reset path status is to be re-checked so that the reset path can be restored after an error is detected by a reset path health check. The interval can be specified in minutes in the range from 10 to 1,440.

This operand takes effect only if the resetpath_retry operand is omitted or if use is specified for the resetpath_retry operand. If nouse is specified for the resetpath_retry operand, the specification of this operand is ignored.

(n) multistandby

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether the multi-standby function that enables multiple standby servers to be defined for a single active server is to be used.

Specify the same value for all hosts in the hot-standby configuration. For details about the multi-standby function, see 4.5 Managing servers and hosts when using the multi-standby function.

use: Uses the multi-standby function.

nouse: Does not use the multi-standby function.

(o) deviceoff_order

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

order|reverse

--

(Units: --)

Yes

order

Description

Specifies the order in which shared resources are disconnected.

Specify the same value for all hosts in the hot-standby configuration. This operand applies to all servers defined in the HA Monitor.

You can also specify a shared resource separation order for each server. For details about how to specify a shared resource separation order for each server, see 8.4.1 Server environment definition (servers). For details about the order in which shared resources are disconnected, see 4.8.7 Processing flow for disconnecting shared resources in the reverse order from which they were connected.

order: Disconnects shared resources in the same order in which they were connected.

reverse: Disconnects shared resources in the reverse order from which they were connected.

(p) reset_type

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

server|host

--

(Units: --)

Yes

server

Description

Specifies how the reset issuing host is determined.

Specify the same value for all hosts in the hot-standby configuration. If use is specified for the multistandby operand in the HA Monitor environment settings, host is assumed regardless of this operand's setting.

server: Determines the reset issuing host in advance according to a process that determines both the reset issuing host and the host that has reset priority.

host: Determines the reset issuing host at the time the host reset is performed, based on the host that has the highest reset priority. The smaller the host address specified in the address operand of the HA Monitor environment settings, the higher the reset priority of the host.

(q) partition_reset

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether the physical partition reset function is to be used.

This operand is applicable in virtualization environments in which Hitachi server virtualization (Virtage) or VMware ESXi is used. In a hot-standby configuration that includes virtualization and non-virtualization environments, this operand does not have to be specified for the hosts in non-virtualization environments. Specify this operand only in a virtualization environment.

use: Uses the physical partition reset function. Specify use if each processor is to be reset during hot standby processing in the event of a reset error on the host in a virtualization environment.

nouse: Does not use the physical partition reset function. Specify nouse if the server is to be placed in the hot-standby wait state in the event of a reset error on the host in the virtualization environment.

(r) jp1_event

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether the JP1 event notification function is to be used.

You must specify the same value for all hosts in the hot-standby configuration. To use the JP1 event notification function, you need JP1/Base for the physical host environment in your system.

When use is specified in the jp1_event operand, HA Monitor issues JP1 events. When the jp1_event operand is omitted, HA Monitor does not issue JP1 events.

use: Uses the JP1 event notification function.

nouse: Does not use the JP1 event notification function.

(s) ph_log_size

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1024 to 10485760

(Units: bytes)

Yes

--

Description

Specifies the size of a monitoring history file that is used to record monitoring history information about slowdowns of servers and hosts.

You must specify the same value for all hosts in the hot-standby configuration.

When this operand is specified, HA Monitor creates a monitoring history file of the specified size when HA Monitor starts, and then collects monitoring history for servers and hosts. When the monitoring history file becomes full, its contents are stored in a backup file and the monitoring history file is wrapped around for reuse. The following shows the formula for estimating the size of a monitoring history file:

Number of times the servers and hosts for which monitoring history is collected are started × 180 + number of slowdowns to be recorded in the monitoring history file × 90

(t) ph_threshold

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

5 to 559

(Units: seconds)

Yes

--#

#: If this operand is omitted, the host monitoring history is not collected.

Description

When information about slowdowns of local and remote hosts is to be collected as monitoring history, this operand specifies the amount of time (time when host monitoring history is to be obtained) it takes from the time transmission of alive messages stops to the time collection of monitoring history begins.

You must specify the same value for all hosts in the hot-standby configuration. The value for the time when host monitoring history is to be obtained must be less than the value for the host failure monitoring time specified in the patrol operand in the HA Monitor environment settings.

When you specify this operand to collect a host monitoring history, you must also specify the ph_log_size operand.

(u) termcmd_at_abort

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether postprocessing by the server termination command is to be enabled if server startup processing is cancelled due to an error in the server start command and hot standby or grouped-system switchover processing is performed in the event of a server failure.

Specify the same value for all hosts in the hot-standby configuration.

use: Enables postprocessing by the server termination command.

If server startup processing is canceled due to an error in the server start command or if hot-standby switchover or grouped-system switchover is performed, the server termination command is executed with the -c argument.

You must create the server termination command in such a manner that the command performs postprocessing when it is executed with the -c argument.

nouse: Does not enable postprocessing by the server termination command.

If server startup processing is cancelled due to an error in the server start command, the server termination command is not executed.

If hot standby or grouped-system switchover processing is to be performed in the event of a server failure, the server termination command is executed with the -w or -e argument in the same manner as with normal termination of a server.

For details about the server termination command, see 6.12.2 Creating a server termination command.

(v) alive_interval

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 10

(Units: 100 milliseconds)

Yes

10

Description

Specifies a new (changed) default value for the alive message transmission interval.

The normal default alive message transmission interval is 1,000 milliseconds. If you set the host failure monitoring time to a small value, we recommend that you change the alive message transmission interval. Specify the same value for all hosts in the hot-standby configuration.

If this operand is omitted, the normal default alive message transmission interval is assumed. If you use the normal default alive message transmission interval, we recommend that you omit this operand (there is no need to specify 10 in this operand).

When you specify this operand, a preferred monitoring path cannot be specified because alive messages are sent to all monitoring paths.

If this operand is not specified when the patrol value is small, HA Monitor is likely to detect a disruption of alive messages, which might lead to false detection of a failure.

(w) alive_multicast

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse#

#: The system assumes that use is specified only if 33 or a larger value is specified for the hostmax operand.

Description

Specifies whether alive messages are to be transmitted by multicasting.

Specify the same value for all hosts in the hot-standby configuration.

use: Multicasts alive messages.

nouse: Does not multicast alive messages (transmits by unicasting).

When you specify use in this operand, alive messages are transmitted to all monitoring paths that use the same communication method.

When alive messages are transmitted by multicasting, 239.0.0.1 is used as the multicast group ID. If necessary, you can use the multicast_lan operand to specify a different multicast group ID.

(x) multicast_lan

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 32 alphanumeric characters

--

(Units: --)

Yes

--#

#: If you omit this operand, 239.0.0.1 is assumed as the multicast group ID in the /etc/hosts file.

Description

Specifies the host name corresponding to the multicasting group ID that is to be used when alive messages are transmitted by multicasting.

Specify the same value for all hosts in the hot-standby configuration.

This host name must be a host name specified in the /etc/hosts file. For the IP address to be specified in the /etc/hosts file, specify a multicast address of class D.

Specification of this operand takes effect only if alive messages are to be transmitted by multicasting. In other cases, this operand is ignored even if it is specified.

(y) lanfailswitch

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether LANs are to be monitored.

Make sure that the same value is specified on all hosts for which LANs are to be monitored. (The same value does not need to be specified on all hosts in the hot-standby switchover configuration.)

use: Monitors LANs.

nouse: Does not monitor LANs.

When you specify this operand, you must also specify in the switchbyfail operand in the server environment definition the name of the LAN interface to be monitored. If the server specified in the switchbyfail operand is grouped and a failure occurs on the specified LAN interface, hot standby processing will be performed on all servers in the group.

(z) lancheck_patrol

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 180

(Units: seconds)

Yes

15

Description

Specifies the monitoring interval when LANs are to be monitored.

Make sure that the same value is specified on all hosts for which LANs are to be monitored. (The same value does not need to be specified on all hosts in the hot-standby switchover configuration.)

If you omit the lanfailswitch operand, this operand is ignored, if specified.

Make sure that this operand's value is greater than the execution time for the LAN monitoring script. For details about the LAN monitoring script, see (2) Specifying a LAN monitoring script.

Note that if only the hbonding status is to be monitored, there is no need to specify this operand because the monitoring interval is fixed at one second.

(aa) lancheck_mode

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

packet|route

--

(Units: --)

Yes

packet

Description

Specifies the LAN monitoring method that HA Monitor uses when monitoring LANs.

Make sure that the same value is specified on all hosts for which LANs are to be monitored. (The same value does not need to be specified on all hosts in the hot-standby switchover configuration.)

This operand is ignored if the lanfailswitch operand is specified with the value nouse or is omitted. If you use a LAN monitoring script (lanpatrol.sh) that was created for HA Monitor whose version is earlier than 01-68, make sure that this operand is omitted or specified with the value packet.

packet: HA Monitor monitors the number of packets received by the monitoring-target LAN adapter.

route: HA Monitor monitors ping responses that arrive at the address specified in the LAN monitor definition file.

(ab) disk_ptrl

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether to monitor the system disk.

Specify the same values for all hosts in the hot-standby configuration.

use: HA Monitor monitors the system disk.

nouse: HA Monitor does not monitor the system disk.

If you specify use for this operand, you must set up the monitoring definition file for the system disk. For details about how to specify the settings in the monitoring definition file for the system disk, see (1) Settings in the files required for monitoring the system disk in 6.19.2 Settings in the files required for disk monitoring.

If necessary, also specify the following operands in the HA Monitor environment settings:

disk_ptrl_inter

disk_ptrl_retry

disk_log_size

(ac) disk_ptrl_inter

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

3 to 3600

(Units: seconds)

Yes

120

Description

Specifies the interval at which the status of the system disk is to be checked.

Specify the same values for all hosts in the hot-standby configuration.

Determine the value to be specified for this operand taking into account the following items specified for the disk_ptrl_retry operand in the HA Monitor environment settings:

Check retry interval for the system disk

Check retry count for the system disk

Specify a value that meets the following condition:

system-disk-check-interval ≥ system-disk-check-retry-interval × (system-disk-check-retry-count + 1)

(ad) disk_ptrl_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

system-disk-check-interval: Unsigned integer

1 to 3600

(Units: seconds)

Yes

60

system-disk-check-retry-count: Unsigned integer

0 to 60

(Units: number of times)

Yes

1

Description

Use this operand to specify the interval at which HA Monitor retries checking the system disk and the number of times HA Monitor can retry it if access to the system disk fails. Specify the retry interval and count with an intervening colon (:). If the number of times access to the system disk fails exceeds the specified count, HA Monitor judges that the system disk has failed.

Specify the same values for all hosts in the hot-standby configuration.

system-disk-check-retry-interval

Specify the interval at which HA Monitor retries checking the system disk. Make sure that the value you specify is equal to or larger than the disk I/O timeout value. The disk I/O timeout value means the time for which an application that requests an I/O operation can wait before receiving a response. This value is determined from the driver settings, OS kernel parameters, and multipath software settings.

system-disk-check-retry-count

Specify the number of times HA Monitor can retry checking the system disk. If you specify 0, HA Monitor does not retry checking the system disk.

(ae) disk_log_size

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

65535 to 10485760

(Units: bytes)

Yes

1048576

Description

Specifies in bytes the maximum size of log files that are used for disk monitoring. This operand is applied to both of the following types of log file monitoring:

Monitoring of the system disk

Monitoring of disks for business use

Specify the same values for all hosts in the hot-standby configuration.

Normally, you do not need to change the value of this operand.

(af) patrol_100ms

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies a new (changed) default value for the alive message check interval.

The normal default alive message check interval is one second. When you specify this operand, HA Monitor will check for alive messages every 100 milliseconds. If you set the host failure monitoring time to a small value, we recommend that you check for alive messages every 100 milliseconds. Specify the same value for all hosts in the hot-standby configuration.

use: Checks alive messages every 100 milliseconds.

nouse: Checks alive messages every second.

(ag) suppress_reset

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 32

(Units: --)

Yes

--#

#: If this operand is omitted, host reset is not suppressed.

Description

If hosts are to be reset, this operand specifies the minimum number of active hosts as a reference for suppressing host resets.

As the minimum number of active hosts, specify a value that is greater than half the total number of hosts that constitute the system (minimum number of active hosts > total number of hosts in the entire system ÷ 2) and equal to or less than the value of the hostmax operand. You must specify the same value for all hosts in the hot-standby configuration.

For details about suppressing host resets, see 3.3.4 Suppressing host resets.

(ah) exitcode

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

type1|type2

--

(Units: --)

Yes

type1

Description

Specifies the version of the codes to be returned by HA Monitor's commands:

type1: Returns the codes described in 9. Commands.

type2: Returns the codes for P-9S2C-E111 HA Monitor version 01-41 or earlier.

(ai) vg_off

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

sequential|parallel

--

(Units: --)

Yes

parallel

Description

Specifies whether disconnection processing is to be performed for volume groups sequentially or in parallel, when volume group disconnection processing is to be performed more than once (such as when multiple servers are terminated).

Specify the same value for all hosts in the hot-standby configuration.

Note that if the OS is RHEL 6.2 or later or RHEL 7 or later, you do not need to specify this operand.

sequential: Performs volume group disconnection processing separately for each volume group (performs the processing sequentially).

If there are multiple server and resource definition statements with the disk operand specified in the server environment definition, we recommend that you specify sequential in order to avoid disconnection errors that might occur when multiple volume groups are disconnected at the same time (in parallel). Note that if the OS is RHEL 6.2 or later or RHEL 7 or later, a disconnection error does not occur when multiple volume groups are disconnected simultaneously (in parallel).

parallel: Performs volume group disconnection processing in parallel.

(aj) vmware_env

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether hot standby processing is to be performed in a VMware ESXi-based virtualization environment.

Specify the same value for all hosts in the hot-standby configuration.

use: Performs hot standby processing in a VMware ESXi-based virtualization environment.

nouse: Does not perform hot standby processing in a VMware ESXi-based virtualization environment.

If you have specified that a VMware ESXi virtual machine is to be used in the reset path setting by using the monsetup -resetpath command, use is assumed.

If SCSI reservation for shared disk is performed, you do not need to specify this operand even in a VMware ESXi-based virtualization environment.

For examples of the values to be specified for the reset path settings, see 6.6.2 Examples of settings (BladeSymphony), 6.7.2 Examples of settings (model HA8000xM or earlier), or 6.8.2 Examples of settings (model HA8000xN or later).

(ak) patrol_type

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

server|host

--

(Units: --)

Yes

server

Description

Specifies when hosts are to be monitored.

You must specify the same value for all hosts in the hot-standby configuration.

server: Monitors hosts while hot standby processing can be performed.

host: Monitors hosts while HA Monitor's hosts are connected.

Specify host when either of the following conditions is satisfied:

The suppress_reset operand is specified.

The servers are not monitoring each other on all hosts in the hot-standby switchover configuration.

If neither of the above conditions is satisfied, specify server or omit this operand.

(al) resetpatrol_mode

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

mode1|mode2

--

(Units: --)

Yes

mode1

Description

When the system being used is BladeSymphony, specifies whether a reset path failure is to be detected when an SVP cannot respond temporarily to a reset path health check.

In BladeSymphony, specify mode2. In HA8000, either omit this operand or specify mode1. If this operand is omitted, mode1 is assumed.

This operand is applicable only to reset path health checking.

mode1: Detects as a reset path failure the status in which an SVP cannot respond temporarily to a reset path health check.#

mode2: Does not detect as a reset path failure the status in which an SVP cannot respond temporarily to a reset path health check.#

#

A BladeSymphony SVP might not be able to respond to a reset path health check temporarily (for up to two minutes) when SVPs are switched (for example, for maintenance purposes).

(am) fence_reset

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

use

Description

Specifies whether host reset is to be used as the method for protecting data on a shared disk.

Specify the same value for all hosts in the hot-standby configuration.

use: Resets hosts.

nouse: Does not reset hosts.

The value to be specified for this operand differs depending on the function to be used.

If hybrid fencing is used

Omit this operand or specify use. In addition, specify use for the fence_scsi operand.

If hybrid fencing is not used

Specify use or nouse for this operand. If this operand is omitted, the system assumes that use is specified for this operand. If you omit this operand or specify use for this operand, you must specify nouse for the fence_scsi and fence_lan operands. Conversely, if you specify nouse for this operand, you must specify use for the fence_scsi and fence_lan operands.

If nouse is specified in this operand, the reset path settings specified by the HA Monitor environment setup command (monsetup command) are ignored, if specified.

For details about host reset, see 2.3.5 Host reset. For examples of the values to be specified for the reset path settings, see 6.6.2 Examples of settings (BladeSymphony), 6.7.2 Examples of settings (model HA8000xM or earlier), or 6.8.2 Examples of settings (model HA8000xN or later).

(an) fence_scsi

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether SCSI reservation for shared disk is to be used as the method for protecting data on a shared disk.

Specify the same value for all hosts in the hot-standby configuration.

use: Uses SCSI reservation for shared disk.

nouse: Does not use SCSI reservation for shared disk.

The value to be specified for this operand differs depending on the function to be used.

If hybrid fencing is used

Specify use for this operand. In addition, specify use for the fence_reset operand or omit the operand.

If hybrid fencing is not used

Specify use or nouse for this operand. If this operand is omitted, the system assumes that nouse is specified for this operand. When you specify use in this operand, the following settings are also required:

• Specifying nouse in the fence_reset and fence_lan operands

• Specifying the scsi_device or dmmp_device operand in the server environment definition on the servers that use a shared disk#

#: If you do not specify the scsi_device or dmmp_device operand, active servers might start on multiple hosts, causing the data on the shared disk to be corrupted.

For details about SCSI reservation for shared disk, see 2.3.6 SCSI reservation for shared disk.

(ao) fence_lan

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether the function for controlling hot standby based on the availability of LAN communications is to be used.

You must specify the same value for all hosts in the hot-standby configuration.

use: Uses the function for controlling hot standby based on the availability of LAN communications.

nouse: Does not use the function for controlling hot standby based on the availability of LAN communications.

If you specify use in this operand, specify nouse in the fence_reset and fence_scsi operands. Also specify use in the lanfailswitch operand and the name of a LAN interface whose availability is to be checked in the switch_judge operand in the server environment definition.

For details about the function for controlling hot standby based on the availability of LAN communications, see 2.3.7 Function for controlling hot standby based on the availability of LAN communications.

(ap) scsi_check

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 600

(Units: seconds)

Yes

5

Description

Specifies the interval at which the shared disk reservation status is to be monitored from the active system when SCSI reservation for shared disk is used.

Specify the same value for all hosts in the hot-standby configuration.

If this operand is omitted, five seconds is assumed. Normally, there is no need to change this operand's default value.

When nouse is specified in the fence_scsi operand or the fence_scsi operand is omitted, this operand is ignored.

(aq) scsi_pathcheck

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 60

(Units: minutes)

Yes

1

Description

Specifies the interval at which a shared disk path status is to be monitored from the standby system when SCSI reservation for shared disk is used.

Specify the same value for all hosts in the hot-standby configuration.

When nouse is specified in the fence_scsi operand or the fence_scsi operand is omitted, this operand is ignored.

(ar) scsi_timeout

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 3600

(Units: seconds)

Yes

30

Description

Specifies the response wait time for the SCSI command that is issued to a physical device when SCSI reservation for shared disk is used.

Specify the same value for all hosts in the hot-standby configuration.

I/O timeout values, such as the timeout value between RAID controller and Linux SCSI subsystem or the timeout value for a Linux block device, might be specified in OS settings. For these cases, specify a value that is larger than the I/O timeout value specified here in the scsi_timeout operand.

When nouse is specified in the fence_scsi operand or the fence_scsi operand is omitted, this operand is ignored.

(as) scsi_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 60

(Units: number of times)

Yes

2

Description

Specifies the number of times SCSI command issuance is to be retried when SCSI reservation for shared disk is used and the SCSI command issued to a physical device fails or results in a timeout.

Specify the same value for all hosts in the hot-standby configuration.

If this operand is omitted, 2 (twice) is assumed. In this case, after an attempt is made to execute the SCSI command, retry is performed twice. Therefore, a maximum of three attempts are made to execute the SCSI command.

When nouse is specified in the fence_scsi operand or the fence_scsi operand is omitted, this operand is ignored.

(at) hbond_lacp

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 32 alphanumeric characters

--

(Units: --)

Yes

--#

#: If this operand is omitted, HA Monitor monitors only hbonding physical link status.

Description

Specifies the name of an interface for hbonding when hbonding with the LACP monitoring method is to be monitored in 802.3ad mode by using HA Monitor's LAN monitoring function.

You can specify a maximum of 32 interface names delimited by the colon (:). Specifying this operand enables you to detect failures in network switches and physical links based on the hbonding status. When this operand is omitted, HA Monitor monitors only hbonding physical link status.

For details about the settings required to monitor hbonding, see (4) Required environment settings.

(au) vg_offskip

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether volume group disconnection processing is to be skipped when resource disconnection processing that occurs before volume group disconnection processing fails on the active system that is the source system for hot standby processing.

use: Skips volume group disconnection processing.

nouse: Does not skip volume group disconnection processing.

When use is specified in this operand and shared resource disconnection processing fails on the active system that is the source system for hot standby processing, the time required for hot standby processing can be reduced because volume group disconnection processing will be skipped. Volume group disconnection processing is skipped for the resources listed below when shared resource disconnection processing fails.

When the deviceoff_order operand value is order

File system disconnection

Volume group disconnection

When the deviceoff_order operand value is reverse

File system disconnection

Volume group disconnection

If host reset is enabled for the active system, the active system is reset and, if SCSI reservation for shared disk is used, kernel panic is generated; therefore, volume groups are disconnected.

(av) notswitch_notify

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

use

Description

Specifies whether a warning message is to be issued every hour while hot standby processing is disabled.

use: Issues a warning message repeatedly.

nouse: Does not issue a warning message.

If you specify use in this operand, you can obtain early warning of hot standby-disabled status.

The following table shows the types of hot standby-disabled status that result in issuance of a warning message and the warning message IDs.

Table 8‒3: Hot standby-disabled statuses resulting in issuance of warning messages and the warning message IDs

Hot standby-disabled status resulting in issuance of a warning message

Warning message ID

The active server has started but it is not ready for hot standby processing.

KAMN288-W

The number of active hosts is equal to or less than the minimum number of active hosts.#

KAMN447-W

The server monitoring is temporarily stopped.

KAMN521-W

The system disk monitoring is temporarily stopped.

KAMN540-W

An error was detected during reset path health check.

KAMN644-W

While SCSI reservation is being used, a disk path error was detected when the shared disk path was checked from the standby system.

KAMN729-W

#

This includes HA Monitors whose servers are not active.

(aw) servcomplete_msg

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether the KAMN298-I and the KAMN496-I messages are to be issued. The KAMN298-I message is issued when server termination processing is completed, and the KAMN496-I message is issued when grouped-system switchover processing is completed.

use: Issues a message when server termination processing is completed and when grouped-system switchover processing is completed.

nouse: Does not issue a message when server termination processing is completed or when grouped-system switchover processing is completed.

If grouped-system switchover processing fails on even one of the target servers, the KAMN496-I message is not issued. For a server for which the switch_error operand is specified in the server environment definition to perform start retries, the message is not issued because the server is treated as having been terminated abnormally.

(3) options definition statement

This definition statement defines the options related to HA Monitor operation. Specification of this definition statement is optional. If you omit this definition statement, all the default values are assumed. The following explains the operands of the options definition statement.

(a) clearcheck

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 600

(Units: seconds)

Yes

20

Description

This operand specifies the host reset end time (in seconds).

If host reset does not finish within a specified time when hot-standby switchover occurs, the system outputs an error message and enters the hot-standby switchover wait state. After that, when host reset finishes, the system restarts hot-standby switchover.

For details about specification of this operand, see the following table:

Model

Environment

Server configuration

Specification details

HA8000xN or later

VMware ESXi not used

Management server not used

Set a value for this operand according to the documentation for HA Monitor Connector. If the documentation for HA Monitor Connector does not include descriptions of this operand, do not specify this operand.

Management server used

Do not specify this operand.

VMware ESXi used

Management server not used

Specify a value that is larger by 20 (seconds) than the value indicated in the documentation for HA Monitor Connector. If the documentation for HA Monitor Connector does not include descriptions of this operand, do not specify this operand.

Management server used

Do not specify this operand.

Other models

--

--

Do not specify this operand.

Legend:

--: Not applicable

The following shows the points that you must note when specifying this operand.

Specify the same value for all hosts in the hot-standby configuration. Even if the models of servers in the hot-standby switchover configuration include ones that do not require setting of this operand, make sure that the operand is specified with the same value for all servers.

Specify the same value that is set for the clearwait operand.

8.4 Configuring a server environment

In the configuration of a server environment, you define an environment for the active and standby servers that run on a host.

You can also specify environment settings for locked servers. When you have defined servers as locked servers and a standby server has been started as the active server, you can then terminate the other standby servers on the same host. The servers that are to be terminated are called the locked servers. The reason for doing this is to prevent more than one active server from being run on the same host.

8.4.1 Server environment definition (servers)

You create a definition file containing the server environment definitions under the file name servers in the directory for HA Monitor environment settings.

A sample servers file is provided under the directory for HA Monitor sample files. If you copy this file to the directory for HA Monitor environment settings and then edit it, you can save time over creating the file from scratch.

You must create this definition file for each host. Bold type indicates a value that must be the same on all hosts. Care must be taken in specifying the other items so that there will be no conflicts between the hosts.

The following shows the format of the definition file used to specify server environment definitions:

/*  Server environment definition  */
server  name
             program-name
       ,alias
             server's-alias-name
     [,initial
           {online|standby}]
     [,standbypri
           {priority-of-each-standby-server|1}]
     [,group
             group-name[:{exchange|no_exchange[:{cancel|no_cancel}]}]]
     [,parent
             parent-server's-alias-name]
     [,acttype
           {server|monitor}]

     [,program
           {use|nouse}]
     [,patrol
             server-failure-monitoring-time]
     [,ph_threshold
             time-when-server-monitoring-history-was-obtained]
     [,actcommand
             server-start-command's-absolute-path-name]
     [,termcommand
             server-termination-command's-absolute-path-name]
     [,patrolcommand
             server-monitoring-command's-absolute-path-name]
     [,ptrlcmd_ex
             server-monitoring-command's-absolute-path-name]
     [,ptrlcmd_ex_inter
             server-monitoring-command-execution-interval]
     [,ptrlcmd_ex_retry
           {server-monitoring-command-execution-retry-count|2}]
     [,ptrlcmd_ex_tmout
             server-monitoring-command-timeout]
     [,start_timeout
           timeout-for-server-startup]
     [,stop_timeout
           timeout-for-server-termination]
     [,stop_timeout_sw
             timeout-for-server-termination]
     {,waitserv_exec
           {yes|no}]
     [,servexec_retry
           {server-startup-retry-count|2}]
     [,retry_stable
           {startup-completion-monitoring-time|60}]
     [,sby_actcommand
            standby-server-start-command's-path-name]
     [,sby_termcommand
            standby-server-termination-command's-path-name]
     [,sby_ptrlcmd_ex
            standby-server-monitoring-command's-absolute-path-name]
     [,sby_ptrlcmd_ex_inter
            standby-server-monitoring-command-execution-interval]
     [,sby_ptrlcmd_ex_retry
            standby-server-monitoring-command-retry-count|2]
    [,sby_ptrlcmd_ex_tmout
            standby-server-monitoring-command-execution-timeout]
     [,sby_start_timeout
            standby-server-start-command-processing-timeout]
     [,sby_stop_timeout
            standby-server-termination-command-processing-timeout]
     [,sby_servexec_retry
            standby-server-start-retry-count|0]
     [,switchtype
           {switch|restart[:restart-monitoring-time]|manual}]
     [,switch_error
           {retry|ignore}]
     [,switch_retry
           {start-retry-count|2}]
     [,switchbyfail
             LAN-interface-name[:LAN-interface-name...]]
     [,disk_ptrl
           {use|nouse}]
     [,disk_ptrl_inter
           {check-interval-for-disk-for-business-use|120}]
     [,disk_ptrl_retry
           {check-retry-interval-for-disk-for-business-use|60}:
           {check-retry-count-for-disk-for-business-use|1}]
     [,disk_ptrl_act
           {switch|pairdown|noswitch}]
     [,switch_judge
             LAN-interface-name]
     [,pairdown
           {use[:serv_slow]|nouse}]
     [,uoc_neck
           {use|nouse}]
     [,uoc_pairdown
           {use|nouse}]
     [,term_pairdown
           {use|nouse}]
     [,lan_updown
           {use|nouse}]
     [,ip_neck
           {use|nouse}]]
     [,disk
             volume-group's-absolute-path-name[:volume-group's-absolute-path-name...]]
     [,vg_on_opt
             volume-group-connection-option[:option...]]
     [,vg_neck
           {use|nouse}[: {use|nouse}...]]
     [,fs_name
             logical-volume's-absolute-path-name[:logical-volume's-absolute-path-name...]]
     [,fs_mount_dir
             directory-name[:directory-name...]]
     [,fs_mount_optoption
             option[:option...]]
     [,fs_umount_retry
           {retry-count|10}]
     [,fs_neck
           {use|nouse}[: {use|nouse}...]]
     [,scsi_device
             device-name[:device-name...]]
     [,dmmp_device
             device-name[:device-name...]]

     [,deviceoff_order
           {order|reverse}]
     [,dev_timelimit
             shared-resource-switchover-timeout]
     [,dev_offlimit
             timeout-for-disconnecting-resources-shared-by-servers]
     [,dev_onlimit
             timeout-for-connecting-resources-shared-by-servers];
/*  Resource server environment definition  */
[resource    alias
                 resource-server's-alias-name
           [,initial
               {online|standby}]
           [,standbypri
               {priority-of-each-standby-server|1}]
             ,group
                 group-name
           [,switch_error
                {retry|ignore}]
           [,switch_retry
                {start-retry-count|2}]
           [,switchbyfail
                 LAN-interface-name[:LAN-interface-name...]]
           [,switch_judge
                  LAN-interface-name]
           [,uoc_neck
               {use|nouse}]
           [,uoc_pairdown
                {use|nouse}]
           [,lan_updown
               {use|nouse}]
           [,ip_neck
               {use|nouse}]
           [,disk
                 volume-group's-absolute-path-name[:volume-group's-absolute-path-name...]}
           [,vg_on_opt
                 volume-group-connection-option[:option...]]
           [,vg_neck
               {use|nouse}[: {use|nouse}...]]
           [,fs_name
                 logical-volume's-absolute-path-name[:logical-volume's-absolute-path-name...]]
           [,fs_mount_dir
                 directory-name[:directory-name...]]
           [,fs_mount_opt
                 option[:option...]]
           [,fs_umount_retry
               {retry-count\10}]
           [,fs_neck
               {use|nouse}[: {use|nouse}...]]
           [,scsi_device
                device-name[:device-name...]]
           [,dmmp_device
                 device-name[:device-name...]]
           [,deviceoff_order
               {order|reverse}]
           [,dev_timelimit
                 shared-resource-switchover-timeout]
           [,dev_offlimit
                  timeout-for-disconnecting-resources-shared-by-servers]
           [,dev_onlimit
                 timeout-for-connecting-resources-shared-by-servers]
           [,disk_ptrl
                {use|nouse}]
           [,disk_ptrl_inter
                {check-interval-for-disk-for-business-use|120}]
           [,disk_ptrl_retry
                {check-retry-interval-for-disk-for-business-use|60}:
                {check-retry-count-for-disk-for-business-use|1}]
           [,disk_ptrl_act
                {switch|pairdown|noswitch}];

Do not specify any of the following operands when the function for controlling hot standby based on the availability of LAN communications is used by business applications (use is specified in the fence_lan operand):

disk

vg_neck

fs_name

fs_mount_dir

fs_mount_opt

fs_umount_retry

fs_neck

scsi_device

dmmp_device

(1) List of server types and operands to be defined

The operands to be defined in a server environment definition depend on the server type. The server types are listed below:

Server in the server mode

Server in the monitor mode

Resource server

The following table shows the applicable operands for each server type.

Table 8‒4: Operands applicable to each server type

Operand

Server mode

Monitor mode

Resource server

For TP1/Server Base or HiRDB

For TP1/EE

name

R

R

--

alias

R

R

R

initial

D

D

D

standbypri

D

D

D

group

O

O

R

parent

O

O

--

acttype

D

R

--

program

--

D

--

patrol

R

--

--

--

ph_threshold

O

--

--

--

actcommand

--

O

--

termcommand

--

O

--

patrolcommand

--

O

--

ptrlcmd_ex

--

O

--

ptrlcmd_ex_inter

--

O#2

--

ptrlcmd_ex_retry

--

D

--

ptrlcmd_ex_tmout

--

O

--

start_timeout

--

O

--

stop_timeout

--

O

--

stop_timeout_sw

--

O

--

waitserv_exec

--

D

--

servexec_retry

--

D

--

retry_stable

--

D#3

--

sby_actcommand

--

O

--

sby_termcommand

--

O

--

sby_ptrlcmd_ex

--

O

--

sby_ptrlcmd_ex_inter

--

O#4

--

sby_ptrlcmd_ex_retry

--

D

--

sby_ptrlcmd_ex_tmout

--

O

--

sby_start_timeout

--

O

--

sby_stop_timeout

--

O

--

sby_servexec_retry

--

D

--

switchtype

D

--

--

switch_error

D

D

D

switch_retry

--

D

D

switchbyfail

O

O

O

disk_ptrl

D

D

D

disk_ptrl_inter

D

D

D

disk_ptrl_retry

D

D

D

disk_ptrl_act

D

D

D

switch_judge

O

O

O

pairdown

D

--#1

--

uoc_neck

D

D

D

uoc_pairdown

D

D

D

term_pairdown

--

O

--

lan_updown

D

D

D

ip_neck

D

D

D

disk

O

O

O

vg_on_opt

D

D

D

vg_neck

D

D

D

fs_name

O

O

O

fs_mount_dir

O

O

O

fs_mount_opt

O

O

O

fs_umount_retry

D

D

D

fs_neck

D

D

D

scsi_device

O

O

O

dmmp_device

O

O

O

deviceoff_order

D

D

D

dev_timelimit

O

O

O

dev_offlimit

O

O

O

dev_onlimit

O

O

O

Legend:

R: Required (cannot be omitted).

D: Default value is assumed if specification is omitted.

O: Optional (no value is assumed if left unspecified).

--: Not applicable (ignored, if defined).

#1

O applies if ptrlcmd_ex is specified.

#2

R applies if ptrlcmd_ex is specified.

#3

-- applies if ptrlcmd_ex is specified.

#4

R applies if sby_ptrlcmd_ex is specified.

(2) server definition statement

The server definition statement defines a server's operating environment. The following explains the operands of the server definition statement.

(a) name

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Server path name having 1 to 1000 characters

--

(Units: --)

No

--

Description

Specifies a name used to identify the program.

Specify the same value for all hosts in the hot-standby configuration.

For a server in the server mode:

When using TP1/Server Base

Specify the absolute path name of the OpenTP1 directory. For details about the OpenTP1 directory, see the manual OpenTP1 Operation.

When using HiRDB

The value depends on the HiRDB system switchover facility that is being used. For details about the value to be specified, see the HiRDB System Operation Guide (UNIX).

For a server in the monitor mode:

Specify the absolute path name of the server start command. If the actcommand operand is specified, specify either the absolute path name of server start command or a unique name that can identify the program.

Notes

Do not specify an asterisk (*) at the end of the operand when using the online serverless mode.

(b) alias

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 8 alphanumeric characters

--

(Units: --)

No

--

Description

Specifies the server alias name that is to be used in HA Monitor commands and in the messages that are issued.

This name must be unique within each host. Specify the same value among all hosts in a hot-standby configuration.

(c) initial

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

online|standby

--

(Units: --)

Yes

online

Description

Specifies the server's start status, which establishes the server's type at the time of startup.

Specify online for the primary system and standby for the secondary system.

online: Starts the server as the active server.

standby: Starts the server as a standby server.

(d) standbypri

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 9999

(Units: --)

Yes

1

Description

This operand specifies the priority of each standby server if the multi-standby function is used.

The smaller the value of this operand, the higher the priority of the host. You must specify a different value for each host. This operand becomes effective only when use is specified in the multistandby operand in the HA Monitor environment settings and standby is specified in the initial operand in the server environment definition. If online is specified in the initial operand, the highest priority is assumed. For details about the multi-standby function, see 4.5 Managing servers and hosts when using the multi-standby function.

(e) group

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Server group name: 1 to 8 alphanumeric characters

--

(Units: --)

Yes

--

Server switchover type: exchange|no_exchange

--

(Units: --)

Yes

no_exchange#

#: For monitor-mode servers, no default value is provided.

Description

Specifies the name of the server group to be switched over in the batch mode during grouped-system switchover and the server's switchover type.

Specify the same value for all hosts in the hot-standby configuration. When you specify the group operand, you must specify the same initial operand value for all grouped servers on the same host.

You cannot specify exchange, no_exchange, cancel, or no_cancel for a server group name.

For the server switchover type, specify whether a grouped-system switchover is to be performed on the server in the event of a server failure. You can specify the server switchover type for servers in the server mode.

The server switchover type is valid only for servers in the server mode. The default server switchover type is no_exchange. Specify exchange if you want grouped-system switchover to be performed in the event of a server failure.

The server switchover type cannot be specified for a server in the monitor mode. In the event of a server failure on a server in the monitor mode, if a server monitoring command has been created, grouped-system switchover will be performed. If no server monitoring command has been created, grouped-system switchover will not be performed.

Specification of a server switchover type is optional. If you specify a server switchover type, you must also specify a group name. If you omit the group name, the specified switchover type for the server will be ignored. You can specify either of the following values for the server switchover type:

exchange: If a server failure occurs on the server (when this value is specified), grouped-system switchover is performed on all the servers in the same server group in the batch mode. This value and cancel or no_cancel are mutually exclusive.

no_exchange: If a server failure occurs on the server (when this value is specified), grouped-system switchover is not performed. The corresponding standby server is placed in the grouped-system switchover wait state. However, grouped-system switchover is performed in the following cases:

no_exchange is specified for all servers of the server group, and a server failure occurs on all the servers.

The server group includes a resource server and all the resource server's child servers fail.

If TP1/Server Base or HiRDB is used as the server, you can specify one of the values shown later after no_exchange. If you specify both no_exchange and cancel or no_cancel, separate the two values with a colon (:).

cancel

If HA Monitor detects a slowdown on the server when this value is specified, HA Monitor terminates the server abnormally. The corresponding standby server is placed in the grouped-system switchover wait state. However, if manual is specified for the switchtype operand, HA Monitor continues to monitor the status of this server.

no_cancel

If HA Monitor detects slowdown on the server when this value is specified, HA Monitor continues monitoring the server. The corresponding standby server is not placed in the grouped-system switchover wait state.

If no_exchange:no_cancel is specified for all servers in the server group and HA Monitor detects a slowdown on all servers, HA Monitor continues monitoring server status without performing hot standby processing.

(f) parent

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 8 alphanumeric characters

--

(Units: --)

Yes

--

Description

Specifies the parent server required for starting the local server when the server switchover order is to be controlled.

Specify the same value for all hosts in the hot-standby configuration. In this operand, specify the server alias name (alias) of the parent server. When this operand is specified, HA Monitor controls the start order in such a manner that a local server is not started until its parent server's startup processing has been completed. You can specify this operand only if the group operand is specified.

Do not change this operand's value while a server in the server group is running. If it is changed, HA Monitor can no longer control the start order correctly.

(g) acttype

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

server|monitor

--

(Units: --)

Yes

server

Description

Specifies the server's start method.

For a server in the server mode, specify server; for a server in the monitor mode, specify monitor. Specify the same value for all hosts in the hot-standby configuration.

server: Starts the server by using the server start command.

monitor: Starts the server by using the HA Monitor command.

(h) program

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether the monitor-mode program management function is to be used.

You must specify the same value for all hosts in the hot-standby configuration.

use: Uses the monitor-mode program management function.

nouse: Does not use the monitor-mode program management function.

If you specify use in this operand, also specify the following definitions:

pgmmax operand in the HA Monitor environment settings

Monitor-mode program environment definition

For details about the monitor-mode program environment definition, see 8.5.1 Monitor-mode program environment definition (programs).

For a server in the server mode, this operand is ignored, if specified.

(i) patrol

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

5 to 600

(Units: seconds)

Yes

--

Description

Specifies the interval (server failure monitoring time) at which HA Monitor is to check for an operation report from the server (an operation report provides criteria for detecting server failures).

You must specify the same value for all hosts in the hot-standby switchover configuration. If no operation report is received from the server within the amount of time specified in the patrol operand, HA Monitor assumes that a server failure occurred.

If TP1/EE is used as a server-mode server or if a monitor-mode server is used, specification of this operand is ignored.

(j) ph_threshold

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

5 to 559

(Units: seconds)

Yes

--#

#: If this operand is omitted, server monitoring history is not collected.

Description

When information about slowdowns of the local host is to be collected as monitoring history, this operand specifies the amount of time before collection of monitoring history is to begin after the active server stops sending operation reports (time when server monitoring history is to be obtained).

You must specify the same value for all hosts in the hot-standby configuration.

The value for the time when server monitoring history is to be obtained must be less than the value for the server failure monitoring time specified in the patrol operand in the server environment definition.

If a monitor-mode server is used, specification of this operand is ignored.

When you specify this operand to collect server monitoring history, also specify the ph_log_size operand.

(k) actcommand

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 1000 characters

--

(Units: --)

Yes

--

Description

Specifies the absolute path name of the server's start command.

As with the termcommand operand, you can specify arguments by enclosing the operand value in double quotation marks ("). A double quotation mark is not counted as a character. Specify the same value for all hosts in the hot-standby configuration.

For a server in the monitor mode, if you want to specify arguments in the server start command, specify this operand. If you specify this operand, there is no need to specify the absolute path name of the server start command in the name operand. In the name operand, specify a unique name that can identify the program

For a server in the server mode, this operand is ignored, if specified.

(l) termcommand

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 1000 characters

--

(Units: --)

Yes

--

Description

Specifies the absolute path name of a server termination command that has been created by using a shell script, for example.

Specify the same value for all hosts in the hot-standby configuration.

If you specify this operand for a server in the monitor mode, you can have planned termination performed automatically on the active server at any time simply by executing the server hot-standby switchover command (monswap command). You can also have the active server terminated normally at any time simply by executing the monitor-mode server termination command (monend command).

For a server in the server mode, this operand is ignored, if specified.

You can include spaces in the absolute path name by enclosing the entire name in double quotation marks ("). By using spaces, you can specify arguments for the command. A double quotation mark (") is not counted as a character. If you use a specification that includes spaces, the first character string before the first space is recognized as the path name of the server termination command. The character strings after the first space are recognized as arguments. An argument passed by HA Monitor (-e, -w, -c) is added and executed as the last argument. For details about the arguments that are passed by HA Monitor, see 6.12.2 Creating a server termination command.

(m) patrolcommand

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 1000 characters

--

(Units: --)

Yes

--#

#

If this operand is omitted, HA Monitor does not monitor the active server.

However, if the ptrlcmd_ex operand is specified, HA Monitor monitors the active server.

Description

Specifies the absolute path name of a server monitoring command to be used to monitor the server in the monitor mode.

Specify the same value for all hosts in the hot-standby configuration.

For a server in the server mode, this operand is ignored, if specified.

As with the termcommand operand, you can specify arguments in this operand by enclosing the operand value in double quotation marks (").

(n) ptrlcmd_ex

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 1000 characters

--

(Units: --)

Yes

--#

#

If this operand is omitted, HA Monitor does not monitor the active server.

However, if the patrolcommand operand is specified, HA Monitor monitors the active server.

Description

Specifies the absolute path name of a server monitoring command to be used to monitor the server in the monitor mode.

Specify the same value for all hosts in the hot-standby configuration.

For a server in the server mode, this operand is ignored, if specified.

Note that you can include arguments in the path name by enclosing it in double quotation marks (").

If you specify this operand, you must also specify the ptrlcmd_ex_inter operand. If necessary, specify the ptrlcmd_ex_tmout operand. Note that if you specify this operand, you cannot specify the patrolcommand operand. Also, you cannot specify the program operand.

(o) ptrlcmd_ex_inter

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

3 to 180

(Units: seconds)

Yes#

--

#

You must specify this operand if you specify the ptrlcmd_ex operand.

Description

Specifies the interval at which the server monitoring command is executed to monitor the server in the monitor mode. Make sure that the interval you specify is longer than the time period from the start to end of the server monitoring command.

You must specify the same value for all hosts in the hot-standby switchover configuration.

For a server in the server mode, this operand is ignored, if specified.

(p) ptrlcmd_ex_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

0 to 19

(Units: number of times)

Yes

2

Description

If the server monitoring command executed to monitor a monitor-mode server fails (with the EXIT code 1 to 9), command execution is retried by the number of times specified for this operand.

You must specify the same value for all hosts in the hot-standby switchover configuration.

If an attempt to execute the command fails the specified number of times in succession, HA Monitor assumes that the active server failed and operates as specified by the servexec_retry operand.

(q) ptrlcmd_ex_tmout

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

5 to 3600

(Units: seconds)

Yes

--

Description

Specifies the timeout value for the server monitoring command that is executed to monitor the server in the monitor mode. Make sure that the timeout value you specify is equal to or longer than the time period from the start to end of the server monitoring command. If you specify a value shorter than the time period from the start to end of the monitoring command, monitoring cannot be performed appropriately because HA Monitor might perform a false detection. If the command times out, HA Monitor assumes that a server failure occurred, and operates as specified by the servexec_retry operand.

You must specify the same value for all hosts in the hot-standby switchover configuration.

(r) start_timeout

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

60 to 3600

(Units: seconds)

Yes

--#

#: If this operand is omitted, HA Monitor does not monitor the timeout for startup of the server in the monitor mode.

Description

Specifies a timeout value for the server's startup processing in the monitor mode.

We recommend that you specify this operand to prevent indefinite waiting for server startup.

Specify the same value for all hosts in the hot-standby configuration. When this operand is specified, the target server's startup processing is also subject to this timeout monitoring.

If the server is to be restarted according to the specification of the switch_retry operand, the processing is also subject to this timeout monitoring.

If a server's startup processing is retried according to the specification of the servexec_retry operand, its startup processing is not subject to timeout monitoring.

If you specify too small a timeout value, hot-standby switchover might fail to start the other server. To avoid this, specify a sufficiently large value such as a few minutes or the maximum value (3600).

If a timeout occurs, HA Monitor forcibly terminates the server start command and cancels the server's startup processing. If you want to execute a server termination command after a timeout has occurred, specify use in the termcmd_at_abort operand in the HA Monitor environment settings.

If you have specified yes in the waitserv_exec operand, specify the amount of time from start to termination of the server start command. If you have specified use in the program operand, include the amount of time before the hamon_patrolstart function is issued as the timeout value. HA Monitor monitors only once the amount of time before the hamon_patrolstart function is issued. Even when a UAP consists of multiple processes, HA Monitor monitors only once the amount of time before the hamon_patrolstart function is issued.

This operand is ignored if specified for a server in the server mode or when no is specified in the waitserv_exec operand and nouse is specified in the program operand.

The following table shows the correspondence between the waitserv_exec and program operand values and the monitoring range to be specified.

Table 8‒5: Correspondence between the waitserv_exec and program operand values and the monitoring range to be specified

Operand value

Monitoring range to be specified

waitserv_exec

program

yes

use

From the start to termination of a server start command and the amount of time before the hamon_patrolstart function is issued

nouse

From the start to termination of a server start command

no

use

Length of time before the hamon_patrolstart function is issued

nouse

Timeout is not monitored (the operand is ignored, if specified).

(s) stop_timeout

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

60 to 3600

(Units: seconds)

Yes

--#

#: If this operand is omitted, HA Monitor does not monitor the timeout for termination of the server in the monitor mode.

Description

Specifies a timeout value for the server's termination processing in the monitor mode.

We recommend that you specify this operand to prevent indefinite waiting for server termination.

Specify the same value for all hosts in the hot-standby configuration.

If the server is to be restarted according to the specification of the switch_retry operand, the processing is also subject to this timeout monitoring.

If the server's startup processing is retried according to the specification of the servexec_retry operand, the processing is not subject to timeout monitoring.

For the timeout value, specify the maximum wait time that is allowable for server termination. Make sure that the time you specify is equal to or longer than the time period from the start to end of the server termination command. The value specified here takes effect for normal server termination and for server termination at hot-standby switchover.

If you want to set a smaller timeout value for server termination at hot-standby switchover than for normal server termination, specify the timeout value for server termination at hot-standby switchover for the stop_timeout_sw operand. If you specify both the stop_timeout operand and the stop_timeout_sw operand, the specification of the stop_timeout operand is applied to normal termination, and the stop_timeout_sw operand is applied to termination at hot-standby switchover.

If you have specified use in the program operand, include in the timeout value the amount of time before the hamon_patrolstop function is issued. HA Monitor monitors for the amount of time before the hamon_patrolstop function is issued from all programs that have issued the hamon_patrolstart function.

If a timeout occurs, HA Monitor forcibly terminates the server termination command. If a UAP process is running, HA Monitor also terminates the UAP process.

For a server in the server mode, this operand is ignored, if specified.

(t) stop_timeout_sw

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 3600

(Units: seconds)

Yes

--

Description

Specifies the timeout value for termination of the monitor-mode server at hot-standby switchover.

You must specify the same value for all hosts in the hot-standby switchover configuration.

For the timeout value, specify the maximum wait time that is allowable for server termination at hot-standby switchover. Make sure that the time you specify is equal to or longer than the time period from the start to end of the server termination command.

If you want to set for server termination at hot-standby switchover a smaller timeout value than for normal server termination, specify this operand. If you specify both the stop_timeout operand and the stop_timeout_sw operand, the specification of the stop_timeout operand is applied to normal termination, and the stop_timeout_sw operand is applied to termination at hot-standby switchover.

(u) waitserv_exec

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

yes|no

--

(Units: --)

Yes

no

Description

Specifies whether HA Monitor is to wait until execution of the server start command specified in the name operand has been completed before it performs active server start completion processing when a server in the monitor mode is to be started as the active server.

Specify the same value for all hosts in the hot-standby configuration.

yes: Waits until execution of the server start command is completed and then performs active server start completion processing.

no: Performs the active server start completion processing without waiting for completion of the server start command.

If you are monitoring a server in the monitor mode, determine the value of this operand based on the information provided in (2) How to create a server start command.

For a server in the server mode, this operand is ignored, if specified.

(v) servexec_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

0 to 9999

(Units: number of times)

Yes

2

Description

Specifies the number of times server startup processing is to be re-executed if a server failure is detected on the active server in the monitor mode.

Specify the same value for all hosts in the hot-standby configuration.

If HA Monitor detects a server failure on the active server, it retries execution of the server start command for the server specified in the name operand as many times as specified in this operand. If the server completes startup processing successfully within the retry count, HA Monitor treats it as a normal operation and continues processing. When the number of server failures detected exceeds the specified count, HA Monitor performs automatic planned hot standby processing.

If the server start command returns a return value other than 0 and server restart fails, hot-standby switchover does not occur. If you want to perform hot standby processing when a restart fails, create the start command in such a manner that it always returns a value of 0 and use the monitoring command to detect server start errors.

If 0 is specified in this operand, HA Monitor performs automatic planned hot standby processing immediately after detecting completion of the server monitoring command without the server restarting.

If 9999 is specified in this operand, HA Monitor retries until the server startup processing is completed or the active server is terminated. In such a case, the server might not start successfully for reasons such as that the path specified in the name operand in the server environment definition cannot be found or the required execution permissions are not granted. Therefore, if you specify 9999, it is especially important that you ensure that the specified environment settings and server start command are correct.

Note that specification of this operand is ignored in the following cases:

The server has a program for which the restartcommand operand is specified in the monitor-mode program environment definition.

server is specified in the acttype operand.

No server monitoring command is used (no server monitoring command is specified for the ptrlcmd_ex and patrolcommand operands).

If the switch_retry operand is specified and the target server has already started, the switch_retry operand takes effect. If the server becomes ready for hot standby processing, the servexec_retry operand takes effect.

(w) retry_stable

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

60 to 3600

(Units: seconds)

Yes

60

Description

Specifies a monitoring interval (in seconds) for determining whether operation of a server in the monitor mode has become stable after the server has been restarted successfully.

Specify the same value for all hosts in the hot-standby configuration.

Note that specification of this operand is ignored if the ptrlcmd_ex operand is specified.

If no failure is detected within the time specified in this operand after the server restarted, HA Monitor determines that the server's operation has become stable and resets the restart count to 0. After a successful restart, HA Monitor starts monitoring for stable operation beginning at second 0.

If a failure occurs within the time specified in this operand after the server was restarted, the restart count is updated and then the server is restarted again. If the updated restart count exceeds the servexec_retry operand value specified in the server environment definition, HA Monitor performs hot standby processing without restarting the server.

(x) sby_actcommand

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 1000 characters

--

(Units: --)

Yes

--

Description

This operand specifies the absolute path of the command that is to be executed when the standby server starts in the case where HA Monitor performs monitoring in the standby system.

For a server in the server mode, this operand is ignored, if specified.

You can include arguments in the value specified for this operand by enclosing the value in double quotation marks ("). A double quotation mark is not counted as a character. Specify the same value for all hosts in the hot-standby configuration.

(y) sby_termcommand

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 1000 characters

--

(Units: --)

Yes

--

Description

This operand specifies the absolute path of the command that is to be executed when the standby server terminates in the case where HA Monitor performs monitoring in the standby system.

For a server in the server mode, this operand is ignored, if specified.

You can include spaces in the absolute path name by enclosing the entire name in double quotation marks ("). By using spaces, you can specify arguments for the command. A double quotation mark is not counted as a character. If you use a specification that includes spaces, the first character string before the first space is recognized as the path name of the server termination command. The character strings after the first space are recognized as arguments. An argument passed by HA Monitor (-e or -c) is added to the server termination command as the last argument when the command is executed. For details about the arguments that are passed by HA Monitor, see 6.13.2 Creating a command to be executed when the standby server stops.

(z) sby_ptrlcmd_ex

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 1000 characters

--

(Units: --)

Yes

--#

#

If this operand is omitted, HA Monitor does not monitor the standby server.

Description

This operand specifies the absolute path of the command that is to be executed for a monitoring purpose in the case where HA Monitor performs monitoring in the standby system.

You can monitor, for example, processes started by sby_actcommand and resources.

You must specify the same value for all hosts in the hot-standby switchover configuration.

For a server in the server mode, this operand is ignored, if specified.

Note that you can include arguments in the path name by enclosing it in double quotation marks (").

If you specify this operand, you must also specify the sby_ptrlcmd_ex_inter operand. If necessary, specify the sby_ptrlcmd_ex_tmout operand.

(aa) sby_ptrlcmd_ex_inter

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

3 to 180

(Units: seconds)

Yes#

--

#

You must specify this operand if you specify the sby_ptrlcmd_ex operand.

Description

This operand specifies the interval at which the monitoring command is to be executed if sby_ptrlcmd_ex is specified. Make sure that the interval you specify is longer than the time period from the start to end of the monitoring command.

You must specify the same value for all hosts in the hot-standby switchover configuration.

For a server in the server mode, this operand is ignored, if specified.

(ab) sby_ptrlcmd_ex_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

0 to 19

(Units: number of times)

Yes

2

Description

This operand specifies the number of times HA Monitor can retry execution of the monitoring command specified for sby_ptrlcmd_ex if the command fails (with the EXIT code 1 to 9).

You must specify the same value for all hosts in the hot-standby switchover configuration.

If an attempt to execute the command fails the specified number of times in succession, HA Monitor assumes that a failure occurred and operates as specified by the sby_servexec_retry operand.

(ac) sby_ptrlcmd_ex_tmout

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

5 to 3600

(Units: seconds)

Yes

--

Description

This operand specifies the timeout value for execution of the monitoring command specified for sby_ptrlcmd_ex. Make sure that the timeout value you specify is equal to or longer than the time period from the start to end of the monitoring command. If you specify a value shorter than the time period from the start to end of the monitoring command, monitoring cannot be performed appropriately because HA Monitor might perform a false detection.

If the command times out, HA Monitor assumes that a server failure occurred, and operates as specified by the sby_servexec_retry operand.

You must specify the same value for all hosts in the hot-standby switchover configuration.

(ad) sby_start_timeout

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

60 to 3600

(Units: seconds)

Yes

--

Description

This operand specifies the timeout value for the command specified for sby_actcommand.

You must specify the same value for all hosts in the hot-standby switchover configuration.

Note that HA Monitor does not monitor the timeout for the command started when the standby server's startup processing is retried according to the specification of the sby_servexec_retry operand.

If a timeout occurs, HA Monitor forcibly terminates the command specified for sby_actcommand and cancels the standby server's startup processing. After the monitoring-target command times out, the command specified for sby_termcommand starts. Make sure that the timeout value you specify is equal to the time period from the start to end of the command specified for sby_actcommand.

For a server in the server mode, this operand is ignored, if specified.

(ae) sby_stop_timeout

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

60 to 3600

(Units: seconds)

Yes

--

Description

This operand specifies the timeout value for the command specified for sby_termcommand.

You must specify the same value for all hosts in the hot-standby switchover configuration.

HA Monitor does not monitor the timeout for the following command:

The command that is specified for sby_termcommand to be executed when the standby server is restarted according to the specification of the sby_servexec_retry operand

Make sure that the timeout value you specify is equal to the time period from the start to end of the command specified for sby_termcommand.

If a timeout occurs, HA Monitor forcibly terminates the command specified for sby_termcommand.

For a server in the server mode, this operand is ignored, if specified.

(af) sby_servexec_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

0 to 9999

(Units: number of times)

Yes

0

Description

This operand specifies the number of times the standby server's startup processing is retried if a failure is detected by the monitoring command specified for sby_ptrlcmd_ex.

You must specify the same value for all hosts in the hot-standby switchover configuration.

Upon detecting a failure, HA Monitor retries execution of the command specified for the sby_actcommand operand as many times as specified in this operand. If startup processing of the standby server is completed successfully within the retry count, HA Monitor treats it as a normal operation and continues processing. If a failure is still detected after command execution is retried the specified number of times, HA Monitor terminates the standby server.

In the following cases, HA Monitor does not restart the standby server and terminates the standby server immediately upon detecting termination of the monitoring command:

This operand is omitted.

0 is specified for this operand.

No command is specified for the sby_actcommand operand.

If 9999 is specified in this operand, HA Monitor retries until the standby server startup processing is completed or the standby server is terminated. In such a case, the standby server might not start successfully for reasons such as that the path specified in the sby_actcommand operand in the server environment definition cannot be found or the required execution permissions are not granted. Therefore, it is especially important that you ensure that the specified environment settings and the command specified for the sby_actcommand operand are correct.

For a server in the server mode, this operand is ignored, if specified.

(ag) switchtype

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

switch|restart|manual

--

(Units: --)

Yes

switch

Description

Specifies the processing to be performed when a server failure is detected on the server in the server mode.

Specify the same value for all hosts in the hot-standby configuration. For a server in the monitor mode, this operand is ignored, if specified.

switch: Terminates the active server and performs hot standby processing.

restart: Places the active server in the active-server-restart wait state and waits until the active server is restarted automatically. HA Monitor does not perform hot standby processing at this point. If restart of the active server fails, HA Monitor retries the restart processing as many times as specified for the server (retry count). If the restart is not successful after the specified number of retries has been attempted, HA Monitor starts hot standby processing.

When you specify restart, you can also specify an active-server-restart monitoring time by separating the values with a colon (:). If the active server has not been restarted successfully after the specified restart monitoring time has been reached, HA Monitor starts hot standby processing. For the restart monitoring time, you can specify a value in the range from 60 to 3,600 seconds. If you omit the restart monitoring time, HA Monitor does not monitor for restart of the active server.

manual: Places the active server in the active-server-restart wait state and waits until the active server is restarted automatically. HA Monitor does not perform hot standby processing at this point. If restart of the active server fails, HA Monitor retries the restart processing as many times as specified for the server (retry count). If the restart is not successful after the specified number of retries has been attempted, HA Monitor terminates the active server, terminates the corresponding standby server, restarts the standby server, then places the standby server in the active server start wait state.

If HA Monitor detects a slowdown on the server when manual is specified, HA Monitor continues monitoring the server without performing any processing.

(ah) switch_error

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

retry|ignore

--

(Units: --)

Yes

ignore

Description

Specifies whether startup of a server is to be retried automatically as the active system if a shared resource connection error occurs while the target active server is starting or the active server fails to start and there is no available switchover host.

You must specify the same value for all hosts in the hot-standby configuration.

retry: Automatically retries startup of server as the active system.

ignore: Cancels server startup.

To have HA Monitor retry startup processing in the server mode in the event of a hot standby error, you must specify the following settings on the server:

HiRDB

Specify retry in the pd_ha_switch_error operand in the HiRDB system definition (pdsys or pdutsys).

TP1/Server Base

Specify a desired retry count in the ha_switch_error_retry_count operand in the TP1/Server Base system environment definition.

For a server in the monitor mode or a resource server, you can specify the start retry count in the switch_retry operand.

The following table describes the definition of the switch_error and switch_retry operands and the HA Monitor processing.

Table 8‒6: Definition of switch_error and switch_retry operands and HA Monitor processing

Server type

Definition of switch_error

Definition of switch_retry

HA Monitor processing

Server mode

Omitted or ignore

--

Continues moncheck.

Ignores the start retry function during hot standby processing.

retry

--

Continues moncheck.

Enables the start retry function during hot standby processing.

Monitor mode or resource server

Omitted or ignore

Omitted

Continues moncheck.

Disables the start retry function during hot standby processing.

Specified

Issues the KAMN514-E message and stops moncheck.

retry

Omitted or specified

Continues moncheck.

Enables the start retry function during hot standby processing.

Legend:

--: Undefined (ignored if defined)

The switch_error operand is ignored, if specified, for a server that contains a program for which the restartcommand operand is specified in the monitor-mode program environment definition.

For details about start retries during hot standby processing, see 3.2.7 Performing start retries during hot standby processing.

(ai) switch_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 9999

(Units: number of times)

Yes

2

Description

Specifies the number of times startup of a server in the monitor mode or a resource server is to be attempted automatically when a shared resource connection error occurs while the target active server is starting or the active server fails to start and there is no available switchover host.

You must specify the same value for all hosts in the hot-standby configuration.

If 9999 is specified in this operand, HA Monitor attempts to restart the server indefinitely until the server startup processing is completed or the start retry state is released forcibly.

To perform start retries in the event of a hot standby error, specify retry in the switch_error operand.

(aj) switchbyfail

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 32 alphanumeric characters

--

(Units: --)

Yes

--

Description

Specifies the names of the LAN interfaces that are to be monitored.

If a failure occurs on any LAN interface specified in this operand, HA Monitor performs hot standby processing on the active server. If an alias IP address is assigned to a LAN interface that is to be monitored, specify the alias IP address's interface name without the index number.

You can specify a maximum of 32 LAN interface names by delimiting them with the colon. When you specify this operand, you must create a LAN monitor definition file for each LAN interface. If a server for which this operand is specified is grouped with other servers and a failure occurs on a specified LAN interface, all servers in the group will be switched over.

For details about the LAN monitor definition file, see 6.19.1 Settings in the files required for monitoring a LAN.

(ak) disk_ptrl

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether to monitor the disks for business use on the active server.

Specify the same values for all hosts in the hot-standby configuration.

use: HA Monitor monitors the disks for business use.

nouse: HA Monitor does not monitor the disks for business use.

If you specify use for this operand, you must set up the monitoring definition file for the disk for business use. For details about how to specify the settings in a monitoring definition file for the disk for business use, see (2) Settings in the files required for monitoring disks for business use in 6.19.2 Settings in the files required for disk monitoring.

If necessary, also specify the following operands.

Server environment definition:

disk_ptrl_inter

disk_ptrl_retry

disk_ptrl_act

HA Monitor environment settings:

disk_log_size

(al) disk_ptrl_inter

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

3 to 3600

(Units: seconds)

Yes

120

Description

Specifies the interval at which HA Monitor checks a disk for business use on the active server.

Specify the same values for all hosts in the hot-standby configuration.

Determine the value to be specified for this operand taking into account the following items specified for the disk_ptrl_retry operand in the server environment definition:

Check retry interval for a disk for business use

Check retry count for a disk for business use

Specify a value that meets the following condition:

check-interval-for-disk-for-business-use ≥ check-retry-interval-for-disk-for-business-use × (check-retry-count-for-disk-for-business-use + 1)

(am) disk_ptrl_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

check-interval-for-disk-for-business-use: Unsigned integer

1 to 3600

(Units: seconds)

Yes

60

check-retry-count-for-disk-for-business-use: Unsigned integer

0 to 60

(Units: number of times)

Yes

1

Description

Use this operand to specify the interval at which HA Monitor retries checking a disk for business use on the active server and the number of times HA Monitor can retry checking it if access to the disk fails. Specify the retry interval and count with an intervening colon (:). If the number of times access to a disk for business use fails exceeds the specified count, HA Monitor judges that the disk has failed.

Specify the same values for all hosts in the hot-standby configuration.

Check retry interval for the disks for business use

Specify the interval at which HA Monitor retries checking a disk for business use. Make sure that the value you specify is equal to or larger than the disk I/O timeout value. The disk I/O timeout value means the time for which an application that requests an I/O operation can wait before receiving a response. This value is determined from the driver settings, OS kernel parameters, and multipath software settings.

Check retry count for a disk for business use

Specify the number of times HA Monitor can retry checking a disk for business use. If you specify 0, HA Monitor does not retry checking a disk for business use.

(an) disk_ptrl_act

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

switch|pairdown|noswitch

--

(Units: --)

Yes

switch

Description

Specifies the action that HA Monitor takes when a failure occurs on a disk for business use on the active server.

Specify the same values for all hosts in the hot-standby configuration.

switch: HA Monitor performs planned hot-standby switchover for servers.

pairdown: HA Monitor causes host pair shutdown to perform hot-standby switchover as a host failure.

noswitch: HA Monitor does not perform hot-standby switchover and continues monitoring of the disk for business use.

(ao) switch_judge

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 32 alphanumeric characters

--

(Units: --)

Yes

--

Description

Specifies the name of a LAN interface whose availability is to be checked during hot standby processing resulting from a host failure.

This operand can be specified only when the fence_lan operand is specified in the HA Monitor environment settings.

HA Monitor controls in such a manner that the server that can establish communication via the LAN interface specified in this operand becomes the active server. If an alias IP address has been assigned to the target LAN interface, specify the interface name without the index number for the alias IP address.

If you group servers, specify this operand only for the server at the top of the hierarchy or a resource server. If there are multiple servers at the top in the group, specify the same LAN interface name for all such servers.

When you specify this operand, you must create a LAN monitor definition file for the LAN interface.

For details about the function for controlling hot standby based on the availability of LAN communications, see 2.3.7 Function for controlling hot standby based on the availability of LAN communications.

For details about LAN monitor definition files, see 6.19.1 Settings in the files required for monitoring a LAN.

(ap) pairdown

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether the pair shutdown function that terminates HA Monitor when a server failure is detected is to be used.

Specify the same value for all hosts in the hot-standby configuration. For a server in the monitor mode, specification of this operand is ignored unless the ptrlcmd_ex operand is specified.

use: Uses the pair shutdown function. If a failure occurs on the active server, pair shutdown is performed. If TP1/Server Base or HiRDB is used as the server, you can add the following value after use. To do so, use a colon (:) as a separator.

serv_slow

Performs pair shutdown only when server slowdown of the active server is detected. Pair shutdown is not performed when the server itself detects a failure. You can specify serv_slow only if switch is specified for the switchtype operand. In the case where B is specified for the server_type operand or where the ptrlcmd_ex operand is specified, if you specify serv_slow, HA monitor outputs KAMN163-E and stops parsing the definitions.

nouse: Does not use the pair shutdown function.

(aq) uoc_neck

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether the termination code is to be checked for the user command that is executed when server startup processing or hot standby processing is started.

Specify the same value for all hosts in the hot-standby configuration.

use: Cancels server startup processing if the user command's termination code is not 0.

nouse: Continues server startup processing without checking the user command's termination code.

Note that termination code checking is not performed when no user command is executed at the time that server startup processing or hot standby processing is started.

(ar) uoc_pairdown

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether the pair shutdown function is to be used based on the termination code of the user command that is executed at the following time:

When hot standby processing begins due to a server failure

(server type: online; server status parameter: -a; start or end parameter: start)

When planned hot standby processing begins

(server type: online; server status parameter: -w; start or end parameter: start)

You must specify the same value for all hosts in the hot-standby configuration.

use: Uses the pair shutdown function that also terminates HA Monitor when the user command's termination code is not 0.

nouse: Resumes server hot standby processing without checking the user command's termination code.

Note that if use is specified, pair shutdown is also performed in the following cases:

The user command cannot be executed.

Processing is canceled according to the specification of the dev_timelimit operand in the server environment definition.

Processing is canceled according to the specification of the dev_offlimit operand in the server environment definition.

For details about host pair shutdown performed for each host in the event of a user command error, see (2) Host pair shutdown performed for each host in the event of a user command error.

(as) term_pairdown

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

This operand specifies whether to use the pair shutdown function if the server termination command returns a termination code other than 0.

You must specify the same value for all hosts in the hot-standby switchover configuration.

use: Uses the pair shutdown function, which also terminates HA Monitor, if the server termination command returns a termination code other than 0.

nouse: Resumes server hot-standby switchover without checking the termination code of the server termination command.

For details about host pair shutdown performed for each host if the server termination command fails, see (3) Host pair shutdown for each host in the event of a server termination command error.

For a server in the server mode, this operand is ignored, if specified.

(at) lan_updown

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

--#

#: If this operand is omitted but server-alias-name.up and server-alias-name.down files exist under the directory for HA Monitor environment settings, HA Monitor uses those files as LAN status settings files anyway.

Description

Specifies whether LAN status settings files are to be used.

Specify the same value for all hosts in the hot-standby configuration.

use: Uses LAN status settings files (server-alias-name.up and server-alias-name.down files).

nouse: Does not use LAN status settings files.

(au) ip_neck

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies whether the server-alias-name.up file's termination code is to be checked when the file is executed during server startup processing or hot standby processing.

Specify the same value for all hosts in the hot-standby configuration.

use: Cancels server startup processing if the server-alias-name.up file's termination code is not 0. If you specify this value, specify LAN status settings files by referencing the example of a server-alias-name.up file provided in 6.11.2 Specifying LAN status settings files.

nouse: Continues server startup processing without checking the termination code from the server-alias-name.up file. When you specify this value, specify LAN status settings files by referencing the example of a server-alias-name.up file provided in 6.11.2 Specifying LAN status settings files.

(av) disk

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 256 characters

--

(Units: --)

Yes

--#

#: If this operand is omitted, HA Monitor does not switch shared disks.

Description

Specifies the absolute path names of volume groups defined on a shared disk that are to be switched over when a shared disk is used.

You can specify a maximum of 3,000 path names. Specify the same values for all hosts in the hot-standby configuration.

(aw) vg_on_opt

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Character string having 1 to 256 characters

--

(Units: --)

Yes

-aΔy

Description

Specifies for the vgchange command run-time options that are to be used when volume groups to be switched over are connected.

Specify the same values for all hosts in the hot-standby configuration.

If this operand is omitted, the -aΔy option is assumed.

If an option contains a space (Δ), enclose the entire option in double quotation marks (").

The options in the vg_on_opt operand must be specified so that they correspond in order and number to the volume groups specified in the disk operand. You must specify as many sets of options as there are values in the disk operand. To omit an option for a volume group at any intermediate location, specify a colon (:) without specifying any value.

The following shows an example of specifying vgchange command run-time options:

    disk               /dev/vg01      :/dev/vg02  ,
                       ↓
    vg_on_opt          "-aΔyΔ-qΔn" :  ,

In the preceding example, for the /dev/vg01 volume group specified by the disk operand, the vgchange command is executed with the -aΔyΔ-qΔn option. For the /dev/vg02 volume group, the vgchange command is executed with the -aΔy option (default option).

If the vg_on_opt operand is omitted, the default value is assumed as the option for all volume groups.

(ax) vg_neck

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies for each volume group whether the server startup processing is to be cancelled if the volume group's connection processing fails during server startup processing or hot standby processing.

Specify the same value for all hosts in the hot-standby configuration.

use: Cancels server startup processing if connection fails.

nouse: Continues server startup processing even if connection fails.

As with the vg_on_opt operand, specify the options in the vg_neck operand so that they corresponding in order and number to the volume groups specified in the disk operand. You must specify as many options as there are values in the disk operand. To omit the option for a volume group at any intermediate location, specify a colon (:) without specifying any value (such as vg_neck use::nouse,). If the vg_neck operand is omitted, nouse is assumed for all volume groups.

(ay) fs_name

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 256 characters

--

(Units: --)

Yes

--#

#: If this operand is omitted, HA Monitor does not switch over file systems on the shared disk.

Description

This operand specifies the absolute path name of the logical volume that corresponds to a file system to be switched over when a file system on the shared disk is used. The value of this operand is specified in the /dev/volume-group-name/logical-volume-name format.

You can specify a maximum of 3,000 path names. Specify the same values for all hosts in the hot-standby configuration.

When you specify the fs_name operand, you must also specify the fs_mount_dir operand.

(az) fs_mount_dir

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 256 characters

--

(Units: --)

Yes

--

Description

Specifies the absolute path names of the directories where the file systems to be switched over are to be mounted when file systems on the shared disk are used.

Specify the same values for all hosts in the hot-standby configuration.

You must specify this operand if you have specified the fs_name operand.

The directories in the fs_mount_dir operand must be specified so they correspond in order and number to the logical volumes specified in the fs_name operand. Make sure that you specify the same number of mount target directories as there are path names in the fs_name operand. The following shows an example of the fs_name and fs_mount_dir operands:

      fs_name    /dev/vg01/lvol1:/dev/vg02/lvol1,
                      ↓
      fs_mount_dir  /home1         :/home2,

In this example, /dev/vg01/lvol1 in the fs_name operand corresponds to /home1 in the fs_mount_dir operand, and /dev/vg02/lvol1 in the fs_name operand corresponds to /home2 in the fs_mount_dir operand.

File systems specified in this operand must not be set to be mounted automatically during system startup.

(ba) fs_mount_opt

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Character string having 1 to 256 characters

--

(Units: --)

Yes

--#

#: If this operand is omitted, the run-time options are omitted from the mount command for the file system.

Description

Specifies mount command run-time options for the file systems that are to be switched over (when file systems on a shared disk are used).

Specify the same values for all hosts in the hot-standby configuration. If an option contains a space, comma, colon, semicolon, forward slash, or asterisk, enclose the option in double quotation marks (").

The options in the fs_mount_opt operand must be specified so that they correspond in order and number to the file systems specified in the fs_name and fs_mount_dir operands. You must specify as many sets of options as there are values in the fs_name and fs_mount_dir operands. To omit an option for a file system at any intermediate location, specify a colon (:) without specifying any value. The following shows an example of fs_name, fs_mount_dir, and fs_mount_opt operands:

      fs_name      /dev/vg01/lvol1:/dev/vg02/lvol1,
                         ↓
      fs_mount_dir     /home1         :/home2,
                         ↓
      fs_mount_opt     "-o rw"        :,

In this example, -o rw in the fs_mount_opt operand corresponds to /dev/vg01/lvol1 in the fs_name operand and to /home1 in the fs_mount_dir operand. No fs_mount_opt operand value is set for /dev/vg02/lvol1 in the fs_name operand and for /home2 in the fs_mount_dir operand.

(bb) fs_umount_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

0 to 9999

(Units: number of times)

Yes

10

Description

Specifies a retry count to be applied if an unmount operation fails when file systems are disconnected.

Specify the same value for all hosts in the hot-standby configuration.

If 0 is specified, an unmount operation is not retried. If 9999 is specified, the unmount operation is retried until it is successful. The retry interval is one second.

(bc) fs_neck

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

use|nouse

--

(Units: --)

Yes

nouse

Description

Specifies for each file system whether the server startup processing is to be cancelled if the file system's connection processing fails during server start or hot standby processing.

Specify the same value for all hosts in the hot-standby configuration.

use: Cancels server startup processing if connection fails.

nouse: Continues server startup processing even if connection fails.

As with the fs_mount_opt operand, specify the options in the fs_neck operand so that they correspond in order and number to the file systems specified in the fs_name operand. You must specify as many options as there are values in the fs_name operand. You can omit options in a list of colon-separated values. To omit options, only omit specifying the values and do not omit colons (:) in a format such as fs_neck use::nouse,. If the fs_neck operand is omitted, HA Monitor assumes that nouse is specified for all file systems.

In RHEL7 or a later version, if XFS is used as the file system of the shared disk, HA Monitor uses the xfs_repair command to check the consistency of the file system. Before the consistency of the file system is checked, mount and unmount processing is performed. If unmount processing fails, HA Monitor stops starting the server regardless of the values specified for this operand.

(bd) scsi_device

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 384 characters

--

(Units: --)

Yes

--

Description

When SCSI reservation for shared disk is used, this operand specifies the absolute path of a device subject to SCSI reservation. For notes on specifying the SCSI reservation settings, see 6.4.4 Notes about SCSI reservation for shared disk.

Express a path name as 1 to 384 alphabetic characters, numeric characters, forward slashes (/), hyphens (-), underscores (_), and periods (.). You can specify a maximum of 3,000 path names.

Specify a path name value as follows:

If HFC-PCM or EMPD is used with a single-path configuration in a VMware ESXi-based virtualization environment

Specify the absolute path of the symbolic link under /dev/disk/by-id/ that corresponds to the physical device used for the shared disk. Specify the same names in the same order on all hosts in the hot-standby configuration. For details about the symbolic link, see (a) If HFC-PCM is used with a single-path configuration in a VMware ESXi-based virtualization environment.

Note that this operand is not needed for a redundant configuration using DMMP even when a VMware ESXi-based virtualization environment is used. Specify the dmmp_device operand.

Redundant configuration using HDLM

Specify the absolute path of the HDLM logical device that is used as the shared disk. Specify the same absolute paths in the same order on all hosts in the hot-standby configuration. If the same disk is used for all hosts in the hot-standby configuration, you can specify different names. For details about the HDLM logical devices that make up a shared disk, see (b) Redundant configuration using multipath software (HDLM).

When you specify this operand, you must specify use in the fence_scsi operand in the HA Monitor environment settings. The server's startup processing will fail if the specified value is neither a symbolic link nor HDLM logical device.

In this case, before you restart the server, correct the operand value and then execute the definition check command (moncheck command) to make sure that neither the KAMN113-E message nor the KAMN723-E message is issued.

(be) dmmp_device

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 256 characters

--

(Units: --)

Yes

--

Description

When SCSI reservation for shared disk is used in a multi-path configuration, specifies a device subject to SCSI reservation, expressed as the absolute path of the logical device of DMMP. For notes on specifying the SCSI reservation settings, see 6.4.4 Notes about SCSI reservation for shared disk.

Specify the same names in the same order on all hosts in the hot-standby configuration. For details about the logical device name of DMMP, see (2) dmmp_device operand value (redundant configuration using multipath software (DMMP)).

Express a path name as 1 to 256 alphabetic characters, numeric characters, forward slashes (/), hyphens (-), underscores (_), and periods (.). You can specify a maximum of 3,000 path names.

When you specify this operand, you must specify use in the fence_scsi operand in the HA Monitor environment settings.

The server's startup processing will fail in the following cases:

use is not specified in the fence_scsi operand in the HA Monitor environment settings.

The specified name is not the logical device name of DMMP.

In this case, before you restart the server, correct the operand value and then execute the definition check command (moncheck command) to make sure that neither the KAMN113-E message nor the KAMN723-E message is issued.

(bf) deviceoff_order

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

order|reverse

--

(Units: --)

Yes

order

Description

Specifies the order in which shared resources are to be disconnected.

Specify the same value for all hosts in the hot-standby configuration. In contrast to the deviceoff_order operand in the HA Monitor environment settings, this operand is applicable only to the specified server.

If the deviceoff_order operand is specified in the HA Monitor environment settings, this operand is ignored, in which case the specified HA Monitor environment setting is effective. If you want to change the order in which shared resources are to be disconnected on a per-server basis, specify this operand and do not specify the deviceoff_order operand in the HA Monitor environment settings.

For details about the order in which shared resources are disconnected, see 4.8.7 Processing flow for disconnecting shared resources in the reverse order from which they were connected.

order: Disconnects the shared resources in the same order as when they were connected.

reverse: Disconnects the shared resources in the reverse order from which they were connected.

(bg) dev_timelimit

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

60 to 3600

(Units: seconds)

Yes

--

Description

Specifies a timeout value for shared resource connection and disconnection processing for the server during hot standby processing.

We recommend that you use the dev_offlimit and dev_onlimit operands to specify the timeout values for disconnection and connection of resources shared by servers at the time of hot-standby switchover. We do not recommend that you use the dev_timelimit operand for that purpose. This is because the system might behave as follows if the dev_timelimit operand is used:

If too small a timeout value is specified, hot-standby switchover might fail.

If too large a timeout value is specified and disconnection of shared resources takes time, hot-standby switchover also takes time.

Specify the same value for all hosts in the hot-standby configuration.

If too small a timeout value is specified and restoration or mounting of a file system takes time at the hot-standby switchover destination, switchover might fail. To avoid this, specify a sufficiently large value such as a few minutes or the maximum value (3600).

If a timeout occurs, HA Monitor treats the shared resource connection or disconnection processing as having failed and cancels the processing. It then continues the server start or termination processing. If this operand is omitted, HA Monitor does not monitor the timer and places server start or termination processing on hold until connection or disconnection processing is completed.

You cannot use the dev_timelimit operand to specify different timeout values for shared resource connection processing and for shared resource disconnection processing. Specify a timeout value for the following operands:

The dev_onlimit operand in the server environment definition (timeout value for shared resource connection)

The dev_offlimit operand in the server environment definition (timeout value for shared resource disconnection)

(bh) dev_offlimit

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

1 to 3600

(Units: seconds)

Yes

--

Description

Specifies a timeout value for the shared resource disconnection processing performed for the server when hot-standby switchover occurs.

We recommend that you specify this operand (or the dev_timelimit operand) to prevent indefinite waiting for disconnection of shared resources.

You must specify the same value for all hosts in the hot-standby switchover configuration.

If a timeout occurs, HA Monitor treats the processing performed for shared resources as having failed and cancels the processing. HA Monitor then performs SCSI reservation for shared disk and continues hot-standby switchover. If this operand is omitted, HA Monitor does not monitor the timer and places server termination processing on hold until disconnection processing is completed.

You cannot use this operand if you have specified the dev_timelimit operand.

(bi) dev_onlimit

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

60 to 3600

(Units: seconds)

Yes

--

Description

Specifies a timeout value for the shared resource connection processing performed for the server when hot-standby switchover occurs.

We recommend that you specify this operand (or the dev_timelimit operand) to prevent indefinite waiting for connection of shared resources.

You must specify the same value for all hosts in the hot-standby switchover configuration.

If too small a timeout value is specified and restoration or mounting of a file system takes time at the hot-standby switchover destination, switchover might fail. To avoid this, specify a sufficiently large value such as a few minutes or the maximum value (3600).

If a timeout occurs, HA Monitor treats the processing performed for shared resources as having failed and cancels the processing. HA Monitor then continues the server start processing. If this operand is omitted, HA Monitor does not monitor the timer and places server start processing on hold until connection processing is completed.

You cannot use this operand if you have specified the dev_timelimit operand.

(3) resource definition statement

The resource definition statement defines a resource server. The operands of the resource definition statement are explained below. For details about resource servers, see 4.6 Managing a resource server.

The resource definition statement's operands are the same as the operands of the server statement in the server environment definition. For details of the operands, see the server definition statement. This subsection explains only those operands that differ from when they are specified in a server definition statement.

(a) alias

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 8 alphanumeric characters

--

(Units: --)

No

--

Description

Specifies the alias name of the resource server (server's alias) that is used in HA Monitor commands and in the messages that are issued.

This is the same as the alias operand in the server definition statement.

(b) group

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 8 alphanumeric characters

--

(Units: --)

No

--

Description

Specifies the name of the server group to be switched over in the batch mode during grouped-system switchover.

Specify the same value for all hosts in the hot-standby configuration.

This operand is required for a resource server. The names exchange, no_exchange, cancel, and no_cancel are ignored, if specified. The other information is the same as for the group operand in the server definition statement.

Note that only one resource server can be defined per group.

8.4.2 Locked server environment settings (servers_opt)

This subsection explains the format of the locked server environment definition file and how to define the file.

(1) Format of definition

The following shows the format of a definition:

Δ₀exclusive_serversΔ₀:Δ₀server-alias-nameΔ₀:Δ₀server-alias-nameΔ₀↓

Legend:

Δ₀: Zero or more spaces or tabs

↓: End-of-line code

Important

A hash mark (#) at the beginning of a line indicates that the codes from the hash mark through the end-of-line code constitute a comment.

Specify one operand per line. The maximum length of a line is 2,048 characters.

The definition is checked when HA Monitor is started, when a server is started, and when the definition check command (moncheck command) is executed. If a definition error is detected, the startup processing is terminated.

If this definition file contains only comments or this definition file is missing, no definition error results.

(2) Definition procedure

You create the file that defines the locked server environment under the file name servers_opt in the directory for HA Monitor environment settings.

We recommend that you create the locked server environment definition file on a host that contains only standby servers.

The following shows a locked server environment definition file:

# Locked server environment definition : server alias : server alias
  exclusive_servers    :server-alias-name  :server-alias-name[:server-alias-name...]

The following explains the operands used to define locked servers.

(a) exclusive_servers

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

--

--

(Units: --)

Yes

--

Description

Specifies all the servers that are to be locked, expressed by the server alias names specified in the alias operand in the server environment definition.

To lock servers on a server group basis, specify the server alias names of all servers that belong to all groups to be locked.

If one of the specified servers becomes the active server, all the remaining specified servers that satisfy the following conditions are terminated:

The server is starting as a standby server, is on standby, is waiting for restart of the active server, is waiting for the active server to be started, is waiting for hot standby processing, or is waiting for grouped-system switchover.

The server is not defined in the group operand in the server environment definition under the same group name as the server that will become the active server.

The maximum number of server alias names that can be specified depends on the maximum number of servers that can be run on one host. The maximum number of server alias names that can be specified is shown below for each maximum number of servers that can be run per host. Note that the number of specifiable server alias names includes the resource server.

2 to 24 alias names when the maximum number of servers is 16

2 to 96 alias names when the maximum number of servers is 64

2 to 192 alias names when the maximum number of servers is 128 or more

You can specify more than one exclusive_servers operand.

If a server with a specified server alias name is already running as the active server, the other servers that are also specified can no longer be started. To start any of those servers, take either of the following actions:

Terminate the active server and then start the desired server.

Switch over the active server to the remote host by performing planned hot standby processing and then start the desired server.

8.5 Configuring an environment for programs in the monitor mode

The monitor-mode program environment settings specify definitions required for running programs that use the monitor-mode program management function.

You need to specify these environment settings only when you will be using the monitor-mode program management function.

8.5.1 Monitor-mode program environment definition (programs)

You create a definition file containing the monitor-mode program environment definitions under the file name programs in the directory for HA Monitor environment settings.

A sample programs file is provided under the directory for HA Monitor sample files. Instead of creating a new file from scratch, you can save time by copying this sample file to the directory for HA Monitor environment settings and then editing it.

You must create this definition file for each host. Bold type indicates a value that must be the same on all hosts. Care must be taken in specifying the other items so that there will be no conflicts between the hosts.

/*    monitor-mode program environment definition    */
program  name  monitor-mode-program-name
        ,alias monitor-mode-program-alias-name
        ,server_alias
               monitor-mode-server-alias-name
      [,restartcommand
               absolute-path-name-of-program-restart-command]
      [,patrol
               program-slowdown-monitoring-interval]
      [,restart_timeout
               program-restart-monitoring-time]
      [,exec_retry
             {program-restart-retry-count|2}]
      [,retry_stable
             {duration-of-stable-program-operation|60}];

(1) program definition statement

You use the program definition statement to define an operating environment for programs that use the program management function. The following explains the operands of the program definition statement.

(a) name

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 1,000 characters

--

(Units: --)

No

--

Description

Specifies a name used to identify the monitor-mode program.

You can also specify a path name. This name must be unique within each host. If this program is a UAP, specify the name specified in this operand in the HAMON_UAPNAME environment variable.

Specify the same value among all hosts in a hot-standby configuration.

(b) alias

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 8 alphanumeric characters

--

(Units: --)

No

--

Description

Specifies the monitor-mode program alias name that is to be used in HA Monitor commands and in the messages that are issued.

This name must be unique within each host. Specify the same value among all hosts in a hot-standby configuration.

(c) server_alias

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

1 to 8 alphanumeric characters

--

(Units: --)

No

--

Description

Specifies the server alias name for the monitor mode that is managed by this program (alias operand value in the server environment definition).

Specify the same value for all hosts in the hot-standby configuration.

(d) restartcommand

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Path name having 1 to 1,000 characters

--

(Units: --)

Yes

--#

#: If this operand is omitted, HA Monitor performs hot-standby switchover on a server basis in the event of a program error.

Description

Specifies the absolute path name of the program restart command that has been created by using a shell script, for example.

You must specify the same value for all hosts in the hot-standby configuration.

You can specify tab and end-of-line codes in this operand by enclosing the operand value in double quotation marks ("). The double quotation marks are not included in the character count.

(e) patrol

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

60 to 3600

(Units: seconds)

Yes

--#

#: If this operand is omitted, HA Monitor does not monitor for UAP operation reports.

Description

Specifies the amount of time HA Monitor is to monitor for an operation report from the UAP (program slowdown monitoring interval).

You must specify the same value for all hosts in the hot-standby configuration. If no operation report is received from the UAP within the amount of time specified in the patrol operand, HA Monitor detects a UAP error.

(f) restart_timeout

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

60 to 3600

(Units: seconds)

Yes

--

Description

Specifies a timeout value for program restart processing.

This is the amount of time from the start to the termination of a program restart command or to issuance of the hamon_patrolstart function. You must specify the same value for all hosts in the hot-standby configuration.

When a timeout occurs, HA Monitor re-executes the program restart command.

If the restartcommand operand is omitted, this operand is ignored.

(g) exec_retry

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

0 to 9999

(Units: number of times)

Yes

2#

#: If this operand is omitted, HA Monitor restarts the server according to the setting of the servexec_retry operand in the server environment definition.

Description

Specifies the number of times the program restart command is to be re-executed when a program error has been detected on the active server.

You must specify the same value for all hosts in the hot-standby configuration.

When HA Monitor detects a program error on the active server, it retries execution of the program restart command specified in the restartcommand operand as many times as specified in this operand. If the program restarts successfully within the retry count, HA Monitor treats it as a normal operation and continues processing. If the program does not restart within the specified number of retry attempts, HA Monitor treats it as a program error, stops the program restart processing, and performs hot standby processing.

If 9999 is specified in this operand, HA Monitor retries until program restart processing is completed or the active server is terminated.

If 0 is specified for this operand, HA Monitor immediately performs hot-standby switchover without attempting to restart the program.

If the restartcommand operand is omitted, specification of this operand is ignored.

(h) retry_stable

User-specified values

Value type

Range of numerical values

Can be omitted

Assumed value when this value is omitted

Unsigned integer

60 to 9999

(Units: seconds)

Yes

60

Description

Specifies a monitoring interval (in seconds) for determining whether program operation has become stable after the program has restarted successfully.

You must specify the same value for all hosts in the hot-standby configuration.

If no failure is detected within the amount of time specified in this operand after the program restarts, HA Monitor determines that the program's operation is stable and resets the restart count to 0. After a successful restart, HA Monitor starts monitoring for stable operation beginning at second 0.

If 9999 is specified in this operand, HA Monitor does not reset the retry count to 0 until the program terminates normally or is terminated by hot standby processing.

If an error occurs within the amount of time specified in this operand after the program has restarted, the restart count is updated and the program is restarted again. If the updated restart count exceeds the exec_retry operand value specified in the monitor-mode program environment definition, HA Monitor performs hot standby processing without restarting the program.

8.6 Examples of environment settings

This section explains the following examples of environment settings:

1-to-1 switchover configuration

Multi-host hot-standby configuration

VMware ESXi-based virtualization environment

Hot-standby configuration with a locked server specified

8.6.1 Examples of environment settings for a 1-to-1 switchover configuration

When a 1-to-1 switchover configuration is used, the settings to be specified in the definition files depend on the resources and functions that are used. This subsection explains typical examples of 1-to-1 switchover system configurations and the settings to be specified in the definition files for each system configuration.

(1) Example of environment settings when using independent servers

The following figure shows a system configuration for an example of environment settings when independent servers are used.

Figure 8‒1: System configuration when using independent servers

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when using independent servers)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name     /users/server1,
        alias    server1,
        acttype  server,
        patrol   60,
        initial  online,
        disk     /dev/vg01;

Environment settings for the secondary system (when using independent servers)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name     /users/server1,
        alias    server1,
        acttype  server,
        patrol   60,
        initial  standby,
        disk     /dev/vg01;

(2) Example of environment settings when grouping servers

The following figure shows a system configuration for an example of environment settings when servers are grouped.

Figure 8‒2: System configuration when grouping servers

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when grouping servers)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name     /users/server1,
        alias    server1,
        acttype  server,
        patrol   60,
        initial  online,
        group    groupA:exchange,
        disk     /dev/vg01;

/*  server environment definition (server 2)  */
server  name     /users/server2,
        alias    server2,
        acttype  server,
        patrol   60,
        initial  online,
        group    groupA:no_exchange:cancel,
        disk     /dev/vg02;

Environment settings for the secondary system (when grouping servers)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name     /users/server1,
        alias    server1,
        acttype  server,
        patrol   60,
        initial  standby,
        group    groupA:exchange,
        disk     /dev/vg01;

/*  server environment definition (server 2)  */
server  name     /users/server2,
        alias    server2,
        acttype  server,
        patrol   60,
        initial  standby,
        group    groupA:no_exchange:cancel,
        disk     /dev/vg02;

(3) Example of environment settings when using a mutual hot-standby configuration

The following figure shows a system configuration for an example of environment settings when using a mutual hot-standby configuration.

Figure 8‒3: System configuration when using a mutual hot-standby configuration

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when using a mutual hot-standby configuration)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name     /users/server1,
        alias    server1,
        acttype  server,
        patrol   60,
        initial  online,
        disk     /dev/vg01;

/*  server environment definition (server 2)  */
server  name     /users/server2,
        alias    server2,
        acttype  server,
        patrol   60,
        initial  standby,
        disk     /dev/vg02;

Environment settings for the secondary system (when using a mutual hot-standby configuration)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name     /users/server1,
        alias    server1,
        acttype  server,
        patrol   60,
        initial  standby,
        disk     /dev/vg01;

/*  server environment definition (server 2)  */
server  name     /users/server2,
        alias    server2,
        acttype  server,
        patrol   60,
        initial  online,
        disk     /dev/vg02;

(4) Example of environment settings when monitoring LANs

The following figure shows a system configuration for an example of environment settings when monitoring LANs.

Figure 8‒4: System configuration when monitoring LANs

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when monitoring LANs)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             lanfailswitch    use,
             lancheck_patrol  20;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name          /users/server1,
        alias         server1,
        acttype       server,
        patrol        60,
        initial       online,
        switchbyfail  hbond0;

LAN monitor definition file

LAN monitor definition file (definition file: /opt/hitachi/HAmon/etc/hbond0.conf)

192.168.0.110
192.168.0.101
192.168.0.102
192.168.0.103
nodeD

Environment settings for the secondary system (when monitoring LANs)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             lanfailswitch    use,
             lancheck_patrol  20;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name          /users/server1,
        alias         server1,
        acttype       server,
        patrol        60,
        initial       standby,
        switchbyfail  hbond0;

LAN monitor definition file

LAN monitor definition file (definition file: /opt/hitachi/HAmon/etc/hbond0.conf)

192.168.0.100
192.168.0.101
192.168.0.102
192.168.0.103
nodeD

(5) Example of environment settings when using file systems

The following figure shows a system configuration for an example of environment settings when using file systems.

Figure 8‒5: System configuration when using file systems

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when using file systems)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name         /users/server1,
        alias        server1,
        acttype      monitor,
        initial      online,
        disk         /dev/vg01,
        fs_name      /dev/lvol1:/dev/lvol2,
        fs_mount_dir /mnt/fs1:/mnt/fs2;

Environment settings for the secondary system (when using file systems)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name         /users/server1,
        alias        server1,
        acttype      monitor,
        initial      standby,
        disk         /dev/vg01,
        fs_name      /dev/lvol1:/dev/lvol2,
        fs_mount_dir /mnt/fs1:/mnt/fs2;

(6) Example of environment settings when controlling the server switchover order

The following figure shows a system configuration for an example of environment settings when controlling the server switchover order.

Figure 8‒6: System configuration when controlling the server switchover order

This example assumes the following configuration:

server1 and server2 are grouped together.

server1 is server2's parent server in the server start order during hot standby processing.

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when controlling the server switchover order)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name         /users/server1,
        alias        server1,
        acttype      server,
        patrol       60,
        initial      online,
        group        groupA:exchange,
        disk         /dev/vg01,
        lan_updown   use;

/*  server environment definition (server 2)  */
server  name         /users/server2,
        alias        server2,
        acttype      monitor,
        initial      online,
        group        groupA,
        disk         /dev/vg02,
        lan_updown   use,
        parent       server1;

LAN status settings files

server-alias-name.up (definition file: /opt/hitachi/HAmon/etc/server1.up)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 inet 1.2.3.4 netmask 255.255.255.0
/sbin/arping -U -c 2 -I eth1 1.2.3.4
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

server-alias-name.down (definition file: /opt/hitachi/HAmon/etc/server1.down)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 down
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

server-alias-name.up (definition file: /opt/hitachi/HAmon/etc/server2.up)

#!/bin/sh
set -x
/sbin/ifconfig eth1:2 inet 1.2.3.5 netmask 255.255.255.0
/sbin/arping -U -c 2 -I eth1 1.2.3.5
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

server-alias-name.down (definition file: /opt/hitachi/HAmon/etc/server2.down)

#!/bin/sh
set -x
/sbin/ifconfig eth1:2 down
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

Environment settings for the secondary system (when controlling the server switchover order)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name         /users/server1,
        alias        server1,
        acttype      server,
        patrol       60,
        initial      standby,
        group        groupA:exchange,
        disk         /dev/vg01,
        lan_updown   use;

/*  server environment definition (server 2)  */
server  name         /users/server2,
        alias        server2,
        acttype      monitor,
        initial      standby,
        group        groupA,
        disk         /dev/vg02,
        lan_updown   use,
        parent       server1;

LAN status settings files

server-alias-name.up (definition file: /opt/hitachi/HAmon/etc/server1.up)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 inet 1.2.3.4 netmask 255.255.255.0
/sbin/arping -U -c 2 -I eth1 1.2.3.4
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

server-alias-name.down (definition file: /opt/hitachi/HAmon/etc/server1.down)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 down
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

server-alias-name.up (definition file: /opt/hitachi/HAmon/etc/server2.up)

#!/bin/sh
set -x
/sbin/ifconfig eth1:2 inet 1.2.3.5 netmask 255.255.255.0
/sbin/arping -U -c 2 -I eth1 1.2.3.5
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

server-alias-name.down (definition file: /opt/hitachi/HAmon/etc/server2.down)

#!/bin/sh
set -x
/sbin/ifconfig eth1:2 down
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

(7) Example of environment settings when sharing resources among multiple servers

The following figure shows a system configuration for an example of environment settings when sharing resources among multiple servers.

Figure 8‒7: System configuration when sharing resources among multiple servers

This example assumes the following configuration:

The resource server, server1, and server2 are grouped together.

server1 and server2 share /dev/vg01 on the shared disk and alias IP address 1.2.3.4.

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when sharing resources among multiple servers)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name         /users/server1,
        alias        server1,
        acttype      server,
        patrol       60,
        initial      online,
        group        groupA:exchange,
        parent       rc0;

/*  server environment definition (server 2)  */
server  name         /users/server2,
        alias        server2,
        acttype      monitor,
        initial      online,
        group        groupA,
        parent       rc0;

/*  resource server environment definition  */
resource  alias        rc0,
          initial      online,
          group        groupA,
          disk         /dev/vg01,
          lan_updown   use;

LAN status settings files

server-alias-name.up (definition file: /opt/hitachi/HAmon/etc/rc0.up)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 inet 1.2.3.4 netmask 255.255.255.0
/sbin/arping -U -c 2 -I eth1 1.2.3.4
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

server-alias-name.down (definition file: /opt/hitachi/HAmon/etc/rc0.down)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 down
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

Environment settings for the secondary system (when sharing resources among multiple servers)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name         /users/server1,
        alias        server1,
        acttype      server,
        patrol       60,
        initial      standby,
        group        groupA:exchange,
        parent       rc0;

/*  server environment definition (server 2)  */
server  name         /users/server2,
        alias        server2,
        acttype      monitor,
        initial      standby,
        group        groupA,
        parent       rc0;

/*  resource server environment definition  */
resource  alias        rc0,
          initial      standby,
          group        groupA,
          disk         /dev/vg01,
          lan_updown   use;

LAN status settings files

server-alias-name.up (definition file: /opt/hitachi/HAmon/etc/rc0.up)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 inet 1.2.3.4 netmask 255.255.255.0
/sbin/arping -U -c 2 -I eth1 1.2.3.4
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

server-alias-name.down (definition file: /opt/hitachi/HAmon/etc/rc0.down)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 down
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

(8) Example of environment settings when using the monitor-mode program management function

The following figure shows a system configuration for an example of environment settings when the monitor-mode program management function is used.

Figure 8‒8: System configuration when using the monitor-mode program management function

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when using the monitor-mode program management function)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2,
             pgmmax           10;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name     /users/server1,
        alias    server1,
        acttype  monitor,
        initial  online,
        disk     /dev/vg01,
        program  use;

Monitor-mode program environment definition (definition file: /opt/hitachi/HAmon/etc/programs)

/*  monitor-mode program environment definition  */
program  name           UAP1,
         alias          UAP1,
         server_alias   server1,
         patrol         180;
program  name           UAP2,
         alias          UAP2,
         server_alias   server1,
         patrol         180;

Environment settings for the secondary system (when using the monitor-mode program management function)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2,
             pgmmax           10;
function     pathpatrol       1,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name     /users/server1,
        alias    server1,
        acttype  monitor,
        initial  standby,
        disk     /dev/vg01,
        program  use;

Monitor-mode program environment definition (definition file: /opt/hitachi/HAmon/etc/programs)

/*  monitor-mode program environment definition  */
program  name           UAP1,
         alias          UAP1,
         server_alias   server1,
         patrol         180;
program  name           UAP2,
         alias          UAP2,
         server_alias   server1,
         patrol         180;

(9) Example of environment settings when using SCSI reservation for shared disk (single-path configuration or a redundant configuration using multipath software)

The following figure shows a system configuration for an example of environment settings when using SCSI reservation for shared disk with a single-path configuration or a redundant configuration using multipath software.

Figure 8‒9: System configuration when using SCSI reservation for shared disk

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when using SCSI reservation for shared disk)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             fence_reset      nouse#,
             fence_scsi       use;

#: To use hybrid fencing, specify use (rather than nouse) for the fence_reset operand.

Server environment definition (when using a single-path configuration or when specifying symbolic links in a redundant configuration) (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name        /users/server1,
        alias       server1,
        acttype     server,
        patrol      60,
        initial     online,
        disk        /dev/vg01,
        scsi_device /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065;

Server environment definition (redundant configuration using multipath software (HDLM)) (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name        /users/server1,
        alias       server1,
        acttype     server,
        patrol      60,
        initial     online,
        disk        /dev/vg01,
        scsi_device /dev/sddlmaa;

Server environment definition (redundant configuration using multipath software (DMMP)) (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name        /users/server1,
        alias       server1,
        acttype     server,
        patrol      60,
        initial     online,
        disk        /dev/vg01,
        dmmp_device /dev/mapper/mpath1;

Environment settings for the secondary system (when using SCSI reservation for shared disk)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             fence_reset      nouse,
             fence_scsi       use;

Server environment definition (when using a single-path configuration or when specifying symbolic links in a redundant configuration) (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name        /users/server1,
        alias       server1,
        acttype     server,
        patrol      60,
        initial     standby,
        disk        /dev/vg01,
        scsi_device /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065;

Server environment definition (redundant configuration using multipath software (HDLM)) (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name        /users/server1,
        alias       server1,
        acttype     server,
        patrol      60,
        initial     standby,
        disk        /dev/vg01,
        scsi_device /dev/sddlmaa;

Server environment definition (redundant configuration using multipath software (DMMP)) (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name        /users/server1,
        alias       server1,
        acttype     server,
        patrol      60,
        initial     standby,
        disk        /dev/vg01,
        dmmp_device /dev/mapper/mpath1;

(10) Example of environment settings when using the function for controlling hot standby based on the availability of LAN communications

The following figure shows a system configuration for an example of the environment settings when using the function for controlling hot standby based on the availability of LAN communications.

Figure 8‒10: System configuration when the function for controlling hot standby based on the availability of LAN communications is used

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when using the function for controlling hot standby based on the availability of LAN communications)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment name             HAmon1
            address          1,
            patrol           60,
            lan              path11: path12,
            lanport          HAmon1:HAmon2:
function    pathpatrol       1,
            pathpatrol_retry 3:3,
            alive_interval   1,
            patrol_100ms     use,
            fence_reset      nouse,
            fence_scsi       nouse,
            fence_lan        use,
            lanfailswitch    use,
            lancheck_patrol  15,
            :

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server name         /users/server1,
       alias        server1,
       acttype      monitor,
       initial      online,
       switch_judge bond0,
       switchbyfail bond0,
       :

LAN monitor definition file

LAN monitor definition file (definition file: /opt/hitachi/HAmon/etc/bond0.conf)

192.168.0.110
192.168.0.101

Environment settings for the secondary system (when using the function for controlling hot standby based on the availability of LAN communications)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment name             HAmon2
            address          2,
            patrol           60,
            lan              path21: path22,
            lanport          HAmon1:HAmon2:
function    pathpatrol       1,
            pathpatrol_retry 3:3,
            alive_interval   1,
            patrol_100ms     use,
            fence_reset      nouse,
            fence_scsi       nouse,
            fence_lan        use,
            lanfailswitch    use,
            lancheck_patrol  15,
            :

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server name         /users/server1,
       alias        server1,
       acttype      monitor,
       initial      standby,
       switch_judge bond0,
       switchbyfail bond0,
            :

LAN monitor definition file

LAN monitor definition file (definition file: /opt/hitachi/HAmon/etc/bond0.conf)

192.168.0.100
192.168.0.101

(11) Example of environment settings in the case where the server model is HA8000xN or later and a management server is not used as a failure management processor

The following figure shows a system configuration for an example of environment settings in the case where the server model is HA8000xN or later and a management server is not used as a failure management processor.

Figure 8‒11: System configuration in the case where the server model is HA8000xN or later and a management server is not used as a failure management processor

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (in the case where the server model is HA8000xN or later and a management server is not used as a failure management processor)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             clearwait        35;
options      clearcheck       35;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name     /users/server1,
        alias    server1,
        acttype  server,
        patrol   60,
        initial  online,
        disk     /dev/vg01;

Environment settings for the secondary system (in the case where a management server is not used as a failure management processor with model HA8000xN or later)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             clearwait        35;
options      clearcheck       35;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name     /users/server1,
        alias    server1,
        acttype  server,
        patrol   60,
        initial  standby,
        disk     /dev/vg01;

(12) Example of environment settings in the case where hot-standby switchover based on hybrid fencing is used

For an example of environment settings in the case where hot-standby switchover based on hybrid fencing is used, see (9) Example of environment settings when using SCSI reservation for shared disk (single-path configuration or a redundant configuration using multipath software). Note that for the fence_reset operand, you must specify use although nouse is specified in the example shown.

8.6.2 Examples of environment settings for a multi-host hot-standby configuration

When a multi-host hot-standby configuration is used, the settings to be specified in the definition files depend on the configuration used, such as a 2-to-1 switchover configuration or a cluster hot-standby configuration. This subsection explains typical examples of multi-host hot-standby configurations and the settings to be specified in the definition files for each system configuration.

When the system configuration consists of multiple hosts, such as 2-to-1 switchover configuration or a cluster hot-standby configuration, specify the HA Monitor environment settings for each host and a server environment definition for each server.

The HA Monitor environment settings must be the same on all hosts.

The server environment definitions must be consistent between paired servers (active server and standby server).

A shared disk that is shared between active and standby servers is connected to a volume group with the same special file name on the paired servers. Therefore, specify the same special file name on the paired active and standby servers.

If the same special file name is specified for an active server and a standby server that are not paired, hot standby processing might not be performed correctly. For shared resources, do not specify the same value for the active and standby servers unless these servers are paired.

(1) Example of environment settings when using a 2-to-1 switchover configuration

The following figure shows a system configuration for an example of environment settings when using a 2-to-1 switchover configuration.

Figure 8‒12: System configuration when using a 2-to-1 switchover configuration

The following table shows the prerequisites for this example system configuration.

Table 8‒7: Prerequisites for environment settings (HA Monitors are in 2-to-1 switchover configuration)

Prerequisite

Primary system 1

Secondary system

Primary system 2

Host name

host1

host2

host3

Host address

1

2

3

Host failure monitoring time

60 seconds

60 seconds

60 seconds

Server failure monitoring time

60 seconds

60 seconds

60 seconds

Host that has reset priority

online

online

online

Monitoring path health check interval

120 minutes

120 minutes

120 minutes

Reset path health check interval

2 minutes

2 minutes

2 minutes

Table 8‒8: Prerequisites for environment settings (servers are in 2-to-1 switchover configuration)

Prerequisite

Server 1

Server 2

Primary system 1

Secondary system

Secondary system

Primary system 2

Program name

/users/server1

/users/server1

/users/server2

/users/server2

Server alias name

server1

server1

server2

server2

Server start method

server

server

server

server

Start type

online

standby

standby

online

An example of the environment settings for this system configuration is shown below. Bold type indicates a value that must be the same in all HA Monitors and paired servers on all hosts.

Environment settings for primary system 1

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     cpudown          online,
             pathpatrol       120,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name         /users/server1,
        alias        server1,
        acttype      server,
        patrol       60,
        initial      online,
        disk         /dev/vg01;

Environment settings for secondary system

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     cpudown          online,
             pathpatrol       120,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name         /users/server1,
        alias        server1,
        acttype      server,
        patrol       60,
        initial      standby,
        disk         /dev/vg01;

/*  server environment definition (server 2)  */
server  name         /users/server2,
        alias        server2,
        acttype      server,
        patrol       60,
        initial      standby,
        disk         /dev/vg02;

Environment settings for primary system 2

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host3,
             address          3,
             patrol           60,
             lan              path31:path32,
             lanport          HAmon1:HAmon2;
function     cpudown          online,
             pathpatrol       120,
             pathpatrol_retry 3:3;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 2)  */
server  name         /users/server2,
        alias        server2,
        acttype      server,
        patrol       60,
        initial      online,
        disk         /dev/vg02;

(2) Example of environment settings when using a multi-standby configuration (TP1/Server Base or HiRDB)

The following figure shows a system configuration for an example of environment settings using TP1/Server Base or HiRDB as server-mode servers to create a multi-standby configuration.

Figure 8‒13: System configuration when using a multi-standby configuration (TP1/Server Base or HiRDB)

The following table shows the prerequisites for this example system configuration.

Table 8‒9: Prerequisites for environment settings (HA Monitors in a multi-standby configuration using TP1/Server Base or HiRDB)

Prerequisite

Primary system

Secondary system 1

Secondary system 2

Host name

host1

host2

host3

Host address

1

2

3

Host failure monitoring time

60 seconds

60 seconds

60 seconds

Server failure monitoring time

60 seconds

60 seconds

60 seconds

Host that has reset priority

online

online

online

Monitoring path health check interval

1 minute

1 minute

1 minute

Monitoring path recheck interval

3 seconds

3 seconds

3 seconds

Monitoring path recheck count

3

3

3

Whether the multi-standby function is used

Used

Used

Used

Table 8‒10: Prerequisites for environment settings (servers in a multi-standby configuration using TP1/Server Base or HiRDB)

Prerequisite

Primary system

Secondary system 1

Secondary system 2

Program name

/users/server1

/users/server1

/users/server1

Server alias name

server1

server1

server1

Server start method

server

server

server

Start type

online

standby

standby

Whether LAN status settings files are used

Used

Used

Used

Standby server priority

Not specified.

1

2

An example of the environment settings for this system configuration is shown below. Bold type indicates a value that must be the same in all HA Monitors and paired servers on all hosts.

Environment settings for the primary system

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             multistandby     use;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name            /users/server1,
        alias           server1,
        acttype         server,
        patrol          60,
        initial         online,
        disk            /dev/vg01,
        lan_updown      use;

LAN status settings files

server-alias-name.up (definition file: /opt/hitachi/HAmon/etc/server1.up)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 inet 1.2.3.4 netmask 255.255.255.0 broadcast 1.2.3.255
/sbin/arping -U -c 2 -I eth1 1.2.3.4
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

server-alias-name.down (definition file: /opt/hitachi/HAmon/etc/server1.down)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 down
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

Environment settings for secondary system 1

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             multistandby     use;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name            /users/server1,
        alias           server1,
        acttype         server,
        patrol          60,
        initial         standby,
        disk            /dev/vg01,
        lan_updown      use,
        standbypri      1;

LAN status settings files

server-alias-name.up (definition file: /opt/hitachi/HAmon/etc/server1.up)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 inet 1.2.3.4 netmask 255.255.255.0 broadcast 1.2.3.255
/sbin/arping -U -c 2 -I eth1 1.2.3.4
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

server-alias-name.down (definition file: /opt/hitachi/HAmon/etc/server1.down)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 down
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

Environment settings for secondary system 2

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host3,
             address          3,
             patrol           60,
             lan              path31:path32,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             multistandby     use;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name            /users/server1,
        alias           server1,
        acttype         server,
        patrol          60,
        initial         standby,
        disk            /dev/vg01,
        lan_updown      use,
        standbypri      2;

LAN status settings files

server-alias-name.up (definition file: /opt/hitachi/HAmon/etc/server1.up)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 inet 1.2.3.4 netmask 255.255.255.0 broadcast 1.2.3.255
/sbin/arping -U -c 2 -I eth1 1.2.3.4
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

server-alias-name.down (definition file: /opt/hitachi/HAmon/etc/server1.down)

#!/bin/sh
set -x
/sbin/ifconfig eth1:1 down
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

(3) Example of environment settings when using a cluster hot-standby configuration

The following figure shows a system configuration for an example of environment settings when using a cluster hot-standby configuration.

Figure 8‒14: System configuration when a cluster hot-standby configuration is used

The following table shows the prerequisites for this example system configuration.

Table 8‒11: Prerequisites for environment settings (HA Monitors in a cluster hot-standby configuration)

Prerequisite

Primary system 1

Secondary system 1

Primary system 2

(secondary system 3)

Primary system 3

(secondary system 2)

Host name

host1

host2

host3

host4

Host address

1

2

3

4

Host failure monitoring time

60 seconds

60 seconds

60 seconds

60 seconds

Server failure monitoring time

60 seconds

60 seconds

60 seconds

60 seconds

Host that has reset priority

online

online

online

online

Monitoring path health check interval

120 minutes

120 minutes

120 minutes

120 minutes

Reset path health check interval

2 minutes

2 minutes

2 minutes

2 minutes

Table 8‒12: Prerequisites for environment settings (servers in a cluster hot-standby configuration)

Prerequisite

Server 1

Server 2

Server 3

Primary system 1

Secondary system 1

Primary system 2

(secondary system 3)

Primary system 3

(secondary system 2)

Primary system 2

(secondary system 3)

Primary system 3

(secondary system 2)

Program name

/users/server1

/users/server1

/users/server2

/users/server2

/users/server3

/users/server3

Server alias name

server1

server1

server2

server2

server3

server3

Server start method

server

server

server

server

server

server

Start type

online

standby

online

standby

standby

online

An example of the environment settings for this system configuration is shown below. Bold type indicates a value that must be the same in all HA Monitors and paired servers on all hosts.

Environment settings for primary system 1

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     cpudown          online,
             pathpatrol       120;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name         /users/server1,
        alias        server1,
        acttype      server,
        patrol       60,
        initial      online,
        disk         /dev/vg01;

Environment settings for secondary system 1

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     cpudown          online,
             pathpatrol       120;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 1)  */
server  name         /users/server1,
        alias        server1,
        acttype      server,
        patrol       60,
        initial      standby,
        disk         /dev/vg01;

Environment settings for primary system 2 (secondary system 3)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host3,
             address          3,
             patrol           60,
             lan              path31:path32,
             lanport          HAmon1:HAmon2;
function     cpudown          online,
             pathpatrol       120;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 2)  */
server  name         /users/server2,
        alias        server2,
        acttype      server,
        patrol       60,
        initial      online,
        disk         /dev/vg02;

/*  server environment definition (server 3)  */
server  name         /users/server3,
        alias        server3,
        acttype      server,
        patrol       60,
        initial      standby,
        disk         /dev/vg03;

Environment settings for primary system 3 (secondary system 2)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host4,
             address          4,
             patrol           60,
             lan              path41:path42,
             lanport          HAmon1:HAmon2;
function     cpudown          online,
             pathpatrol       120;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition (server 2)  */
server  name         /users/server2,
        alias        server2,
        acttype      server,
        patrol       60,
        initial      standby,
        disk         /dev/vg02;

/*  server environment definition (server 3)  */
server  name         /users/server3,
        alias        server3,
        acttype      server,
        patrol       60,
        initial      online,
        disk         /dev/vg03;

8.6.3 Examples of environment settings for a VMware ESXi-based virtualization environment

When a VMware ESXi-based virtualization environment is used, the settings to be specified in the definition files depend on the resources and functions that are used. This subsection explains typical examples of system configurations for VMware ESXi-based virtualization environments and the settings to be specified in the definition files for each system configuration.

(1) Example of environment settings when resetting hosts

The following figure shows a system configuration for an example of environment settings when resetting hosts.

Figure 8‒15: System configuration when resetting hosts

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when resetting hosts)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             vmware_env       use,
             fence_reset      use,
             fence_scsi       nouse;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name        /users/server1,
        alias       server1,
        acttype     server,
        patrol      60,
        initial     online,
        disk        /dev/vg01;

Environment settings for the secondary system (when resetting hosts)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             vmware_env       use,
             fence_reset      use,
             fence_scsi       nouse;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name        /users/server1,
        alias       server1,
        acttype     server,
        patrol      60,
        initial     standby,
        disk        /dev/vg01;

(2) Example of environment settings when using SCSI reservation for shared disk (redundant configuration using multipath software)

The following figure shows a system configuration for an example of the environment settings when using SCSI reservation for shared disk in a redundant configuration using multipath software.

Figure 8‒16: System configuration when using SCSI reservation for shared disk (redundant configuration using multipath software)

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when using SCSI reservation for shared disk (redundant configuration using multipath software))

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             vmware_env       nouse,
             fence_reset      nouse,
             fence_scsi       use;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name        /users/server1,
        alias       server1,
        acttype     server,
        patrol      60,
        initial     online,
        disk        /dev/vg01,
        scsi_device /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065;

Environment settings for the secondary system (when using SCSI reservation for shared disk (redundant configuration using multipath software))

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             vmware_env       nouse,
             fence_reset      nouse,
             fence_scsi       use;

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name        /users/server1,
        alias       server1,
        acttype     server,
        patrol      60,
        initial     standby,
        disk        /dev/vg01,
        scsi_device /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000065;

(3) Example of environment settings using the function for controlling hot standby based on the availability of LAN communications

The following figure shows a system configuration for an example of environment settings when using the function for controlling hot standby based on the availability of LAN communications.

Figure 8‒17: System configuration using the function for controlling hot standby based on the availability of LAN communications

The following shows an example of the environment settings for the primary system and the secondary system in this system configuration.

Environment settings for the primary system (when using the function for controlling hot standby based on the availability of LAN communications)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host1,
             address          1,
             patrol           60,
             lan              path11:path12,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             fence_reset      nouse,
             fence_scsi       nouse,
             fence_lan        use,
             lanfailswitch    use,
             lancheck_patrol  15;
            :

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name         /users/server1,
        alias        server1,
        acttype      monitor,
        initial      online,
        switch_judge eth1,
        switchbyfail eth1,
            :

LAN monitor definition file

LAN monitor definition file (definition file: /opt/hitachi/HAmon/etc/eth1.conf)

192.168.0.110
192.168.0.101

Environment settings for the secondary system (when using the function for controlling hot standby based on the availability of LAN communications)

HA Monitor environment settings (definition file: /opt/hitachi/HAmon/etc/sysdef)

/*  HA Monitor environment settings  */
environment  name             host2,
             address          2,
             patrol           60,
             lan              path21:path22,
             lanport          HAmon1:HAmon2;
function     pathpatrol       1,
             pathpatrol_retry 3:3,
             fence_reset      nouse,
             fence_scsi       nouse,
             fence_lan        use,
             lanfailswitch    use,
             lancheck_patrol  15;
            :

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers)

/*  server environment definition  */
server  name         /users/server1,
        alias        server1,
        acttype      monitor,
        initial      standby,
        switch_judge eth1,
        switchbyfail eth1,
            :

LAN monitor definition file

LAN monitor definition file (definition file: /opt/hitachi/HAmon/etc/eth1.conf)

192.168.0.100
192.168.0.101

8.6.4 Examples of environment settings for a hot-standby configuration with a locked server specified

This subsection explains examples of the settings to be specified in the definition files when a locked server is specified for each lock relationship on the locked server.

(1) Example of definition when servers have a lock relationship with each other

The following figure shows an example of the environment settings when servers have a lock relationship with each other.

Figure 8‒18: Example of environment settings when servers have a lock relationship with each other

(2) Example of definition when groups have a lock relationship with each other

The following figure shows an example of the environment settings when server groups have a lock relationship with each other.

Figure 8‒19: Example of environment settings when server groups have a lock relationship with each other

(3) Example of definition when some of the servers in one group have a lock relationship with some of the servers in another group

The following figure shows an example of the environment settings when some of the servers in one group have a lock relationship with some of the servers in another group.

Figure 8‒20: Example of environment settings when some of the servers in one group have a lock relationship with some of the servers in another group

(4) Example of definition when servers have (multiple) lock relationships

The following figure shows an example of environment settings when servers have (multiple) lock relationships.

Figure 8‒21: Example of environment settings when servers have (multiple) lock relationships

(5) Example of definition when groups have (multiple) lock relationships

The following figure shows an example of environment settings when server groups have (multiple) lock relationships.

Figure 8‒22: Example of environment settings when groups have (multiple) lock relationships

(6) Example of definition when a group has a lock relationship with individual servers

The following figure shows an example of environment settings when a server group has a lock relationship with individual servers.

Figure 8‒23: Example of environment settings when a group has a lock relationship with individual servers

9 Commands

This chapter explains the commands used in HA Monitor.

9.1 List of commands

The following table lists and describes the commands used in HA Monitor.

Table 9‒1: List of HA Monitor commands

Command name

Description

Executor

Superuser

General user

Commands for configuring and changing the environment

monsetup

Specifies HA Monitor environment settings.

Y

N

monlistip

Detects IP addresses.

Y

N

moncheck

Checks definitions.

Y

Y

mondevice

Changes shared resources while the active server is running.

Y

N

monchange

Changes settings while HA Monitor and servers are running.

Y

N

HA Monitor operation commands

monstart

Starts HA Monitor.

Y

Y

monstop

Terminates HA Monitor.

Y

Y

monlink

Connects HA Monitors manually.

Y

Y

Server operation commands

monbegin

Starts a server in the monitor mode.

Y

Y

monend

Terminates the active server in the monitor mode.

Y

Y

monsbystp

Terminates a standby server.

Y

N

monresbgn

Starts a resource server.

Y

Y

monresend

Terminates an active resource server.

Y

Y

monressbystp

Terminates a standby resource server.

Y

N

monact

Starts a server in the wait state as the active server.

Y

N

mondeact

Terminates a server in the wait state.

Y

N

Status checking commands

monshow

Displays the status of servers and hosts.

Y

Y

monrp

Displays the status of the reset path.

Y

N

monpath

Displays the status of monitoring paths.

Y

N

monodrshw

Displays the server switchover order control status.

Y

Y

Command for performing hot standby processing

monswap

Performs planned hot-standby switchover or forced hot-standby switchover.

Y

N

Command issued from within user commands

moninfo

Specifies server information in the active system that is to be inherited, or references and displays inherited server information in the standby system.

Y

Y

Command for collecting troubleshooting information

monts

Collects HA Monitor troubleshooting information.

Y

N

Command for manipulating shared resources

monscsiclr

Releases reservations for shared disks.

Y

N

Legend:

Y: Can execute

N: Cannot execute

9.2 Headings used in the explanations of commands

This section explains the headings used in the explanations of the commands. Only the applicable heading items are provided for each command. For some commands, command-specific information is also provided in addition to these items.

Format

Shows the command's format, using syntax symbols. For details about the syntax symbols, see 9.3 Syntax symbols in 9. Commands.

Users who can execute the command

Shows the types of users who are authorized to execute the command.

Description

Provides a description of the command.

Options

Describes the options and provides notes about specifying the options.

Return codes

Explains the command's return codes.

Notes

Provides notes about executing the command.

Examples

Provides examples of executing the command.

The results displayed after an example of a command's execution might include message IDs. For details about message IDs displayed with results, see the manual HA Monitor Cluster Software Messages.

9.3 Syntax symbols

This section explains the syntax symbols used in the explanations of commands' specification formats. The following table lists and describes the syntax symbols.

Table 9‒2: List of command syntax symbols

Syntax symbol

Meaning

[ ]

Square brackets indicate that specification of the enclosed item or items is optional. For example:

monshow [server-alias-name] means that either a server alias name or nothing is to be specified in the monshow command.

{ }

Curly brackets indicate that only one of the enclosed items must be selected for specification. For example:

monodrshw { -s server-alias-name | -g group-name } means that either -s server-alias-name or -g group-name must be specified in the monodrshw command.

|

A vertical bar separates multiple items that are enclosed in curly brackets. For example:

moninfo server-alias-name { -p inherited-server-information | -g } means that either -p inherited-server-information or -g must be specified after server-alias-name in the moninfo command.

9.4 monact (starts a server in the wait state as the active server)

Format

monact server-alias-name

Users who can execute the command

Superuser

Description

The monact command starts one of the following types of servers as the active server:

Standby server placed in the active server start wait state

If a server defined in the server environment definition as a standby server cannot check the status of the active system during startup processing due to an error (such as a communication error), it is placed in the wait state until the active server in the active system has started. This command forcibly starts such a standby server that is in the wait state as the active server.

Standby server placed in the grouped-system switchover wait state

If a failure occurs on a server for which no_exchange is specified in the group operand in the server environment definition, the standby servers in the standby system are placed in the grouped-system switchover wait state. This command forcibly starts such a standby server that is in the wait state as the active server.

Active server placed in the hot-standby wait state

If host reset fails, the standby server in the standby system is switched in as the active server but then is placed in the hot-standby wait state. This command forcibly starts this active server that is in the wait state as the active server.

Standby server started in the online serverless mode (startup complete state)

This command starts a standby server started in the online serverless mode as the active server.

Options

server-alias-name

Specifies the server alias name of the server that is to be started as the active server.

Return codes

Value

Meaning

0

Normal termination (the request to start as the active server has been accepted)

Other than 0

Abnormal termination

Notes

Before you execute the monact command, make sure that no active server is running in the remote host. If you execute the monact command without checking for an active server in the remote host, you might end up with two active servers running at the same time.

You can execute the monact command only on servers in the following statuses:

Standby server in the active server start wait state

Standby server in the grouped-system switchover wait state

Active server in the hot-standby wait state

Standby server started in the online serverless mode

The monact command cannot be executed on a server in the resource server start wait state.

The return code is also set to 0 when any of the following messages is issued while type2 is specified in the exitcode operand in the HA Monitor environment settings:

KAMN227-E, KAMN322-E, KAMN339-E, KAMN456-E, KAMN465-E

The active server does not start at the same time that the monact command's return code is received.

9.5 monbegin (starts a server in the monitor mode)

Format

monbegin server-alias-name

Users who can execute the command

Superuser and general user

Description

The monbegin command starts a server in the monitor mode.

Options

server-alias-name

Specifies the server alias name of the server that is to be started.

Return codes

Value

Meaning

0

Normal termination (the request to start a server in the monitor mode has been accepted)

Other than 0

Abnormal termination

Notes

The monbegin command can be used only for servers in the monitor mode.

The server does not start at the same time that the monbegin command's return code is received.

Upon receiving a server start request, the monbegin command outputs the KAMN376-I message, and returns the return value 0 regardless of the status of the server specified by a server alias name.

9.6 monchange (changes settings and operations while HA Monitor and servers are running)

Format

Changing an HA Monitor setting:

monchange -m operand-name value

Changing the operations of monitoring the system disk:

monchange –m operand-name {-stop_patrol | -start_patrol}

Setting and releasing the online serverless mode:

monchange –m operand-name {-use | -nouse}

Changing a server setting:

monchange -s server-alias-name operand-name value

Changing the operations of monitoring servers:

monchange –s server-alias-name operand-name {-stop_patrol | -start_patrol}

Changing a monitor-mode program setting:

monchange -p monitor-mode-program-alias-name operand-name value

Users who can execute the command

Superuser

Description

While HA Monitor and servers are running, you can change the following settings and operations:

HA Monitor setting:

You can change the host failure monitoring time specified in the patrol operand in the HA Monitor environment settings.

Operation of system disk monitoring:

You can temporarily stop or restart the monitoring of the system disk specified in the disk_ptrl operand in the HA Monitor environment settings. When the monitoring of the system disk is restarted, the monitoring definition file for the system disk that currently exists takes effect.

Setting and releasing the online serverless mode:

You can set or release the online serverless mode.

To set the online serverless mode, all servers must be terminated. If setting of the online serverless mode is successful, the KAMN545-I message is output. If the setting fails, the KAMN547-E message is output.

To release the online serverless mode, all servers must be placed in one of the following statuses:

The server has terminated.

The active server has finished startup or is ready for hot-standby switchover (server status: ONL).

The standby server is ready for hot-standby switchover (server status: SBY).

If releasing of the online serverless mode is successful, the KAMN546-I message is output. If the releasing fails, the KAMN548-E message is output.

Changing a server setting:

You can change the server failure monitoring time specified in the patrol operand in the server environment definition.

Operation of server monitoring:

You can temporarily stop or restart the monitoring of the monitor-mode server specified in the ptrlcmd_ex or sby_ptrlcmd_ex operand in the server environment definition. You can also temporarily stop or restart the monitoring of the disks for business use on the server specified in the disk_ptrl operand in the server environment definition. When the monitoring of disks for business use on the server is restarted, the monitoring definition file for the disk for business use that currently exists takes effect.

Changing a monitor-mode program setting:

You can change the program slowdown monitoring interval specified in the patrol operand in the monitor-mode program environment definition.

Options

Changing the HA Monitor setting:

-m

Specifies that the HA Monitor setting is to be changed.

operand-name

Specifies patrol as the name of the operand whose HA Monitor setting to be changed.

value

Specifies a new value for the patrol operand. For details about the patrol operand value, see 8.3.1 HA Monitor environment settings (sysdef).

Changing the method of monitoring the system disk:

-m

Specify this option when changing the method of monitoring the system disk.

operand-name

Specify disk_ptrl as the name of the operand for the monitoring of the system disk.

value

To temporarily stop the monitoring, specify -stop_patrol. To restart the monitoring that is temporarily stopped, specify -start_patrol.

Setting and releasing the online serverless mode:

-m

Specify this option when setting or releasing the online serverless mode.

operand-name

Specifies onlless_mode as the name of the operand setting or releasing the online serverless mode.

value

Specify -use to set the online serverless mode. Specify -nouse to release the online serverless mode.

Changing a server setting:

-s

Specifies that the server setting is to be changed.

server-alias-name

Specifies the server alias name of the server whose setting is to be changed.

operand-name

Specifies patrol as the name of the operand whose server setting is to be changed.

value

Specifies a new value for the patrol operand. For details about the patrol operand value, see 8.4.1 Server environment definition (servers).

Changing the method of server monitoring:

-s

Specify this option when changing the method of server monitoring.

server-alias-name

Specifies the server alias name of the server for which you want to change the method of monitoring.

operand-name

The value to be specified differs depending on the monitoring target for which you want to change the method of monitoring. Specify one of the following values:

If the monitoring target is the active server, specify ptrlcmd_ex.

If the monitoring target is the standby server, specify sby_ptrlcmd_ex.

value

To temporarily stop the monitoring, specify -stop_patrol. To restart the monitoring that is temporarily stopped, specify -start_patrol.

Changing a monitor-mode program setting:

-p

Specifies that a setting for a monitor-mode program that uses the program management function is to be changed.

monitor-mode-program-alias-name

Specifies the alias name of the monitor-mode program whose setting is to be changed.

operand-name

Specifies patrol as the name of the operand whose value is to be changed for the monitor-mode program.

value

Specifies a new value for the patrol operand. For details about the patrol operand value, see 8.5.1 Monitor-mode program environment definition (programs).

Return codes

Value

Meaning

0

Normal termination (Changing the settings and operations while HA Monitor and servers are running was completed.)

Other than 0

Abnormal termination

Notes

The monchange command does not change the contents of the definition file. When you have used this command to change a setting and then restart HA Monitor, the setting reverts to its initial value. If necessary, change the environment setting and then restart HA Monitor.

In the following cases, temporary monitoring stoppage executed by the monchange command is canceled and the monitoring is restarted:

When the host or server is restarted

When hot-standby switchover is performed

The following notes apply to changing the HA Monitor setting:

Before you execute the command, make sure that all hosts on which the setting is to be changed are connected.

You change the host failure monitoring time for all connected hosts when you execute the monchange command on any one of the hosts.

Do not execute this command from multiple hosts at the same time.

If you have restarted a host after changing its host failure monitoring time or if the monitoring time no longer matches between hosts due to an error such as a communication error, re-execute the monchange command to synchronize the host failure monitoring time between hosts.

If you collect host monitoring history, specify a value that is greater than the current value of the host's monitoring history time. If the specified value is smaller than the host's monitoring history time, monitoring history will not be collected.

The following notes apply to changing the method of monitoring the system disk:

The monchange command changes the method of monitoring the system disk on only the host on which the command is executed.

After executing the monchange command, to check whether the method of monitoring the system disk has changed, execute monshow -o -m.

If use is specified for the notswitch_notify operand in the HA Monitor environment settings, the KAMN540-W message is output each time one hour elapses in the status in which the monitoring is temporarily stopped.

The following notes apply when setting or releasing the online serverless mode:

The monchange command sets or releases the online serverless mode only for the host on which the command is executed.

After executing the monchange command, to check whether the online serverless mode was set or released, execute monshow -j.

When HA Monitor stops in the online serverless mode, the online serverless mode is released. If necessary, set the online serverless mode again after restarting HA Monitor.

If you are using the multi-standby function (use is specified in the multistandby operand), you cannot set the online serverless mode. In this case, the KAMN547-E message is output.

The following notes apply to changing the server setting:

The monchange command changes the server failure monitoring time only for the host on which the command is executed. You must execute the command separately on the host where the standby server is running.

If you collect server monitoring history, specify a value that is greater than the current value of the server's monitoring history time. If the specified value is smaller than the server's monitoring history time, monitoring history will not be collected.

The following notes apply to changing the operation of server monitoring:

The monchange command changes the operation of monitoring the server that is running on only the host on which the command is executed.

After executing the monchange command, to check whether the operation of server monitoring has changed, execute monshow -o -s.

If use is specified for the notswitch_notify operand in the HA Monitor environment settings, the KAMN521-W message is output each time one hour elapses in the status in which the monitoring is temporarily stopped.

If the monitoring is temporarily stopped, the monitoring command cannot detect a failure. If necessary, restart the monitoring.

Even when the monitoring is temporarily stopped, if hot-standby switchover is possible, you can use one of the following methods to perform hot-standby switchover:

• Hot-standby switchover by using the server hot-standby switchover command (monswap command)

• Grouped-system switchover

• Hot-standby switchover caused by a host failure

To prevent hot-standby switchover from occurring when the monitoring is temporarily stopped, terminate the standby server.

The following notes apply to changing the settings for a monitor-mode program:

The monchange command changes the program slowdown monitoring interval only on the host on which the command is executed. You must also execute the command on the host where the standby server runs.

In a multi-process configuration where multiple processes run under the same program alias name, this command changes the program slowdown monitoring interval for all the processes.

Examples

Changing the HA Monitor setting:

monchange -m patrol 60

Changing the operations of monitoring the system disk:

To temporarily stop the monitoring of the system disk, execute the following command:

monchange –m disk_ptrl -stop_patrol

To restart the monitoring of the system disk, execute the following command:

monchange –m disk_ptrl -start_patrol

Setting and releasing the online serverless mode:

To set the online serverless mode, execute the following command:

monchange -m onlless_mode -use

To release the online serverless mode, execute the following command:

monchange -m onlless_mode -nouse

Changing a server setting:

monchange -s server1 patrol 5

Changing the operation of server monitoring:

To temporarily stop the monitoring on the active server, execute the following command:

monchange –s server1 ptrlcmd_ex -stop_patrol

To restart the monitoring on the active server, execute the following command:

monchange –s server1 ptrlcmd_ex -start_patrol

Changing a monitor-mode program setting:

monchange -p program1 patrol 600

9.7 moncheck (checks definitions)

Format

moncheck

Users who can execute the command

Superuser and general user

Description

This command checks definitions. The moncheck command checks the following items:

Whether the definitions in the definition files are consistent with the specification rules and conventions

Whether the following resources are available:

Resources specified in operands in the environment definition file

LAN status settings files

LAN monitor definition files for the LAN interfaces specified in the switchbyfail operand in the server environment definition

If these resources are not available, not available is displayed.

The following table shows the resources that the moncheck command checks for availability and the meaning of not available.

Table 9‒3: Resources that the moncheck command checks for availability and the meaning of not available

Resource whose availability is checked

Meaning of not available

HA Monitor environment settings (sysdef)

lan operand

The specified host name is not defined in /etc/hosts.

usrcommand operand

No file corresponds to the specified path, or the file owner does not have execution permissions.

multicast_lan operand

The host name of the specified host group is not defined in /etc/hosts. Alternatively, the specified IP address is not a class-D address.

Server environment definition (servers)

disk operand

The specified volume group does not exist.

At least one of the disk devices that make up the specified volume group is incorrectly included in a control group of HA Booster.

fs_name operand

No file system corresponds to the specified logical volume.

fs_mount_dir operand

No directory corresponds to the specified path.

hab_gid operand

HA Booster cannot be used.

ptrlcmd_ex operand

sby_ptrlcmd_ex operand

patrolcommand operand

No file corresponds to the specified path, or the file owner does not have execution permissions.

scsi_device operand

No SCSI device name is specified.

If the specified device name belongs to a symbolic link, no symbolic link name is specified.

If the specified device name belongs to an HDLM logical device, no HDLM logical device name is specified.

This includes when HDLM has not been installed and when HDLM services are stopped.

dmmp_device operand

No DMMP logical device name is specified.

DMMP has not been installed.

The DMMP services are stopped.

Monitor-mode program environment definition (programs)

restartcommand operand

No file corresponds to the specified path, or the file owner does not have execution permissions.

LAN status settings file

The server-alias-name.up or server-alias-name.down file does not exist in /opt/hitachi/HAmon/etc. Alternatively, the file owner does not have execution permissions.

LAN monitor definition files for the LAN interfaces specified in the switchbyfail operand in the server environment definition

There is no LAN monitor definition file under /opt/hitachi/HAmon/etc/ that corresponds to the LAN interface specified for the switchbyfail operand. Alternatively, the file owner does not have read permissions.

Monitoring definition file for the system disk (if use is specified for the disk_ptrl operand in the HA Monitor environment settings)

The /opt/hitachi/HAmon/etc/system_patrol.disk file does not exist. Alternatively, the file owner does not have read permissions.

The monitoring definition file for the system disk does not satisfy any of the following conditions:

• The monitoring-target files exist.

• The monitoring targets are files.

• The absolute path of each monitoring-target file has a valid length.

• The number of monitoring-target files is valid.

• All monitoring-target files are unique.

Monitoring definition file for the disk for business use (if use is specified for the disk_ptrl operand in the server environment definition)

The server-alias-name.disk file does not exist in /opt/hitachi/HAmon/etc/. Alternatively, the file owner does not have read permissions.

The monitoring definition file for the disk for business use does not satisfy any of the following conditions:

• The absolute path of each monitoring-target file has a valid length.

• The number of monitoring-target files is valid.

• All monitoring-target files are unique.

When the HA Monitor Extension functions are set up, whether the monsetup -ex command is executed.

When SCSI reservation for shared disk is used, whether the following operands are consistent with each other:

fence_reset and fence_scsi operands in the HA Monitor environment settings

fence_scsi operand in the HA Monitor environment settings and the scsi_device or dmmp_device operand in the server environment definition

Options

None

Return codes

Value

Meaning

0

Normal termination (checking of the definitions has been completed)

Other than 0

Abnormal termination

9.8 mondeact (terminates a server in the wait state)

Format

mondeact server-alias-name

Users who can execute the command

Superuser

Description

The mondeact command terminates the following servers:

Standby server in the active server start wait state

If a server defined as a standby server in the server environment definition cannot check the status of the active system during startup processing due to an error (such as a communication error), it is placed in the wait state until the active server in the active system has started. This command forcibly terminates such a standby server that is in this status.

Active server in the restart wait state

An active server for which restart or manual is specified in the switchtype operand in the server environment definition is placed in the restart wait state without being switched over in the event of a server failure. This command forcibly terminates such an active server that is in this wait state. When an active server in the restart wait state is terminated by execution of the mondeact command, the corresponding standby server is also terminated automatically.

Active server in the hot-standby wait state

If host reset fails, the standby server in the standby system is switched in as the active server but then is placed in the hot-standby wait state. This command forcibly terminates such an active server that is in this wait state.

If the multi-standby function is used and the active server in the hot-standby wait state is terminated by executing the mondeact command, the corresponding standby servers on all hosts are also terminated automatically.

Active server in the start retry state

If an active server for which retry is specified in the switch_error operand in the server environment definition fails to start at the destination during hot standby processing, it is placed in the start retry state. The mondeact command forcibly releases the active server from this status.

The mondeact command can release an active server during the following time periods:

Server in the server mode

From the time the KAMN310-I message is issued to the time the KAMN251-I or KAMN254-I message is issued

Server in the monitor mode

From the time the KAMN310-I message is issued to the time the KAMN252-I message is issued

Options

server-alias-name

Specifies the alias name of the server that is to be terminated.

Return codes

Value

Meaning

0

Normal termination (the request to terminate the server in the wait state has been accepted)

Other than 0

Abnormal termination

Notes

You can execute the mondeact command only on servers in the following statuses:

Standby server in the active server start wait state

Active server in the restart wait state

Active server in the hot-standby wait state

Active server in the start retry state

The mondeact command cannot be executed on a server in the resource server start wait state.

If a server is waiting for startup of a resource server and the resource server is also in the start wait state, you can terminate both the server and the resource server by executing the mondeact command on the resource server. The KAMN377-E message will be issued in such a case, but there is no problem.

To terminate a server while the server and the resource server are both in the hot-standby wait state, execute the mondeact command on the server. The KAMN377-E message will be issued in such a case, but there is no problem.

The return code is also set to 0 when any of the following messages is issued while type2 is specified in the exitcode operand in the HA Monitor environment settings:

KAMN322-E, KAMN337-E, KAMN339-E, KAMN465-E

The wait-state server does not terminate at the same time that the mondeact command's return code is received.

9.9 mondevice (changes shared resources while the active server is running)

Format

Adding a shared resource:

mondevice server-alias-name -a device-type resource-name

Deleting a shared resource:

mondevice server-alias-name -d device-type resource-name

Changing a shared resource:

mondevice server-alias-name -c device-type resource-name change-type attribute

Executing the command interactively:

mondevice -Q

Users who can execute the command

Superuser

Description

The mondevice command adds and deletes shared resources while the active server is running. It also changes the attributes of shared resources. This command can also be used to change server environment definitions automatically.

Options

Adding a shared resource:

server-alias-name

Specifies the alias name of the server for which a shared resource is to be added.

-a

Specifies that a shared resource is to be added.

device-type

Specifies the type of the shared resource that is to be added.

disk: Adds a shared disk.

resource-name

Specifies the name of the shared resource on the local host that is to be added.

Deleting a shared resource:

server-alias-name

Specifies the alias name of the server for which a shared resources is to be deleted.

-d

Specifies that a shared resource is to be deleted.

device-type

Specifies the type of the shared resource that is to be deleted.

disk: Deletes a shared disk.

resource-name

Specifies the name of the shared resource on the local host that is to be deleted.

Changing a shared resource:

server-alias-name

Specifies the alias name of the server for which a shared resource is to be changed.

-c

Specifies that an attribute of a shared resource is to be changed.

device-type

Specifies the type of the shared resource that is to be changed.

disk: Changes a shared disk.

resource-name

Specifies the name of that shared resource on the local host that is to be changed.

change-type

Specifies the type of change, as listed below, that is to be made to the shared resource on the local host. This option is applicable only when the -c option is specified. When you specify this option, you must specify the new attribute option following this option.

vg_neck

Changes the setting of whether server startup processing is to be cancelled when placement of a volume group online has failed. This option corresponds to the vg_neck operand in the server environment definition.

vg_on_opt

Changes the command run-time option for placing a volume group online. This option corresponds to the vg_on_opt operand in the server environment definition.

attribute

Specifies the new attribute for the shared resource on the local host that is to be changed. You specify this option only when you have specified the -c option.

If the change-type option is vg_neck:

use: Cancels server startup processing when placement online of a disk resource has failed.

nouse: Continues processing without canceling the server startup processing when placement online of a disk resource has failed.

If the change-type option is vg_on_opt:

Specifies the new command run-time options that will be applied when a disk resource is placed online. To change the existing option(s) to no option, specify two consecutive double quotation marks ("") as the attribute. To change the existing option(s) to the default, specify a comma (,). To specify multiple options, enclose the entire attribute in double quotation marks (").

Executing the command interactively:

-Q

Executes the mondevice command interactively.

Return codes

Value

Meaning

0

Normal termination (the shared resource has been changed while the active server was running)

Other than 0

Abnormal termination

Notes

You can execute the mondevice command after the active and standby servers have both completed their startup processing or after only the active server has completed its startup processing. If you execute the command after only the active server has completed its startup processing (before the standby server completes its startup processing), the server environment definition on the standby server will not be changed automatically. In this case, you must terminate the standby server, change its environment definition so that it matches that of the active server, and then restart the standby server.

The original (pre-change) server environment definition is saved in the servers.bac file.

If the server environment definition is invalid during command execution, the definition file will not be updated. In this case, you must use a program such as a text editor to edit the file.

The mondevice command cannot be used to add or delete LANs. To add and delete LANs:

Adding a LAN

Connect the desired LAN in the active system to the network, and then add information about the connected LAN to the LAN status settings files on both hosts.

Deleting a LAN

Disconnect the desired LAN in the active system from the network, and then delete the information about the disconnected LAN from the LAN status settings files on both hosts.

The return code is also set to 0 when any of the following messages is issued while type2 is specified in the exitcode operand in the HA Monitor environment settings:

KAMN320-E, KAMN322-E, KAMN451-E, KAMN452-E, KAMN453-E, KAMN454-E, KAMN455-E, KAMN456-E, KAMN460-E, KAMN465-E

Examples

Adding a shared resource:

In the server environment definition, a shared resource is added by applying the xxx_add operand, where xxx indicates the device type of the added resource.

The following example shows the contents of the server environment definition when mondevice server1 -a disk /dev/vg01 has been executed:

/*  server environment definition  */
 server    name       /users/server1,
           alias      server1,
             :
             :
             :
             :
           disk_add   /dev/vg01;

Deleting a shared resource:

In the server environment definition, a shared resource is deleted by applying the xxx_del operand, where xxx indicates the device type of the deleted resource.

The following example shows the contents of the server environment definition when mondevice server1 -d disk /dev/vg01 has been executed:

/*  server environment definition  */
 server    name       /users/server1,
           alias      server1,
             :
             :
             :
             :
           disk_del   /dev/vg01;

Adding and deleting shared resources:

Shared resources added by applying the xxx_add operand and deleted by applying the xxx_del operand are added and deleted automatically the next time the server is started. However, the user must correct the server environment definition when needed.

The following shows an example of the server environment definition when one disk has been deleted and another disk has been added:

Before change:

        disk         /dev/vg:/dev/vg2,
        disk_add     /dev/vg3,
        disk_del     /dev/vg2;

After change:

        disk         /dev/vg:/dev/vg3;

Changing a shared resource:

In the server environment definition, an attribute of a shared resource is changed by applying the xxx_chg operand, where xxx indicates the device type of the resource whose attribute is being changed.

The following example shows the contents of the server environment definition when mondevice server1 -c disk /dev/vg01 vg_neck use has been executed:

/*  server environment definition  */
 server    name       /users/server1,
           alias      server1,
             :
             :
             :
             :
           disk       /dev/vg01,
           disk_chg   /dev/vg01:vg_neck:use;

Adding, changing, and deleting shared resources:

Shared resources added by applying the xxx_add operand, deleted by applying the xxx_del operand, and whose attribute values were changed by applying the xxx_chg operand are added, deleted, and changed automatically the next time the server is started. However, the user must correct the server environment definition when needed.

The following example shows the contents of the server environment definition when /dev/vg04 is added to the disk operand, the vg_neck operand value for /dev/vg03 is changed to use, and /dev/vg01 is deleted:

Before change:

        disk       /dev/vg01:/dev/vg02:/dev/vg03,
        vg_neck    use:nouse:nouse,
        disk_add   /dev/vg04,
        disk_chg   /dev/vg03:vg_neck:use,
        disk_del   /dev/vg01;

After change:

        disk       /dev/vg02:/dev/vg03:/dev/vg04,
        vg_neck    nouse:use:nouse;

If you change the disk operand value, you must also change the operand value specified in change-type at the same time.

Executing the command interactively:

You can execute the command interactively by specifying the -Q option. If you want to cancel execution of the command for a reason such as a specification error, press the C key while holding down the Ctrl key (CTRL+C). The following example adds disk for the server whose server alias name is server1:

> mondevice -Q
  This command changes a resource.
  If processing is stopped, Input the [CTRL]+[C].

  Q. Input alias that is processed.
>  [alias]:
>  server1
  Q. Select the processing that is executed by number.
   (1)Addition of a device
   (2)Deletion of a device
   (3)Change attribute of a device
>  [Addition(1) /Deletion(2) /Change(3) ]:
>  1
  Q. Select the kind of a device by number:
   (1) disk (Shared disk unit)
>  [Device kind]:
>  1
  Q. Input the device is added, is deleted or is changed.
>  [Device name]:
>  /dev/vg01
  Edited command is.: mondevice server1 -a disk /dev/vg01
  Q. Is a command executed ?
  If a command is executed, input the [ENTER].
>
  A command is executed.

Legend:

>: Indicates a user operation.

9.10 monend (terminates servers in the monitor mode)

Format

monend server-alias-name

Users who can execute the command

Superuser and general user

Description

If the active server in the monitor mode stops, this command notifies HA Monitor of the event so that HA Monitor stops host monitoring and terminates the corresponding standby server.

Options

server-alias-name

Specifies the alias name of the active server that is to be terminated.

Return codes

Value

Meaning

0

Normal termination (Termination of servers in the monitor mode was completed regardless of whether the termination command was successful.)

Other than 0

Abnormal termination

Notes

The monend command can be used only for servers in the monitor mode.

The monend command can be executed only from the active system.

9.11 moninfo (specifies server information in the active system that is to be inherited, or references and displays inherited server information in the standby system)

Format

moninfo server-alias-name {-p inherited-server-information|-g}

Users who can execute the command

Superuser and general user

Description

The moninfo command specifies server information that is to be passed from the active server to the paired standby server (inherited by the standby server) when pairing is established between an active server and a standby server. It is also used to reference and display server information in the standby system that has been inherited from the active server.

The moninfo command can be issued only from within a user command.

Options

server-alias-name

Specifies the alias name of the active server whose specified server information is to be passed to (inherited by) the paired standby server. This option also specifies the alias name of the server whose inherited server information is to be referenced and displayed.

-p inherited-server-information

Specifies the inherited server information that is to be passed to (inherited by) the paired standby server. You can specify inherited server information as a path name consisting of up to 80 single-byte characters. For details about how to specify path names, see (2) Component symbols.

If you specify the -p option, you cannot omit specification of inherited server information.

-g

Specifies that existing inherited server information from the specified server is to be referenced or output to the standard output.

Return codes

Value

Meaning

0

Normal termination (inherited server information has been specified, referenced, or displayed)

Other than 0

Abnormal termination

Notes

This command cannot be used in the online serverless mode.

The specified inherited server information is inherited when pairing is established between active and standby servers.

If the specified inherited server information consists of 81 or more single-byte characters, it will not be processed or inherited.

If the specified server alias name belongs to a standby server, the information will not be processed or inherited.

You specify inherited server information for a specific server. Only one instance of inherited server information can be specified per server.

Execute the command that specifies the inherited server information when the active server's startup processing begins (online -s start).

The return code is also set to 0 when any of the following messages is issued while type2 is specified in the exitcode operand in the HA Monitor environment settings:

KAMN021-E, KAMN322-E, KAMN372-E, KAMN465-E

Examples

The following shows examples of the moninfo command in a user command.

Example 1

This example uses the moninfo -p command to specify an alias IP address that is to be used as a shared resource.

Target server alias name: server1

Alias IP address: 1.2.3.4

Timing of specifying the inherited server information: When the active server's startup processing begins (online -s start)

A portion of the user command is shown below. In this example, the specified alias IP address is passed to (inherited by) the standby server when pairing is established between the active and standby servers.

case "$2" in
"server1")
    case "$4" in
    "online")
        case "$5" in
        "-s")
        if [ "$6" = "start" ]
        then
#server1 online start!!
              moninfo server1 -p 1.2.3.4
        fi

Legend: Bold: Inherited server information that is specified

Example 2

This example uses the moninfo -g command to reference inherited server information and then passes it to a command that specifies an alias IP address.

Target server alias name: server1

Name of command that specifies an alias IP address: /sbin/ifconfig

Timing of passing the inherited server information: When hot standby processing is started on the standby server (standby -a start)

The following shows a portion of the user command:

case "$2" in
"server1")
    case "$4" in
    "standby")
        case "$5" in
        "-a")
        if [ "$6" = "start" ]
        then
#server1 system exchange start!!
               IPADDR='moninfo server1 -g'
               /sbin/ifconfig eth1:1 inet $IPADDR netmask 255.255.255.0 broadcast 1.2.3.255
               /sbin/arping -U -c 2 -I eth1 $IPADDR
               /bin/echo 0 > /proc/sys/net/ipv4/route/flush
        fi

Legend: Bold: Inherited server information that is referenced

9.12 monlink (connects HA Monitors manually)

Format

monlink

Users who can execute the command

Superuser and general user

Description

Normally, HA Monitor is connected automatically to the HA Monitors on the remote hosts at the time of startup.

The monlink command connects the HA Monitors manually when automatic connection establishment between the HA Monitors has failed.

Options

None

Return codes

Value

Meaning

0

Normal termination (the request to manually connect the HA Monitors has been accepted)

Other than 0

Abnormal termination

Notes

After you have executed the monlink command, use the monshow -c or monpath command to check whether connection was established with the remote hosts successfully.

Connection between HA Monitors is not established at the same time that the monlink command's return code is received.

9.13 monlistip (detects IP addresses)

Format

monlistip output-file-name

Users who can execute the command

Superuser

Description

The monlistip command detects the devices in the reset path's network and outputs a list of their IP addresses for HA Monitor on the host where the command was executed. The command outputs the IP addresses that responded to the ICMP echo packet sent by the command.

Before you execute this command, check that the kernel parameter has been set to a value that is at least the number of host sections in the network of the reset path. For details, see Notes for this command.

Options

output-file-name

Specifies a name for the CSV file to which the list of IP addresses is to be output. If a file with the specified name already exists, the command cancels file output processing, so first check that the file name you intend to specify is unique.

An example of the CSV file is shown below. This example specifies IP_list.csv as the output file name.

Search IP address in a range from 1.1.2.1 to 1.1.2.254
IP address,machine,HAmon,address,resetpath,SVP(or HRL),resetpath(2nd),HRL(2nd)
1.1.2.131
1.1.2.141
1.1.2.111
1.1.2.151
1.1.2.161
1.1.2.121
1.1.2.171
1.1.2.172

Return codes

Value

Meaning

0

Normal termination (detection of IP addresses has been completed)

Other than 0

Abnormal termination

Messages

The following table lists and describes the messages that might be output to the standard error output during command execution.

Table 9‒4: Messages that might be output to the standard error output by the monlistip command

Message

Description

Action

Usage :aa....aa outfile

The option specified in the monlistip command is invalid.

aa....aa: Path used during monlistip command execution

Correct the option specification and re-execute the command.

The specified file (aa....aa) already exists.

A file with the output file name specified in the monlistip command already exists.

aa....aa: Output file name

Either specify a different output file name in the option or delete the existing file with the specified file name.

Settings of the reset path have not been configured.

The reset path has not been configured.

Complete reset path configuration on the host on which the monlistip command is being executed.

This message is also output if you have executed the monlistip command on a system other than BladeSymphony or HA8000. In such a case, execute the monlistip command on a BladeSymphony or HA8000 system.

An error occurred in the aa....aa system call.

Errno : bb....bb.

Information : cc....cc.

A system call error occurred during monlistip command execution.

aa....aa: System call name

bb....bb: System call's errno

cc....cc: Detail information

If the displayed system call name (aa....aa) is sendto and the system call's errno (bb....bb) is 105, the kernel parameter setting is invalid. See Notes for the command.

If any other error has occurred, contact maintenance personnel.

The netmask cannot be acquired.

The netmask setting of the reset path could not be obtained.

Eliminate the cause of the error according to the error message that was issued immediately before this message.

If no error message has been issued before this message, check the IP address of the reset path specified in the monsetup -resetpath command and correct it if there is an error.

Processing will now stop.

Processing is being cancelled due to an error.

Eliminate the cause of the error according to the error message that was issued immediately before this message, and then re-execute the monlistip command.

Notes

The monlistip command can be executed only when the system being used is BladeSymphony or HA8000.

Execute the monlistip command on a host on which an HA Monitor environment has been configured.

Execution of the monlistip command results in approximately 20 kilobytes of load per second. Execute the command when the processing load is light.

Before you execute the monlistip command, make sure that the kernel parameter shown below is set to a value that is at least the number of host sections in the network of the reset path. You can use the sysctl OS command to display and change the kernel parameter.

Kernel parameter to be changed:

net.ipv4.neigh.default.gc_thresh3

Value:

Specify a value that is at least the number of host sections in the network of the reset path. You can also check the number of host sections in the network of the reset path in the number of IP addresses to be checked that is displayed when you execute the monlistip command.

Example of sysctl command (when the address class is B):

/sbin/sysctl -w net.ipv4.neigh.default.gc_thresh3= 65536

Detection of IP addresses requires the following number of seconds at the most:

Number of host sections in the network of the reset path ÷ 100 × 3 (the value is rounded up)

Examples

This example specifies IP_list.csv as the output file name:

> monlistip IP_list.csv
Now processing.
100/254 39% complete.
200/254 78% complete.
254/254 100% complete.
Processing ended.

The following table lists and describes the messages that are displayed during command execution.

Table 9‒5: Messages that are displayed during monlistip command execution

Item that is displayed

Description

Now processing.

Command processing has started.

aa....aa / bb....bb cc....cc% complete.

Shows the progress of the command's processing:

aa....aa: Number of IP addresses that have been checked

bb....bb: Number of IP addresses to be checked

cc....cc: Percentage of IP addresses that have been checked

Processing ended.

Command processing has terminated.

9.14 monodrshw (displays the server order control status)

Format

monodrshw { -s server-alias-name | -g group-name }

Users who can execute the command

Superuser and general user

Description

The monodrshw command displays parent servers. If no parent server has been defined, the command displays none.

Options

-s server-alias-name

Specifies the server whose parent servers are to be displayed, if the specified server is running. If the specified server is not running, the command adds the KAMN322-E message to the results that are displayed. If the specified server is not grouped, the command adds the KAMN944-E message to the results that are displayed.

-g group-name

Specifies the group whose parent servers are to be displayed, if any servers in the specified group are running. If the specified group contains no servers that are currently running, the command adds the KAMN428-E message to the results that are displayed.

Return codes

Value

Meaning

0

Normal termination (the server order control status has been displayed)

Other than 0

Abnormal termination

Examples

This example specifies a server alias name and executes the command while the server is running:

> monodrshw -s server2
KAMN950-I Order Information
group name:groupA
alias:server2             parent:server1

Legend: >: Indicates a user operation.

Example of when a group name is specified:

> monodrshw -g groupA
KAMN950-I Order Information
group name:groupA
alias:server1             parent:none
alias:server2             parent:server1
alias:server3             parent:server2
alias:server4             parent:server2
alias:server5             parent:none

Legend: >: Indicates a user operation.

9.15 monpath (displays the status of monitoring paths)

Format

monpath [-i] [check-interval]

Users who can execute the command

Superuser

Description

The monpath command checks the status of all monitoring paths and displays the results. You use the monpath command when a failure on a monitoring path has occurred and when the failure has been recovered.

The command displays the following information:

device name: TCP/IP host name of each monitoring path

host name: Host names of the remote hosts

status: Status of communication with each remote host

A monitoring path that is currently in use is indicated by an asterisk (*) following the status of communication with each remote host. If there is an error in a monitoring path, the monitoring path's status is displayed in the status column corresponding to the host name.

The following table lists and describes the items that are displayed.

Table 9‒6: Items that are displayed by the monpath command

Displayed item

Meaning

device name

host name

status

Monitoring path's TCP/IP host name

Host name of a remote host

OK

Normal

NO RESPONSE

There was no response for one of the following reasons:

Slowdown of the host

There is no HA Monitor on the remote host.

Failure or disconnection of the monitoring path

**not connect**

Connection has not been established with the HA Monitor on this remote host.

**open err**

Open error

**LAN err**

Transmission error

Options

-i

Specifies that information about a TCP/IP LAN on the remote host is to be displayed in addition to the information that is displayed when the -i option is omitted.

The following is the additional information that is displayed:

host: TCP/IP host name of the monitoring path on the remote host

This option displays the TCP/IP host name of the monitoring path on the remote host that is specified in the /etc/hosts file. If no TCP/IP host name has been specified, ---- is displayed.

(IP_address): IP address of the monitoring path on the remote host

This option displays the IP address corresponding to the TCP/IP host name of the monitoring path on the remote host that is specified in the /etc/hosts file.

check-interval

Specifies the interval at which monitoring paths are to be checked. The permitted value range is 3 to 30 seconds. If this option is omitted, 3 seconds is assumed.

If a failure is detected on any monitoring path during a check, the monitoring paths are checked again at the specified interval. The number of re-checks to be performed depends on the pathpatrol_retry operand value specified in the HA Monitor environment settings.

Normally, a check interval of three seconds is sufficient. If (number of hosts × number of monitoring paths) is large or when multiple servers are engaged in startup processing, you might need to specify a larger check interval.

Return codes

Value

Meaning

0

Normal termination (status of monitoring paths has been displayed)

Other than 0

Abnormal termination

Notes

The extent of a monitoring path that is subject to this checking is from the TCP/IP LAN adapter on the local host to the TCP/IP LAN adapter on the remote host.

The monpath command is not accepted if another monpath command is already executing or HA Monitor is performing a health check on the reset path.

The return code is also set to 0 when any of the following messages is issued while type2 is specified in the exitcode operand in the HA Monitor environment settings:

KAMN206-E, KAMN320-E, KAMN393-E

Examples

This example executes the monpath command on host 1 (host1) in the hot-standby configuration shown below:

Figure 9‒1: Example of hot-standby configuration in which the monpath command is used

In this example, the -i option is omitted and a failure has occurred on monitoring path path32 on host 3 (host name: host3):

> monpath
KAMN390-I Path status Display
device name  host name  status
path11       host2      OK  *
             host3      OK  *
path12       host2      OK
             host3      NO RESPONSE

Legend: >: Indicates a user operation.

In this example, the -i option is specified and monitoring path path32 on host 3 (host name: host3) has not been specified in the /etc/hosts file:

> monpath -i
KAMN390-I Path status Display
device name  host name  host   (IP_address)  status
path11       host2      path21 (100.2.1.132) OK  *
             host3      path31 (100.2.1.133) OK  *
path12       host2      path22 (100.2.2.132) OK
             host3      ----   (100.2.2.133) OK

Legend >: Indicates a user operation.

9.16 monresbgn (starts a resource server)

Format

monresbgn resource-server-alias-name

Users who can execute the command

Superuser and general user

Description

The monresbgn command starts a resource server.

Options

resource-server-alias-name

Specifies the alias name of the resource server that is to be started.

Return codes

Value

Meaning

0

Normal termination (the resource server start request has been accepted)

Other than 0

Abnormal termination

Notes

The monresbgn command can be used only for resource servers.

Normally, a resource server is started automatically when any of its child servers is started. Execute the monresbgn command as needed.

The resource server is not started at the same time that the monresbgn command's return code is received.

Upon receiving a resource server start request, the monresbgn command outputs the KAMN376-I message, and returns the return value 0 regardless of the status of the resource server specified by a resource server alias name.

9.17 monresend (terminates an active resource server)

Format

monresend resource-server-alias-name

Users who can execute the command

Superuser and general user

Description

The monresend command terminates an active resource server. When an active resource server is terminated, the corresponding standby resource server, if any, is terminated automatically.

Options

resource-server-alias-name

Specifies the alias name of the active resource server that is to be terminated.

Return codes

Value

Meaning

0

Normal termination (the active resource server has been terminated)

Other than 0

Abnormal termination

Notes

The monresend command can be used only for active resource servers. You can execute the monresend command only from the active system.

Normally, there is no need to use the monresend command because a resource server is terminated automatically when all its child servers have terminated.

When a resource server is terminated by the monresend command, its resources are disconnected. Before you execute the command, check that no servers that use those resources are running. If any child server of the specified resource server is running, execution of the monresend command will fail.

9.18 monressbystp (terminates a standby resource server)

Format

monressbystp resource-server-alias-name

Users who can execute the command

Superuser

Description

The monressbystp command terminates the following standby resource servers:

Resource server that has finished startup or that is in the hot-standby wait state

Resource server in the grouped-system switchover wait state

Resource server that has finished startup in the online serverless mode

Options

resource-server-alias-name

Specifies the alias name of the resource server on standby that is to be terminated.

Return codes

Value

Meaning

0

Normal termination (the resource server on standby has been terminated)

Other than 0

Abnormal termination

Notes

The monressbystp command can be used only for resource servers that are on standby. You can execute the monressbystp command only from a standby system.

Normally, there is no need to use the monressbystp command because a resource server on standby is terminated automatically when all its child servers have terminated.

Use the monressbystp command only when hot standby processing can be performed on the resource server and you want to terminate only the resource server that is on standby. Note that if any child server of the resource server on standby is running, execution of the monressbystp command will fail.

The return code is also set to 0 when any of the following messages is issued while type2 is specified in the exitcode operand in the HA Monitor environment settings:

KAMN221-E, AMN322-E, KAMN329-E, KAMN465-E, KAMN958-E, KAMN963-E

9.19 monrp (displays the status of the reset path)

Format

monrp [remote-host-name]

Users who can execute the command

Superuser

Description

The monrp command checks the status of the reset path by communicating with the failure management processors, and displays the results. If the reset path for the local host has not been initialized, the monrp command initializes the reset path. You use the monrp command in the event of a failure on a failure management processor or on the reset path when the failure is to be recovered. The monrp command displays the following information:

host name: Host name

status: Status of the reset path

The following table lists and describes the items that are displayed.

Table 9‒7: Items that are displayed by the monrp command

Displayed item

Meaning

host name

status

host-name

OK

Normal

NG

Error (failure management processor error or error in the path connecting failure management processors)

OK(Partition)

Physical partition can be reset.

NG(Partition)

Physical partition cannot be reset.

**nothing**

Not connected with the remote host.

Options

remote-host-name

Specifies the name of the remote host that is connected to the local host whose reset path is to be checked. If this option is omitted, the command checks all hosts connected to the local host.

Return codes

Value

Meaning

0

Normal termination (the reset path status has been displayed)

Other than 0

Abnormal termination

Notes

It takes a maximum of (number of hosts × 15) seconds to check the reset path. While the monrp command is executing, hot standby processing will be delayed in the event of a failure because a host reset by HA Monitor will be placed in the wait state. Execute this command only when it is necessary to do so.

The monrp command can be executed only when the system used to execute the command is BladeSymphony or HA8000.

The monrp command will not be accepted if another monrp command is already executing or HA Monitor is performing a health check on the reset path.

The return code is also set to 0 when any of the following messages is issued while type2 is specified in the exitcode operand in the HA Monitor environment settings:

KAMN320-E, KAMN392-E, KAMN397-E, KAMN398-E, KAMN399-E, KAMN460-E

Examples

In this example, the local host is connected to two remote hosts:

> monrp
KAMN395-I Reset Path Status Display
host name                         status
host1                             OK
host2                             NG

Legend:

>: Indicates a user operation.

In this example, the local host is not connected to any remote host:

> monrp
KAMN395-I Reset Path Status Display
host name                         status
**nothing**

Legend:

>: Indicates a user operation.

In this example, the local host is connected to remote hosts (using the physical partition reset function):

> monrp
KAMN395-I Reset Path Status Display
host name                         status
host1                             OK
                                  OK(Partition)
host2                             NG
                                  OK(Partition)

Legend:

>: Indicates a user operation.

9.20 monsbystp (terminates a standby server)

Format

monsbystp server-alias-name

Users who can execute the command

Superuser

Description

The monsbystp command terminates a standby server that is in one of the following statuses:

Standby server that has finished startup or that is ready for hot-standby switchover

Standby server placed in the grouped-system switchover wait state

Standby server started in the online serverless mode (startup complete state)

Note that if you use the monsbystp command to terminate the standby server that corresponds to an active server that is in the restart wait state, the active server in the restart wait state is also terminated automatically.

If the multi-standby function is used, the monsbystp command terminates only the standby server on the host on which the command is executed. Note that the command's processing depends on the status of the standby server, as described below.

If you use the monsbystp command to terminate a standby server that is in the grouped-system switchover wait state, the command also terminates automatically all the corresponding standby servers on all hosts.

If you terminate the standby server that corresponds to an active server that is in the restart wait state and there is no other standby server on standby, the command also terminates automatically the active server that is in the restart wait state.

Options

server-alias-name

Specifies the alias name of the server that is to be terminated.

Return codes

Value

Meaning

0

Normal termination (the standby server has been terminated)

Other than 0

Abnormal termination

Notes

The monsbystp command can be executed only from a standby system.

The return code is also set to 0 when any of the following messages is issued while type2 is specified in the exitcode operand in the HA Monitor environment settings:

KAMN323-E, KAMN329-E, KAMN465-E, and KAMN957-E

9.21 monscsiclr (releases reservations for shared disks)

Format

monscsiclr server-alias-name

Users who can execute the command

Superuser

Description

The monscsiclr command releases reservations for the devices specified in the scsi_device and dmmp_device operands in the server environment definition (servers).

If the active server cannot be started because shared disks are reserved, use this command to release the reservations so that the active server can be started. You can execute this command regardless of whether HA Monitor is running.

If all the reservations are released successfully, the KAMN108-I message with detail code SUCCESS is issued. If the command fails to release the reservation for any shared disk, the KAMN108-I message with detail code FAILURE is issued. The command skips a device whose reservation could not be released and continues processing.

Options

server-alias-name

Specifies the alias name of the server for which shared disk reservations are to be released.

Return codes

Value

Meaning

0

Normal termination (Reservations for all shared disks used by the server have been released.)

Other than 0

Abnormal termination

Notes

Do not execute the monscsiclr command while the active server whose alias name is specified in the option is running on any of the hosts. If the monscsiclr command is executed in such a case, the shared disks might no longer be locked.

This command ignores the scsi_timeout and scsi_retry operand settings in the HA Monitor environment settings (sysdef) and assumes the default values.

Example

Display during command execution

This example assumes the following:

/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000000 is reserved.

/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000001 is reserved, but its open processing fails when the monscsiclr command is executed.

/dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000002 is not reserved.

> monscsiclr server1
KAMN107-I Reservation release processing for server server1 will now start.
  Device                                                        Status
  /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000000        CLEARED
KAMN601-W An error occurred in the open system call. Error details: 2 Functional name: scsi_cmd_ctrl.
KAMN725-W Execution of the SCSI command on the device /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000001 ended abnormally. (details code: OPEN, server: server1, host address: *(*)).
KAMN726-E Errors occurred on all the paths to the device /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000001.
  /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000001        FAILED
  /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000002        NO_RESERVE
KAMN108-I Reservation release processing for server server1 is completed. Details: FAILURE

Legend:

>: Indicates a user operation.

The value displayed for Device consists of a maximum of 61 single-byte characters. If the path of a shared disk consists of more than 61 characters, the excess characters are not displayed.

The following table shows the Status values and their meanings:

Status

Meaning

CLEARED

Reservation was released.

FAILED

Reservation release processing failed.

See the displayed message and take appropriate action.

NO_RESERVE

Reservation had not been made.

9.22 monsetup (specifies HA Monitor environment settings)

Format

Changing how to start and terminate HA Monitor:

monsetup [{-start|-stop}]

Specifying reset path settings:

monsetup -resetpath

Specifying advanced SSH connection settings:

monsetup -resetpath ssh

Specifying and displaying HA Monitor Extension function settings:

monsetup -ex [sysmax {on|off}]

Users who can execute the command

Superuser

Description

Changing the settings for how to start and terminate HA Monitor:

Normally, HA Monitor is set to start automatically when the system is started and to terminate automatically when the system is terminated.

The monsetup command enables you to change these settings so that you can start and terminate HA Monitor by using the monstart and monstop commands. If you execute the monsetup command with no option specified, the command displays the current settings for the start and termination methods.

The command displays the following information:

Start mode: Current setting for the start method

Stop mode: Current setting for the termination method

Specifying reset path settings:

The monsetup command enables you to specify reset path settings interactively, such as IP addresses. This command also enables you to display the current settings. You can execute the reset path setting command if you use a host reset on a BladeSymphony, HA8000, or RV3000.

Specifying advanced SSH connection settings:

If you use VMware ESXi-based virtualization, the monsetup command enables you to specify advanced SSH connection settings for VMware ESXi. If there is a device specifying a no-communication monitoring interval on the reset path for VMware ESXi connection, you must specify these SSH connection settings.

Note that you do not need to specify these settings if vSphere with high-availability functionality added is used on an RV3000.

Specifying and displaying HA Monitor Extension function settings:

The monsetup command enables you to specify whether the functions of HA Monitor Extension are to be used. This command also enables you to display the current function settings for HA Monitor Extension.

Options

Changing the settings for how to start and terminate HA Monitor:

The option specifies whether the setting for the start method or the setting for the termination method is to be changed. If no option is specified, the monsetup command displays the current settings for the start and termination methods.

-start

Changes the setting for the start method. If the current setting is auto (start automatically), the command changes it to manual (start by execution of the monstart command). If the current setting is manual, the command changes it to auto.

-stop

Changes the setting for the termination method. If the current setting is auto (terminate automatically), the command changes it to manual (terminate by execution of the monstop command). If the current setting is manual, the command changes it to auto.

Specifying reset path settings:

-resetpath

Specifies the settings required in order to use the reset path interactively.

Important

If you specify the -resetpath option, after all reset path settings have been specified completely, HA Monitor checks whether the system can communicate with the failure management processor. In a VMware environment in which vSphere with high-availability functionality added is not used on an RV3000, HA Monitor also checks whether the system can communicate with VMware ESXi. If a communication error occurs, remove the cause of the error, and then set the reset path again.

Note that in the following cases, HA Monitor does not check whether communication is possible after the reset path is set:

• The N+M cold-standby linkage function is used. (In this case, HA Monitor checks whether communication with VMware ESXi in a VMware environment is possible.)

• HA Monitor is running.

• The reset path is set after HA Monitor terminates abnormally.

To set the reset path after HA Monitor terminates abnormally, restart HA Monitor, terminate HA Monitor, and then set the reset path again.

The settings depend on the system being used.

If the RV3000 system model is used

The following describes the settings to be specified when the RV3000 system model is used.

Table 9‒8: Settings to be specified if the RV3000 system model is used

Setting item

Value to be set

System's partition name

Set the same name as the management system name (System name) set in HA Monitor Connector.

Virtual machine name

Set this item only when vSphere with high-availability functionality added is used on an RV3000.

Set the name of the virtual machine set in VMware ESXi. A virtual machine name can have a maximum of 31 characters. Note that FullSystemPartition cannot be specified as the virtual machine name.

Reset path's IP address and port number

Specify any IP address that belongs to the reset path's network. For the port number, specify a number in the range from 5001 to 65535 that is not in use. Make sure that the IP address and port number you set for this item are different from HA Monitor Connector's IP address and port number.

HA Monitor Connector's IP address and port number

Set the same value as IP address of the interface for the reset path used by HA Connector set in HA Monitor Connector. For the port number, specify a number in the range from 5001 to 65535 that is not in use. Set the same value as Port number used by HA Connector set in HA Monitor Connector.

If the system being used is BladeSymphony

If the system being used is BladeSymphony, the switch and management module displayed when the settings are displayed shows the SVPs and the hardware used to manage SVPs. For details, see the hardware documentation. The settings for BladeSymphony are described below.

Table 9‒9: Settings to be specified if the system being used is BladeSymphony

Setting item

Value to be set

System's partition name

Set the same name as the system name of the CPU module in the system set in the HA Monitor linkage settings of the switch and management module.

For model BS2000 or earlier, specify this setting by using the HA command in the SVP command mode.

LPAR name

Specify this setting only if you use Hitachi server virtualization to run LPARs. For the LPAR name, specify an LPAR name that was specified on the HVM control screen.

Do not specify FullSystemPartition as the LPAR name.

Reset path's IP address and port number

Specify any IP address that belongs to the reset path's network. The port number must be the same for all hosts. For the port number, specify a number in the range from 5001 to 65535 that is not in use. The combination of IP address and port number must differ from that of the switch and management module.

Switch and management module's IP address

Specify the same value as SVP IP address that was specified with the LC command for the switch and management module in the SVP command mode. For an N+M cold standby configuration, specify auto.

Switch and management module's port number

Specify a port number in the range from 5001 to 65535 that is not in use. Set the same value as the port number that is set in the HA Monitor linkage settings of the switch and management module.

For model BS2000 or earlier, specify this setting by using the HA command in the SVP command mode.

If the system model being used is HA8000xM or earlier

If the system model being used is HA8000xM or earlier, the switch and management module that is displayed during setup is the SVP or BMC. The following describes the settings to be specified if the system model being used is HA8000xM or earlier.

Table 9‒10: Settings to be specified if the system model being used is HA8000xM or earlier

Setting item

Value to be set

System's partition name

Specify any name consisting of 1 to 31 alphanumeric characters. This name is used by HA Monitor and SVPs as an interface to identify hosts.

Reset path's IP address and port number

Specify any IP address that belongs to the reset path's network. For the port number, specify a number in the range from 5001 to 65535 that is not in use. This combination of IP address and port number must differ from that of the switch and management module.

Switch and management module's IP address

Specify the same value as SVP IP Address that was specified by using the HA Monitor setup utility provided with the SVP microprogram.

Switch and management module's port number

Specify a port number in the range from 5001 to 65535 that is not in use. This port number must be the same as Port no that was specified with the HA Monitor setup utility provided with the SVP microprogram.

If the system model being used is HA8000xN or later

If the system model being used is HA8000xN or later, the switch and management module that is displayed during setup is HA Monitor Option (HA Monitor Connector) that will be used as a failure management processor. The following describes the settings to be specified if the system model being used is HA8000xN or later.

Table 9‒11: Settings to be specified if the system model being used is HA8000xN or later

Setting item

Value to be set

System's partition name

Specify the same value that is set for System name (management system name) in the HA Monitor Connector configuration window. The value you specify can consist of 1 to 31 alphanumeric characters. This name is used by HA Monitor and HA Monitor Option as an interface to identify hosts.

Reset path's IP address and port number

Specify any IP address that belongs to the reset path's network. For the port number, specify a number in the range from 5001 to 65535 that is not in use. Note, however, that the port number you specify must be different from the port number that is set in the HA Monitor Connector configuration window.

Switch and management module's IP address

If you use a management server as the failure management processor, specify the IP address of the management server. If you do not use a management server as the failure management processor, specify the IP address of the reset path. However, in a VMware-based virtualization environment in which HA Monitor and HA Monitor Option exist on separate virtual machines, do not specify the IP address of the reset path. In this case, specify the IP address of the reset path over the network to the virtual machine on which HA Monitor Option operates.

Switch and management module's port number

Specify the same value that is specified for Port (port number of the management server for communication with HA Monitor) in the HA Monitor Connector configuration window.

If VMware ESXi-based virtualization is to be used (except when vSphere with high-availability functionality added is used on an RV3000)

The following describes the settings to be specified only if VMware ESXi-based virtualization is to be used (except when vSphere with high-availability functionality added is used on an RV3000).

Table 9‒12: Settings to be specified if VMware ESXi-based virtualization is to be used (except when vSphere with high-availability functionality added is used on an RV3000)

Setting item

Value to be set

Virtual machine name

Specify the name of the virtual machine to be managed by VMware ESXi. This is information that you specified when you created the virtual machine in VMware ESXi.

IP address of VMware ESXi

Specify the IP address of VMware ESXi for management.

Port number of VMware ESXi for SSH connection

Specify the port number that HA Monitor is to use to establish SSH connection with VMware ESXi. If you specify default, port 22 will be used.

Authentication password of VMware ESXi for SSH connection

Specify the password required for authentication when SSH connection is established with VMware ESXi on the local host. This is the root user's password specified in VMware ESXi.

Note that the password is not displayed when you specify and display the settings.

Machine address

Set any numeric value that can identify the physical partition. You can set a value in the range from 0 to 9999.

Specify the same machine address for all hosts in the same physical partition. For a host in a different physical partition, specify a unique machine address for each physical partition. If the system contains multiple hot-standby configurations that do not monitor each other and they share a reset path, specify each machine address so that it is unique in all physical partitions. If the same machine address is used for multiple physical partitions, HA Monitor might malfunction.

Specifying advanced SSH connection settings:

-resetpath ssh

Specifies the advanced settings required for HA Monitor to establish SSH connection with VMware ESXi interactively. Specify this option only if you use VMware ESXi-based virtualization.

If you use a firewall or router on the reset path for VMware ESXi connection and the no-communication status lasts for a specific period of time between HA Monitor and VMware ESXi, the firewall or router function might disconnect the SSH connection to VMware ESXi. When HA Monitor establishes SSH connection with VMware ESXi, it sends a packet requesting a response from VMware ESXi (connection-maintenance packet) so that the no-communication status does not last for more than the specific period of time. If there is no response from VMware ESXi, HA Monitor releases the SSH connection.

The following describes the settings:

Table 9‒13: Specifying advanced SSH connection settings:

Setting item

Value to be set

Connection-maintenance packet transmission interval

Specify the interval at which connection-maintenance packets are to be sent. You can specify a value in the range from 0 to 2147483647. The value you specify must be shorter than the no-communication monitoring interval of the firewall or router. The value is in seconds. If you do not specify this value, the default value (60) is assumed.

If 0 is specified, HA Monitor does not send connection-maintenance packets.

Connection-maintenance packet transmission count

Specify the number of times connection-maintenance packets are to be sent. The permitted value is from 0 to 2147483647. If you do not specify this value, the default value (3) is assumed.

If 0 is specified, HA Monitor does not send connection-maintenance packets.

If there is no response from VMware ESXi after the specified number of connection-maintenance packets have been sent, SSH connection is released.

Specifying and displaying HA Monitor Extension function setting:

-ex

Specifies that the functions of HA Monitor Extension are to be used. If all the operands explained below are omitted, HA Monitor displays the current function settings for HA Monitor Extension.

sysmax

Specifies that HA Monitor Extension function settings are to be set or released. You must use this option together with one of the options described below.

on

Enables use of HA Monitor Extension functions. Specifically, you will be able to specify the following settings:

•Specification of the hostmax operand in the HA Monitor environment settings. The hostmax operand enables you to increase the maximum number of hosts to 33 or more.

• Specification of a value greater than 128 in the servmax operand in the HA Monitor environment settings.

off

Specifies that the following functions of HA Monitor Extension are to be disabled:

•Specification of the hostmax operand in the HA Monitor environment settings.

•Specification of a value greater than 128 in the servmax operand in the HA Monitor environment settings.

Return codes

Value

Meaning

0

Normal termination (the HA Monitor environment settings have been specified)

Other than 0

Abnormal termination

Notes

If you specify the reset path settings while HA Monitor is running, the settings are not enabled. In such a case, terminate and restart HA Monitor. The reset path settings will then be enabled.

If initialization of the reset path fails while HA Monitor is being started, the KAMN684-E message is output. In such a case, respecify the reset path settings, and then execute the monrp command. The reset path settings will then be enabled. If the KAMN684-E message is output, always execute the monrp command on all hosts to confirm that the status of all connected hosts is OK.

Do not start HA Monitor while the reset path settings are being specified by using the monsetup -resetpath command. If HA Monitor is checking whether communication with the failure management processor and VMware ESXi is possible, communication fails.

Do not use the monsetup -resetpath command to specify the reset path settings on multiple PCs concurrently. If you do so, you might be unable to correctly specify settings or check communication.

You cannot configure HA Monitor Extension functions while HA Monitor is operating. If HA Monitor is running, terminate it before executing the monsetup command.

HA Monitor Extension must be installed in order to specify or display HA Monitor Extension function settings.

Examples

Displaying the settings for how to start and terminate HA Monitor:

>monsetup
Start mode : auto   Stop mode : auto

Legend: >: Indicates a user operation.

Changing the settings for how to start HA Monitor:

>monsetup -start
Start mode : manual

Legend: >: Indicates a user operation.

Changing the settings for how to terminate HA Monitor:

>monsetup -stop
Stop  mode : auto

Legend: >: Indicates a user operation.

Specifying reset path settings (when HA8000 or BladeSymphony is used and LPARs are not run):

> monsetup -resetpath
   The reset path for system reset will be set up.
   Please push [CTRL]+[D] key to stop setup.

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->1
Please select the model.
1. HA8000 or BladeSymphony
2. BladeSymphony (LPAR)
3. Virtual machine on VMware ESXi
4. RV3000
---> 1
Please input the partition name of the system by which HA monitor works.
Current Setting : not configured
--->MS0001
Please input the IP address of the reset path.
Current Setting : not configured
--->192.168.0.1
Please input the port number of the reset path.
Current Setting : not configured
--->10000
Please input the IP address of the switch and management module.
Please input "auto" to cooperate with N+M Cold Standby function.
Current Setting : not configured
--->192.168.0.2
Please input the port number of the switch and management module.
Current Setting : not configured
--->15000
Communication with the switch and management module will be checked.
Now checking communication...
Communication was successful.
Settings of the reset path was completed. The present setting information is displayed.

Switch and management module is used.
partition name ........................................: MS0001
The IP address of the reset path ......................: 192.168.0.1
The PORT number of the reset path .....................: 10000
The IP address of the switch and management module ....: 192.168.0.2
The PORT number of the switch and management module ...: 15000

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->2
Switch and management module is used.
partition name ........................................: MS0001
The IP address of the reset path ......................: 192.168.0.1
The PORT number of the reset path .....................: 10000
The IP address of the switch and management module ....: 192.168.0.2
The PORT number of the switch and management module ...: 15000

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->3
Command will be terminated.

Legend: >: Indicates a user operation.

Specifying reset path settings (when BladeSymphony is used and LPARs are run):

> monsetup -resetpath
   The reset path for system reset will be set up.
   Please push [CTRL]+[D] key to stop setup.

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->1
Please select the model.
1. HA8000 or BladeSymphony
2. BladeSymphony (LPAR)
3. Virtual machine on VMware ESXi
4. RV3000
---> 2
Please input the partition name of the system by which HA monitor works.
Current Setting : not configured
--->MS0001
Please input the LPAR name of the system by which HA monitor works.
Current Setting : not configured
--->LPAR1
Please input the IP address of the reset path.
Current Setting : not configured
--->192.168.0.1
Please input the port number of the reset path.
Current Setting : not configured
--->10000
Please input the IP address of the switch and management module.
Please input "auto" to cooperate with N+M Cold Standby function.
Current Setting : not configured
--->192.168.0.2
Please input the port number of the switch and management module.
Current Setting : not configured
--->15000
Communication with the switch and management module will be checked.
Now checking communication...
Communication was successful.
Settings of the reset path was completed. The present setting information is displayed.

Switch and management module is used.
partition name ........................................: MS0001
LPAR name .............................................: LPAR1
The IP address of the reset path ......................: 192.168.0.1
The PORT number of the reset path .....................: 10000
The IP address of the switch and management module ....: 192.168.0.2
The PORT number of the switch and management module ...: 15000

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->2
Switch and management module is used.
partition name ........................................: MS0001
LPAR name .............................................: LPAR1
The IP address of the reset path ......................: 192.168.0.1
The PORT number of the reset path .....................: 10000
The IP address of the switch and management module ....: 192.168.0.2
The PORT number of the switch and management module ...: 15000

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->3
Command will be terminated.

Legend: >: Indicates a user operation.

Specifying reset path settings (VMware ESXi-based virtualization is used in a system other than RV3000):

> monsetup -resetpath
  The reset path for system reset will be set up.
   Please push [CTRL]+[D] key to stop setup.

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->1
Please select the model.
1. HA8000 or BladeSymphony
2. BladeSymphony (LPAR)
3. Virtual machine on VMware ESXi
4. RV3000
---> 3
Please input the partition name of the system by which HA monitor works.
Current Setting : not configured
--->MS0001
Please input the IP address of the reset path.
Current Setting : not configured
--->192.168.0.1
Please input the port number of the reset path.
Current Setting : not configured
--->10000
Please input the IP address of the switch and management module.
Please input "auto" to cooperate with N+M Cold Standby function.
Current Setting : not configured
--->192.168.0.2
Please input the port number of the switch and management module.
Current Setting : not configured
--->15000
Please input the virtual machine name by which HA monitor works.
Current Setting : not configured
--->VM0001
Please input the IP address of the VMware ESXi.
Current Setting : not configured
--->192.168.0.3
Please input the port number of the SSH connection of VMware ESXi.
Please input "default" when you use port number (22) for the SSH connection.
When the value is not input, current setting is used. Current Setting : default
--->
Please input the attestation password of the SSH connection of VMware ESXi.
Current Setting : not configured
--->
Please input the password again.
--->
Please input the machine address of the machine by which HA monitor works.
Current Setting : not configured
--->1
Communication with VMware ESXi and the switch and management module will be checked.
Now checking communication...
Communication was successful.
Settings of the reset path was completed. The present setting information is displayed.

Switch and management module is used.
partition name ........................................: MS0001
The IP address of the reset path ......................: 192.168.0.1
The PORT number of the reset path .....................: 10000
The IP address of the switch and management module ....: 192.168.0.2
The PORT number of the switch and management module ...: 15000
Virtual machine name ..................................: VM0001
The IP address of the VMware ESXi .....................: 192.168.0.3
The PORT number of the VMware ESXi ....................: default
The PASSWORD of the VMware ESXi .......................: Setting OK
Machine address .......................................: 1

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->2
Switch and management module is used.
partition name ........................................: MS0001
The IP address of the reset path ......................: 192.168.0.1
The PORT number of the reset path .....................: 10000
The IP address of the switch and management module ....: 192.168.0.2
The PORT number of the switch and management module ...: 15000
Virtual machine name ..................................: VM0001
The IP address of the VMware ESXi .....................: 192.168.0.3
The PORT number of the VMware ESXi ....................: default
The PASSWORD of the VMware ESXi .......................: Setting OK
Machine address .......................................: 1

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->3
Command will be terminated.

Legend: >: Indicates a user operation.

Specifying reset path settings (when the system is to be operated on an RV3000 in an environment other than a VMware environment):

> monsetup –resetpath
  The reset path for system reset will be set up.
Please push [CTRL]+[D] key to stop setup.

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->1
Please select the model.
1. HA8000 or BladeSymphony
2. BladeSymphony (LPAR)
3. Virtual machine on VMware ESXi (other than RV3000)
4. RV3000
--->4
Is it a virtual machine on VMware ESXi?
1. Yes
2. No.
--->2
Please input the partition name of the system by which HA monitor works.
Current Setting : not configured
--->MS0001
Please input the IP address of the reset path.
Current Setting : not configured
--->192.168.0.1
Please input the port number of the reset path.
Current Setting: not configured
--->10000
Please input the IP address of HA Monitor Connector.
Current Setting: not configured
--->192.168.0.2
Please input the port number of HA Monitor Connector.
Current Setting: not configured
--->15000
The system will now check whether communication with HA Monitor Connector is possible.
Now checking...
The system confirmed that communication is possible.
Settings of the reset path was completed. The present setting information is displayed.

HA Monitor Connector is in use.
partition name ...............................: MS0001
The IP address of the reset path .............: 192.168.0.1
The PORT number of the reset path ............: 10000
The IP address of HA Monitor Connector .......: 192.168.0.2
The PORT number of HA Monitor Connector ......: 15000

Please choose the processing to perform by the number.
1. Specify settings.
2. Display the current settings.
3. Exit
...

Legend: >: Indicates a user operation.

Specifying reset path settings (in a configuration where a virtual machine on VMware ESXi with an RV3000 and the management server is used):

> monsetup –resetpath
  The reset path for system reset will be set up.
Please push [CTRL]+[D] key to stop setup.

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->1
Please select the model.
1. HA8000 or BladeSymphony
2. BladeSymphony (LPAR)
3. Virtual machine on VMware ESXi (other than RV3000)
4. RV3000
--->4
Is it a virtual machine on VMware ESXi?
1. Yes
2. No
--->1
Are you using an RV3000 high-availability vSphere environment?
1. Yes
2. No
--->1
Please input the partition name of the system by which HA monitor works.
Current Setting: not configured
--->MS0001
Please input the virtual machine name of the system by which HA Monitor works.
Current Setting: not configured
--->VM1
Please input the IP address of the reset path.
Current Setting: not configured
--->192.168.0.1
Please input the port number of the reset path.
Current Setting: not configured
--->10000
Please input the IP address of HA Monitor Connector.
Current Setting: not configured
--->192.168.0.2
Please input the port number of HA Monitor Connector.
Current Setting: not configured
--->15000
The system will now check whether communication with HA Monitor Connector is possible.
Now checking...
The system confirmed that communication is possible.
Settings of the reset path was completed. The present setting information is displayed.

HA Monitor Connector is in use.
partition name ...............................: MS0001
Virtual machine name .........................: VM1
The IP address of the reset path .............: 192.168.0.1
The PORT number of the reset path ............: 10000
The IP address of HA Monitor Connector .......: 192.168.0.2
The PORT number of HA Monitor Connector ......: 15000

Please choose the processing to perform by the number.
1. Specify settings.
2. Display the current settings.
3. Exit
...

Legend: >: Indicates a user operation.

Specifying advanced SSH connection settings (when VMware ESXi-based virtualization is used):

> monsetup -resetpath ssh
   The SSH connection will be set up.
   Please push [CTRL]+[D] key to stop setup.

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->1
Please input the transmission interval of connected maintenance packet. (per second)
Please input "default" when you use default setting value (60).
When the value is not input, current setting is used. Current Setting : default
--->30
Please input the transmission frequency of connected maintenance packet. (per times)
Please input "default" when you use default setting value (3).
When the value is not input, current setting is used. Current Setting : default
--->2
The setting was completed.

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->2
Transmission interval of connected maintenance packet (per second)...: 30
Transmission frequency of connected maintenance packet (per times)...: 2

Please choose the processing to perform by the number.
 1. Setup
 2. The present setting information is displayed.
 3. End
--->3
Command will be terminated.

Legend: >: Indicates a user operation.

Specifying HA Monitor Extension function settings:

> monsetup -ex sysmax on
sysmax : on

Legend: >: Indicates a user operation.

Releasing HA Monitor Extension function settings:

> monsetup -ex sysmax off
sysmax : off

Legend: >: Indicates a user operation.

Displaying HA Monitor Extension function settings:

> monsetup -ex
sysmax : on

Legend: >: Indicates a user operation.

9.23 monshow (displays the status of servers and hosts)

Format

Displaying server status:

monshow [server-alias-name]

Displaying host status:

monshow -c

Displaying server group status:

monshow -g [group-name]

Displaying information about shared resources:

monshow -d [server-alias-name]

Displaying resource server status:

monshow -r

Displaying server priority when the multi-standby function is used:

monshow -p [server-alias-name]

Displaying the server failure monitoring time:

monshow -t [server-alias-name]

Displaying monitor-mode program status:

monshow -u [server-alias-name]

Displaying the server monitoring status and disk monitoring status:

monshow -o { -s [server-alias-name] | -m }

Displaying the host setting status:

monshow -j

Users who can execute the command

Superuser and general user

Description

Displaying server status:

The monshow command displays the status of the servers running on the host (local host) where this command is executed and the status of servers on paired hosts (remote hosts). The command does not display the status of inactive servers. If the multi-standby function is used, the command displays the status of all servers running on remote hosts. The command displays the following information:

Own host name: Host name of the local host

Own Server: Information about servers on the local host

For the information about servers on the local host, the command displays the following:

•Alias: Alias name of a server on the local host

•Status: Status of a server on the local host

The following table describes the local host server statuses that are displayed by the command.

Table 9‒14: Server statuses on the local host

Status

Meaning

ONL

Startup completed as the active server, or ready for hot-standby switchover

SBY

Startup completed as a standby server, or ready for hot-standby switchover

(ONL)

Starting as the active server

(SBY)

Starting as a standby server

*ONL*

Waiting for restart

*SBY*

Waiting for the active server or resource server to start

ONL??

Waiting for hot standby processing

SBY??

Waiting for grouped-system switchover

ONL>>

Terminating as the active server

SBY>>

Terminating as a standby server

>ONL

Retrying startup

+SBY+

Startup completed as a standby server in the online serverless mode

Pair Server: Information about servers on remote hosts

For the information about servers on remote hosts, the command displays the following:

•Status: Status of a server on a remote host

The following table describes the remote host server statuses that are displayed by the command.

Table 9‒15: Server statuses on a remote host

Status

Meaning

ONL

Startup completed as the active server, or ready for hot-standby switchover

SBY

Startup completed as a standby server, or ready for hot-standby switchover

(ONL)

Starting as the active server

(SBY)

Starting as a standby server

*ONL*

Waiting for restart

ONL??

Waiting for hot standby processing

SBY??

Waiting for grouped-system switchover

+SBY+

Startup completed as a standby server in the online serverless mode

•Host name: Host name of the remote host

KAMN360-I ---------------- Detailed Information ----------------: Detailed information message

Some of the following messages are displayed as detailed command information:

KAMN242-D, KAMN243-D, KAMN244-D, KAMN364-D, KAMN423-E, KAMN931-I, KAMN932-I, KAMN936-I

Displaying host status:

The command displays the status of remote hosts that are connected to the local host. The command displays the following information:

Host name: Host name

Host address: Host address

Patrol time: Host failure monitoring time

Displaying server group status:

The command displays the status of grouped servers (server groups) that are running on the host (local host) on which the command is executed. The command displays the following information:

Own host name: Host name of the local host

Group name: Group name of the server group

The command also displays the information listed below for each server in the server group. If the server group contains a resource server, the command also displays information about the resource server.

Alias: Server alias name

Status: The following information is displayed:

•Whether grouped-system switchover can be performed on the server

possible: Grouped-system switchover can be performed

impossible: Grouped-system switchover cannot be performed

•Switchover type

exchange: Switchover is enabled

no_exchange cancel: Switchover is disabled

When information about an active server is displayed, the switchover type is displayed, but when information about a standby server is displayed, the switchover type is not displayed.

Displaying information about shared resources:

The command displays information about the shared resources that are used by the active server on the host (local host) on which the command is executed. The command displays the following information:

Own host name: Host name of the local host

Server: Server alias name on the local host

*** DISK information ***: Information about a shared disk

The command displays the following information about a shared disk:

•Special file names of volume groups

•Whether the function for canceling server startup processing in the event of a connection error on a shared resource is to be used

•Options to be applied when volume groups are placed online

*** File system information ***: Information about file systems

The command displays the following information about a file system:

•File system name

•Target directory for mounting

•Mounting options

*** SCSI device information ***: Symbolic link or logical device name (when multipath software is HDLM)

*** DMMP device information ***: Logical device name (when multipath software is DMMP)

Displaying resource server status:

The command displays a resource server's status in addition to the information provided in Displaying server status.

Displaying server priority when the multi-standby function is used:

If the multi-standby function is used, the command displays the priority of a specified server or of all servers for each host, in addition to the information provided in Displaying server status. status is followed by the following information:

pri: Host's priority

Displaying the server failure monitoring time:

The command displays the server failure monitoring time of the servers that are running on the host where the command is executed.

The command displays the following information:

Own host name: Host name of the local host

Alias: Server alias name

Patrol time: Server failure monitoring time

For a server for which no server failure monitoring time has been specified, hyphens (---) are displayed.

Displaying monitor-mode program status:

The command displays the status of a program that uses the monitor-mode program management function and is running on the host (local host) where the command was executed:

Alias: Server alias name

Program Alias: Alias of the monitor-mode program

Status: Status of the monitor-mode program

The following table lists the statuses that are displayed for monitor-mode programs.

Table 9‒16: Statuses of monitor-mode programs

Status

Meaning

(ACT)

The program is being started.

ACT

The program has started.

ACT>>

The program is being terminated.

The command does not display status for a terminated program.

Patrol time: Program slowdown monitoring interval

--- is displayed for a UAP for which no program slowdown monitoring interval has been specified.

Displaying the server monitoring status and disk monitoring status:

The command displays the following monitoring statuses:

Monitoring status of the server# and monitoring status of disks for business use

Monitoring status of the system disk

#

The status of the monitoring for the patrolcommand operand in the server environment definition is not displayed.

The following items are displayed as the monitoring status of the server and disks for business use:

Own host name: Host name of the local host

Alias: Server alias name

Function: Type of monitoring

The following items are displayed:

• ptrlcmd_ex: Monitoring of the server

• sby_ptrlcmd_ex: Monitoring of the server in the standby system

• disk_ptrl: Monitoring of disks for business use

Status: Monitoring status

One of the following strings is displayed:

• patrol: Now monitoring

• patrol paused: Monitoring temporarily stopped

• ----: Monitoring not performed

The following items are displayed as the monitoring status of the system disk:

Own host name: Host name of the local host

Host name: Host name of the local host

Function: Type of monitoring

The following items are displayed:

• disk_ptrl: Monitoring of the system disk

Status: Monitoring status

One of the following strings is displayed:

• patrol: Now monitoring

• patrol paused: Monitoring temporarily stopped

• ----: Monitoring not performed

Displaying the host setting status:

The following items are displayed as the setting status of the host:

Own host name: Host name of the local host

Host name: Host name of the local host

Function: The following items are displayed:

Online serverless mode: onlless_mode

Status: Setting status

Either of the following strings is displayed:

• use: Set

• nouse: Released

Options

If no option is specified, the command displays the server status.

Displaying server status:

server-alias-name

Specifies the alias name of a server whose status is to be displayed. If this option is omitted, the command displays the status of all servers running on the host where the command is executed.

Displaying host status:

-c

Specifies that host status is to be displayed.

Displaying server group status:

-g

Specifies that server group status is to be displayed.

If a group contains a resource server, the command also displays the status of the resource server.

group-name

Specifies the group name of the server group whose status is to be displayed. If this option is omitted, the command displays the status of all server groups that are running on the host where the command is executed.

Displaying information about shared resources:

-d

Specifies that information about shared resources is to be displayed.

server-alias-name

Specifies the alias name of a server for which shared resource information is to be displayed. If this option is omitted, the command displays shared resource information for all servers.

Displaying resource server status:

-r

Specifies that resource server status information is to be displayed, in addition to the information shown in Displaying server status.

Displaying server priority when the multi-standby function is used:

-p

Specifies that when the multi-standby function is used, the priority of a specified server or of all servers on the host is to be displayed.

server-alias-name

Specifies the alias name of a server whose status and priority are to be displayed. If this option is omitted, the command displays the status and priorities of all servers that are running on the host where the command is executed. The order in which information about servers on remote hosts is displayed depends on the order in which hosts and servers were started. This order is not always the same.

Displaying the server failure monitoring time:

-t

Specifies that the server failure monitoring time of the servers running on the host where the monshow command is executed is to be displayed.

server-alias-name

Specifies the alias name of a server whose monitoring interval is to be displayed. If this option is omitted, the command displays the server failure monitoring time for all servers running on the host where the command is executed.

Displaying monitor-mode program status:

-u

Displays the status of a program that uses the monitor-mode program management function and has been running on the host where the monshow command is executed.

server-alias-name

Specifies the alias of the server whose program status is to be displayed. If this operand is omitted, the command displays the statuses of all programs that are running on the host where the command is executed.

Displaying the server monitoring status and disk monitoring status:

-o -s server-alias-name

The monitoring status of the server and disks for business use are displayed. If a server alias name is specified, the monitoring status of the server and the monitoring status of the disks for business use on that server are displayed. If a server alias name is not specified, the monitoring status of all servers and the monitoring status of the disks for business use on those servers are displayed.

-o -m

The monitoring status of the system disk on the host on which the monshow was executed is displayed.

Displaying the host setting status:

-j

The host setting status is displayed.

Return codes

Value

Meaning

0

Normal termination (the status of servers and hosts has been displayed)

Other than 0

Abnormal termination

Notes

The return code is also set to 0 when any of the following messages is issued while type2 is specified in the exitcode operand in the HA Monitor environment settings:

KAMN320-E, KAMN322-E, KAMN428-E

Examples

Displaying server status:

> monshow
KAMN213-I Own host name : host1
     Own servers           Pair servers
     Alias     Status      Status    Host name
     server1    ONL         SBY      host2
     server2    SBY        (ONL)     host2
     server3   *SBY*
     server4    ONL??
     server5   *ONL*        SBY      host2
     server6   >ONL
KAMN360-I ---------------- Detailed Information ----------------
KAMN243-D There is a fail of system reset or the Server who stops abnormally in the own host:host1. Online server:server3 has the possibility that is starting in other hosts. The initial state was changed to Standby server, and it was made a start wait state.
KAMN364-D Server:server4 is Hotstandby wait state, because system reset is failed.

Legend: >: Indicates a user operation.

Displaying server status (when using the multi-standby function):

> monshow
KAMN213-I Own host name : host1
     Own servers           Pair servers
     Alias     Status      Status    Host name
     server1    ONL         SBY      host3
                            SBY      host2
                            SBY      host4
     server2    ONL         SBY      host2
     server3   *SBY*
     server4    SBY        (SBY)     host4
                           (ONL)     host2
                            SBY      host3
     server6   >ONL

Legend: >: Indicates a user operation.

The order in which information about servers on remote hosts is displayed depends on the order in which the hosts and servers were started. This order is not always the same.

Displaying host status:

> monshow -c
KAMN335-I Connected host information
     Host name                           Host address     Patrol time
     host1                                     100             10
     host2                                     200             10

Legend: >: Indicates a user operation.

Displaying server group status:

> monshow -g
KAMN213-I Own host name : host1
      Group name : groupA
         Alias : server1   Status :   possible       exchange
         Alias : server2   Status :   possible    no_exchange       cancel
      Group name : groupB
         Alias : server3   Status : impossible       exchange
         Alias : server4   Status : impossible    no_exchange       cancel

Legend: >: Indicates a user operation.

Displaying information about shared resources:

> monshow -d
KAMN213-I Own host name : host1
 Server  :  server1
     ***   DISK   information   ***
     -NAME----------------------------
      /dev/vg01
        neck
        on_opt="-a y -q n"
     ***   File system information  ***
     -NAME----------------------------
      /dev/vg01/lv_fs1
        mount dir=/fs1
        mount opt=-o log

     ***   SCSI device information  ***
     -NAME----------------------------
      /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000000
      /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000001
      /dev/disk/by-id/scsi-360060e8010462fe004f2b6ae00000002
     ***   DMMP device information  ***
     -NAME----------------------------
      /dev/mapper/mpath1
      /dev/mapper/mpath2
      /dev/mapper/mpath3

Legend: >: Indicates a user operation.

Displaying resource server status (ressrv0 is the alias name of a resource server):

> monshow -r
KAMN213-I Own host name : host1
     Own servers           Pair servers
     Alias     Status      Status    Host name
     ressrv0    ONL
     server1    ONL
     server2    ONL
     server6   >ONL

Legend: >: Indicates a user operation.

Displaying server priority when the multi-standby function is used:

> monshow -p
KAMN213-I Own host name : host1
     Own servers               Pair servers
     Alias     Status   pri    Status   pri   Host name
     server1    ONL     0       SBY     2     host3
                                SBY     1     host2
                                SBY     3     host4
     server2    ONL     0       SBY     1     host2
     server3    SBY     1       ONL     0     host3
     server4    SBY     1       SBY     3     host4
                                ONL     0     host2
                                SBY     2     host3
     server6   >ONL     0

Legend: >: Indicates a user operation.

The order in which information about servers on remote hosts is displayed depends on the order in which the hosts and servers were started. This order is not always the same.

Displaying the server failure monitoring time:

> monshow -t
KAMN213-I   Own host name : host1
     Alias     Patrol time
     server1        60
     server2         5
     server3       600
     server4       ---

Legend: >: Indicates a user operation.

Displaying monitor-mode program status:

> monshow -u
KAMN213-I Own host name : host1
     Alias     Program Alias  Status   Patrol time
     server1   UAP1            ACT        3600
               UAP2           (ACT)         60
     server2   UAP3            ACT>>      ----

Legend: >: Indicates a user operation.

Displaying the server monitoring status and disk monitoring status:

> monshow -o -s
KAMN213-I Own host name : host1
     Alias     Function             Status
     server1   ptrlcmd_ex           patrol
               sby_ptrlcmd_ex       ----
               disk_ptrl            ----
     server2   ptrlcmd_ex           patrol paused
               sby_ptrlcmd_ex       ----
               disk_ptrl            patrol
     server3   ptrlcmd_ex           ----
               sby_ptrlcmd_ex       ----
               disk_ptrl            patrol paused

> monshow -o -s server1
KAMN213-I Own host name : host1
     Alias     Function             Status
     server1   ptrlcmd_ex           patrol
               sby_ptrlcmd_ex       ----
               disk_ptrl            ----

> monshow -o -m
KAMN213-I Own host name : host1
     Host name Function             Status
     host1     disk_ptrl            patrol

Legend: >: Indicates a user operation.

Displaying the host setting status:

> monshow -j
KAMN213-I Own host name : host1
     Host name Function             Status
     host1     onlless_mode         use

Legend: >: Indicates a user operation.

9.24 monstart (starts HA Monitor)

Format

monstart

Users who can execute the command

Superuser and general user

Description

The monstart command starts HA Monitor.

Options

None

Return codes

Value

Meaning

0

Normal termination (HA Monitor has started)

Other than 0

Abnormal termination

9.25 monstop (terminates HA Monitor)

Format

monstop

Users who can execute the command

Superuser and general user

Description

The monstop command terminates HA Monitor.

Options

None

Return codes

Value

Meaning

0

Normal termination (HA Monitor has terminated)

Other than 0

Abnormal termination

Notes

The monstop command is not accepted if a server is running.

9.26 monswap (performs server hot-standby switchover)

Format

Planned server hot-standby switchover

monswap {server-alias-name|-g group-name}

Forced server hot-standby switchover

monswap -f {server-alias-name|-g group-name}

Users who can execute the command

Superuser

Description

This command performs planned or forced server hot-standby switchover.

Planned server hot-standby switchover

The monswap -g command performs planned hot-standby switchover processing on the entire server group for grouped-system switchover. If the server group in the standby system contains a standby server that is in the grouped-system switchover wait state, you can start that standby server as the active server by executing the monswap -g command to perform hot standby processing on the entire server group.

When the multi-standby function is used, the monswap command performs planned hot standby processing on the standby server that has the highest priority. If a standby server is running in the primary system (the server for which online is specified in the initial operand in the server environment definition is running as a standby server), the command treats that standby server as having the highest priority.

Note that you can execute this command on a host on which an active server subject to planned hot standby is running.

Forced server hot-standby switchover

The monswap -f command forcibly terminates the OS on which the active server is operating, and then performs hot-standby switchover.

If the multi-standby function is used, the monswap command performs forced hot-standby switchover to the standby server that has the highest priority. If a standby server is running in the primary system (the server for which online is specified in the initial operand in the server environment definition is running as a standby server), the command treats that standby server as having the highest priority.

Note that you can execute this command from any of the following components: the forced hot-standby switchover target server, the active system in which the forced hot-standby switchover target group exists, and the standby system in which the forced hot-standby switchover target group exists.

Options

server-alias-name

Specifies the alias name of the server on which planned hot standby processing is to be performed. Note that this option is applicable to a server that is not grouped.

-g group-name

Specifies the group name of a server group on which planned hot standby processing is to be performed.

You can use this option for a server group that is ready for grouped-system switchover.

You cannot use this option if only one server in the group is ready for hot-standby switchover. In this case, execute the monswap command with the server alias name specified.

-f server-alias-name

Specifies the alias name of the server on which forced hot-standby switchover is to be performed. You can use this option for a server that is paired with another server.

-f -g group-name

Specifies the group name of a server group on which forced hot-standby switchover is to be performed. You can use this option for the server group that is ready for grouped-system switchover and that contains servers paired with other servers.

You cannot use this option if only one server in the group is ready for hot-standby switchover. In this case, execute the monswap -f command with the server alias name specified.

Return codes

Value

Meaning

0

Normal termination (the planned hot standby request has been accepted)

Other than 0

Abnormal termination

Examples

Performing forced server hot-standby switchover (monswap -f server-alias-name):

> monswap -f server1
To perform forced system switching of server:server1, a forced stop will be performed on host:host1.
To continue this process, enter YES (in capital letters).
If you enter anything other than YES, this command processing will stop.
--->YES
KAMN703-I Performing a forced stop on host:host1.
KAMN459-I Command:monswap was terminated.

Legend: >: Indicates a user operation.

Performing forced server hot-standby switchover (monswap -f -g group-name):

> monswap -f -g groupA
To perform forced system switching of group:groupA, a forced stop will be performed on host:host1.
To continue this process, enter YES (in capital letters).
If you enter anything other than YES, this command processing will stop.
--->YES
KAMN703-I Performing a forced stop on host:host1.
KAMN459-I Command:monswap was terminated.

Legend: >: Indicates a user operation.

Notes

The return code is also set to 0 when any of the following messages is issued while type2 is specified in the exitcode operand in the HA Monitor environment settings:

KAMN212-E, KAMN215-E, KAMN216-E, KAMN322-E, KAMN323-E, KAMN328-E, KAMN330-E, KAMN426-E, and KAMN431-E

Planned or forced hot-standby switchover is performed asynchronously with the time when the monswap command returns a result.

If forced hot-standby switchover is performed, all servers that are operating on the same host terminate to forcibly terminate the OS of the host on which the target server or group is operating as the active server.

Note that only a server that is ready for hot-standby switchover (standby server on which the KAMN252-I message has been output) is subject to forced hot-standby switchover.

9.27 monts (collects HA Monitor troubleshooting information)

Format

monts

Users who can execute the command

Superuser

Description

The monts command collects in a regular file information required for troubleshooting HA Monitor. The file is output in tar format.

Options

None

Return codes

Value

Meaning

0

Normal termination (HA Monitor troubleshooting information has been collected)

Other than 0

Abnormal termination

Example

>monts

### Collect HAmonitor's trouble-shoot informations DATE:2012/04/19 20:54:07 ###

This command is collecting HAmonitor's trouble-shoot informations to the file.
   Input the file name. If specified file name only, output file will be made on /tmp.
   (If you Hit only Enter-key, assumed /tmp/monts.tar)
=>
>/tmp/monts.tar
Please push a return key.
   =>
>
   Now outputting error information to the file. Wait for a while.
   Execution is Completed.

Legend: >: Indicates a user operation.

Notes

This command is executed interactively. If you do not want to execute it interactively (you want to execute it non-interactively), create a shell as shown below:

#!/bin/sh
# syntax : monts.sh outputfile-pathname
TARFILE="$1"
MONTS_CMD="/opt/hitachi/HAmon/bin/monts"
echo "${TARFILE}" | ${MONTS_CMD}
exit 0

This example specifies the absolute path of the output destination file in the first argument when the shell is executed.

10 APIs

This chapter explains the APIs that can be included in UAPs.

10.1 List of APIs

The following table lists and describes the APIs that are used by HA Monitor.

Table 10‒1: List of APIs used by HA Monitor

API

Description

hamon_patrolstart

Issues a request to HA Monitor to start monitoring of the UAP.

hamon_patrolstop

Issues a request to HA Monitor to terminate monitoring of the UAP.

10.2 Headings used in the explanation of APIs

This section explains the headings used in the explanations of the APIs.

Format

Indicates the header file required for the API and the format of the API.

Description

Describes the API.

Arguments whose values are to be set by the UAP

Explains the arguments whose values are to be set by the UAP.

Arguments whose values are returned by HA Monitor

Explains the arguments whose values are returned by HA Monitor.

Return codes

Explains the return codes of the API.

Notes

Provides notes about using the API.

10.3 hamon_patrolstart (starts UAP monitoring)

Format

#include <HA_mon_uap.h>
long hamon_patrolstart(int signum,int *inf,void *opt);

Description

The hamon_patrolstart function issues a request to HA Monitor to start monitoring of the UAP.

For a UAP to be started, the program name specified in the name operand in the monitor-mode program environment definition must be set in the HAMON_UAPNAME environment variable.

If nouse is set in the HAMON_API_PATROL environment variable, the function returns with return code 0 without issuing a request to HA Monitor.

For details about the UAP environment variables, see 6.15.2 Specifying UAP environment variables.

Arguments whose values are to be set by the UAP

signum

Specifies a signal number. Specify either 0 or the number of a signal that can be acquired for this value. If you want to monitor UAP processes as well as the UAP operation reports, specify a nonzero value. If you want to monitor only UAP processes, specify 0.

If a nonzero value is specified in this argument but the patrol operand is omitted in the monitor-mode program environment definition, HA Monitor monitors only the UAP processes.

opt

This is a reserved area. Specify NULL.

Arguments whose values are returned by HA Monitor

inf

This area is used for returning a detail code in the event of an error. For details about the detail codes, see 10.5 List of detail codes below.

Return codes

Value

Meaning

0

HA Monitor will begin monitoring the UAP that issued this function.

-1

An error occurred. HA Monitor will not monitor the UAP that issued this function. If HA Monitor is already monitoring the UAP, it continues to monitor it. For details, see 10.5 List of detail codes below.

Notes

Do not issue this function multiple times consecutively from the same process.

Only a single thread can issue the API per process.

10.4 hamon_patrolstop (terminates UAP monitoring)

Format

#include <HA_mon_uap.h>
long hamon_patrolstop(int *inf, void *opt);

Description

The hamon_patrolstop function issues a request to HA Monitor to terminate monitoring of the UAP. If the UAP is terminated while it is being monitored, HA Monitor detects a UAP error. For this reason, you must issue this function before you terminate the UAP.

For a UAP to be started, the program name specified in the name operand in the monitor-mode program environment definition must be set in the HAMON_UAPNAME environment variable.

If nouse is set in the HAMON_API_PATROL environment variable, the function returns with return code 0 without issuing a request to HA Monitor.

For details about the UAP environment variables, see 6.15.2 Specifying UAP environment variables.

Arguments whose values are to be set by the UAP

opt

This is a reserved area. Specify NULL.

Arguments whose values are returned by HA Monitor

inf

This area is used for returning a detail code in the event of an error. For details about the detail codes, see 10.5 List of detail codes below.

Return codes

Value

Meaning

0

HA Monitor will stop monitoring the UAP that issued this function.

-1

An error occurred. For details, see 10.5 List of detail codes below.

Notes

Issue this function from the process (thread) that issued the hamon_patrolstart function.

Do not issue this function multiple times consecutively from the same process.

Only a single thread can issue the API per process.

10.5 List of detail codes

The table below lists and describes the detail codes that the APIs return to the inf argument

Note that a detail code is set only when the pointer specified in the inf argument is not NULL and the return code is -1.

Table 10‒2: List of detail codes

Detail code (numeric value)

Meaning

Message output to syslog

1

One of the following:

HA Monitor has not started.

The version of the HA Monitor being used is invalid.

Y

2

An invalid signal number was specified in the signum argument in the hamon_patrolstart function.

Y

3

HA Monitor has already started monitoring the UAP that issued the API.

N

4

One of the following:

The HAMON_UAPNAME environment variable has not been specified.

The program name specified in the HAMON_UAPNAME environment variable is null or consists of more than 1,000 characters.

The UAP corresponding to the program name specified in the HAMON_UAPNAME environment variable is not defined in the monitor-mode program environment definition.

The server managing the UAP that corresponds to the program name specified in the HAMON_UAPNAME environment variable is not running as the active server.

Y

5

One of the following:

HA Monitor is not monitoring the UAP that issued the API.

The number of active programs has reached the number specified in the pgmmax operand in the HA Monitor environment settings.

N

6

System call error

Y

Legend:

Y: The API outputs a message to syslog.

N: The API does not output a message to syslog.

10.6 Coding example

The following shows a coding example for a UAP.

#include <stdio.h>
#include <signal.h>
#include <stdlib.h>
#include "HA_mon_uap.h"

int main(int argc, char *argv[])
{

/*  define variables  */
    int     rt = 0;
    long    rc1 = 1;
    long    rc2 = 1;
    int     signum;
    int     inf;

/*  issue the hamon_patrolstart function  */
    signum = SIGUSR2;
    rc1 = hamon_patrolstart(signum, &inf, NULL);
    if(rc1 != 0){
            rt = rc1;
            goto END_OF_FUNC;
    }
    else;

    while (1) {

            :
            :
         job-processing
            :
            :
    }

/*  issue the hamon_patrolstop function  */
    rc2 = hamon_patrolstop(&inf, NULL);
    if(rc2 != 0){
                rt = rc2;
                goto END_OF_FUNC;
    }
    else;

    return(rt);
}

Appendixes

A. Event IDs Used by HA Monitor

You can link HA Monitor to JP1 products to achieve system automation. If events such as HA Monitor startup or server abnormal termination occur, HA Monitor issues events to notify JP1.

The events issued by HA Monitor are managed by JP1/Base. For details about event management, see the JP1/Base User's Guide.

This section explains the JP1 events that are issued by HA Monitor. You can use a program such as the integrated JP1 management window to check the items listed below.

Figure A‒1: List of items in a JP1 event that is issued by HA Monitor

Event ID

Displays the JP1 event ID assigned to HA Monitor.

Corresponding HA Monitor message

Displays the HA Monitor message ID and message text that correspond to the event.

Severity

Displays the severity level of the event in JP1. One of the following strings is displayed:

Error

Error

Warning

Warning

Information

Information

User name

Displays a superuser name.

Product name

Displays /HITACHI/HAmon.

Object type

Displays the type of the HA Monitor object that caused the JP1 event to be issued. One of the following strings is displayed:

HAHOST

Event related to HA Monitor itself

HASERV

Event related to a server

HARESET

Event related to a reset path

HAPATH

Event related to a monitoring path

HASERVPATH

Event related to a server path error

Object name

Displays the object name that corresponds to the object type. One of the following strings is displayed:

HA Monitor's host name

This is an object that corresponds to HAHOST.

Server alias name or server path name

This is an object that corresponds to HASERV.

HA Monitor's host name at the reset target

This is an object that corresponds to HARESET.

Host name of the monitoring path

This is an object that corresponds to HAPATH.

Server path name

This is an object that corresponds to HASERVPATH.

Registration name type

Displays HAmon.

Registration name

Displays the host name for the HA Monitor that issued the event.

Event type

Displays the status of the object set by the object type in which the event is issued. One of the following strings is displayed:

START

The object was started.

END

The object was terminated

NOTSTART

The object could not be started.

RESTART

Re-execution of the object was started.

SWITCH

Hot standby processing occurred.

NOTICE

The user is being notified of the event.

EXCEPTION

Other error occurred.

Start time

Displays the time the event occurred.

The following table lists the event items whose contents depend on the event.

Table A‒1: Contents of events issued by HA Monitor

Event issuance timing

Event ID

Corresponding HA Monitor message

Severity

Object type

Object name

Event type

When HA Monitor starts

00010200

KAMN002-I

Information

HAHOST

HA Monitor's host name

START

When HA Monitor terminates normally

00010201

KAMN050-I

Information

END

KAMN056-I

When HA Monitor terminates abnormally

00010202

KAMN003-E#1

Error

NOTSTART

KAMN205-E

KAMN207-E

KAMN208-E

KAMN617-E

EXCEPTION

When a server starts

00010203

KAMN241-I

Information

HASERV

Server alias name

START

KAMN251-I

KAMN254-I

RESTART

KAMN274-I

KAMN310-I

SWITCH

KAMN311-I

When a server terminates normally

00010204

KAMN280-I

Information

END

KAMN281-I

KAMN290-I

When a server error has occurred or a server terminates abnormally

00010205

KAMN253-E

Error

EXCEPTION

KAMN266-I

KAMN272-E

KAMN273-E#2

KAMN275-E

KAMN276-E

KAMN295-I

KAMN300-E

KAMN301-E

KAMN517-E

KAMN305-E

NOTSTART

KAMN306-E

KAMN307-E

KAMN312-E

KAMN315-E

KAMN316-E

KAMN384-E

KAMN423-E

KAMN237-E

HASERVPATH

Server path name

When a failure is detected on the remote host

00010206

KAMN340-E

Error

HAHOST

HA Monitor's host name on the host resulting in the failure

EXCEPTION

When a reset path failure or a reset error has occurred

00010207

KAMN347-E

Error

HARESET

HA Monitor's host name at the reset target

KAMN399-E

When a monitoring path failure has occurred

00010208

KAMN177-E

Error

HAPATH

Host name of the monitoring path

KAMN607-E

KAMN640-E

KAMN174-W

Warning

KAMN641-W

When HA Monitor is waiting for operator intervention#3

00010209

KAMN238-D

Warning

HASERV

Server alias name

WAIT

KAMN258-D

KAMN368-D

#1: This event is not issued at the following times: when HA Monitor starts, before the KAMN001-I message is issued, when HA Monitor fails to start due to an analysis error in the HA Monitor definition file.

#2: If the KAMN273-E message is output together with the KAMN300-E, KAMN305-E, KAMN312-E, or KAMN315-E message, the KAMN273-E message is not issued as an event. Only the messages that are output together with that message are issued as events.

#3: An operator intervention event might occur multiple times. In such a case, only the first such event that occurs is issued.

A.1 Event issuance timing when HA Monitor starts

The following figure shows the event issuance timing when HA Monitor starts.

Figure A‒2: Event issuance timing when HA Monitor starts

A.2 Event issuance timing when HA Monitor terminates normally

The following figure shows the event issuance timing when HA Monitor terminates normally.

Figure A‒3: Event issuance timing when HA Monitor terminates normally

The following figure shows the event issuance timing when automatic termination has occurred due to HA Monitor shutdown.

Figure A‒4: Event issuance timing when automatic termination has occurred due to HA Monitor shutdown.

A.3 Event issuance timing when HA Monitor terminates abnormally

The following figure shows the event issuance timing when HA Monitor terminates abnormally.

Figure A‒5: Event issuance timing when HA Monitor terminates abnormally

The following figure shows the event issuance timing when HA Monitor startup has failed.

Figure A‒6: Event issuance timing when HA Monitor startup has failed

The following figure shows the event issuance timing when multiple HA Monitors have failed to start.

Figure A‒7: Event issuance timing when multiple HA Monitors have failed to start

A.4 Event issuance timing when a server starts

The following figure shows the event issuance timing when a server starts.

Figure A‒8: Event issuance timing when a server starts

A.5 Event issuance timing when a server terminates normally

The following figure shows the event issuance timing when a server terminates normally.

Figure A‒9: Event issuance timing when a server terminates normally

The following figure shows the event issuance timing when planned server termination occurs.

Figure A‒10: Event issuance timing when planned server termination occurs

A.6 Event issuance timing when server startup fails

The following figure shows the event issuance timing when server startup has failed.

Figure A‒11: Event issuance timing when server startup has failed

A.7 Event issuance timing when a server failure has occurred

The following figure shows the event issuance timing when a server failure has occurred while switch is specified in the switchtype operand in the server environment definition.

Figure A‒12: Event issuance timing when a server failure has occurred (switchtype=switch)

The following figure shows the event issuance timing when a server failure has occurred while restart is specified in the switchtype operand in the server environment definition.

Figure A‒13: Event issuance timing when a server failure has occurred (switchtype=restart)

The following figure shows the event issuance timing when a server failure has occurred while manual is specified in the switchtype operand in the server environment definition.

Figure A‒14: Event issuance timing when a server failure has occurred (switchtype=manual)

The following figure shows the event issuance timing when a server restart limit is reached while restart is specified in the switchtype operand in the server environment definition.

Figure A‒15: Event issuance timing when a server restart limit is reached (switchtype=restart)

The following figure shows the event issuance timing when a server restart limit is reached while manual is specified in the switchtype operand in the server environment definition.

Figure A‒16: Event issuance timing when a server restart limit is reached (switchtype=manual)

The following figure shows the event issuance timing when a standby server terminates abnormally.

Figure A‒17: Event issuance timing when a standby server terminates abnormally.

The following figure shows the event issuance timing when a failure is detected on a server in the monitor mode while server processes are being monitored by using the server monitoring command.

Figure A‒18: Event issuance timing when a failure is detected on a server in the monitor mode

The following figure shows the event issuance timing when a duplicate active server is started.

Figure A‒19: Event issuance timing when a duplicate active server is started

The following figure shows the event issuance timing when an automatic standby server start request has failed on a server in the monitor mode.

Figure A‒20: Event issuance timing when an automatic standby server start request has failed on a server in the monitor mode

The following figure shows the event issuance timing when a server group is placed in the grouped-system switchover wait state while no_exchange is specified in the group operand in the server environment definition.

Figure A‒21: Event issuance timing when a server failure has occurred (group=no_exchange)

A.8 Event issuance timing when a failure is detected on the remote host

The following figure shows the event issuance timing when a host failure has occurred.

Figure A‒22: Event issuance timing when a host failure has occurred

A.9 Event issuance timing when a reset path failure has occurred

The following figure shows the event issuance timing when a reset path failure has occurred.

Figure A‒23: Event issuance timing when a reset path failure has occurred

A.10 Event issuance timing when a host reset has failed

The following figure shows the event issuance timing when a host reset has failed due to a host failure.

Figure A‒24: Event issuance timing when a host reset has failed

A.11 Event issuance timing when a monitoring path failure has occurred

The following figure shows the event issuance timing when a monitoring path failure has occurred while HA Monitor was starting.

Figure A‒25: Event issuance timing when a monitoring path failure has occurred (while HA Monitor was starting)

The following figure shows the event issuance timing when a monitoring path failure has occurred while HA Monitor was running.

Figure A‒26: Event issuance timing when a monitoring path failure has occurred (while HA Monitor was running)

A.12 Event issuance timing when HA Monitor is waiting for operator intervention

The following figure shows the event issuance timing when HA Monitor is waiting for operator intervention because the active server is inactive.

Figure A‒27: Event issuance timing when HA Monitor is waiting for operator intervention because the active server is inactive

B. Messages Output as Monitoring History

When information about server and host slowdowns is collected as monitoring history, the collected information is output to a monitoring history file. This section explains the messages that are issued as monitoring history and the monitoring history output timing. For details about the output format for a monitoring history file, see (1) Output format for a monitoring history file.

The following table lists and describes the messages that are issued as monitoring history.

Table B‒1: List of messages that are issued as monitoring history

ID

Type

Message text

Explanation of the message

0101

H

Acquisition of the patrol history has started.

Collection of monitoring history has started for the remote host indicated by its host name.

0102

H

Acquisition of the patrol history has ended.

Collection of monitoring history has ended for the remote host indicated by its host name.

0103

H

The alive message was not received from the other system.

No alive message was received from the remote host indicated by its host name within the period indicated (in seconds).

0201

S

Acquisition of the patrol history has started.

Collection of monitoring history has started for the active server indicated by its server alias name.

0202

S

Acquisition of the patrol history has ended.

Collection of monitoring history has ended for the active server indicated by its server alias name.

0203

S

Slowdown of Online server was detected.

No operation report was received from the active server indicated by its server alias name within the period indicated (in seconds).

0301

O

Acquisition of the patrol history has started.

Collection of monitoring history has started for the local host.

0302

O

Acquisition of the patrol history has ended.

Collection of monitoring history has ended for the local host.

0303

O

Slowdown of HAmonitor in the own host was detected.

Slowdown had occurred on the active system for the period indicated (in seconds).

Legend:

H: Remote host

S: Active server running on the local host

O: Local host

B.1 Server monitoring history output timing

This subsection explains the times at which HA Monitor outputs active server monitoring history.

When the active server starts on the local host

HA Monitor issues a notification message that collection of server monitoring history has started.

When HA Monitor has received an operation report from the active server after the amount of time set for collection of server monitoring history

A message indicating occurrence of a slowdown is issued.

When a failure has occurred on the active server

A message indicating occurrence of slowdown and a message indicating that collection of monitoring history has ended are issued.

When the active server or HA Monitor on the local host has terminated

A message indicating that collection of monitoring history has ended is issued.

The following figure shows an example of server monitoring history output timing.

Figure B‒1: Server monitoring history output timing

The following explains the server status at the times indicated by numbers in the figure:

The active server starts on the local host.

HA Monitor starts monitoring the active server and collection of monitoring history (ID: 0201).

A slowdown occurs that lasts longer than the amount of time set for collection of server monitoring history.

The slowdown status is resolved before the server failure monitoring time is reached.

HA Monitor collects monitoring history (ID: 0203) when it has received the server's operation report.

The active server is terminated.

HA Monitor terminates monitoring of the active server and collects monitoring history (ID: 0202).

B.2 Remote host monitoring history output timing

This subsection explains the times at which HA Monitor outputs remote host monitoring history.

When HA Monitor on the local host starts monitoring between hosts

HA Monitor issues a message reporting that collection of monitoring history has started.

When HA Monitor has received an alive message from the remote host after the amount of time set for collection of host monitoring history

A message indicating occurrence of a slowdown is issued.

When the remote host is terminated abnormally due to a failure on the remote host

A message indicating that collection of monitoring history has ended is issued. If the cause of the failure is a slowdown, a message indicating occurrence of the slowdown is also issued.

When servers on the local host or the remote host have terminated and no other server is available for hot standby processing

A message indicating that collection of monitoring history has ended is issued.

When HA Monitor on the local host or the remote host has terminated

A message indicating that collection of monitoring history has ended is issued.

The following figure shows an example of remote host monitoring history output timing.

Figure B‒2: Remote host monitoring history output timing

The following explains the remote host status at the times indicated by numbers in the figure:

The standby server starts.

HA Monitor on the local host starts monitoring operations between hosts because the hosts are now ready for hot standby processing and starts collection of monitoring history (ID: 0101).

A slowdown occurs that lasts longer than the amount of time set for collection of host monitoring history.

The slowdown status is resolved before the host failure monitoring time is reached.

HA Monitor on the local host collects monitoring history (ID: 0103) when it has received an alive message.

The active server is terminated.

Because hot standby processing can no longer be performed, HA Monitor on the local host terminates monitoring of operations between hosts and collects monitoring history (ID: 0102).

B.3 Local host monitoring history output timing

This subsection explains the times at which HA Monitor outputs local host monitoring history.

When HA Monitor starts on the local host

HA Monitor issues a message reporting that collection of monitoring history has started.

When the local host has recovered from a slowdown that lasted longer than the amount of time set for collection of host monitoring history

A message indicating occurrence of slowdown is issued.

When the local host has been restarted after hot standby processing resulting from a slowdown of the local host

A message indicating occurrence of a slowdown is issued.

When HA Monitor on the local host has terminated

A message indicating that collection of monitoring history has ended is issued.

The following figure shows an example of local host monitoring history output timing.

Figure B‒3: Local host monitoring history output timing

The following explains the local host status at the times indicated by numbers in the figure:

HA Monitor on the local host starts.

HA Monitor starts monitoring the local host and starts collection of monitoring history (ID: 0301).

A slowdown occurs that lasts longer than the amount of time set for collection of host monitoring history.

The slowdown status is resolved before the host failure monitoring time is reached.

HA Monitor on the local host collects monitoring history (ID: 0303) when the local host has recovered from the slowdown.

HA Monitor on the local host is terminated.

HA Monitor on the local host terminates monitoring of the local host and collects monitoring history (ID: 0302).

C. List of Files Output by HA Monitor

The table below lists the files that are output by HA Monitor.

The history, etc, and spool paths in file path names indicate locations under /opt/hitachi/HAmon.

Table C‒1: List of files output by HA Monitor

No.

Name

File path name

Creation and deletion timing

Management method

Supported version#1

Format

Maximum file size (bytes)

Spool file that can be deleted during operation#10

1

Error information transfer

Any path name

(the default during execution of the monts command is under /tmp)

When the monts command is executed

--

01-10 or later

Binary

Variable#2

--

2

Module trace

All files beginning with /opt/hitachi/HAmon/core

When HA Monitor terminates abnormally

If there are two or more files under /opt/hitachi/HAmon when HA Monitor is started, after the contents of spool/save are cleared, core under /opt/hitachi/HAmon is moved.

01-10 or later

Binary

675,840,000

--

3

Reset path configuration file

etc/.CPSS_RESET

When the reset path is set by using the monsetup command

Manages one generation.

01-62 or earlier

Text

108

--

Manages two generations: One with no suffix and one with suffix _old (generation with suffix _old is output under spool/).

01-63 or later

4

HA Monitor start information control file

etc/.CTLFILE

When HA Monitor is started

Manages one generation.

01-10 or later

Binary

32

--

5

Configuration check file for HA Monitor start/termination methods

etc/.HApwon_manual

When automatic start and termination are configured by the monsetup command

Manages one generation.

01-30 or later

Text

0

--

6

etc/.HApwoff_manual

--

7

Reset path option configuration file

etc/.RSTDENV

When the reset path is set by using the monsetup command

Manages one generation.

01-62 or earlier

Text

42

--

Manages two generations: One with no suffix and one with suffix _old (generation with suffix _old is output under spool/).

01-63 or later

8

Connection settings

etc/connection

When HA Monitor is terminated

Manages one generation.

01-10 or later

Text

Variable#3

--

9

Backup copy of server environment definition file

etc/servers.bac

When the mondevice command is executed (except when the -c option is specified)

Manages one generation.

01-10 or later

Text

Variable#4

--

10

Monitoring history

history/patrol_history

When server or host slowdown is recovered while server monitoring history is being collected (with the ph_log_size operand specified)

Manages two _old generations

(wraparound and round-robin methods).

01-12 or later

Text

10,485,760

--

11

HA Monitor start message skip information

spool/.dup_start

When an old HA Monitor start message is received from the other host

Manages two .bak generations.

01-10 or later

Binary

73,400,320

Y

12

fuser execution results

All files beginning with spool/.ERRdisk_

When a disk disconnection error occurs

Number of devices × Management of two .bak generations.

01-10 or later

Text

199,680

Y

13

haiod troubleshooting information

spool/.haiod_trace

When a slowdown or another failure occurs#6

21-generation management under spool/ and spool/haiod_trace/

01-11 or later

Text

Through 01-33

4,142,080

01-40 or later

4,142,080 to 8,749,056#5

Y

14

halrstd troubleshooting information

spool/.halrstd_trace

When a slowdown or another failure occurs#6

21-generation management under spool/ and spool/halrstd_trace/

01-50 or later

Text

4,178,944

Y

15

VMware ESXi response information

spool/.halrstdlog

When HA Monitor starts, connection is established between hosts, or a host failure is detected

Manages one generation (wraparound method).

01-50 or later

Text

1,049,120

Y

16

hcmalive troubleshooting information

spool/.hcmalive_trace

When a slowdown or another failure occurs#6

21-generation management under spool/ and spool/hcmalive_trace/

01-32 or later

Text

4,178,944

Y

17

hcmread troubleshooting information

spool/.hcmr_trace

When a slowdown or another failure occurs#6

21-generation management under spool/ and spool/hcmr_trace/

01-10 or later

Text

01-30 or earlier

20,480

01-31 to 01-33

4,178,944

01-40 or later

4,178,944 to 8,749,056#5

Y

18

JP1 error information

spool/.jp1log

When JP1 event registration processing results in an error while the JP1 event notification function is being used (jp1_event operand is specified)

Manages one generation.

01-10 or later

Text

4,096

Y

19

HA Monitor's internal message log

spool/.msglog

When an HA Monitor message is issued

Manages one generation

(wraparound method).

01-10 or later

Text

Through 01-33

917,504

01-40 or later

917,504 (if HA Monitor Extension is used, 26,240,000)

N

20

Socket file for hcmalive process control

spool/.s_alive

When HA Monitor is started or terminated

Manages one generation.

01-30 or later

Binary

0

N

21

Socket file for hcmalive command processing

spool/.s_alivecmd

When HA Monitor is started or terminated

Manages one generation.

01-31 or later

Binary

0

N

22

Socket file for commands

spool/.s_cmnd

When HA Monitor is started or terminated

Manages one generation.

01-10 or later

Binary

0

N

23

Socket file for internal event control

spool/.s_event

When HA Monitor is started or terminated

Manages one generation.

01-10 or later

Binary

0

N

24

Socket file for halrstd process control

spool/.s_halrstd

When HA Monitor is started or terminated

Manages one generation.

01-50 or later

Binary

0

N

25

Socket file for hcmread command processing

spool/.s_hcmrcmd

When HA Monitor is started or terminated

Manages one generation.

01-31 or later

Binary

0

N

26

Socket file for haiod process control

spool/.s_iod

When HA Monitor is started or terminated

Manages one generation.

01-10 or later

Binary

0

N

27

spool/.s_iod2

28

spool/.s_iodother

29

spool/.s_iodcmd

30

API interface socket for haiod UAPs

spool/.s_iod_up

When HA Monitor is started or terminated

Manages one generation.

01-42 or later

Binary

0

N

31

Socket file for prcmain process external event control

spool/.s_other

When HA Monitor is started or terminated

Manages one generation.

01-10 or later

Binary

0

N

32

Socket file for server control

spool/.s_path

When HA Monitor is started or terminated

Manages one generation.

01-10 or later

Binary

0

N

33

Socket file for haresetd process control

spool/.s_resetd

When HA Monitor is started or terminated

Manages one generation.

01-10 or later

Binary

0

N

34

Failure management processor response information

spool/.sphralog

When HA Monitor starts, when the reset path fails, or when failure management processor commands are sent or received

Manages one generation

(wraparound method).

01-10 or later

Text

Through 01-33

21,248

01-40 or later

21,248 (if HA Monitor Extension is used, 1,049,120)

01-53 or later

524,560

N

35

Command log

spool/cmdlog/.cmdlog1

When HA Monitor's commands and user commands are executed or terminated

Manages one generation

(wraparound method).

01-11 or later

Text

917,504

N

36

spool/cmdlog/.cmdlog2

Through 01-33

917,504

01-40 or later

917,504 (if HA Monitor Extension is used, 3,145,728)

37

spool/cmdlog/.uoclog

Through 01-33

917,504

01-40 or later

917,504 (if HA Monitor Extension is used, 83,886,080)

38

spool/cmdlog/.unamewrap.log

When moncheck and HA Monitor is started

Manages two .bak generations.

01-60 or later

Text

10,240

N

39

spool/cmdlog/.unamewrap_ins.log

When installation is performed

Text

10,240

N

40

Extended troubleshooting information

spool/DUPSTART_EX

When an old HA Monitor start message is received from the other host

Manages two .bak generations.

01-11 or later

Text

199,680

Y

41

spool/DUPSTART_EX_TOP#7

20,480

42

spool/ERRPATH_*_EX

When a monitoring path failure has occurred

199,680

43

spool/ERRPATH_*_EX_TOP#7

20,480

44

spool/HOSTSLOW_remote-host-name_EX

When a 70% slowdown has occurred on the remote host

199,680

45

spool/HOSTSLOW_remote-host-name_EX_TOP#7

20,480

46

spool/HOSTSLOW_local-host-name_EX

When the local host has recovered from a slowdown

199,680

47

spool/HOSTSLOW_local-host-name_EX_TOP#7

20,480

48

spool/MONTS_EX

When the monts command is executed

199,680

49

spool/MONTS_EX_TOP#7

20,480

50

spool/RESET_EX

When the system is reset

199,680

51

spool/RESET_EX_TOP#7

20,480

52

spool/SERVSLOW_server-name_EX

When a 70% slowdown has occurred on the server

199,680

53

spool/SERVSLOW_server-name_EX_TOP#7

20,480

54

spool/PTRLCMD_EX_SERVSLOW_server-name_EX

When the monitor-mode server monitoring function is used and a slowdown continues for 70% of the time specified (in seconds) for ptrlcmd_ex_tmout

01-65 or later

199,680

55

spool/PTRLCMD_EX_SERVSLOW_server-name_EX_TOP#7

20,480

56

Monitoring path failure information

All files beginning with spool/ERRPATH_

When a monitoring path failure has occurred

Manages the number of monitoring paths × two .bak generations.

01-10 or later

Binary

73,400,320

Y

57

Debug information during slowdown

spool/HOSTSLOW_remote-host-name

When a 70% slowdown has occurred on the remote host

Manages the number of hosts × two .bak generations.

01-10 or later

Binary

73,400,320

Y

58

spool/HOSTSLOW_local-host-name

When the local host has recovered from a slowdown

Manages two .bak generations.

59

spool/SERVSLOW_server-name

When a 70% slowdown has occurred on the server

Manages the number of servers × two .bak generations.

60

spool/UAPSLOW_UAP-alias-name

When a 70% slowdown has occurred on the UAP

Manages the number of programs × one generation.

01-42 or later

61

spool/PTRLCMD_EX_SERVSLOW_server-name

When the monitor-mode server monitoring function is used and a slowdown continues for 70% of the time specified (in seconds) for ptrlcmd_ex_tmout

Manages two .bak generations.

01-65 or later

62

LAN monitoring status

spool/lanpatrol/IF-name.log

When the LAN monitoring function is used

Manages one generation.

(wraparound method).

If the KAMN529-I or KAMN530-E message is output, the information is backed up to _judge.#8

01-40 or later

and

01-56 or later#8

Text

917,504

Y

63

spool/lanpatrol/IF-name_stat.log

4,096

64

Internal execution logs for LAN monitoring script

spool/lanpatrol/lanpatrol_IF-name.log

When the LAN monitoring function is used

Manages two _old generations

(If the LAN monitoring function is used, the information is backed up to _err when the execution time of the LAN monitoring script exceeds the LAN monitoring interval.)

If the KAMN529-I or KAMN530-E message is output, the information is backed up to _judge.#8

01-52 or later

and

01-56 or later#8

Text

5,242,880

Y

65

Error information during system reset

spool/RESET

When the system is reset

Manages two .bak generations.

01-10 or later

Binary

73,400,320

Y

66

SCSI reservation and path check error information#9

spool/scsi3PR_check_err_server-alias-name.scsilog

When a SCSI command results in an error while SCSI reservation and SCSI connection path are checked

Manages two _old generations.

01-52 or later

Text

10,485,760

Y

67

SCSI reservation information#9

spool/scsi3PR_server-alias-name.scsilog

When the server starts

When the server terminates

(when the SCSI command is issued)

Manages two generations: One with no suffix and one with suffix _old

01-52 or later

Text

10,485,760

Y

spool/scsi3PR_scsiclr_server-alias-name.scsilog

When the monscsiclr command is executed

01-58 or later

68

Statistical information

spool/sms

(spool/oldsms)

When a host or server slowdown has occurred

01-10 or earlier:

Manages two generations: One with no suffix and one with suffix _old

(Continuous increase mode)

01-11 or later:

Manages two _old generations

(wraparound and round-robin methods).

01-10 or later

Text

01-10 or earlier

Continuous increase

01-11 or later

102,400

Y

69

Available resource control information

spool/ulimit.log

When HA Monitor is started

Manages one generation.

01-22 or later

Text

703

Y

70

File system switchover information

spool/server-alias-name.fslog

When file systems are switched

Manages the number of servers × two _old generations.

01-10 or later

Text

2,147,483,647

Y

71

Multipath command execution information#9

spool/server-alias-name.mplog

When the server starts

Manages two _old generations.

01-52 or later

Text

65,535

Y

72

LAN status settings file execution results

spool/server-alias-name.up.log

When the server starts or terminates

Manages the number of servers × two .bak generations.

01-12 or later

Text

5,120

Y

73

spool/server-alias-name.down.log

74

Volume group connection and disconnection information

spool/volume-group-name.vglog

When the vgchange -a y command is executed

Manages the number of volume groups × two _old generations.

01-10 or later

Text

65,535

Y

75

Message log

All files beginning with /var/log/messages

When an HA Monitor message is issued

--

01-10 or later

Text

Size specified in /etc/syslog.conf

--

76

Settings for handling a communication error occurring when the reset path is being set

spool/.CPSS_RESET_err

When the reset path is set by using monsetup

Manages one generation.

01-63 or later

Text

108

Y

spool/.RSTDENV_err

42

77

Execution log for the monitoring command

spool/ptrlcmd_ex/server-alias-name_ptrlcmdlog

When the server whose monitoring command is specified for the ptrlcmd_ex operand is started as the active server

When the server whose monitoring command is specified for the sby_ptrlcmd_ex operand is started as the standby server

Manages three generations: One with no suffix, one with suffix .bak, and one with suffix .bak2.

01-65 or later

Text

917,504

N

78

Details log for the monitoring command

spool/ptrlcmd_ex/server-alias-name.log

When the server whose monitoring command is specified for the ptrlcmd_ex operand is started as the active server

When the server whose monitoring command is specified for the sby_ptrlcmd_ex operand is started as the standby server

Manages one generation.

01-65 or later

Text

Depends on the condition (for example, the maximum size of the sample monitoring command (patrol_ex.sh) is about 1,000).

Y

spool/ptrlcmd_ex/server-alias-name.log_err

When the server whose monitoring command is specified for the ptrlcmd_ex operand detects a server failure while operating as the active server

When the server whose monitoring command is specified for the sby_ptrlcmd_ex operand detects a server failure while operating as the standby server

79

Information returned when the reset path is initialized

spool/.gtstpxlog

When HA Monitor is started

When the reset path is restored after initialization of the reset path failed

Manages two generations: One with no suffix and one with suffix .bak

01-68 or later

Text

1,537

N

80

Disk monitoring execution status log

spool/disk_patrol/disk-monitoring-target-file's-absolute-path#11_log

When disk monitoring is started

Manages one generation.

01-69 or later

Text

10,485,760

Y

spool/disk_patrol/disk-monitoring-target-file's-absolute-path#11_log_err

When disk access for disk monitoring fails

Manages one generation.

Legend:

Y: Can be deleted.

N: Cannot be deleted.

--: Not applicable

#1

Files created by version 01-41 or earlier are supported by P-9S2C-E111 HA Monitor.

#2

The maximum file size varies because data under a directory such as /opt/hitachi/HAmon/etc is acquired in batch mode and files that are not for HA Monitor are also transferred.

#3

The maximum file size depends on the number of connected hosts and the number of monitoring paths.

#4

The maximum file size varies according to the number of device definitions.

#5

The maximum file size depends on the servmax and hostmax operand values.

#6

The file is output at one of the following times:

• When HA Monitor's troubleshooting information collection command (monts command) is executed (corresponds to No. 1 in the table)

• When a monitoring path failure is detected (corresponds to No. 52 in the table)

• When a slowdown of the other host is detected (corresponds to No. 53 in the table)

• When a slowdown of the local host is detected (corresponds to No. 54 in the table)

• When a server slowdown is detected on the local host (corresponds to No. 55 in the table)

• When a UAP slowdown is detected on the local host (corresponds to No. 56 in the table)

• When the remote host is reset (corresponds to No. 60 in the table)

#7

The top command's results are set in xxx_EX_TOP.

#8

Information is backed up to _judge in version 01-56 or later. If the function for controlling hot standby based on the availability of LAN communications is not used with version 01-55 or earlier or version 01-56 or later, no data is backed up to _judge.

#9

This is output when SCSI reservation for shared disk is used.

#10

Delete files after terminating HA Monitor. Do not delete directories.

#11

Compose the disk monitoring target file's absolute path by replacing a slash (/) with a hyphen (-).

D. List of HA Monitor Processes

The following table lists and describes the HA Monitor processes.

Table D‒1: List of HA Monitor processes

No.

Process name

Function

Number of active processes

Remarks

1

haiod

Device processing, log output, and file I/O processing

≥ 1

--

2

halrstd

(01-50 or later)

Reset processing by virtualization software from another company

0 or 1

This process is run only when use is specified in the vmware_env operand.

3

haptrld

Monitoring servers in the monitor mode

≥ 0

The process operates in the following cases:

A server for which the ptrlcmd_ex operand is specified is operating as the active server.

A server for which the sby_ptrlcmd_ex operand is specified is operating as the standby server.

4

haresetd

Reset processing by failure management processors

≥ 0

Normally there is one process. The number of processes increases by forking only when receive message logs are output.

This process is not run when nouse is specified in the fence_reset operand.

5

hcmalive

(01-30 or later)

Transmission of alive messages

0 or 1

This process is run only when the alive_interval operand is specified.

6

hcmread

Reception of host-to-host communication messages

1

--

7

mon_lanpt

(01-40 or later)

Monitoring of LANs

≥ 0

This process is not run when the LAN monitoring function is not used.

8

mondsk_mgr

(01-69 or later)

Disk monitoring

≥ 0

The process operates in the following cases:

Case where use is specified for the disk_ptrl operand in the HA Monitor environment settings and HA Monitor is running

Case where use is specified for the disk_ptrl operand in the server environment definition and the server that the monitored disks for business use belong to is running as the active server

9

mondsk_pt

(01-69 or later)

10

prcmain

Monitoring of hosts and servers and transmission of alive messages

1

--

Legend: --: Not applicable

E. Reference Material for This Manual

This appendix provides reference information, including various conventions, for this manual.

E.1 Related publications

This manual is part of a related set of manuals. The other manual in the set is listed below (with the manual number):

HA Monitor Cluster Software Messages (3000-9-203(E))

Manuals (other than HA Monitor manuals) related to this manual are listed below. Refer to these manuals as necessary.

If you are using OpenTP1

OpenTP1 Version 7 Description (3000-3-D50(E))

OpenTP1 Version 7 System Definition (3000-3-D52(E))

OpenTP1 Version 7 Operation (3000-3-D53(E))

OpenTP1 Version 7 Messages (3000-3-D56(E))

OpenTP1 Version 7 TP1/Client User's Guide TP1/Client/W.P (3000-3-D58(E))

OpenTP1 Version 7 TP1/Client User's Guide TP1/Client/J (3000-3-D59(E))

If you are using HiRDB

HiRDB Version 9 Description (3000-6-450(E)), for UNIX systems

HiRDB Version 9 Installation and Design Guide (3000-6-452(E)), for UNIX systems

HiRDB Version 9 System Definition (3000-6-453(E)), for UNIX systems

HiRDB Version 9 System Operation Guide (3000-6-454(E)), for UNIX systems

HiRDB Version 9 Command Reference (3000-6-455(E)), for UNIX systems

HiRDB Version 9 Messages (3020-6-458(E))

If you are automating system operations by event management

JP1 Version 11 JP1/Base User's Guide (3021-3-A01(E))

JP1 Version 11 JP1/Base Messages (3021-3-A02(E))

JP1 Version 12 JP1/Base User's Guide (3021-3-D65(E))

JP1 Version 12 JP1/Base Messages (3021-3-D66(E))

For disk path redundancy

Hitachi Dynamic Link Manager Software User's Guide (3000-3-F04(E)), for Linux(R) systems

This manual abbreviates the manual titles listed below. The following table lists the full title of each of these manuals together with the convention used to abbreviate its title.

Convention used for these manuals

Full title of the manual

JP1/Base User's Guide

JP1 Version 11 JP1/Base User's Guide

JP1 Version 12 JP1/Base User's Guide

OpenTP1 System Definition

OpenTP1 Version 7 System Definition

OpenTP1 Operation

OpenTP1 Version 7 Operation

HiRDB System Definition

HiRDB Version 9 System Definition (for UNIX systems)

HiRDB System Operation Guide (UNIX)

HiRDB Version 9 System Operation Guide (for UNIX systems)

E.2 Conventions: Abbreviations for product names

This manual uses the following abbreviations for product names:

Abbreviation

Full name or meaning

AIX

AIX 5L V5.1

AIX 5L V5.2

AIX 5L V5.3

AIX V6.1

AIX V7.1

AIX V7.2

EMPD

ETERNUS Multipath Drive

HCSM

Hitachi Compute Systems Manager

HDLM

Hitachi Dynamic Link Manager Software

HFC-PCM

Hitachi Fiber Channel - Path Control Manager

HI-UX/WE2

HI-UX/workstation Extended Version 2

HiRDB

HiRDB Version 9

HiRDB Accelerator Version 9

HiRDB Advanced High Availability Version 9

HiRDB Server Version 9

HiRDB Server with Additional Function Version 9

JP1/ServerConductor

JP1/ServerConductor/Advanced Agent

JP1/ServerConductor/Agent

JP1/ServerConductor/Blade Server Manager

JP1/ServerConductor/Blade Server Manager Plus

JP1/ServerConductor/Server Manager

Linux

Linux(R)

Linux

Linux (x86)

Oracle Linux 7(x86_64)

Oracle Linux 8(x86_64)

Red Hat(R) Enterprise Linux(R) Server 6 (x86)

Red Hat(R) Enterprise Linux(R) Server 6 (x86_64)

Red Hat(R) Enterprise Linux(R) Server7(x86_64)

RHEL6

Red Hat(R) Enterprise Linux(R) server 6(x86)

Red Hat(R) Enterprise Linux(R) server 6(x86_64)

RHEL7

Red Hat(R) Enterprise Linux(R) Server7(x86_64)

RHEL8

Red Hat(R) Enterprise Linux(R) Server8(x86_64)

OpenTP1

TP1/High Availability

TP1/High Availability

uCosminexus TP1/High Availability

uCosminexus TP1/High Availability (64)

TP1/Message Control

TP1/Message Control

uCosminexus TP1/Message Control

uCosminexus TP1/Message Control (64)

TP1/NET/High Availability

TP1/NET/High Availability

uCosminexus TP1/NET/High Availability

uCosminexus TP1/NET/High Availability (64)

TP1/NET/Library

TP1/NET/Library

uCosminexus TP1/NET/Library

uCosminexus TP1/NET/Library (64)

TP1/NET/TCP/IP

TP1/NET/TCP/IP

uCosminexus TP1/NET/TCP/IP

uCosminexus TP1/NET/TCP/IP (64)

TP1/Server Base

TP1/Server Base

uCosminexus TP1/Server Base

uCosminexus TP1/Server Base Enterprise Option

UNIX

UNIX(R)

vCenter

VMware vCenter Server(R)

VMware ESXi

VMware ESXi(TM)

VMware vSphere(R) ESXi(TM)

VMware vSAN

VMware vSAN(TM)

VMware vSphere Client

VMware vSphere(R) Client

E.3 Conventions: Acronyms

This manual also uses the following acronyms:

Acronym

Full name or meaning

API

Application Programming Interface

ARP

Address Resolution Protocol

BMC

Baseboard Management Controller

C/S

Client/Server

CPU

Central Processing Unit

DB

Database

DHCP

Dynamic Host Configuration Protocol

DMMP

Device Mapper Multipath

DNS

Domain Name System

EUC

Extended UNIX Code

FC

Fibre Channel

FCSW

Fibre Channel Switch

FEP

Front-End Processor

FS

File System

HA

High Availability

HBA

Host Bus Adapter

HDLM

Hitachi Dynamic Link Manager Software

HVM

Hitachi Virtage Manage

I/O

Input/Output

ICMP

Internet Control Message Protocol

IPF

Itanium Processor Family

iSCSI

Internet Small Computer System Interface

LAN

Local Area Network

LPAR

Logical Partition

LU

Logical Unit

LVM

Logical Volume Manager

MAC

Media Access Control

NFS

Network File System

OLTP

Online Transaction Processing

OS

Operating System

OSI

Open Systems Interconnection

PC

Personal Computer

PC/AT

Personal Computer/Advanced Technology

RAID

Redundant Arrays of Inexpensive Disks

SAS

Serial Attached SCSI

SCSI

Small Computer System Interface

SCSI-3 PR

Small Computer System Interface 3 Persistent Reservation

SOA

Service Oriented Architecture

SPP

Service Providing Program

SSH

Secure Shell

SVP

Service Processor

TCP/IP

Transmission Control Protocol/Internet Protocol

UAP

User Application Program

UTF-8

8-bit UCS Transformation Format

WS

Workstation

WWID

World Wide Identifier

XNF

Extended Hitachi Network Architecture Based Communication Networking Facility

E.4 Conventions: KB, MB, GB, and TB

This manual uses the following conventions:

1 KB (kilobyte) is 1,024 bytes.

1 MB (megabyte) is 1,024² bytes.

1 GB (gigabyte) is 1,024³ bytes.

1 TB (terabyte) is 1,024⁴ bytes.

Glossary

A

active server

The server that is currently performing job processing.

active server start wait state

When a standby server is to be started, it might not always be possible for some reason to confirm that the active server has started on the remote host. In such a case, HA Monitor places the startup processing on the standby server in the wait state until the status of the active server can be confirmed. Such a standby server is called being in the active server start wait state. A standby server in the start wait state waits for a user intervention. When the active server startup processing is confirmed to have started, HA Monitor then starts the standby server.

active system

The system (host) that is performing job processing by running an active server.

alias IP address function

A function that enables multiple IP addresses to be assigned to a single LAN adapter so that the LAN adapter can be shared through the use of different IP addresses.

alive message

A communication that is issued between hosts at a specified interval to determine whether the target host is running normally.

C

chassis

A rack in which a computer and its peripheral equipment are installed. When the system being used is BladeSymphony, this rack is called a chassis. Multiple systems can be configured in one chassis.

child server

A server whose startup processing begins after its parent server has started. This corresponds to a child in the parent-child relationship in server groups.

client

A machine (node) that uses various services provided by programs.

cluster configuration

A system configuration that is larger than one server system. Multiple server systems are connected via high-speed LANs and processing is distributed to different server systems. A client can perform processing by treating these server systems as a single server system.

core file

An OS-specific file in which memory information (module trace information) for a process is stored in the event the program terminates abnormally. Core files might not always be created. If an error occurs in a user-created program and its core file is available, the user can use OS commands to debug the program.

D

dual server system

A system that improves reliability and availability of the entire system by providing two sets of machines, programs, and resources for the system on which a server is run (or by sharing dual resources).

E

event ID

A number assigned to each event in order to manage the events that occur in a UNIX system individually.

G

grouped-system switchover

A function that groups multiple servers together beforehand and switches operation over to a standby server for that group if a failure occurs on any of the active servers in the group (server group). HA Monitor enables the user to specify hot standby processing for each server within the server group.

grouped-system switchover wait state

If a server failure occurs on an active server for which no_exchange is specified in the group operand in the server environment definition, HA Monitor delays hot standby processing on the standby server in the standby system. This status for a standby server is called the grouped-system switchover wait state. A standby server in the grouped-system switchover wait state waits for a user intervention. However, if a server failure occurs on an active server for which exchange is specified in the group, grouped-system switchover is also performed on the standby server in the grouped-system switchover wait state.

H

HA Monitor Extension

An HA Monitor optional product. When HA Monitor Extension is installed, hot standby control in a large system configuration can be supported.

host

A unit for a system in which one server is run per CPU. The hardware making up a system and the programs that are run on the system are referred to collectively as a host.

host slowdown

An event in which the execution time on the entire host becomes longer than usual. Its causes include concurrent execution of too many programs or a failure in communication between programs.

hot standby

A function for switching jobs to a standby system (host) or server if a failure occurs on the primary system (host) or server.

hot-standby wait state

If the standby system fails to reset the active system after a failure on the active system, and the active server is started on the standby system while the status of the active server cannot be verified, two active servers might result. To avoid this, HA Monitor delays startup of the active server on the standby system. This status at the active server is called the hot-standby wait state. A server in the hot-standby wait state cannot be started as an active server without user intervention.

I

inherited server information

Information that is inherited from an active server to a standby server when pairing is established between the active server and the standby server. If information needs to be transmitted between servers within a user command, HA Monitor's moninfo -p command (command for specifying and displaying server inheritance information) is used to specify the information beforehand and then the moninfo -g command is used to reference and display the information.

IP address

An address used with the IP protocol. The IP protocol corresponds to the network layer in the OSI basic reference model. The network layer manages addresses used to establish data transfer paths and to determine paths.

K

kernel

A core program of an OS. Its roles include management of tasks, memory, and I/O operations.

L

LAN adapter

Data conversion hardware used to connect a computer and a LAN.

lock

A control to prevent concurrent updating and deadlock on system resources when multiple requests for the same system resources result in contention. In HA Monitor, a function that prevents shared disks from being updated concurrently from both the active server and a standby server is called the lock function.

locked server

If multiple standby servers provide separate services to the standby system, the user can prevent multiple active servers from being run concurrently after hot standby processing.

When a standby server starts operation as the active server, HA Monitor stops all other standby servers running on the same host. Such a standby server that is terminated by HA Monitor is called a locked server.

LVM

Abbreviation for Logical Volume Manager, which is a kernel functions. LVM enables the user to group several physical disk devices together as a volume group and assign to them any number and size of logical volumes.

In this manual, LVM means LVM2.

M

message log

An OS function that stores issued messages in a specific file (message log file).

module trace information

A flow of module processing in HA Monitor that is collected in a module trace buffer (core file). The module trace information is first transferred to a portable medium and then is analyzed.

monitor mode

One of the server operation modes. A server in the monitor mode cannot use some of the HA Monitor functions, such as starting a server on the standby server beforehand or monitoring for server failures. However, the monitor-mode program management function makes some of those unavailable HA Monitor functions available.

multi-standby function

A function for providing multiple standby servers for one active server.

This function can protect against a system failure while the primary system is recovering from a failure compared to when only one standby server is provided per active server.

N

N+M cold standby configuration

A system configuration for BladeSymphony in which JP1/ServerConductor or HCSM is used to switch over to a standby server module when a hardware failure occurs on the primary server module that is running jobs. Because one or more standby server modules are provided, reliability is improved for handling failures.

O

operation report

Information that is sent from a server to HA Monitor at specific intervals. HA Monitor monitors the server based on this information.

OS panic

An OS kernel panic.

P

parent server

A server that must be active so that another server can be started or hot standby processing can be started. This corresponds to the parent in the parent-child relationship in server groups.

partition

A function that divides a server machine into multiple sections and runs each section as a virtual server. Dividing a server machine into multiple sections is called partitioning, and each such section is called a partition.

In BladeSymphony, which enables multiple processors to be installed in a single chassis, each processor is sometimes called a partition. Hitachi server virtualization (Virtage) enables logical partitions (LPARs) to be configured within a processor.

portable medium

A memory medium such as DAT that enables programs and data to be recorded and transported.

primary system

The system (host) that performs job processing when it is started.

program

The program (application) that actually executes jobs. HA Monitor improves system reliability by employing hot-standby configurations to achieve dual programs.

Programs can be classified into two types, programs with an HA Monitor interface and programs without an HA Monitor interface.

program with an HA Monitor interface

A program that has a dedicated HA Monitor interface. When a program with an HA Monitor interface is run in the server mode, HA Monitor monitors both host failures and server failures.

HA Monitor monitors a program with an HA Monitor interface and performs hot standby processing in the event of a failure that cannot be detected by the program.

program without an HA Monitor interface

A program that does not have a dedicated HA Monitor interface. Some of the HA Monitor functions are not available, including the function for starting programs on the standby system in advance and the function for monitoring server failures.

There is no difference in functional limitations between using programs without an HA Monitor interface and running programs with an HA Monitor interface in the monitor mode.

R

resource server

A server used only for sharing resources among multiple servers. A resource server does not provide server functions.

Shared resources are controlled for each server when no resource server is used, while shared resources are controlled for each server group when a resource server is used.

restart wait state

If a server failure occurs on an active server for which restart or manual is specified in the switchtype operand in the server environment definition, HA Monitor waits until the active server is restarted without terminating it. Such a status for an active server is called the restart wait state.

S

secondary system

A system (host) that is placed on standby mode when it is started.

server

A service that processes jobs in accordance with requests. In this manual, a program as a unit of hot standby processing is called a server.

Servers are classified mainly into servers in server mode and servers in monitor mode.

server mode

A server operation mode that can be selected when the program has an interface with HA Monitor. When a server is run in the server mode, HA Monitor monitors both host failures and server failures.

server slowdown

An event where the server execution time becomes longer than usual. Its causes include program looping and resource contention.

shared resource

A resource, such as shared disks and LANs, that can be shared between the active and a standby system. The shared resources controlled by HA Monitor include shared disks, file systems, and LANs. HA Monitor controls shared resources for each server.

A resource server can also be used to share resources among server groups.

standby server

A server currently on standby in the event of a failure on the active server.

standby system

A system (host) that is running a standby server that is on standby ready to take over in the event of a failure.

start retry function

If resource connection or server startup fails on the target host during hot standby processing and there is no available switchover host, the start retry function automatically retries startup processing on the target host.

This function retries the server startup processing until hot standby processing becomes available or as many time as specified for the retry count.

In the monitor mode, this start retry processing can be attempted even when a failure has occurred on the active server after hot standby processing.

system dump

An OS function for storing error information that cannot be narrowed down to a specific program. Memory information, information about swap area (virtual memory), and processor-specific information can be collected in a system dump. In general, a system dump is used when the cause of an error in the system cannot be identified.

T

TCP/IP

A standard communication protocol used for connection between UNIX computers. TCP/IP supports both the TCP protocol and the IP protocol.

U

UAP

A user's job created as a program. UAPs can be used as programs on a server in the monitor mode. HA Monitor can monitor UAPs by having the UAPs issue APIs.

UDP

A standard communication protocol used for connection between UNIX computers. TCP is also used for this purpose, but UDP provides a faster transfer rate and lower reliability than TCP.

user command

A command created by a user. A user command that has been registered into HA Monitor in advance can be issued automatically when HA Monitor processing is triggered by a change in the server status. User commands enable the user to use as shared resources various resources that are not controlled by HA Monitor.

Index

Numerics

1-to-1 switchover configuration[1]

example of environment settings[1]

example of hardware configuration (BladeSymphony (Basic mode)) (1 chassis)[1]

example of hardware configuration (BladeSymphony (Basic mode)) (2 chassis)[1]

example of hardware configuration (BladeSymphony (LPAR mode)) (1 chassis)[1]

example of hardware configuration (BladeSymphony (LPAR mode)) (2 chassis)[1]

example of hardware configuration (function for controlling hot standby based on availability of LAN communication)[1]

example of hardware configuration (PC/AT-compatible system from another company)[1]

hardware configuration example (model HA8000xM or earlier)[1]

hardware configuration example (model HA8000xN or later)[1]

2-to-1 switchover configuration[1]

example of environment settings[1]

example of hardware configuration (BladeSymphony (Basic mode))[1]

example of hardware configuration (BladeSymphony (LPAR mode))[1]

example of hardware configuration (function for controlling hot standby based on availability of LAN communication)[1]

example of hardware configuration (PC/AT-compatible system from another company)[1]

hardware configuration example (model HA8000xM or earlier)[1]

hardware configuration example (model HA8000xN or later)[1]

A

abbreviations for products[1]

acronyms[1]

actcommand[1]

active server[1], [2]

hot-standby switchover wait state[1]

restart limit[1]

restart monitoring time[1]

restart wait state (description)[1]

restart wait state (operation)[1]

start wait state[1]

active system[1], [2]

acttype[1]

address[1]

handling duplicated[1]

alias (program definition statement)[1]

alias (resource definition statement)[1]

alias (server definition statement)[1]

alias IP address function[1]

alive message[1], [2]

alive_interval (transmission interval)[1]

alive_multicast (multicast transmission)[1]

multicast_lan (multicast transmission)[1]

suppress_reset (suppressing host reset)[1]

transmission method[1]

alive_interval[1]

alive_multicast[1]

API[1]

coding example[1]

detail codes, list of[1]

including in UAP[1]

APIs, list of[1]

ARP request[1]

B

BMC[1]

C

cancel[1]

changing settings and operations while HA Monitor and servers are running[1]

changing settings and operations while HA Monitor and servers are running (command)[1]

chassis[1]

checking definition (command)[1]

child server[1], [2]

clearcheck (specifying host reset end time)[1]

clearwait (specifying reset publication wait time)[1]

client[1]

cluster configuration[1]

cluster hot-standby configuration[1]

example of environment settings[1]

example of hardware configuration (BladeSymphony (Basic mode))[1]

example of hardware configuration (BladeSymphony (LPAR mode))[1]

example of hardware configuration (function for controlling hot standby based on availability of LAN communication)[1]

example of hardware configuration (PC/AT-compatible system from another company)[1]

cluster hot-standby switchover configuration

hardware configuration example (model HA8000xM or earlier)[1]

hardware configuration example (model HA8000xN or later)[1]

command

changing shared resources while active server is running[1]

collecting HA Monitor troubleshooting information[1]

detecting IP address[1]

displaying status of reset path[1]

releasing reservation for shared disk[1]

terminating server in wait state[1]

commands, list of[1]

configuration

character special file[1]

DMMP (multipath software)[1]

DNS[1]

file system[1]

HA Monitor connection configuration[1]

HDLM (multipath software)[1]

kernel parameter[1]

LAN[1]

LANG environment variable[1]

monitoring path[1]

multipath software[1]

output language[1]

output of Japanese messages when HA Monitor is automatically started at system startup[1]

reset path (BladeSymphony)[1]

reset path (model HA8000xM or earlier)[1]

reset path (model HA8000xN or later)[1]

security level[1]

shared disk[1]

system clock[1]

system file[1]

system log file[1]

connect_retry[1]

connecting HA Monitors manually (command)[1]

connection[1]

connection interval and count,

connect_retry[1]

console[1]

conventions[1]

abbreviations for products[1]

acronyms[1]

diagrams[1]

fonts and symbols[1]

KB, MB, GB, and TB[1]

version numbers[1]

core file[1]

cpudown[1]

D

data redundancy using RAID[1]

definition file[1]

definition, checking[1]

dev_offlimit[1]

dev_onlimit[1]

dev_timelimit[1]

deviceoff_order (function definition statement)[1]

deviceoff_order (servers definition statement)[1]

diagram conventions[1]

directory structure[1]

disk[1]

disk failure[1]

disk for business use[1]

disk monitoring[1]

disk for business use[1]

monitoring of disk for business use[1], [2], [3], [4], [5]

monitoring of system disk[1], [2], [3], [4], [5]

system disk[1]

disk_log_size[1]

disk_ptrl (function definition statement)[1]

disk_ptrl (server definition statement)[1]

disk_ptrl_act[1]

disk_ptrl_inter (function definition statement)[1]

disk_ptrl_inter (server definition statement)[1]

disk_ptrl_retry (function definition statement)[1]

disk_ptrl_retry (server definition statement)[1]

displaying server order control status (command)[1]

displaying status of monitoring path (command)[1]

displaying status of servers and hosts (command)[1]

dmmp_device[1]

dual server system[1], [2]

E

environment definition statement[1]

error information, using monts command to collect[1]

event ID[1], [2]

exchange[1]

exclusive_servers[1]

exec_retry[1]

exitcode[1]

F

failure

collecting error information[1]

disk failure (resource failure)[1]

host failure[1]

LAN failure (resource failure)[1]

preventing slowdown of server and host due to heavy load[1]

resource failure[1]

server failure[1]

failure management processor[1]

fence_lan[1]

fence_reset[1]

fence_scsi[1]

file system

configuration[1]

connecting[1]

example of environment settings[1]

fs_log_size[1]

fs_mount_dir[1]

fs_mount_opt[1]

fs_name[1]

fs_neck[1]

fs_umount_retry[1]

handling[1]

inheriting[1]

server-alias-name.fslog[1]

server-alias-name.fslog_old[1]

switching[1]

font conventions[1]

forced hot-standby switchover (command)[1]

fs_log_size[1]

fs_mount_dir[1]

fs_mount_opt[1]

fs_name[1]

fs_neck[1]

fs_umount_retry[1]

fsck command[1]

function definition statement[1]

function for controlling hot standby based on availability of LAN communication[1]

example of environment settings[1]

example of environment settings (VMware ESXi-based virtualization)[1]

system configuration procedure[1]

function for controlling hot-standby switchover based on availability of LAN communication

fence_lan[1]

switch_judge[1]

fuser command[1]

G

GB meaning[1]

group (resource definition statement)[1]

group (server definition statement)[1]

grouped-system switchover[1], [2]

example of environment settings[1]

group (resource definition statement)[1]

group (server definition statement)[1]

planned hot standby[1]

relationship with server switchover type[1]

server group[1]

server switchover type[1]

grouped-system switchover wait state[1]

H

HA Monitor

automating startup processing[1]

checking operation[1]

failure detected by[1]

monitoring range[1]

purpose of[1]

required hardware[1]

required software[1]

starting[1]

terminating[1]

HA Monitor connection configuration file[1]

example[1]

HA Monitor environment settings[1]

changing[1]

HA Monitor Extension[1], [2]

changing maximum number of hosts[1]

changing maximum number of servers[1]

checking operation when changing maximum number of hosts[1]

configuring[1]

consideration for changing maximum number of hosts[1]

example of file settings[1]

function of[1]

hostmax (changing maximum number of hosts)[1]

servmax (changing maximum number of servers)[1]

system configuration procedure[1]

HA Monitor processing after detection of server failure (monitor mode)[1]

HAMON_API_PATROL environment variable[1]

hamon_patrolstart[1]

hamon_patrolstop[1]

HAMON_UAPNAME environment variable[1]

hbond_lacp[1]

HiRDB[1]

Hitachi server virtualization[1]

partition_reset (physical partition reset function)[1]

physical partition reset function[1]

host[1], [2]

adding[1]

address[1]

checking status[1]

collecting monitoring history[1]

cpudown[1]

fence_reset[1]

handling host start error[1]

handling reset error[1]

host failure monitoring time[1]

host pair shutdown[1]

host pair shutdown (server termination command error)[1]

host pair shutdown (user command error)[1]

name (environment definition statement)[1]

pairdown (host pair shutdown)[1]

patrol (host failure monitoring time) (environment definition statement)[1]

reset[1]

reset_type[1]

restarting host where failure occurred as standby system[1]

standbyreset[1]

suppressing reset[1]

switch_error (retry of startup at hot-standby switchover)[1]

switch_retry (retry of startup at hot-standby switchover)[1]

term_pairdown (host pair shutdown in event of server termination command error)[1]

time when host monitoring history was obtained[1]

uoc_pairdown (host pair shutdown in event of user command error)[1]

host failure[1]

automating operation after hot standby processing[1]

condition under which hot standby operation is performed[1]

detecting by using alive message[1]

monbegin_restart (automating operation after hot standby)[1]

host name

lan[1]

name (environment definition statement)[1]

host slowdown[1]

hostmax[1]

hot standby[1], [2]

automatic[1]

flow of[1]

grouped-system switchover[1]

hot-standby enabled state[1]

in N+M cold standby configuration[1]

operation when failure occurs[1]

planned (operation)[1]

planned (overview)[1]

testing[1]

hot standby flow

hot standby operation when user does not specify host that has reset priority[1]

hot standby operation when user specifies host that has reset priority[1]

hot standby operation

in event of host failure[1]

in event of server failure[1]

when multi-standby function is used[1]

when server lock is used[1]

hot standby processing when resource server is used[1]

hot-standby configuration[1]

design considerations[1]

difference in[1]

note about sharing reset path among[1]

type and characteristics[1]

hot-standby configurations, checking settings and definitions between multiple[1]

hot-standby switchover

forced (overview)[1]

forced hot-standby switchover (operation)[1]

hot-standby switchover in online serverless mode[1], [2], [3], [4]

hot-standby switchover methods[1]

hot-standby wait state[1], [2]

how server failures are detected (server mode)[1]

HUB redundancy[1]

hybrid fencing[1]

example of environment settings[1]

I

I/O path redundancy[1]

inherited server information[1]

inheriting[1]

initial[1]

IP address[1], [2]

alias IP address[1]

monitoring path[1]

stationary IP address[1]

targets to which IP addresses are to be assigned (for BladeSymphony, or model HA8000xM or earlier)[1]

targets to which IP addresses are to be assigned (for model HA8000xN or later in case where management server is used as failure management processor)[1]

to be assigned to server[1]

ip_neck[1]

J

JP1[1]

JP1 event[1]

jp1_event[1]

jp1_event[1]

K

KB meaning[1]

kernel[1]

L

lan[1]

LAN[1]

connecting[1]

inheriting[1]

ip_neck (server-alias-name.up file)[1]

server-alias-name.down file[1]

server-alias-name.up file[1]

switching[1]

LAN adapter[1]

LAN adapter redundancy[1]

LAN failure[1]

LAN monitor definition file[1]

specifying[1]

LAN monitoring[1]

hbonding[1]

lancheck_mode[1]

monitoring target[1]

LAN monitoring script, specifying[1]

LAN status settings file[1]

lan_updown[1]

lan_updown[1]

lancheck_mode[1]

lancheck_patrol[1]

lanfailswitch[1]

LANG environment variable[1]

lanport[1]

list of files output by HA Monitor[1]

list of functions[1]

load

operation for preventing failure caused by heavy load[1]

testing[1]

local disk[1]

local system, recognizing slowdown of[1]

lock[1]

locked server[1]

locking servers, example of environment settings[1]

LPAR[1]

consideration[1]

partition_reset (physical partition reset function)[1]

physical partition reset function[1]

LVM (glossary)[1]

M

MAC address[1]

management server[1]

example of environment settings (in case where server model is HA8000xN or later and management server is not used as failure management processor)[1]

HA Monitor Option[1]

HCSM[1]

manual[1]

MB meaning[1]

message log[1]

message_retry[1]

method of monitoring in standby system (monitor mode)[1]

methods of connection between failure management processor and reset path[1]

module trace information[1]

monact[1]

monbegin[1]

monbegin_restart[1]

monchange[1]

moncheck[1]

mondeact[1]

mondevice[1]

monend[1]

moninfo[1]

monitor mode[1], [2]

actcommand[1]

dealing with server failures that disable restart and hot-standby switchover processing[1]

monitoring in standby system[1]

monitoring server[1]

patrolcommand[1]

program management function[1]

ptrlcmd_ex[1]

ptrlcmd_ex_inter[1]

ptrlcmd_ex_retry[1]

ptrlcmd_ex_tmout[1]

retry_stable (server definition statement)[1]

servexec_retry[1]

start_timeout (timeout for server startup)[1]

stop_timeout (timeout for server termination)[1]

stop_timeout_sw (timeout for server termination)[1]

termcmd_at_abort (server termination command)[1]

waitserv_exec[1]

monitor-mode program

checking restart of[1]

number of programs[1]

pgmmax (changing maximum number of programs)[1]

program restart command[1]

program restart limit[1]

program slowdown monitoring interval[1]

restarting[1]

status transition[1]

monitor-mode program environment definition[1]

changing[1]

monitor-mode program management function[1]

alias[1]

example of environment settings[1]

exec_retry[1]

monitoring UAP[1]

name (program definition statement)[1]

patrol (program definition statement)[1]

pgmmax[1]

program[1]

restart_timeout[1]

restartcommand[1]

retry_stable (program definition statement)[1]

server_alias[1]

monitoring history[1]

messages issued[1]

monitoring history file[1]

ph_log_size[1]

ph_threshold (time when server monitoring history is to be obtained) (function definition statement)[1]

ph_threshold (time when server monitoring history is to be obtained) (servers definition statement)[1]

monitoring in standby system

sby_actcommand[1]

sby_ptrlcmd_ex[1]

sby_ptrlcmd_ex_inter[1]

sby_ptrlcmd_ex_retry[1]

sby_ptrlcmd_ex_tmout[1]

sby_servexec_retry[1]

sby_start_timeout[1]

sby_stop_timeout[1]

sby_termcommand[1]

monitoring LAN

example of environment settings[1]

hbond_lacp[1]

LAN monitor definition file[1]

LAN monitoring script[1]

LAN-interface-name.conf[1]

lancheck_patrol (LAN monitoring interval)[1]

lanfailswitch[1]

lanpatrol.sh[1]

patrol_100ms (LAN interface name)[1]

switchbyfail[1]

monitoring of disk for business use

disk_log_size[1]

disk_ptrl (server definition statement)[1]

disk_ptrl_act[1]

disk_ptrl_inter (server definition statement)[1]

disk_ptrl_retry (server definition statement)[1]

monitoring path[1]

/etc/hosts file[1]

/etc/services file[1]

configuration[1]

health check of[1]

host name[1]

how to connect[1]

IP address[1]

lan[1]

multiplexing[1]

pathpatrol (health check)[1]

pathpatrol_retry (re-checking status)[1]

port number[1]

preferred monitoring path[1]

monitoring paths

IP addresses assigned to multiple[1]

monlink[1]

monlistip[1]

monodrshw[1]

monpath[1]

monresbgn[1]

monresend[1]

monressbystp[1]

monrp[1]

monsbystp[1]

monscsiclr[1]

monsetup[1]

monshow[1]

monstart[1]

monstop[1]

monswap[1]

monts[1]

mounting[1]

multi-standby configuration[1]

example of environment settings (TP1/Server Base or HiRDB)[1]

example of hardware configuration (function for controlling hot standby based on availability of LAN communication)[1]

example of hardware configuration (PC/AT-compatible system from another company)[1]

hardware configuration example (model HA8000xM or earlier)[1]

hardware configuration example (model HA8000xN or later)[1]

multistandby[1]

standbypri (priority)[1]

multi-standby function[1], [2]

multistandby[1]

standbypri (priority)[1]

multicast_lan[1]

multistandby[1]

mutual hot-standby configuration[1]

example of environment settings[1]

N

N+M cold standby configuration[1], [2]

name (environment definition statement)[1]

name (program definition statement)[1]

name (server definition statement)[1]

netmask[1]

netmask[1]

network address[1]

no_cancel[1]

no_exchange[1]

notes on performing order control for TP1/Server Base and HiRDB[1]

notswitch_notify[1]

O

oldsms[1]

OpenTP1[1]

operation report[1]

options definition statement[1]

OS panic[1]

P

pairdown[1]

parameters that are passed based on change in server's status[1]

parent[1]

parent server[1], [2]

parent-child relation[1]

parent[1]

partition[1]

partition_reset[1]

PATH environment variable[1]

pathpatrol[1]

pathpatrol_retry[1]

patrol (environment definition statement)[1]

patrol (program definition statement)[1]

patrol (server definition statement)[1]

patrol_100ms[1]

patrol_type[1]

patrolcommand[1]

pgmmax[1]

ph_log_size[1]

ph_threshold (function definition statement)[1]

ph_threshold (servers definition statement)[1]

physical partition reset function[1]

partition_reset[1]

planned hot-standby switchover (command)[1]

planned hot-standby switchover and controlling switchover order after server failure in monitor mode[1]

port number[1]

portable medium[1]

primary system[1], [2]

prior confirmation[1]

processing

hot standby in event of host failure (monitor mode)[1]

hot standby in event of host failure (server mode)[1]

processing flow

for connecting and disconnecting shared resources (resource server)[1]

for hot-standby switchover if server failure occurs (in case where active server is monitored in monitor mode)[1]

for hot-standby switchover if server failure occurs (in case where monitoring is performed in standby system in monitor mode)[1]

for hot-standby switchover in event of server failure (in server mode)[1]

for starting a server (in case where active server is monitored in monitor mode)[1]

for starting a server (in server mode)[1]

for starting server (in case where monitoring is performed in standby system in monitor mode)[1]

for terminating server (in case where active server is monitored in monitor mode)[1]

for terminating server (in server mode)[1]

hot standby error in event of host failure (monitor mode)[1]

hot standby error in event of host failure (monitor mode) (when releasing in reverse order)[1]

hot standby error in event of host failure (server mode)[1]

hot standby error in event of host failure (server mode) (when releasing in reverse order)[1]

hot standby error in event of server failure (monitor mode)[1]

hot standby error in event of server failure (monitor mode) (when releasing in reverse order)[1]

hot standby error in event of server failure (server mode)[1]

hot standby error in event of server failure (server mode) (when releasing in reverse order)[1]

hot standby in event of monitoring path failure (monitor mode) (when SCSI reservation for shared disk is used) (when function for controlling hot standby based on availability of LAN communication is used)[1]

hot standby in event of monitoring path failure (server mode) (when SCSI reservation for shared disk is used) (when function for controlling hot standby based on availability of LAN communication is used)[1]

hot standby processing in event of server failure (monitor mode) (when monitoring UAP)[1]

hot standby processing in event of server failure (monitor mode) (when releasing in reverse order)[1]

hot standby processing in event of server failure (server mode) (when releasing in reverse order)[1]

hot standby processing in event of UAP failure (monitor mode) (when monitoring UAP)[1]

monitoring server in monitor mode[1]

server termination (monitor mode) (when monitoring UAP)[1]

start retries in event of failure on active server after hot standby processing (monitor mode)[1]

start retries in event of host failure (monitor mode)[1]

start retries in event of host failure (server mode)[1]

start retries in event of server failure (monitor mode)[1]

start retries in event of server failure (server mode)[1]

starting server (monitor mode) (when monitoring UAP)[1]

terminating server (in case where monitoring is performed in standby system in monitor mode)[1]

terminating servers (monitor mode) (when releasing in reverse order)[1]

terminating servers (server mode) (when releasing in reverse order)[1]

when server cannot be started by start retries[1]

processor[1]

program[1], [2]

program definition statement[1]

program with HA Monitor interface[1], [2]

program without HA Monitor interface[1], [2]

programs[1]

ptrlcmd_ex[1]

ptrlcmd_ex_inter[1]

ptrlcmd_ex_retry[1]

ptrlcmd_ex_tmout[1]

Q

query response message retry

message_retry[1]

query response message, retrying transmission of[1]

R

referencing and displaying inherited server information in standby system (command)[1]

reservation, canceling[1]

reset[1]

active system[1]

cpudown (preventing concurrent resets)[1]

example of environment settings (VMware ESXi-based virtualization)[1]

host that has reset priority[1]

patrol (environment definition statement)[1]

preventing concurrent host reset[1]

preventing concurrent resets between multiple hosts[1]

preventing multiple resets[1]

processing in event host reset has failed[1]

processing in event host reset has failed (multi-standby)[1]

reset issuing host[1]

reset priority[1]

reset_type (preventing concurrent resets between multiple hosts)[1]

standby system[1]

suppress_reset (suppressing host reset)[1]

suppressing[1]

reset path[1]

configuration[1]

configuration (BladeSymphony)[1]

configuration (model HA8000xM or earlier)[1]

configuration (model HA8000xN or later)[1]

dual reset paths[1]

health check of[1]

IP address[1]

port number[1]

resetpatrol (health check interval)[1]

resetpatrol_mode (health check)[1]

reset_type[1]

resetpath_inter[1]

resetpath_retry[1]

resetpatrol[1]

resetpatrol_mode[1]

resource[1]

evaluating resource sharing method[1]

file system[1]

LAN[1]

maximum configuration[1]

minimum configuration[1]

required for HA Monitor[1]

required for server[1]

shared disk[1]

resource definition statement[1]

resource failure[1]

resource monitoring[1]

resource server[1], [2]

automating startup processing[1]

checking status[1]

determining status[1]

example of environment settings[1]

impossible configuration[1]

possible configuration[1]

sharing resources[1]

starting[1]

terminating[1]

resource server start wait state[1]

resource servers, number of[1]

restart[1]

restart wait state[1]

restart_timeout[1]

restartcommand[1]

retry_stable (program definition statement)[1]

retry_stable (server definition statement)[1]

S

sby_actcommand[1]

sby_ptrlcmd_ex[1]

sby_ptrlcmd_ex_inter[1]

sby_ptrlcmd_ex_retry[1]

sby_ptrlcmd_ex_tmout[1]

sby_servexec_retry[1]

sby_start_timeout[1]

sby_stop_timeout[1]

sby_termcommand[1]

SCSI reservation[1]

SCSI reservation for shared disk[1]

configuration[1]

determining operand value[1]

dmmp_device[1]

example of environment settings[1]

example of environment settings (VMware ESXi-based virtualization)[1]

fence_scsi[1]

scsi_check[1]

scsi_device[1]

scsi_pathcheck[1]

scsi_retry[1]

scsi_timeout[1]

system configuration procedure[1]

scsi_check[1]

scsi_device[1]

scsi_pathcheck[1]

scsi_retry[1]

scsi_timeout[1]

secondary system[1], [2]

servcomplete_msg[1]

server[1], [2]

actcommand (server start command)[1]

acttype (start method)[1]

adding[1]

alias IP address[1]

automating startup processing[1]

checking operation[1]

checking restart of[1]

collecting monitoring history[1]

consideration on using multiple servers[1]

correspondence between start order and parent-child relationship[1]

determining active or standby[1]

determining active or standby (when using multi-standby function)[1]

example of environment settings[1]

group (grouping) (resource definition statement)[1]

group (grouping) (server definition statement)[1]

grouping[1]

handling server restart error[1]

how to monitor (active server in monitor mode)[1]

in wait state[1]

initial (start type)[1]

lock[1]

operands to be defined, list of[1]

operating method[1]

parent (correspondence between start order and parent-child relation)[1]

patrol (server failure monitoring time) (server definition statement)[1]

ph_threshold (time when server monitoring history is to be obtained) (servers definition statement)[1]

relationship of program with[1]

sby_actcommand (command executed when standby server starts)[1]

sby_termcommand (command executed when standby server terminates)[1]

server failure monitoring time[1]

server monitoring command[1]

server start command[1]

server termination command[1]

starting[1]

status transition[1]

switchover order control[1]

termcommand (server termination command)[1]

terminating[1]

that does not release reservation, dealing with (using SCSI reservation for shared disk)[1]

time when server monitoring history is to be obtained[1]

type of[1]

server alias name

alias (resource definition statement)[1]

alias (server definition statement)[1]

server definition statement[1]

server environment definition[1]

changing[1]

server failure[1]

automating operation after hot standby processing[1]

condition under which hot standby operation is performed[1]

detecting (in monitor mode)[1]

detecting (in server mode)[1]

HA Monitor processing after detection of (server mode)[1]

pairdown[1]

switchtype (behavior when server failure is detected)[1]

term_pairdown[1]

server lock[1]

server lock function, locked server environment settings[1]

server mode[1], [2], [3]

server monitoring command[1]

patrolcommand[1]

ptrlcmd_ex[1]

ptrlcmd_ex_inter[1]

ptrlcmd_ex_retry[1]

ptrlcmd_ex_tmout[1]

starting[1]

server slowdown[1]

server start command

actcommand[1]

name (server definition statement)[1]

server switchover order control[1]

after host failure[1]

after server failure in server mode[1]

example of environment settings[1]

processing flow[1]

processing when hot standby operation fails[1]

server termination command

termcommand[1]

server with HA Monitor interface[1]

server without HA Monitor interface[1]

server_alias[1]

server-alias-name.down file, specifying[1]

server-alias-name.up file, specifying[1]

servers[1]

number of[1]

servers on source host[1]

servers_opt[1]

servexec_retry[1]

service name

lanport[1]

servmax[1]

setting whether to issue warning message when hot-standby switchover is impossible

notswitch_notify[1]

setting whether to output message when server termination processing is completed and when grouped-system switchover processing is completed

servcomplete_msg[1]

setting whether to skip disconnection of volume groups

vg_offskip[1]

shared disk[1]

canceling reservation[1]

configuration[1], [2]

connecting[1]

creating volume groups[1]

disk[1]

examples of editing /etc/rc.sysinit file (when using LVM)[1]

handling device failure (during hot standby processing) (using SCSI reservation for shared disk)[1]

handling device failure (while active server is starting) (using SCSI reservation for shared disk)[1]

handling device failure (while active server is terminating) (using SCSI reservation for shared disk)[1]

handling device failures on shared disk (while active server is running) (using SCSI reservation for shared disk)[1]

handling device failures on shared disk (while standby server is starting, on standby, or terminating) (using SCSI reservation for shared disk)[1]

handling, after OS migration[1]

inheriting[1]

SCSI reservation[1]

switching[1]

shared resource[1], [2]

canceling server startup in event of connection error[1]

changing[1]

changing configuration[1]

configuring shared resources used by server[1]

dev_offlimit (timeout)[1]

dev_onlimit (timeout)[1]

dev_timelimit (timeout)[1]

deviceoff_order (separation order) (function definition statement)[1]

deviceoff_order (separation order) (servers definition statement)[1]

disk[1]

dynamic change of[1]

dynamically changing, by using mondevice command[1]

fs_mount_dir[1]

fs_mount_opt[1]

fs_name[1]

fs_neck[1]

fs_umount_retry[1]

handling connection error[1]

handling disconnection error[1]

ip_neck[1]

maintaining[1]

prevention of conflict on[1]

specifying separation order[1]

status[1]

timeout[1]

uoc_neck[1]

user command for controlling[1]

vg_neck[1]

vg_on_opt[1]

slowdown

determining cause[1]

host slowdown[1]

of local system, recognizing[1]

server slowdown[1]

sms[1]

specifying HA Monitor environment settings (command)[1]

specifying server information in active system that is to be inherited (command)[1]

standby server[1], [2]

active server start wait state[1]

command executed when monitoring of standby server is performed[1]

command executed when standby server starts[1]

command executed when standby server stops[1]

grouped-system switchover wait state[1]

standby server[1]

standby system[1], [2]

standbyreset (resetting standby system)[1]

standbypri[1]

standbyreset[1]

start retry function[1]

start retry state[1]

start_timeout[1]

starting

automating operation from system start through server start[1]

server by HA Monitor[1]

server monitoring command[1]

system[1], [2]

starting HA Monitor (command)[1]

starting resource server (command)[1]

starting server in monitor mode (command)[1]

starting server in wait state as active server (command)[1]

status

program[1]

resource server[1]

server[1]

stop_timeout[1]

stop_timeout_sw[1]

suppress_reset[1]

suppressing reset[1]

patrol_type[1]

SVP[1]

IP address[1]

port number[1]

switch[1]

switch_error[1]

switch_judge[1]

switch_retry[1]

switchbyfail[1]

switchtype[1]

symbol conventions[1]

sysdef[1]

system configuration procedure[1]

system disk[1]

system disk monitoring

disk_log_size[1]

disk_ptrl (function definition statement)[1]

disk_ptrl_inter (function definition statement)[1]

disk_ptrl_retry (function definition statement)[1]

system dump[1]

system operation, checking[1]

system start procedure[1]

T

TB meaning[1]

TCP/IP[1]

term_pairdown[1]

termcmd_at_abort[1]

termcommand[1]

terminating

server by HA Monitor[1]

system[1], [2]

terminating active resource server (command)[1]

terminating HA Monitor (command)[1]

terminating servers in monitor mode (command)[1]

terminating standby resource server (command)[1]

terminating standby server (command)[1]

U

UAP[1]

compiling and linking[1]

file provided by HA Monitor[1]

handling errors[1]

including APIs[1]

monitoring[1]

monitoring operation reports[1]

monitoring processes[1]

processing flow[1]

specifying environment variable[1]

UAP monitoring

starting (API)[1]

terminating (API)[1]

UDP[1]

unmounting[1]

uoc_neck[1]

uoc_pairdown[1]

user command[1], [2]

coding example for[1]

how to create[1]

issuance format (when HA Monitor status has changed)[1]

issuance format (when server status has changed)[1]

issuance timing (when HA Monitor status has changed)[1]

issuance timing (when server status has changed)[1]

JP1 event, using[1]

return value, checking[1]

uoc_neck[1]

usrcommand[1]

usrcommand[1]

V

version number conventions[1]

vg_neck[1]

vg_off[1]

vg_offskip[1]

vg_on_opt[1]

vgchange command[1]

vg_on_opt[1]

Virtage[1]

virtual machine[1]

partition_reset (physical partition reset function)[1]

physical partition reset function[1]

VMware ESXi

configuration[1]

consideration[1]

example of hardware configuration (BladeSymphony (Basic mode))[1]

example of hardware configuration (function for controlling hot standby based on availability of LAN communication)[1]

example of hardware configuration (PC/AT-compatible system from another company)[1]

hardware configuration example (model HA8000xM or earlier)[1]

hardware configuration example (model HA8000xN or later)[1]

IP address[1]

partition_reset (physical partition reset function)[1]

physical partition reset function[1]

vmware_env[1]

VMware ESXi-based virtualization, example of environment settings[1]

vmware_env[1]

volume group

connecting for referencing and updating[1]

disconnecting[1]

vg_neck[1]

vg_off[1]

volume-group-name.vglog[1]

volume-group-name.vglog_old[1]

W

wait state[1]

waitserv_exec[1]

when using hybrid fencing

system configuration procedure[1]

Publisher

Hitachi, Ltd.

6-6, Marunouchi 1-chome, Chiyoda-ku, Tokyo, 100-8280 Japan