Contents

• Notices

• Preface

• Part 1: Overview

  • 1. Overview of HA Monitor

    • 1.1 Purpose of HA Monitor

    • 1.2 Hot standby

      • 1.2.1 Hosts

      • 1.2.2 Flow of hot standby

      • 1.2.3 Server

      • 1.2.4 Failures detected by HA Monitor

    • 1.3 Types of hot standby

      • 1.3.1 Automatic hot standby

      • 1.3.2 Planned hot standby

      • 1.3.3 Forced hot-standby switchover

    • 1.4 Hot-standby switchover methods

    • 1.5 System modes with HA Monitor deployed

      • 1.5.1 C/S system mode

      • 1.5.2 FEP mode

      • 1.5.3 Distributed processing system mode

    • 1.6 Operating environment for HA Monitor

      • 1.6.1 Required hardware

      • 1.6.2 Required software

  • 2. Hot Standby Using HA Monitor

    • 2.1 Hot-standby configurations

      • 2.1.1 1-to-1 switchover configuration

      • 2.1.2 Mutual hot-standby configuration

      • 2.1.3 2-to-1 switchover configuration

      • 2.1.4 Multi-standby configuration

      • 2.1.5 Cluster hot-standby configuration

    • 2.2 Operation from failure detection to hot standby

      • 2.2.1 Hot standby operation in the event of a server failure

      • 2.2.2 Hot standby operation in the event of a host failure

    • 2.3 Failure detection by HA Monitor

      • 2.3.1 Conditions under which HA Monitor performs the hot standby operation

      • 2.3.2 Detecting server failures (in the server mode)

      • 2.3.3 Detecting server failures (in the monitor mode)

      • 2.3.4 Detecting host failures

      • 2.3.5 Host reset

      • 2.3.6 SCSI reservation for shared disk

      • 2.3.7 Function for controlling hot standby based on the availability of LAN communications

      • 2.3.8 Hybrid fencing

    • 2.4 Inheriting shared resources

      • 2.4.1 Resources that can be inherited and their units

      • 2.4.2 Inheriting a shared disk

      • 2.4.3 Inheriting a file system

      • 2.4.4 Inheriting a LAN

• Part 2: Description

  • 3. Functions Supported by HA Monitor

    • 3.1 List of functions supported by HA Monitor

    • 3.2 Functions for server control

      • 3.2.1 Monitoring a server in the monitor mode (for the active server)

      • 3.2.2 Monitoring a server in monitor mode (for the standby server)

      • 3.2.3 Grouped-system switchover achieved by grouping servers

      • 3.2.4 Controlling the server switchover order

      • 3.2.5 Multi-standby for providing multiple standby systems

      • 3.2.6 Locking servers to prevent concentration of the processing load after the hot standby operation

      • 3.2.7 Performing start retries during hot standby processing

      • 3.2.8 System operation linked with JP1

      • 3.2.9 Hot-standby switchover in the online serverless mode

    • 3.3 Functions for host control

      • 3.3.1 Preventing concurrent host resets

      • 3.3.2 Preventing concurrent resets between multiple hosts

      • 3.3.3 Preventing multiple resets on a host

      • 3.3.4 Suppressing host resets

      • 3.3.5 Hot-standby switchover caused by a host pair shutdown

      • 3.3.6 Physical partition reset function in a virtualization environment

      • 3.3.7  Hot standby in an N+M cold standby configuration

    • 3.4 Functions for monitoring resources

      • 3.4.1 LAN monitoring and automatic hot standby in the event of a failure

      • 3.4.2 Disk monitoring

    • 3.5 Functions for shared resource control

      • 3.5.1 Using a resource server to share shared resources

      • 3.5.2 Issuing commands when the server and HA Monitor status changes

      • 3.5.3 Canceling server startup in the event of a shared resource connection error

      • 3.5.4 Shared resource inheritance timeout

      • 3.5.5 Specifying the shared resource separation order

      • 3.5.6 Dynamic change of shared resources

    • 3.6 Function for controlling programs (monitor mode)

      • 3.6.1 Monitoring UAPs

    • 3.7 Functions of HA Monitor Extension

      • 3.7.1 Changing the maximum number of servers

      • 3.7.2 Changing the maximum number of hosts

  • 4. System Management

    • 4.1 Managing servers

      • 4.1.1 HA Monitor's control of server startup

      • 4.1.2 HA Monitor's control of server termination

      • 4.1.3 Server status transitions

      • 4.1.4 Controlling the server monitoring command

      • 4.1.5 Processing flow for order control of server switchover

    • 4.2 Managing hosts

      • 4.2.1 How to determine a reset issuing host

      • 4.2.2 Hot standby operation when both hosts detect a failure at the same time

      • 4.2.3 Host reset when there are multiple standby systems

      • 4.2.4 Processing in the event a host reset has failed

    • 4.3 Managing shared resources

      • 4.3.1 Managing shared disks

      • 4.3.2 Managing file systems

      • 4.3.3 Managing a LAN

      • 4.3.4 List of shared resource statuses

    • 4.4 Managing servers when servers are grouped

      • 4.4.1 Server switchover type during grouped-system switchover

      • 4.4.2 Hot standby control on grouped servers

    • 4.5 Managing servers and hosts when using the multi-standby function

      • 4.5.1 Detection of host failures and host reset (multi-standby)

      • 4.5.2 Processing in the event a host reset has failed (multi-standby)

      • 4.5.3 Controlling the server start and termination (multi-standby)

    • 4.6 Managing a resource server

      • 4.6.1 Hot standby when a resource server is used

      • 4.6.2 Processing flow for connecting and disconnecting shared resources when a resource server is used

      • 4.6.3 Determining a resource server's status

    • 4.7 Managing programs (monitor mode)

      • 4.7.1 UAP monitoring control

      • 4.7.2 Program restart control

      • 4.7.3 Program status transitions

      • 4.7.4 Processing flow of UAP monitoring

    • 4.8 Processing flows

      • 4.8.1 Flow of processing for server startup processing

      • 4.8.2 Flow of server termination processing

      • 4.8.3 Hot standby processing in the event of a server failure

      • 4.8.4 Hot standby processing in the event of a host failure

      • 4.8.5 Processing flow for handling a hot standby error in the event of a server failure

      • 4.8.6 Processing flow for handling a hot standby error in the event of a host failure

      • 4.8.7 Processing flow for disconnecting shared resources in the reverse order from which they were connected

      • 4.8.8 Processing flow for start retries in the event of a failure during hot standby processing

      • 4.8.9 Processing flow in the case where hybrid fencing is used

• Part 3: Installation and Operation

  • 5. Installation of HA Monitor and System Design

    • 5.1 Installation and design procedures

    • 5.2 Evaluation of hot-standby configuration

      • 5.2.1 Differences in hot-standby configurations

      • 5.2.2 Considerations for designing a configuration

      • 5.2.3 Considerations for changing the maximum number of hosts (when using HA Monitor Extension)

      • 5.2.4 Considerations for using a hot-standby configuration in a virtualization environment (when using Hitachi server virtualization (Virtage))

      • 5.2.5 Considerations for using a hot-standby configuration in a virtualization environment (when using VMware ESXi)

      • 5.2.6 Considerations for using a hot-standby configuration in a virtualization environment (when using KVM)

    • 5.3 Evaluation of server configuration

      • 5.3.1 Determining the unit of hot standby processing (server)

      • 5.3.2 Considerations on using multiple servers

    • 5.4 Evaluation of resources configuration

      • 5.4.1 Required resources and number of resources

      • 5.4.2 Necessary IP addresses and port numbers

      • 5.4.3 Approach to evaluating the resource sharing method

      • 5.4.4 Configurations using resource servers

    • 5.5 Evaluation of hardware configuration

      • 5.5.1 Shared disk configuration

      • 5.5.2 LAN configuration

      • 5.5.3 Monitoring path configuration

      • 5.5.4 Reset path configuration

      • 5.5.5 Examples of hardware configurations (BladeSymphony (Basic mode))

      • 5.5.6 Examples of hardware configurations (BladeSymphony (LPAR mode))

      • 5.5.7 Examples of hardware configurations (model HA8000xM or earlier)

      • 5.5.8 Example of a hardware configuration (model HA8000xN or later)

      • 5.5.9 Examples of hardware configurations (using SCSI reservation for shared disk and PC/AT-compatible systems from another company)

      • 5.5.10 Examples of hardware configurations (using the function for controlling hot standby based on the availability of LAN communications)

  • 6. System Configuration

    • 6.1 Configuration procedure

      • 6.1.1 When resetting hosts (BladeSymphony or HA8000)

      • 6.1.2 When using SCSI reservation for shared disk

      • 6.1.3 When using hybrid fencing

      • 6.1.4 When using the function for controlling hot standby based on the availability of LAN communications

    • 6.2 Directory structure

    • 6.3 Configuring the OS

      • 6.3.1 Specifying environment variables

      • 6.3.2 Specifying system files

      • 6.3.3 Setting the system clock

      • 6.3.4 Setting the DNS

      • 6.3.5 Setting system log files

      • 6.3.6 Specifying the kernel parameters

      • 6.3.7 Specifying the output language

      • 6.3.8 Setting the security level

      • 6.3.9 Setting the file system

    • 6.4 Specifying settings for using SCSI reservation for shared disk

      • 6.4.1 Setting up multipath software

      • 6.4.2 Determining the values to be specified in the operands required for SCSI reservation of the shared disk if LVM is used

      • 6.4.3 Determining the values to be specified in the operands required for SCSI reservation of the shared disk if LVM is not used

      • 6.4.4 Handling SCSI-related messages output to syslog

      • 6.4.5 Specifying the udev rules

      • 6.4.6 Notes about SCSI reservation for shared disk

    • 6.5 Configuring VMware ESXi for use in HA Monitor operation

      • 6.5.1 Settings required for resetting hosts and using SCSI reservation for shared disk

      • 6.5.2 Settings required for resetting hosts

      • 6.5.3 Settings required for using SCSI reservation for shared disk

    • 6.6 Specifying the KVM-based virtualization environment settings

      • 6.6.1 KVM settings (Setting 1)

      • 6.6.2 Multipath control (DMMP) settings (Setting 2)

    • 6.7 Configuring the reset path (BladeSymphony)

      • 6.7.1 Information required for configuration (BladeSymphony)

      • 6.7.2 Examples of settings (BladeSymphony)

    • 6.8 Configuring the reset path (model HA8000xM or earlier)

      • 6.8.1 Information required for configuration (model HA8000xM or earlier)

      • 6.8.2 Examples of settings (model HA8000xM or earlier)

    • 6.9 Configuring the reset path (model HA8000xN or later)

      • 6.9.1 Information required for configuration (model HA8000xN or later)

      • 6.9.2 Examples of settings (model HA8000xN or later)

      • 6.9.3 Notes on configuration if a management server is used as the failure management processor (model: HA8000xN or later)

    • 6.10 Configuring the monitoring paths

      • 6.10.1 Adding host names and service names

      • 6.10.2 Creating an HA Monitor connection configuration file

    • 6.11 Creating definition files (HA Monitor)

    • 6.12 Configuring the shared resources to be used by servers

      • 6.12.1 Configuring a shared disk

      • 6.12.2 Specifying LAN status settings files

    • 6.13 Creating server start, termination, and monitoring commands (for the active server)

      • 6.13.1 Creating a server start command

      • 6.13.2 Creating a server termination command

      • 6.13.3 Creating a server monitoring command

    • 6.14 Creating commands that will be used to start, stop, and monitor the standby server (in the case where monitoring is performed in the standby system)

      • 6.14.1 Creating a command that is executed when the standby server starts

      • 6.14.2 Creating a command to be executed when the standby server stops

      • 6.14.3 Creating a command to be executed when monitoring of the standby server is performed

    • 6.15 Creating a program restart command (when the program management function is used)

      • 6.15.1 Timing of calling a program restart command

      • 6.15.2 How to create a program restart command

    • 6.16 Creating and configuring UAPs

      • 6.16.1 Including APIs in UAPs

      • 6.16.2 Specifying UAP environment variables

    • 6.17 Creating definition files (server)

    • 6.18 Creating definition files (monitor-mode program)

    • 6.19 Creating user commands

      • 6.19.1 Timing of user command issuance: when the server status has changed

      • 6.19.2 Timing of user command issuance: when the HA Monitor status has changed

      • 6.19.3 User command issuance format: when the server status has changed

      • 6.19.4 User command issuance format: when the HA Monitor status has changed

      • 6.19.5 How to create user commands

      • 6.19.6 Coding example for a user command

    • 6.20 Settings for resource monitoring

      • 6.20.1 Settings in the files required for monitoring a LAN

      • 6.20.2 Settings in the files required for disk monitoring

    • 6.21 Configuring HA Monitor Extension

    • 6.22 Operating environment of commands that are executed from HA Monitor

      • 6.22.1 Environment variables for commands that are executed from HA Monitor

      • 6.22.2 File descriptors for the commands that are executed from HA Monitor

      • 6.22.3 Signals for commands that are executed from HA Monitor

    • 6.23 Checking the settings and definitions for the configured system

      • 6.23.1 Checking the settings and definitions between multiple hot-standby configurations

      • 6.23.2 Checking the definitions

    • 6.24 Settings for collecting error information

    • 6.25 Checking the operation of the configured system

      • 6.25.1 Prior confirmation

      • 6.25.2 Checking the HA Monitor operation

      • 6.25.3 Checking the server operation

      • 6.25.4 Testing hot standby processing

      • 6.25.5 Testing the system load

    • 6.26 Checking the operation when the maximum number of hosts is changed (when using HA Monitor Extension)

  • 7. System Operation

    • 7.1 Operation flow

      • 7.1.1 Preparations for operation

    • 7.2 Starting and terminating

      • 7.2.1 Starting

      • 7.2.2 Terminating

      • 7.2.3 Notes about maintaining shared resources

    • 7.3 Checking the server logs

      • 7.3.1 Checking the execution log for the monitoring command (when the ptrlcmd_ex or sby_ptrlcmd_ex operand is specified on a monitor-mode server)

      • 7.3.2 Checking the details log for the monitoring command (when the ptrlcmd_ex or sby_ptrlcmd_ex operand is specified on a monitor-mode server)

    • 7.4 Hot standby operation when a failure occurs

      • 7.4.1 Starting a server in the wait state and then restarting jobs

      • 7.4.2 Collecting error information

      • 7.4.3 Restarting the host where a failure occurred as the standby system

      • 7.4.4 Checking the status of servers and hosts after handling an error

    • 7.5 Handling errors

      • 7.5.1 Handling host start errors

      • 7.5.2 Handling server restart errors

      • 7.5.3 Handling host reset errors

      • 7.5.4 Handling shared resource connection errors

      • 7.5.5 Handling shared resource disconnection errors

      • 7.5.6 Handling device failures on a shared disk (while the active server is starting) (using SCSI reservation for shared disk)

      • 7.5.7 Handling device failures on a shared disk (while the active server is running) (using SCSI reservation for shared disk)

      • 7.5.8 Handling device failures on a shared disk (while the active server is terminating) (using SCSI reservation for shared disk)

      • 7.5.9 Handling device failures on a shared disk (while the standby server is starting, on standby, or terminating) (using SCSI reservation for shared disk)

      • 7.5.10 Handling device failures on a shared disk (during hot standby processing) (using SCSI reservation for shared disk)

      • 7.5.11 Dealing with the server that does not release the reservation (using SCSI reservation for shared disk)

      • 7.5.12 Canceling SCSI reservation for shared disk

      • 7.5.13 Handling UAP errors

      • 7.5.14 Handling a duplicated address operand value

      • 7.5.15 Dealing with server failures that disable restart and hot standby processing (server in the monitor mode)

      • 7.5.16 Handling a failure in the path to the shared disk (if the path to the shared disk is made redundant in a KVM-based virtualization environment)

    • 7.6 Operation for preventing failures caused by heavy load

      • 7.6.1 Collecting server monitoring history

      • 7.6.2 Collecting host monitoring history

      • 7.6.3 Analyzing the collected monitoring history

      • 7.6.4 Determining the cause of a server slowdown

    • 7.7 Automating operations

      • 7.7.1 Automating the operation from system start through server start

      • 7.7.2 Automating the processing to be performed when the status of a server or HA Monitor changes

      • 7.7.3 Automating an operation after hot standby processing

    • 7.8 Planned hot standby

      • 7.8.1 Performing a planned hot standby operation

    • 7.9 Forced hot-standby switchover

      • 7.9.1 Forcibly performing hot-standby switchover

    • 7.10 Modifying the system

      • 7.10.1 Adding hosts

      • 7.10.2 Adding servers

      • 7.10.3 Changing shared resources

      • 7.10.4 Changing the shared resource configuration while hosts and servers are running

      • 7.10.5 Changing the environment settings for HA Monitor and servers

      • 7.10.6 Changing the settings and operations while HA Monitor and servers are running

• Part 4: Reference

  • 8. Files Defined in Environment Settings

    • 8.1 Overview of definition files

    • 8.2 Definition rules

      • 8.2.1 Specification format

      • 8.2.2 Conventions used in the definitions

    • 8.3 Configuring an environment for HA Monitor

      • 8.3.1 HA Monitor environment settings (sysdef)

    • 8.4 Configuring a server environment

      • 8.4.1 Server environment definition (servers)

      • 8.4.2 Locked server environment settings (servers_opt)

    • 8.5 Configuring an environment for programs in the monitor mode

      • 8.5.1 Monitor-mode program environment definition (programs)

    • 8.6 Examples of environment settings

      • 8.6.1 Examples of environment settings for a 1-to-1 switchover configuration

      • 8.6.2 Examples of environment settings for a multi-host hot-standby configuration

      • 8.6.3 Examples of environment settings for a VMware ESXi-based virtualization environment

      • 8.6.4 Example of environment settings for a KVM-based virtualization environment

      • 8.6.5 Examples of environment settings for a hot-standby configuration with a locked server specified

  • 9. Commands

    • 9.1 List of commands

    • 9.2 Headings used in the explanations of commands

    • 9.3 Syntax symbols

    • 9.4 monact (starts a server in the wait state as the active server)

    • 9.5 monbegin (starts a server in the monitor mode)

    • 9.6 monchange (changes settings and operations while HA Monitor and servers are running)

    • 9.7 moncheck (checks definitions)

    • 9.8 mondeact (terminates a server in the wait state)

    • 9.9 mondevice (changes shared resources while the active server is running)

    • 9.10 monend (terminates servers in the monitor mode)

    • 9.11 moninfo (specifies server information in the active system that is to be inherited, or references and displays inherited server information in the standby system)

    • 9.12 monlink (connects HA Monitors manually)

    • 9.13 monlistip (detects IP addresses)

    • 9.14 monodrshw (displays the server order control status)

    • 9.15 monpath (displays the status of monitoring paths)

    • 9.16 monresbgn (starts a resource server)

    • 9.17 monresend (terminates an active resource server)

    • 9.18 monressbystp (terminates a standby resource server)

    • 9.19 monrp (displays the status of the reset path)

    • 9.20 monsbystp (terminates a standby server)

    • 9.21 monscsiclr (releases reservations for shared disks)

    • 9.22 monsetup (specifies HA Monitor environment settings)

    • 9.23 monshow (displays the status of servers and hosts)

    • 9.24 monstart (starts HA Monitor)

    • 9.25 monstop (terminates HA Monitor)

    • 9.26 monswap (performs server hot-standby switchover)

    • 9.27 monts (collects HA Monitor troubleshooting information)

  • 10. APIs

    • 10.1 List of APIs

    • 10.2 Headings used in the explanation of APIs

    • 10.3 hamon_patrolstart (starts UAP monitoring)

    • 10.4 hamon_patrolstop (terminates UAP monitoring)

    • 10.5 List of detail codes

    • 10.6 Coding example

• Appendixes

  • A. Event IDs Used by HA Monitor

    • A.1 Event issuance timing when HA Monitor starts

    • A.2 Event issuance timing when HA Monitor terminates normally

    • A.3 Event issuance timing when HA Monitor terminates abnormally

    • A.4 Event issuance timing when a server starts

    • A.5 Event issuance timing when a server terminates normally

    • A.6 Event issuance timing when server startup fails

    • A.7 Event issuance timing when a server failure has occurred

    • A.8 Event issuance timing when a failure is detected on the remote host

    • A.9 Event issuance timing when a reset path failure has occurred

    • A.10 Event issuance timing when a host reset has failed

    • A.11 Event issuance timing when a monitoring path failure has occurred

    • A.12 Event issuance timing when HA Monitor is waiting for operator intervention

  • B. Messages Output as Monitoring History

    • B.1 Server monitoring history output timing

    • B.2 Remote host monitoring history output timing

    • B.3 Local host monitoring history output timing

  • C. List of Files Output by HA Monitor

  • D. List of HA Monitor Processes

  • E. Reference Material for This Manual

    • E.1 Related publications

    • E.2 Conventions: Abbreviations for product names

    • E.3 Conventions: Acronyms

    • E.4 Conventions: KB, MB, GB, and TB

• Glossary

• Index

To Page Top