For Linux(R) (x86) Systems

HA Monitor Kit for Oracle

Description and User's Guide

3000-9-135-10(E)

Contents

Notices

Summary of amendments

Preface

1. Overview of HA Monitor Kit

1.1 Purpose of HA Monitor Kit

1.2 Configurations to which HA Monitor Kit can be applied

1.3 Operating environment of HA Monitor Kit

1.3.1 Required hardware

1.3.2 Required software

1.4 Relationship to HA Monitor

2. System Management

2.1 Server management

2.1.1 Error detection framework

2.1.2 Types of errors that can be detected

2.2 Flow of processing

2.2.1 Flow of processing for starting a server

2.2.2 Flow of processing when server startup fails

2.2.3 Flow of processing for stopping a server

2.2.4 Flow of processing for monitoring a server

2.2.5 Flow of processing for performing hot standby when a server failure occurs

3. Designing and Building a System

3.1 Designing a System

3.1.1 Oracle component configuration and hot standby switchable unit considerations

3.1.2 Resource configuration considerations

3.2 Building a system

3.2.1 General configuration procedure

3.2.2 Directory configuration

3.2.3 Relationships among setting items necessary for building a system and setting overview

3.2.4 Oracle installation

3.2.5 Setting up Oracle and building a database

3.2.6 Setting up the shared resources to be used by Oracle

3.2.7 Setting up HA Monitor Kit

3.2.8 Editing HA Monitor Kit Commands

3.2.9 Creating a definition file (server)

3.2.10 Settings for collecting error information

3.2.11 Checking the operation of the system that has been built

3.3 Environment definition example

4. Setting Up the Server Start, Termination, and Monitoring Commands

4.1 Overview of the server start, termination, and monitoring commands

4.2 Server start command setup

4.3 Server termination command setup

4.4 Server monitoring command setup

4.4.1 Server monitoring command

4.4.2 Server monitoring command runtime shell

4.5 Commands used for setting up the server start, termination, and monitoring commands

Headings used for describing the commands

Syntax conventions

haorainsbgn (Starts an Oracle instance)

haorainsend (Terminates an Oracle instance)

haorainsptl (Checks the status of an Oracle instance)

haorainsendsub (Terminates an Oracle instance)

haorainsptlsub (Checks the status of an Oracle instance)

haoralsnrbgn (Starts an Oracle listener)

haoralsnrend (Terminates an Oracle listener)

haoralsnrptl (Checks the status of an Oracle listener)

haoralsnrendsub (Terminates an Oracle listener)

haoralsnrptlsub (Checks the status of an Oracle listener)

haoraenvcheck (Checks a control definition file)

5. System Operation

5.1 Flow of operations

5.2 Starting, terminating, and monitoring Oracle instances and Oracle listeners

5.3 Starting or terminating an Oracle instance or Oracle listener without involving HA Monitor

5.3.1 Not involving HA Monitor when starting an Oracle instance or Oracle listener

5.3.2 Not involving HA Monitor when terminating an Oracle instance or Oracle listener

5.4 Checking the server logs

5.5 Hot standby operation when a failure occurs

5.5.1 Collecting error information

5.6 Error handling

5.7 System modification

5.7.1 Upgrading Oracle

5.7.2 Adding Oracle instances and Oracle listeners

5.7.3 Deleting Oracle instances and Oracle listeners

5.7.4 Modifying the settings related to Oracle instances and Oracle listeners

6. Messages

6.1 Message format

6.1.1 Message output format

6.1.2 Message description format

6.2 Message details

KAMK00021-E (P + L)

KAMK00120-E (P + L)

KAMK00130-E (P + L)

KAMK00201-I (P)

KAMK00202-I (P)

KAMK00203-I (P)

KAMK00204-I (P)

KAMK00205-I (P)

KAMK00206-I (P)

KAMK00301-I (P)

KAMK00302-I (P)

KAMK00303-E (P + L)

KAMK00304-I (P)

KAMK00305-I (P)

KAMK00306-E (P + L)

KAMK00308-E (P + L)

KAMK00309-E (P + L)

KAMK00310-W (P)

KAMK00311-W (P)

KAMK00312-I (P)

KAMK00321-I (P)

KAMK00322-I (P)

KAMK00323-E (P + L)

KAMK00324-I (P)

KAMK00325-I (P)

KAMK00326-E (P + L)

KAMK00328-E (P + L)

KAMK00329-E (P + L)

KAMK00330-W (P)

KAMK00331-W (P)

KAMK00332-I (P)

KAMK00401-I (P)

KAMK00402-I (P)

KAMK00411-I (P)

KAMK00412-I (P)

KAMK00421-I (P)

KAMK00422-I (P)

KAMK00423-E (P + L)

KAMK00601-E (P + L)

Appendix

A. Reference Material for This Manual

A.1 Related publications

A.2 Conventions: Abbreviations for product names

A.3 Conventions: Acronyms

A.4 Conventions: KB, MB, GB, and TB

Index

[Trademarks]

All Rights Reserved. Copyright (C) 2011, 2014, Hitachi, Ltd.

Notices

Relevant program products

P-F9S2C-E1121 HA Monitor Kit for Oracle 01-01 (for Red Hat Enterprise Linux AS 4 (x86), Red Hat Enterprise Linux ES 4 (x86), Red Hat Enterprise Linux AS 4 (AMD64 & Intel EM64T), Red Hat Enterprise Linux ES 4 (AMD64 & Intel EM64T), Red Hat Enterprise Linux 5 (x86), Red Hat Enterprise Linux 5 (AMD/Intel 64), Red Hat Enterprise Linux 5 Advanced Platform (x86), Red Hat Enterprise Linux 5 Advanced Platform (AMD/Intel 64), Red Hat Enterprise Linux Server 6 (x86), and Red Hat Enterprise Linux Server 6 (x86_64))

This manual can be used for products other than the products shown above. For details, see the Release Notes.

Trademarks

AMD, AMD Opteron, and combinations thereof, are trademarks of Advanced Micro Devices, Inc.
Linux(R) is the registered trademark of Linus Torvalds in the U.S. and other countries.
Oracle and Java are registered trademarks of Oracle and/or its affiliates.
Red Hat is a trademark or a registered trademark of Red Hat Inc. in the United States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Other company and product names mentioned in this document may be the trademarks of their respective owners.
Throughout this document Hitachi has attempted to distinguish trademarks from descriptive terms by writing the name with the capitalization used by the manufacturer, or by writing the name with initial capital letters. Hitachi cannot attest to the accuracy of this information. Use of a trademark in this document should not be regarded as affecting the validity of the trademark.

Restrictions

Information in this document is subject to change without notice and does not represent a commitment on the part of Hitachi. The software described in this manual is furnished according to a license agreement with Hitachi. The license agreement contains all of the terms and conditions governing your use of the software and documentation, including all warranty rights, limitations of liability, and disclaimers of warranty.
Material contained in this document may describe Hitachi products not available or features not available in your country.
No part of this material may be reproduced in any form or by any means without permission in writing from the publisher.

Issued

Sept. 2014: 3000-9-135-10(E)

Copyright

All Rights Reserved. Copyright (C) 2011, 2014, Hitachi, Ltd.

Summary of amendments

The following table lists changes in this manual (3000-9-135-10(E)) and product changes related to this manual.

Changes
Location

The description of the flow of processing for HA Monitor and HA Monitor Kit was changed.
2.2.1, 2.2.3, 2.2.4,

HA Monitor Kit commands can now be edited. With this change, the following subcommands were added:

haorainsendsub (terminates an Oracle instance)

haorainsptlsub (checks the status of an Oracle instance)

haoralsnrendsub (terminates an Oracle listener)

haoralsnrptlsub (checks the status of an Oracle listener)

haoraenvcheck (checks the control definition file)

In addition, changes were made to the contents of the sample files for the server start, termination, and monitoring commands.
3.2.1, 3.2.2, 3.2.8, 3.3, 4.2, 4.3, 4.4.1, 4.5

You can now specify any environment variable in a control definition file.
3.2.7(2)

The parameter ORACLE_BASE was added to the parameters that can be specified in a control definition file.
3.2.7(2), 3.3

The operand servexec_retry was added to the server environment definition operands that require special attention.
3.2.9, 3.3, 4.2(2)

Oracle Database 12c is now supported.
A.2, A.3

In addition to the above changes, minor editorial corrections were made.

Preface

This manual describes the functions, setup, and operation of HA Monitor Kit for Oracle. This manual provides the information to enable the reader to design, set up, and operate a system configuration that uses HA Monitor Kit for Oracle, and to monitor a system that uses Oracle.

Intended readers

This manual is intended for system administrators and operators. It assumes that system administrators design, install, and set up the system, and that operators perform the day-to-day operations on the configured system.

Readers of this manual must have:

Knowledge of the applicable hardware

Knowledge of the applicable operator system

Knowledge of HA Monitor

Knowledge of Oracle

Conventions: Diagrams

This manual uses the following conventions in diagrams:

Conventions: Fonts and symbols

The following table explains the text formatting conventions used in this manual:

Text formatting
Convention

Bold
Bold characters indicate text in a window, other than the window title. Such text includes menus, menu options, buttons, radio box options, or explanatory labels. For example:

From the File menu, choose Open.

Click the Cancel button.

In the Enter name entry box, type your name.

Italic
Italic characters indicate a placeholder for some actual text to be provided by the user or system. For example:

Write the command as follows:
copy source-file target-file

The following message appears:
A file was not found. (file = file-name)

Italic characters are also used for emphasis. For example:

Do not delete the configuration file.

Monospace
Monospace characters indicate text that the user enters without change, or text (such as messages) output by the system. For example:

At the prompt, enter dir.

Use the send command to send mail.

The following message is displayed:
The password is incorrect.

The following table explains the symbols used in this manual:

Symbol
Convention

|
In syntax explanations, a vertical bar separates multiple items, and has the meaning of OR. For example:
A|B|C means A, or B, or C.

{ }
In syntax explanations, curly brackets indicate that only one of the enclosed items is to be selected. For example:
{A|B|C} means only one of A, or B, or C.

[ ]
In syntax explanations, square brackets indicate that the enclosed item or items are optional. For example:
[A] means that you can specify A or nothing.
[B|C] means that you can specify B, or C, or nothing.

...
In coding, an ellipsis (...) indicates that one or more lines of coding have been omitted.
In syntax explanations, an ellipsis indicates that the immediately preceding item can be repeated as many times as necessary. For example:
A, B, B, ... means that, after you specify A, B, you can specify B as many times as necessary.

Conventions: Version numbers

The version numbers of Hitachi program products are usually written as two sets of two digits each, separated by a hyphen. For example:

Version 1.00 (or 1.0) is written as 01-00.

Version 2.05 is written as 02-05.

Version 2.50 (or 2.5) is written as 02-50.

Version 12.25 is written as 12-25.

The version number might be shown on the spine of a manual as Ver. 2.00, but the same version number would be written in the program as 02-00.

1. Overview of HA Monitor Kit

This chapter explains the purpose of HA Monitor Kit, the configurations to which it can be applied, its operating environment, and its relationship to HA Monitor. In this manual, HA Monitor Kit for Oracle is referred to as HA Monitor Kit.

Organization of this chapter

1.1 Purpose of HA Monitor Kit

1.2 Configurations to which HA Monitor Kit can be applied

1.3 Operating environment of HA Monitor Kit

1.4 Relationship to HA Monitor

1.1 Purpose of HA Monitor Kit

HA Monitor Kit is a hot-standby system that uses HA Monitor, and is an optional product of HA Monitor that makes it easier to monitor Oracle systems.

To use the basic HA Monitor-based hot-standby system to monitor Oracle, a user has to carefully select a building method and then build a cluster environment, giving special consideration to the structure of the Oracle system. Because the user has to manually create a shell for monitoring Oracle, the investigation and evaluation of implementation methods required extra effort and cost, and there was always a risk of designing defects into the shell.

HA Monitor Kit solves these problems and provides the following benefits:

A user can easily set up a cluster environment even if they do not possess any knowledge about using Oracle in a cluster environment.
Even if a user does not possess any knowledge about using Oracle in a cluster environment, the user can set up a cluster environment to monitor Oracle by following the procedure described in this manual. Furthermore, because the user does not need to be concerned about the structure of Oracle, the effort and cost associated with investigation and evaluation can be reduced.

A user can build an Oracle cluster environment without creating a shell.
Because HA Monitor Kit provides shells for starting, terminating, and monitoring Oracle, there is no need for the user to create separate shells. This reduces the effort and cost associated with shell creation, and also avoids the risk of designing defects into the shell.

The following figure compares a conventional system with a system in which HA Monitor Kit is installed.

Figure 1-1 Comparison between a conventional system and a system in which HA Monitor Kit is installed

1.2 Configurations to which HA Monitor Kit can be applied

To guard against system failures, HA Monitor employs two systems, using one to execute jobs and keeping the other as a standby. These systems are called hosts. A program that executes jobs on a system (host) is called a server. HA Monitor performs hot standby on a server-by-server basis.

This section explains the available Oracle system configurations, Oracle component configurations, and hot-standby configurations, which the reader needs to know when using HA Monitor Kit.

Organization of this section

(1) Oracle system configurations

(2) Oracle component configuration

(3) Hot-standby configuration

(1) Oracle system configurations

Some Oracle system configurations can be made into hot-standby configurations by using HA Monitor Kit while others cannot, as described below.

Configurations with which HA Monitor Kit can be used

You can use HA Monitor Kit with an Oracle single-configuration system (a system that has the primary database only) and with an Oracle system configuration that has a primary database that is configured with a redundant database for disaster recovery to protect the database against disasters. The following figures show examples of these configurations.

Figure 1-2 Configuration in which HA Monitor Kit can be used (Oracle single configuration)

Figure 1-3 Configuration in which HA Monitor Kit can be used (primary database with disaster recovery configuration)

Configuration in which HA Monitor Kit cannot be used

You cannot use HA Monitor Kit with a RAC configuration (one that uses an Oracle RAC). The following figure shows an example of a RAC configuration.

Figure 1-4 Configuration to which HA Monitor Kit cannot be used (RAC configuration)

(2) Oracle component configuration

When HA Monitor Kit is installed on a system, it treats Oracle instances and Oracle listeners as servers that are subject to hot standby. An Oracle instance is a combination of shared memory and processes, and is associated with a database as a component necessary for the client to continue jobs. An Oracle listener is a component that relays a request from the client to an Oracle instance. Both Oracle instances and the Oracle listener that manages these instances must be running on the same system.

The following figure shows an Oracle component configuration.

Figure 1-5 Oracle component configuration

HA Monitor Kit can manage Oracle instances and Oracle listeners as separate servers, or can group multiple such components into a single server for management purposes. For details, see 3.1.1 Oracle component configuration and hot standby switchable unit considerations.

(3) Hot-standby configuration

You use HA Monitor Kit in a hot-standby configuration supported by HA Monitor. Determine which hot-standby configuration you can use based on the version of HA Monitor you are using. For details about the hot-standby configurations of HA Monitor, see the descriptions about hot-standby configurations in the manual HA Monitor (for Linux(R)).

In a hot-standby configuration of HA Monitor, servers are operated in either the server mode or monitor mode. In a configuration in which HA Monitor Kit is used, Oracle servers are operated in the monitor mode.

1.3 Operating environment of HA Monitor Kit

This section explains the hardware and software necessary for operating a system that uses HA Monitor Kit. For environmental issues that are the same as for basic HA Monitor systems, see the description under Operating environment for HA Monitor in the manual HA Monitor (for Linux(R)).

Organization of this section

1.3.1 Required hardware

1.3.2 Required software

1.3.1 Required hardware

The hardware configuration necessary for operating a system that uses HA Monitor Kit depends on HA Monitor. For details, see the description under Required hardware in the manual HA Monitor (for Linux(R)).

This subsection explains the hardware configuration elements that require close attention when using HA Monitor Kit.

Organization of this subsection

(1) Shared disk

(2) Local disk

(3) LAN

(1) Shared disk

A shared disk is used to store the data that the server shares between the active system and the standby system. In Oracle, the following files that make up a database are stored on the shared disk:

Server parameter file

Control file

Data file

REDO log file

Archive log file

Also, the flash recovery area must specify a path on the shared disk.

When you install Oracle on a shared disk, you can choose to share the software between the active system and the standby system. You can also install Oracle on a local disk.

(2) Local disk

A local disk is used as a host-specific disk device not shared between hosts on which Oracle and HA Monitor Kit are installed. You can also install Oracle on a shared disk.

For details about Oracle installation, see 3.2.4 Oracle installation.

(3) LAN

A LAN is used by a client to connect to the database.

1.3.2 Required software

This subsection explains considerations about the software configuration that you must be aware of to use HA Monitor Kit, along with some notes about the software.

Organization of this subsection

(1) Software configuration

(2) Notes on software

(1) Software configuration

The following figure shows the software configuration.

Figure 1-6 Software configuration

(a) Kernel

Linux (x86)

The operating system that forms the system foundation.

(b) Hot-standby control

HA Monitor

Controls switching to a standby host.

HA Monitor Kit

An optional product of HA Monitor that works under HA Monitor.

(c) Program

Oracle

A relational database management system. In a system that uses HA Monitor, Oracle is managed as a monitor-mode server.

(2) Notes on software

Note the following points about the software:

If you are using Automatic Storage Management (ASM) for the file system, you cannot build a cluster environment that uses HA Monitor Kit.

The following optional products and functions cannot be used:

Oracle Clusterware

Oracle Data Guard Broker

Oracle RAC

Oracle Restart

1.4 Relationship to HA Monitor

HA Monitor Kit is an optional product of HA Monitor. This manual describes only those items that are specific to HA Monitor Kit. Therefore, refer to this manual only for issues specific to HA Monitor Kit; refer to the manual HA Monitor (for Linux(R)) for all other issues.

Regarding messages, refer to this manual only for messages specific to HA Monitor Kit; refer to the manual HA Monitor Messages for messages specific to HA Monitor.

The following figure shows the relationship between this manual and the other manuals in the HA Monitor series.

Figure 1-7 Correspondence between this manual and the other manuals in the HA Monitor series

2. System Management

This chapter explains how HA Monitor manages servers in a system that uses HA Monitor Kit. It also explains the detailed operation of a system that uses HA Monitor Kit.

For details about management of hosts and shared resources, see the description under System Management in the manual HA Monitor (for Linux(R)).

Organization of this chapter

2.1 Server management

2.2 Flow of processing

2.1 Server management

When you use HA Monitor Kit, HA Monitor monitors the running server (Oracle instance or Oracle listener) and can detect errors in Oracle. This section explains how errors are detected and the types of errors that can be detected.

Organization of this section

2.1.1 Error detection framework

2.1.2 Types of errors that can be detected

2.1.1 Error detection framework

In a system that uses HA Monitor Kit, a server-monitoring command is used to monitor the active server (Oracle instance or Oracle listener), and HA Monitor detects errors by monitoring the process of this server-monitoring command. When an error is detected, HA Monitor performs hot standby or restart (depending on the setting) in order to continue job execution.

The following figure provides an overview of error detection as performed by the server-monitoring command.

Figure 2-1 Overview of error detection as performed by the server-monitoring command

For Oracle instances, the server-monitoring command provided by HA Monitor Kit monitors status, slowdown, and process operation. For Oracle listeners, the server-monitoring command monitors slowdown and process operation. You specify the monitoring method in the applicable definition control file. For details about how to set up a control definition file, see 3.2.7(2) Setting up control definition files.

2.1.2 Types of errors that can be detected

This subsection explains the types of errors that can be detected in a system that uses HA Monitor Kit.

Organization of this subsection

(1) Errors that HA Monitor Kit detects

(2) Errors that HA Monitor detects

(1) Errors that HA Monitor Kit detects

HA Monitor Kit detects the Oracle errors shown in the following table.

Table 2-1 Errors that HA Monitor Kit detects

Detected error
Description

Oracle instance error
HA Monitor Kit detects the following statuses as Oracle instance errors:

Oracle instance process does not exist.

Database query error by an SQL statement

Oracle instance slowdown
If there is no response to an SQL query for the specified length of time, HA Monitor Kit detects this condition as Oracle instance slowdown#.

Oracle listener error
If there is no Oracle listener process, HA Monitor Kit detects this condition as an Oracle listener error.

Oracle listener slowdown
If there is no response from an Oracle listener process for the specified length of time, HA Monitor Kit detects this condition as Oracle listener slowdown#.

#: Detects the time elapsed following the start of a query by HA Monitor Kit.

(2) Errors that HA Monitor detects

The Oracle errors shown in the following table are detected by HA Monitor and not by HA Monitor Kit.

Table 2-2 Errors that HA Monitor detects

Detected error | Description
Network error | This error is detected by HA Monitor's LAN monitoring function. For details about the LAN monitoring function, see the description under Monitoring of the LAN and automatic hot standby in the event of a failure in the manual HA Monitor (for Linux(R)).
Host failure | This failure is detected by HA Monitor when the alive message is disrupted. For details about how host failures are detected, see Detecting host failures in the manual HA Monitor (for Linux(R)).

2.2 Flow of processing

This section explains the flow of processing for both HA Monitor and HA Monitor Kit, including all shared resources.

In the processing flows explained in this section, the message IDs that HA Monitor and HA Monitor Kit output and the event IDs that they issue are described. You can refer to them when investigating the timing at which a message ID or event is issued. This section also describes the timing at which user commands are executed and the arguments that are relayed to user commands. Refer to these when creating user commands.

The following figure shows the flow of operations in a hot-standby configuration and how it corresponds to the content explained in the processing flows.

Figure 2-2 Flow of operations in a hot-standby configuration and how it corresponds to the content explained in the processing flows

#: For details about the flow of processing when server startup fails, see 2.2.2 Flow of processing when server startup fails.

For details about how to start, terminate, and monitor a server, see 5.2 Starting, terminating, and monitoring Oracle instances and Oracle listeners.

Organization of this section

2.2.1 Flow of processing for starting a server

2.2.2 Flow of processing when server startup fails

2.2.3 Flow of processing for stopping a server

2.2.4 Flow of processing for monitoring a server

2.2.5 Flow of processing for performing hot standby when a server failure occurs

2.2.1 Flow of processing for starting a server

This subsection provides an overview of the processing performed by HA Monitor and HA Monitor Kit when a server (Oracle instance or Oracle listener) is started. Note that the figures used in this subsection show examples in which an Oracle instance or an Oracle listener is associated with a single server.

Organization of this subsection

(1) Flow of processing for starting an Oracle instance

(2) Flow of processing for starting an Oracle listener

(1) Flow of processing for starting an Oracle instance

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when an Oracle instance is started.

Figure 2-3 Overview of processing for starting an Oracle instance

#1: Executed to allow an IP address for client connection to be included in the server resources. For details about the resources required by servers, see 3.1.2 Resource configuration considerations and 3.2.6 Setting up the shared resources to be used by Oracle.

#2: HA Monitor executes the server start command.

#3: When a single server is used to control multiple Oracle instances, the flow of processing described in this subsection is repeated as many times as there are Oracle instances.

(2) Flow of processing for starting an Oracle listener

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when an Oracle listener is started.

Figure 2-4 Overview of processing for starting an Oracle listener

#1: Executed to allow an IP address for client connection to be included in the server resources. For details about the resources required by servers, see 3.1.2 Resource configuration considerations and 3.2.6 Setting up the shared resources to be used by Oracle.

#2: HA Monitor executes the server start command.

#3: When a single server is used to control multiple Oracle listeners, the flow of processing described in this subsection is repeated as many times as there are Oracle listeners.

2.2.2 Flow of processing when server startup fails

This subsection provides an overview of the processing performed by HA Monitor and HA Monitor Kit when startup of a server (Oracle instance or Oracle listener) fails. Note that the figures used in this subsection show examples in which an Oracle instance or an Oracle listener is associated with a single server.

Organization of this subsection

(1) Flow of processing when startup of an Oracle instance fails

(2) Flow of processing when startup of an Oracle instance times out

(3) Flow of processing when startup of an Oracle listener fails

(4) Flow of processing when startup of an Oracle listener times out

(1) Flow of processing when startup of an Oracle instance fails

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when startup of an Oracle instance fails.

Figure 2-5 Overview of processing when startup of an Oracle instance fails

#1: Executed to allow an IP address for client connection to be included in the server resources. For details about the resources required by servers, see 3.1.2 Resource configuration considerations and 3.2.6 Setting up the shared resources to be used by Oracle.

#2: HA Monitor executes the server start command.

(2) Flow of processing when startup of an Oracle instance times out

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when startup of an Oracle instance times out.

Figure 2-6 Overview of processing when startup of an Oracle instance times out

#1: Executed to allow an IP address for client connection to be included in the server resources. For details about the resources required by servers, see 3.1.2 Resource configuration considerations and 3.2.6 Setting up the shared resources to be used by Oracle.

#2: HA Monitor executes the server start command.

#3: HA Monitor executes the server termination command. This occurs only if you specified use in the termcmd_at_abort operand of the HA Monitor environment settings.

#4: When a single server is used to control multiple Oracle instances, the flow of processing described in this subsection is repeated as many times as there are Oracle instances.

(3) Flow of processing when startup of an Oracle listener fails

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when startup of an Oracle listener fails.

Figure 2-7 Overview of processing when startup of an Oracle listener fails

#1: Executed to allow an IP address for client connection to be included in the server resources. For details about the resources required by servers, see 3.1.2 Resource configuration considerations and 3.2.6 Setting up the shared resources to be used by Oracle.

#2: HA Monitor executes the server start command.

(4) Flow of processing when startup of an Oracle listener times out

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when startup of an Oracle listener times out.

Figure 2-8 Overview of processing when startup of an Oracle listener times out

#1: Executed to allow an IP address for client connection to be included in the server resources. For details about the resources required by servers, see 3.1.2 Resource configuration considerations and 3.2.6 Setting up the shared resources to be used by Oracle.

#2: HA Monitor executes the server start command.

#3: HA Monitor executes the server termination command. This occurs only if you specified use in the termcmd_at_abort operand of the HA Monitor environment settings.

#4: When a single server is used to control multiple Oracle listeners, the flow of processing described in this subsection is repeated as many times as there are Oracle listeners.

2.2.3 Flow of processing for stopping a server

This subsection provides an overview of the processing performed by HA Monitor and HA Monitor Kit when a server (Oracle instance or Oracle listener) is stopped. Note that the figures used in this subsection show examples in which an Oracle instance or an Oracle listener is associated with a single server.

Organization of this subsection

(1) Flow of processing for stopping an Oracle instance

(2) Flow of processing for stopping an Oracle listener

(1) Flow of processing for stopping an Oracle instance

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when an Oracle instance is stopped.

Figure 2-9 Overview of processing for stopping an Oracle instance

#1: HA Monitor executes the server termination command.

#2: When a single server is used to control multiple Oracle instances, the flow of processing described in this subsection is repeated as many times as there are Oracle instances.

#3: HA Monitor waits for the server termination command to complete.

#4: Executed to allow an IP address for client connection to be included in the server resources. For details about the resources required by servers, see 3.1.2 Resource configuration considerations and 3.2.6 Setting up the shared resources to be used by Oracle.

(2) Flow of processing for stopping an Oracle listener

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when an Oracle listener is stopped.

Figure 2-10 Overview of processing for stopping an Oracle listener

#1: HA Monitor executes the server termination command.

#2: When a single server is used to control multiple Oracle listeners, the flow of processing described in this subsection is repeated as many times as there are Oracle listeners.

#3: HA Monitor waits for the server termination command to complete.

#4: Executed to allow an IP address for client connection to be included in the server resources. For details about the resources required by servers, see 3.1.2 Resource configuration considerations and 3.2.6 Setting up the shared resources to be used by Oracle.

2.2.4 Flow of processing for monitoring a server

This subsection provides an overview of the processing performed by HA Monitor and HA Monitor Kit when monitoring a server (Oracle instance or Oracle listener). Note that the figures used in this subsection show examples in which an Oracle instance or an Oracle listener is associated with a single server.

Organization of this subsection

(1) Flow of processing for monitoring an Oracle instance

(2) Flow of processing for monitoring an Oracle listener

(1) Flow of processing for monitoring an Oracle instance

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when an Oracle instance is monitored.

Figure 2-11 Overview of processing for monitoring an Oracle instance

#1: HA Monitor executes the server monitoring command.

#2: The processing in this part is repeated at the monitoring interval specified in the PATROL_INTERVAL environment variable of the server monitoring command.

#3: HA Monitor Kit issues SQL queries (database accesses and I/O checks). If there is no response within the specified length of time, a slowdown is considered to have occurred. You specify the time to wait before declaring a slowdown using PATROL in the Oracle instance control definition file.

#4: HA Monitor Kit checks whether the following Oracle instance processes exist:

CKPT, DBW0, LGWR, PMON, SMON

#5: When a single server is used to control multiple Oracle instances, the flow of processing described in this subsection is repeated as many times as there are Oracle instances.

(2) Flow of processing for monitoring an Oracle listener

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when an Oracle listener is monitored.

Figure 2-12 Overview of processing for monitoring an Oracle listener

#1: HA Monitor executes the server monitoring command.

#2: The processing in this part is repeated at the monitoring interval specified in the PATROL_INTERVAL environment variable of the server monitoring command.

#3: HA Monitor Kit executes the listener control utility command and issues a query. You specify the time to wait before declaring a slowdown using PATROL in the Oracle listener control definition file.

#4: HA Monitor Kit checks whether the Oracle listener's TNSLSNR process exists.

#5: When a single server is used to control multiple Oracle instances, the flow of processing described in this subsection is repeated as many times as there are Oracle instances.

2.2.5 Flow of processing for performing hot standby when a server failure occurs

This subsection provides an overview of the processing performed by HA Monitor and HA Monitor Kit when an error occurs in a server (Oracle instance or Oracle listener). Note that the figures used in this subsection show examples in which an Oracle instance or an Oracle listener is associated with a single server.

Organization of this subsection

(1) Flow of processing for performing hot standby when a server failure occurs in an Oracle instance

(2) Flow of processing for performing hot standby when a server failure occurs in an Oracle listener

(1) Flow of processing for performing hot standby when a server failure occurs in an Oracle instance

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when an error occurs in an Oracle instance.

Figure 2-13 Overview of processing for performing hot standby when an error occurs in an Oracle instance

#1: HA Monitor executes the server termination command.

#2: When a single server is used to control multiple Oracle instances, the flow of processing described in this subsection is repeated as many times as there are Oracle instances.

#3: Executed to allow an IP address for client connection to be included in the server resources. For details about the resources required by servers, see 3.1.2 Resource configuration considerations and 3.2.6 Setting up the shared resources to be used by Oracle.

#4: HA Monitor executes the server start command.

(2) Flow of processing for performing hot standby when a server failure occurs in an Oracle listener

The following figure provides an overview of the processing performed by HA Monitor and HA Monitor Kit when an error occurs in an Oracle listener.

Figure 2-14 Overview of processing for performing hot standby when an error occurs in an Oracle listener

#1: HA Monitor executes the server termination command.

#2: When a single server is used to control multiple Oracle listeners, the flow of processing described in this subsection is repeated as many times as there are Oracle listeners.

#3: Executed to allow an IP address for client connection to be included in the server resources. For details about the resources required by servers, see 3.1.2 Resource configuration considerations and 3.2.6 Setting up the shared resources to be used by Oracle.

#4: HA Monitor executes the server start command.

3. Designing and Building a System

This chapter explains how to design and build a system, and provides an example of setting up an environment.

The system design section explains the issues that you need to evaluate before installing HA Monitor Kit in a business system.

The system building section explains the procedure for building a system into which you are going to install HA Monitor Kit, the directory configuration of HA Monitor Kit and the relationships among the setting items necessary for building a system. It then provides an overview of how to set up the environment variables.

Organization of this chapter

3.1 Designing a System

3.2 Building a system

3.3 Environment definition example

3.1 Designing a System

Before building a hot-standby configuration that uses HA Monitor Kit, you need to design a HA Monitor system. For details about the issues that are identical to an HA Monitor systems that do not use HA Monitor Kit, see the description under Installation of HA Monitor and System Design in the manual HA Monitor (for Linux(R)).

This section explains the issues that you need to evaluate in order to manage Oracle as a monitor-mode server.

Organization of this section

3.1.1 Oracle component configuration and hot standby switchable unit considerations

3.1.2 Resource configuration considerations

3.1.1 Oracle component configuration and hot standby switchable unit considerations

HA Monitor performs hot standby on a server-by-server basis. Therefore, you need to consider how you want to map the Oracle instances and Oracle listeners to be controlled by HA Monitor Kit to the servers, which function as the switchable units for hot standby. Consider doing the following:

Set up a server for each switchable operation unit and hot standby unit.
If you plan to operate and execute hot standby for multiple Oracle instances and Oracle listeners at the same time, you can also group them into a single server.

If you wish to operate and execute hot standby for multiple Oracle instances and Oracle listeners at the same time in the following cases, set up multiple servers and then organize them into groups:

You want to execute operations such as startup and termination individually (that is, not to start or stop all servers in a batch).
Divide the servers into groups according to switchable operation units.

You want to separate Oracle users.
Divide the servers into groups according to Oracle user.

You want to set up a different monitoring interval for each Oracle instance or Oracle listener to be monitored.
Divide the servers into groups according to monitoring interval.

You want to set up different server environment definitions for HA Monitor, for example, by setting up a different action for each Oracle instance or Oracle listener to be taken when a failure is detected.
Separate the servers into groups according to the environment setting for server handling.

The following figure shows an example of typical hot-standby configuration.

Figure 3-1 Example of a typical hot-standby configuration

The following figure shows the correspondence between a server and the Oracle instance and listener in the primary system of the above example.

Figure 3-2 Example in which a single server is associated with multiple Oracle instances or listeners, or with multiple Oracle instances and listeners.

Figure 3-3 Example in which servers are associated individually with an Oracle instance and an Oracle listener, and then grouped together

3.1.2 Resource configuration considerations

This subsection explains the resource-related items that you need to consider.

Organization of this subsection

(1) Resource and server correspondence

(2) Resource server consideration

(1) Resource and server correspondence

In an Oracle hot-standby configuration, the resources that HA Monitor needs to switch correspond to servers as follows:

Shared disk
Define a shared disk in the server that is associated with the Oracle instance or Oracle listener that uses the shared disk.

IP address for client connection (alias IP address)
Define the alias IP address needed by a client for connecting to the database in the status-setting file of the LAN of the server that is associated with the Oracle listener. If no Oracle listener is used, define the alias IP address in the status-setting file of the LAN of the server that is associated with the Oracle relational database management system to which the client connects.

For details about the resources that can be controlled by HA Monitor, see the description under Evaluation of resources configuration in the manual HA Monitor (for Linux(R)).

(2) Resource server consideration

In a configuration in which multiple servers share the same resource, consider the use of a resource server.

For details about resource servers, see the description under Configurations using resource servers in the manual HA Monitor (for Linux(R)).

3.2 Building a system

This section explains how to build an HA Monitor Kit system. To build a system, you need to have superuser permissions. For details about the areas that are identical to an HA Monitor system that does not use HA Monitor Kit, such as the procedure for adding a host, see the description under System Configuration in the manual HA Monitor (for Linux(R)).

Note that the following explanation assumes that the hardware necessary for building a system has already been configured.

Organization of this section

3.2.1 General configuration procedure

3.2.2 Directory configuration

3.2.3 Relationships among setting items necessary for building a system and setting overview

3.2.4 Oracle installation

3.2.5 Setting up Oracle and building a database

3.2.6 Setting up the shared resources to be used by Oracle

3.2.7 Setting up HA Monitor Kit

3.2.8 Editing HA Monitor Kit Commands

3.2.9 Creating a definition file (server)

3.2.10 Settings for collecting error information

3.2.11 Checking the operation of the system that has been built

3.2.1 General configuration procedure

This subsection explains the general procedure for building a system when using HA Monitor Kit to set up Oracle as an HA Monitor server. When reading Configuration steps when a server is added under Configuration procedure in the manual HA Monitor (for Linux(R)), replace the first part of that procedure with the procedure described in this subsection.

The tasks explained here are performed by the system administrator, logged on with superuser permissions, after the settings for Configuration steps when a host is added under Configuration procedure in the manual HA Monitor (for Linux(R)) are completed.

The following figure shows the general procedure for building a system.

Figure 3-4 General procedure for building a system

The correspondence between the purposes for building a system and the steps that must be taken in the general procedure for building a system are explained below.

When building a new system by installing HA Monitor and HA Monitor Kit for the first time
Read both this manual and the manual HA Monitor (for Linux(R)), and check and set up all items shown in the figure.

When adding a host
To add a host to an existing hot-standby configuration, see the description under Configuration procedure in the manual HA Monitor (for Linux(R)). Each host requires its own HA Monitor.

When adding a server
To add a server to an existing hot-standby configuration, follow the procedure Settings for adding a server shown in the above figure.

For details about how to add a host or server to an existing system, see the description under Configuration steps when a host is added or Configuration steps when a server is added in the manual HA Monitor (for Linux(R)).

Note the following when executing the individual steps:

Perform the Install Oracle procedure only if an Oracle relational database management system controlled by HA Monitor has not been installed.

Perform the Set up Oracle and build a database procedure for each Oracle instance and Oracle listener to be controlled by HA Monitor.

Perform the Install HA Monitor Kit procedure for each host in the hot-standby configuration when installing HA Monitor Kit for the first time.

3.2.2 Directory configuration

When you install HA Monitor Kit, the HA Monitor Kit directory HAmonOra is created under /opt/hitachi.

The following figure shows the directory configuration after HA Monitor Kit is installed.

Figure 3-5 Directory configuration

Note

Do not edit the files in the command file directory (/opt/hitachi/HAmonOra/bin). If files need to be edited, first copy the relevant files to the edited command file directory (/opt/hitachi/HAmonOra/bin_ex) and then edit the files in that directory. For details, see 3.2.8 Editing HA Monitor Kit Commands.

3.2.3 Relationships among setting items necessary for building a system and setting overview

This subsection explains the relationships among the various setting items necessary for building a system that you must understand before setting up a system that uses HA Monitor Kit, and provides an overview on how to set them up. The following figure shows the relationships among the settings in the system.

Figure 3-6 Relationships among setting items necessary for building a system and setting overview

The relationships among the settings shown in the figure are explained below. The numbers correspond to the numbers shown in the figure.

Copy and set up a sample of the server start command, the server termination command, the server monitoring command, and the server monitoring command runtime shell for each server.
For details, see 3.2.7(1) Setting up the server start, termination, and monitoring commands.

Copy and set up a sample Oracle instance control definition file for each Oracle instance.
For details, see 3.2.7(2) Setting up control definition files.

Copy and set up a sample Oracle listener control definition file for each Oracle listener.
For details, see 3.2.7(2) Setting up control definition files.

Create a server definition in the server environment definition (servers) for each server considered in 3.1.1 Oracle component configuration and hot standby switchable unit considerations.
For details about how to create a definition file, see 3.2.9 Creating a definition file (server).

Create a LAN status setting file for each server that specifies the IP address for client connection.
Determine the servers that will use the IP address for client connection by referencing 3.1.2 Resource configuration considerations. For details about the setting method, see 3.2.6 Setting up the shared resources to be used by Oracle.

3.2.4 Oracle installation

For the Oracle installation procedure, see the Oracle documentation. This subsection explains the issues you need to be aware of when installing Oracle.

When installing Oracle on the local disk, use the same values for the following settings in the primary system and the secondary system:

Oracle user's user name (normally oracle), user ID, group name, and group ID

Oracle home directory

When installing Oracle on a shared disk, use the same values for the following settings in the host used for installation and for all hosts not used for installation:

Oracle user's user name (normally oracle), user ID, group name, and group ID

OS settings such as kernel parameters

If you want to create a database during installation, see 3.2.5(1) Building a database before you start the installation process.

If you want to use Oracle Enterprise Manager Database Control in a hot-standby environment (cluster environment), you must to be aware of certain issues. For details, see the Oracle documentation.

The method of authenticating connection to the Oracle user database must be the OS authentication method. For details about OS authentication, see the Oracle documentation.

Oracle users must have sysdba privileges. Therefore, assign sysdba privileges to all Oracle users.

3.2.5 Setting up Oracle and building a database

Once you have installed Oracle, set up Oracle and build a database. For details about how to set up Oracle and build a database, see the Oracle documentation.

Points to consider when setting up an Oracle listener and building a database are explained below.

Organization of this subsection

(1) Building a database

(2) Building Oracle listeners

(1) Building a database

This subsection explains the items that you need to set up in order to build a database compatible with the hot-standby configuration. Keep in mind the following requirements when building a database:

Build the database from one of the hosts on which Oracle instance will run.

Create the database in a file system that HA Monitor can control.

Store the following files comprising the database on a shared disk:

Server parameter file

Control file

Data file

REDO log file

Archive log file

If the server parameter file (SPFILE) has been created on a local disk, first move the file to the shared disk and then execute either of the following:

Define the path to the server parameter file in the initialization parameter file (PFILE).

Create a link in the server parameter file so that it can be referenced from the path that existed prior to the move.

In the flash recovery area, specify the path on the shared disk.

(2) Building Oracle listeners

Build Oracle listeners on the hosts in all primary and secondary systems.

To use the IP address for client connection (alias IP address), set up the following after you have built Oracle listeners:

Register in the /etc/hosts file the alias IP address and host name to be used. Use the same name in all hosts.

Change the host of the net service name of Oracle's local naming parameter file (tnsnames.ora) to the alias IP address and host name registered in Step 1.

Setting examples follow:

/etc/hosts file example

192.168.0.11  server1    # alias IP address

tnsnames.ora example

ORCL1 =
 (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)(HOST = server1)(PORT = 1521))
   (CONNECT_DATA =
     (SERVER = SHARED)
     (SERVICE_NAME = orcl1)
   )
 )

Reference note

At the client, connect using the tnsnames.ora file modified in Step 2 and the registered host name.

# sqlplus user/passwd@server1/ORCL1

3.2.6 Setting up the shared resources to be used by Oracle

Once you have finished setting up Oracle and building a database, set up the shared resources to be used by Oracle. Set up the following items:

Setting up a shared disk

Set up a shared disk so that it can be controlled by HA Monitor. For details, see the description under Configuring a shared disk in the manual HA Monitor (for Linux(R)).

Setting up a LAN status setting file

Create a LAN status setting file for each server that uses an IP address so that HA Monitor can switch the IP address for client connection. For details, see Specifying LAN status settings files in the manual HA Monitor (for Linux(R)).

3.2.7 Setting up HA Monitor Kit

Once you have finished setting up the shared resources to be used by Oracle, set up HA Monitor Kit. Setting up HA Monitor Kit involves setting up the server start, termination, and monitoring commands, as well as setting up the control definition files.

Organization of this subsection

(1) Setting up the server start, termination, and monitoring commands

(2) Setting up control definition files

(1) Setting up the server start, termination, and monitoring commands

HA Monitor Kit provides sample files for the server start, termination, and monitoring commands. Copy these files to the desired directory in each server and set them up according to the environment. We recommend that you copy these files to HA Monitor's environment settings directory (/opt/hitachi/HAmon/etc).

The following table shows the directories for storing the sample files for the server start, termination, and monitoring commands provided by HA Monitor Kit.

Table 3-1 Directories for storing the sample files for the server start, termination, and monitoring commands

Command | Storage directory
Server start command | /opt/hitachi/HAmonOra/lib/actcommand
Server termination command | /opt/hitachi/HAmonOra/lib/termcommand
Server monitoring command | /opt/hitachi/HAmonOra/lib/patrolcommand
Server monitoring command runtime shell | /opt/hitachi/HAmonOra/lib/patrol.sh

The names of the copied files need not be the same as those of the copying source files. If simply copying files will result in duplicate file names because you are building multiple servers, change the directory at the copying destination or change the file name to avoid duplication.

For details about how to set up each command, see 4. Setting Up the Server Start, Termination, and Monitoring Commands.

(2) Setting up control definition files

You need to set up a control definition file for each Oracle instance or Oracle listener that HA Monitor Kit controls. Copy the sample control definition files provided by HA Monitor Kit, rename them, and then set environment variables and modify the parameter values in the files as needed. If there are multiple Oracle instances or Oracle listeners to be monitored, set up a control definition file for each Oracle instance or Oracle listener.

The following table shows the source control definition files to copy and the copy destination directories.

Table 3-2 Source control definition files to copy and the copy destination directories

Control definition file | Source file | Copy destination directory
Oracle instance control definition file | /opt/hitachi/HAmonOra/lib/orcl1.env | /opt/hitachi/HAmon/etc/HAmonOra_etc/instance
Oracle listener control definition file | /opt/hitachi/HAmonOra/lib/LISTENER.env | /opt/hitachi/HAmon/etc/HAmonOra_etc/listener

Rename each file as shown below. Replace the italicized parts with the applicable values.

Oracle instance control definition file: instance-identifier.env

Oracle listener control definition file: listener-name.env

When copying a file, be careful not to accidentally overwrite an existing file at the copy destination.

The following explains the file details:

Description format

Specify one parameter per line using the format parameter-name=value. Do not enter a blank space before or after parameter, the equal sign (=), and value. You can specify parameters in any order.

If you want to set environment variables, specify them one per line using the format environment-variable-name=value.

Parameter detail

The following table shows the details of the parameters that can be specified.

Table 3-3 Details of the parameters that can be specified in the control definition files

Parameter name
Description

ORACLE_SID
Specify this parameter in the Oracle instance control definition file. Specify the value of the ORACLE_SID environment variable (instance identifier) of the Oracle instance to be controlled.
For details about the ORACLE_SID environment variable, see the Oracle documentation.

LISTENER_NAME
Specify this parameter in the Oracle listener control definition file. Specify the listener name of the Oracle listener to be controlled. For details about listener names, see the Oracle documentation.

ORACLE_BASE
Specify the value of the ORACLE_BASE environment variable (Oracle base directory) of the Oracle instance or Oracle listener to be controlled.
For details about the ORACLE_BASE environment variable, see the Oracle documentation.

ORACLE_HOME
Specify the value of the ORACLE_HOME environment variable (Oracle home directory) of the Oracle instance or Oracle listener to be controlled.
For details about the ORACLE_HOME environment variable, see the Oracle documentation.

ORACLE_USER
Specify the account name of the Oracle user. For details about Oracle users, see the Oracle documentation.

DB_NOACCESS
Specify this parameter in the Oracle instance control definition file. Specify whether SQL will be used to access the Oracle instance. You can specify either of the following values:

yes: SQL is not used to access the Oracle instance. Only process operation is monitored without monitoring the Oracle instance's status or slowdown.

no: SQL is used to access the Oracle instance to monitor the Oracle instance's status, slowdown, and process operation.

If this parameter is omitted, no is assumed. Specify yes if you want to use the following types of operations:

Operation in which HA Monitor Kit is prohibited from accessing the database

Operation in which the ALTER SYSTEM SUSPEND SQL statement is used to prohibit database access

Operation in which you want to put the Oracle instance in a status other than OPEN

LISTENER_NOACCESS
Specify this parameter in the Oracle listener definition file. Specify whether the listener control utility will be used to access Oracle listener. You can specify either of the following values:

yes: The listener control utility is not used. Only process operation is monitored without monitoring the Oracle listener's slowdown.

no: The listener control utility is used to monitor the Oracle listener's slowdown and process operation.

If this parameter is omitted, no is assumed.

PATROL
Specify the length of time (in seconds) after which the Oracle instance or Oracle listener that has been accessed is deemed to have slowed down. You can specify a value between 60 and 3600.
The value you specify for this parameter is ignored if you specified yes for the DB_NOACCESS or LISTENER_NOACCESS parameter.

ABORT_LIMIT
Specify a value between 10 and 3600 (seconds) as the wait time before shifting to forced termination if normal termination does not execute successfully.

Example

Examples of control definition files are shown below. These examples have the same content as the sample files.

Oracle instance control definition file (/opt/hitachi/HAmon/etc/HAmonOra_etc/instance/orcl1.env):

ORACLE_SID=orcl1
ORACLE_BASE=/home/oracle
ORACLE_HOME=/home/oracle/product/10.2.0/db_1
ORACLE_USER=oracle
PATROL=120
ABORT_LIMIT=180

Oracle listener control definition file (/opt/hitachi/HAmon/etc/HAmonOra_etc/listener/LISTENER.env):

LISTENER_NAME=LISTENER
ORACLE_BASE=/home/oracle
ORACLE_HOME=/home/oracle/product/10.2.0/db_1
ORACLE_USER=oracle
PATROL=120
ABORT_LIMIT=180

Notes:

Do not modify an already defined control definition file while an Oracle instance or Oracle listener is running.

If you make a mistake in a definition specification, HA Monitor Kit commands will terminate abnormally and fail to control the Oracle instance or Oracle listener. As a result, server startup or termination fails, or HA Monitor detects an Oracle instance or Oracle listener error.

3.2.8 Editing HA Monitor Kit Commands

Normally there is no need to edit HA Monitor Kit commands, but if necessary you can edit the listener control utility commands and the commands used in SQL*Plus.

Reference note

For example, if SYS audit logging is enabled in the Oracle settings and you connect to Oracle as the SYS user, audit logs will be output for each monitored instance, generating a proliferation of log files. To avoid this, consider editing the HA Monitor Kit commands to change the connecting Oracle user when monitoring an Oracle instance. For details, see Changing the connecting Oracle user when monitoring an Oracle instance in (2) Editing examples.

If you edit HA Monitor Kit commands, carefully follow the steps listed below, taking note of the command structure and editing examples that are shown.

HA Monitor Kit command structure

The HA Monitor Kit command structure is as follows:

Server start command structure
The following figure shows the relationship between the server start command, and the Oracle instance and Oracle listener start commands.

Figure 3-7 Relationship between server start command and Oracle instance and Oracle listener start commands

Server termination command structure
The following figure shows the relationship between the server termination command, and the Oracle instance and Oracle listener termination commands.

Figure 3-8 Relationship between the server termination command, and the Oracle instance and Oracle listener termination commands

Server monitoring command structure
The following figure shows the relationship between the server monitoring command, and the Oracle instance and Oracle listener monitoring commands.

Figure 3-9 Relationship between the server monitoring command and the Oracle instance and Oracle listener monitoring commands

Organization of this subsection

(1) Editing procedure

(2) Editing examples

(1) Editing procedure

The figure below shows the procedure for editing HA Monitor Kit commands. Perform step 3 only if necessary.

Figure 3-10 Procedure for editing HA Monitor Kit commands

Using the cp -p command, copy the files that you need to edit from the command file directory to the edited command file directory.

Command file directory

/opt/hitachi/HAmonOra/bin

Edited command file directory

/opt/hitachi/HAmonOra/bin_ex

Edit the files in the edited command file directory.
In this example, the file haorainsbgn is edited.
For details, see (2) Editing examples.

Perform this step only after upgrading from HA Monitor Kit 01-00. Reconfigure the server start, termination, and monitoring commands using the post-upgrade HA Monitor Kit sample files for those commands.
For details, see 3.2.7(1) Setting up the server start, termination, and monitoring commands.

Note

If you upgrade from HA Monitor Kit 01-01 or later after editing a command, the new version of the file will not be installed in the command file edit directory (/opt/hitachi/HAmonOra/bin_ex). Therefore, you might have to repeat the editing procedure on the new version of the file in the command file directory (/opt/hitachi/HAmonOra/bin). For details about the files that require editing, see Usage Notes in the Release Notes.

(2) Editing examples

The following are examples of how to edit HA Monitor Kit commands when you want to connect to Oracle as a particular Oracle user:

Changing the connecting Oracle user when starting or stopping an Oracle instance
Edit the files as shown below. This example shows how to start and stop an Oracle instance password as the SYS user with password authentication.

Files to edit

haorainsbgn and haorainsendsub

Edits

Before editing

connect/assysdba

After editing

connectsys/passwordassysdba

Legend: : Single-byte space

You must apply these edits at all the relevant places in the file.

Notes

The Oracle user must have been granted sysdba privileges.

If you use password authentication, make sure the password is not set to expire.

Changing the connecting Oracle user when monitoring an Oracle instance
Edit the files according to one of the patterns shown below. However, note that no editing is necessary if DB_NOACCESS is set to yes, because in this case no connection to Oracle is made when an Oracle instance is monitored.

Files to edit

haorainsptlsub and haorainsptl.sql

Edit 1 (to monitor an Oracle instance as the SYSTEM user with password authentication)

Before editing

connect/assysdba

After editing

connectsystem/password

Legend: : Single-byte space

You must apply these edits at all the relevant places in the file.

Notes

If you use password authentication, make sure the password is not set to expire.

Edit 2 (to monitor an Oracle instance as a general DB user with OS authentication)

Before editing

connect/assysdba

After editing

connect/

Legend: : Single-byte space

You must apply these edits at all the relevant places in the file.

Notes

In the server monitoring command runtime shell patrol.sh, set the ORACLE_USER environment variable to the account name of a newly created OS user.

Prepare a general DB user with OS authentication.

Grant the DBA role and CONNECT role to the general DB user.

3.2.9 Creating a definition file (server)

Once you have completed the setup of HA Monitor Kit, create HA Monitor's server environment definition for each server determined in 3.1.1 Oracle component configuration and hot standby switchable unit considerations. For the creation method, see the description under Server environment definition (servers) in the manual HA Monitor (for Linux(R)).

The following table shows the server environment definition operands that require special attention.

Table 3-4 Server environment definition operands that require special attention

Operand name
Note

name
Specify the absolute path to the server start command that was set up in 4.2 Server start command setup.

acttype
Specify monitor.

termcommand
Specify the absolute path to the server termination command that was set up in 4.3 Server termination command setup.

patrolcommand
Specify the absolute path to the server monitoring command runtime shell that was set up in 4.4.2 Server monitoring command runtime shell.

servexec_retry
Specifying 0 is recommended.
However, if you want to restart when a server failure is detected, either do not specify this operand or specify the number of restarts.

waitserv_exec
Specify yes.

group
Specify this operand when you want to divide servers into groups as determined in 3.1.1 Oracle component configuration and hot standby switchable unit considerations. Specify the same group name for servers within the same group.

start_timeout
When specifying this operand, you must also specify yes for the termcmd_at_abort operand of HA Monitor environment settings so that the server can be restarted after executing the termination command when a timeout occurs.

stop_timeout
Do not specify this operand. When the stopping process times out based on the specification of the stop_timeout operand, forced termination of Oracle instance and Oracle listener may not execute correctly in some cases.

3.2.10 Settings for collecting error information

If you are using a configuration in which user application programs are loaded on the server that controls Oracle, you can collect error information for use when investigating the cause of the problem by executing a shell whenever a user application program slows down.

3.2.11 Checking the operation of the system that has been built

Once you have finished building a system and an inspection of its settings and definitions shows no problems, check the operation of the system you built. This subsection explains only those items that must be carefully examined in systems that use HA Monitor Kit. For details about system operation check, see the description under Checking the operation of the configured system in the manual HA Monitor (for Linux(R)).

Organization of this subsection

(1) Checking the operation of servers

(2) Hot-standby test

(1) Checking the operation of servers

This subsection explains how to check the startup and shutdown of the servers necessary for using HA Monitor Kit.

Server startup check

Check the following based on the HA Monitor Kit server log:

For a server that is associated with an Oracle instance, message KAMK00302-I has been output for the applicable Oracle instance.

For a server that is associated with an Oracle listener, message KAMK00322-I has been output for the applicable Oracle listener.

If no message was output, the environment variable that you set in the server start command might be invalid. For details about how to set up the server start command, see 4.2 Server start command setup.

Server stop check

Check the following based on the HA Monitor Kit server log:

For a server that is associated with an Oracle instance, message KAMK00305-I has been output for the applicable Oracle instance.

For a server that is associated with an Oracle listener, message KAMK00325-I has been output for the applicable Oracle listener.

If no message was output, the environment variable that you set in the server termination command might be invalid. For details about how to set up the server termination command, see 4.3 Server termination command setup.

Server monitoring check

Check the following based on the HA Monitor Kit server log:

For a server that is associated with an Oracle instance or Oracle listener, message KAMK00421-I has been output.

If no message was output, the environment variable that you set in the server monitoring command or server monitoring command runtime shell might be invalid. For details about how to set up the server monitoring command and server monitoring command runtime shell, see 4.4 Server monitoring command setup.

(2) Hot-standby test

This subsection explains the hot-standby test to perform when using HA Monitor Kit.

Checking for planned hot standby and checking for automatic hot standby upon server failure

After hot standby has been successful, check the following based on the HA Monitor Kit server log at the switching-destination host:

For a server that is associated with an Oracle instance, message KAMK00302-I has been output for the applicable Oracle instance.

For a server that is associated with an Oracle listener, message KAMK00322-I has been output for the applicable Oracle listener.

If no message was output, the environment variable that you set in the server start command might be invalid. For details about how to set up the server start command, see 4.2 Server start command setup.

3.3 Environment definition example

This section describes an example of environmental definition for a system that uses HA Monitor Kit. This example uses the system configuration shown in the following figure in which an Oracle instance and an Oracle listener are monitored by a single server.

Figure 3-11 System configuration in which an Oracle instance and an Oracle listener are monitored by a single server

#1: The following directory paths are used in the environment definition example:

Server start command: /opt/hitachi/HAmon/etc/orasrv/actcommand_orasrv

Server termination command: /opt/hitachi/HAmon/etc/orasrv/termcommand_orasrv

Server monitoring command runtime shell: /opt/hitachi/HAmon/etc/orasrv/patrol_orasrv.sh

Oracle home directory: /home/oracle/product/10.2.0/db_1

#2: This is the interval at which the server monitoring command performs monitoring.

An environment definition example for this system configuration follows:

/etc/hosts file

192.168.100.1  server1    # alias IP address

Oracle local naming parameter file (definition file: tnsnames.ora):

ORCL1 =
 (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)(HOST = server1)(PORT = 1521))
   (CONNECT_DATA =
     (SERVER = SHARED)
     (SERVICE_NAME = orcl1)
   )
 )

Oracle instance control definition file (definition file: /opt/hitachi/HAmon/etc/HAmonOra_etc/instance/orcl1.env):

ORACLE_SID=orcl1
ORACLE_BASE=/home/oracle
ORACLE_HOME=/home/oracle/product/10.2.0/db_1
ORACLE_USER=oracle
PATROL=120
ABORT_LIMIT=180

Oracle listener control definition file (definition file: /opt/hitachi/HAmon/etc/HAmonOra_etc/listener/LISTENER.env):

LISTENER_NAME=LISTENER
ORACLE_BASE=/home/oracle
ORACLE_HOME=/home/oracle/product/10.2.0/db_1
ORACLE_USER=oracle
PATROL=120
ABORT_LIMIT=180

LAN status setting file

server-alias-name.up (definition file: /opt/hitachi/HAmon/etc/orasrv.up):

#!/bin/sh
set -x

INTERFACE=eth0:1                    # LAN interface name
IPADDR=192.168.100.1                # alias IP address
NETMASK=255.255.255.0               # netmask address
BROADCAST=192.168.100.255           # broadcast address
RETRYCNT=2                          # retry count

# Retry for the alias IP address adding.
exitcode=1
loopcnt=0
while [ $loopcnt -lt $RETRYCNT ]
do
   # The alias IP address is added to the LAN interface.
   /sbin/ifconfig $INTERFACE inet $IPADDR netmask $NETMASK broadcast $BROADCAST

   # The information on old routing cache is deleted.
   /bin/echo 0 > /proc/sys/net/ipv4/route/flush

   # The ARP caches are updated.
   IFNAME=`echo $INTERFACE | /bin/sed -e 's/:[0-9]*$//'`
   /sbin/arping -U -c 2 -I $IFNAME $IPADDR

   # Was the alias IP address registered to the system ?
   RCD=`/sbin/ifconfig -a | /bin/grep ":$IPADDR "`
   if [ "$RCD" = "" ]
   then
       # When it is not registered, "1" is returned as the termination code.
       exitcode=1

       loopcnt=`/usr/bin/expr $loopcnt + 1`
   else
       # When it is registered, "0" is returned as the termination code.
       exitcode=0

       break
   fi
done

exit $exitcode

server-alias-name.down (definition file: /opt/hitachi/HAmon/etc/orasrv.down):

#!/bin/sh
set -x

INTERFACE=eth0:1                    # LAN interface name
RETRYCNT=1                          # retry count

# Retry for the alias IP address delete.
exitcode=1
loopcnt=0
while [ $loopcnt -lt $RETRYCNT ]
do
   # The alias IP address is deleted from the LAN interface.
   /sbin/ifconfig $INTERFACE down

   # Was the alias IP address deleted from the system ?
   RCD=`/sbin/ifconfig -a | /bin/grep "$INTERFACE "`
   if [ "$RCD" = "" ]
   then
       # When it is deleted, "0" is returned as the termination code.
       exitcode=0
       break
   else
       # When it is not deleted, "1" is returned as the termination code.
       exitcode=1
       loopcnt=`/usr/bin/expr $loopcnt + 1`
       sleep 1
   fi
done

# The information on old routing cache is deleted.
/bin/echo 0 > /proc/sys/net/ipv4/route/flush

exit $exitcode

Server environment definition (definition file: /opt/hitachi/HAmon/etc/servers):

/*  Server environment definition  */
server  name         /opt/hitachi/HAmon/etc/orasrv/actcommand_orasrv,
  alias           orasrv,
  acttype         monitor,
  initial         online#,
  termcommand     /opt/hitachi/HAmon/etc/orasrv/termcommand_orasrv,
  patrolcommand   /opt/hitachi/HAmon/etc/orasrv/patrol_orasrv.sh,
  disk            /dev/vg01,
  fs_name         /dev/vg01/lvol1,
  fs_mount_dir    /ora_mnt,
  lan_updown      use,
  waitserv_exec   yes,
  servexec_retry  0,
  start_timeout   180;

#: For a secondary system, change this value to standby.

Server start command:

#! /bin/sh
####################################################################
LOGNAME=orasrv
LOGSIZE=1048576​
####################################################################
act_proc(){
       RT_CD=0
       # Listener
       haoralsnrbgn LISTENER

       # Instance
       haorainsbgn orcl1

       return ${RT_CD}
}
####################################################################
: (omitted)

Server termination command:

#! /bin/sh
####################################################################
LOGNAME=orasrv
LOGSIZE=1048576​
####################################################################
term_proc(){
       RT_CD=0
       case "$1" in
       "-e" )                           # normal end
               # Instance
               haorainsend orcl1
               if [ "$?" != "0" ]
               then
                       RT_CD=1
               fi

               # Listener
               haoralsnrend LISTENER
               if [ "$?" != "0" ]
               then
                       RT_CD=2
               fi
               ;;
       "-w" )                        # monswap
               # Instance
               haorainsend orcl1
               if [ "$?" != "0" ]
               then
                       RT_CD=1
               fi

               # Listener
               haoralsnrend LISTENER
               if [ "$?" != "0" ]
               then
                       RT_CD=2
               fi
               ;;
       "-c" )                        # down
               # Instance
               haorainsend orcl1

               # Listener
               haoralsnrend LISTENER
               ;;
       esac
       return ${RT_CD}
}
####################################################################
: (omitted)

Server monitoring command:

#! /bin/sh
####################################################################
LOGNAME=orasrv
LOGSIZE=1048576​
PATROL_INTERVAL=5
####################################################################
patrol_proc(){
       # Instance
       haorainsptl orcl1
       if [ "$?" != "0" ]
       then
               return 1
       fi

       # Listener
       haoralsnrptl LISTENER
       if [ "$?" != "0" ]
       then
               return 2
       fi

       return 0
}
####################################################################
: (omitted)

Server monitoring command runtime shell:

#! /bin/sh
####################################################################
ORACLE_USER=oracle
PATROL_SHELL=/opt/hitachi/HAmon/etc/patrolcommand_orasrv
####################################################################
: (omitted)

4. Setting Up the Server Start, Termination, and Monitoring Commands

This chapter explains how to set up the server start, termination, and monitoring commands, and the commands to be used.

Organization of this chapter

4.1 Overview of the server start, termination, and monitoring commands

4.2 Server start command setup

4.3 Server termination command setup

4.4 Server monitoring command setup

4.5 Commands used for setting up the server start, termination, and monitoring commands

4.1 Overview of the server start, termination, and monitoring commands

The server start, termination, and monitoring commands are shell commands that describe the processing for starting, terminating, and monitoring programs.

HA Monitor Kit provides sample files of the server start, termination, and monitoring commands. Copy these sample files, set up the server start, termination, and monitoring commands as explained in this chapter, and customize them as needed.

For details about the server start, termination, and monitoring commands, see the description under Creating server start, termination, and monitoring commands in the manual HA Monitor (for Linux(R)).

Messages that are issued by the server start, termination, and monitoring commands (including the standard output and standard errors of the Oracle commands executed as extensions of the server start, termination, and monitoring commands) are output to server logs. For details about server logs, see 5.4 Checking the server logs.

4.2 Server start command setup

HA Monitor Kit provides a sample file of the server start command. Copy the sample file and set it up as explained in this chapter, and customize it as needed. For the storage location and copying destination of the sample file, see 3.2.7(1) Setting up the server start, termination, and monitoring commands.

The server start command sample file executes the act_proc() shell function that describes the startup process to start Oracle listeners and Oracle instances.

The following figure shows the content of the server start command sample file.

Figure 4-1 Content of the server start command sample file

Numbers 1 through 5 in the figure are explained below.

This is the environment variable for setting up the operating environment necessary for starting programs.
For details about the content to be set up here, see (1) Environment variables.

This is the Oracle listener startup process.
The process for executing the haoralsnrbgn command to start the Oracle listener specified by the listener name is described here. For details about the content to be set up here, see (2) Setup method.
For details about the haoralsnrbgn command, see haoralsnrbgn (Starts an Oracle listener) in 4.5 Commands used for setting up the server start, termination, and monitoring commands.

This is the Oracle instance startup process.
The process for executing the haorainsbgn command to start the Oracle instance specified by the instance identifier is described here. For details about the content to be set up here, see (2) Setup method.
For details about the haorainsbgn command, see haorainsbgn (Starts an Oracle instance) in 4.5 Commands used for setting up the server start, termination, and monitoring commands.

This is the return value of the server start command.
For details about the content to be set up here, see (2) Setup method.

This is the process executed by the act_proc() shell function.
To add a user's own process, add it inside this act_proc() shell function. For details about the content to be set up here, see (3) Server start command customization.

Organization of this section

(1) Environment variables

(2) Setup method

(3) Server start command customization

(1) Environment variables

This subsection explains the environment variables indicated by 1 in Figure 4-1 Content of the server start command sample file.

LOGNAME
Specifies a server log file name. Specify a unique file name for each server.
We recommend that you match the server log file name to the server's server alias name. In the provided sample file, orasrv is assumed as the server alias name.
For the server log output destination, see 5.4 Checking the server logs.

LOGSIZE
Specify an integer for the server log file size (in bytes). The default is 1048576. Under normal conditions, use this default file size. Increase the size only if the volume of output is large and may cause the log to run out of space. For details about how to calculate the server log file size, see 5.4(2) File size.
For servers in which you specified the same value for the LOGNAME environment variable, specify the same value for the LOGSIZE environment variable of the server start, termination, and monitoring commands.

(2) Setup method

This subsection explains how to set up the startup processes 2 through 4 in Figure 4-1 Content of the server start command sample file.

Setting up Oracle listener startup process (2 in the figure)

The provided sample file uses a setting that starts a single Oracle listener (listener name: LISTENER) that is associated with a server. If there is no Oracle listener that is associated with a server, comment out the code indicated by 2 in the figure, or delete it. To start multiple Oracle listeners on a single server, duplicate the code indicated by 2 in the figure.

In the provided sample file, LISTENER is assumed as the listener name. Change the underlined argument LISTENER to the listener name of the Oracle listener you want to start.

Setting up Oracle instance startup process (3 in the figure)

The provided sample file uses a setting that starts a single Oracle instance (instance identifier: orcl1) that is associated with a server. If there is no Oracle instance that is associated with a server, comment out the code indicated by 3 in the figure, or delete it. To start multiple Oracle instances on a single server, duplicate the code indicated by 3 in the figure.

In the provided sample file, orcl1 is assumed as the instance identifier. Change the underlined argument orcl1 to the instance identifier of the Oracle instance you want to start.

Setting up server start command return value (4 in the figure)

The provided sample file uses a setting that does not suspend the server startup if the Oracle instance or Oracle listener fails to start. If the Oracle instance or Oracle listener fails to start, a monitoring process will detect the error after server startup is complete.

(3) Server start command customization

You can add non-Oracle application processes necessary for a job, and user-specific processes, to the server start command. When customizing the command, note the following:

Before adding a process, see the description under Creating a server start command in the manual HA Monitor (for Linux(R)).

Add the process in the act_proc() shell function inside the server start command. Do not edit any items other than the act_proc() shell function and the shell variables.
The process described here must be executed with superuser permissions. If you have changed your permissions, change them back to superuser permissions.

Do not use the exit command inside the act_proc() shell function. To terminate the process, use the return command.

The return code of the act_proc() shell function becomes the return value of the server start command. To suspend the server startup, make the act_proc() shell function return a value other than 0.

Do not add a process that does not return control to the server start command, for example, making the executed command resident.

For the output destination for the messages issued by the act_proc() shell function, see 5.4 Checking the server logs.

Specify absolute paths for the commands and files to be input/output inside the act_proc() shell function. (That is, commands other than the ones provided by HA Monitor Kit.)

Do not modify the names of the following shell variables:

LOGNAME

LOGSIZE

Before adding a shell variable, search through the command to make sure it is not already being used.

4.3 Server termination command setup

HA Monitor Kit provides a sample file of the server termination command. Copy the sample file and set it up as explained in this chapter, and customize it as needed. For the storage location and copying destination of the sample file, see 3.2.7(1) Setting up the server start, termination, and monitoring commands.

The server termination command sample file executes the term_proc() shell function that describes the termination process to terminate Oracle listeners and Oracle instances.

The following figure shows the content of the server termination command sample file.

Figure 4-2 Content of the server termination command sample file

Numbers 1 through 5 in the figure are explained below.

This is the environment variable for setting up the operating environment necessary for terminating programs.
For details about the content to be set up here, see (1) Environment variables.

This is the Oracle instance termination process.
The process for executing the haorainsend command to terminated the Oracle instance specified by the instance identifier is described here. For details about the content to be set up here, see (2) Setup method.
For details about the haorainsend command, see haorainsend (Terminates an Oracle instance) in 4.5 Commands used for setting up the server start, termination, and monitoring commands.

This is the Oracle listener termination process.
The process for executing the haoralsnrend command to terminate the Oracle listener specified by the listener name is described here. For details about the content to be set up here, see (2) Setup method.
For details about the haoralsnrend command, see haoralsnrend (Terminates an Oracle listener) in 4.5 Commands used for setting up the server start, termination, and monitoring commands.

This is the process executed by the term_proc() shell function.
To add a user's own process, place it inside this term_proc() shell function. For details about the content to be set up here, see (3) Server termination command customization.

This is the return value of the server termination command.

Organization of this section

(1) Environment variables

(2) Setup method

(3) Server termination command customization

(1) Environment variables

This subsection explains the environment variables indicated by 1 in Figure 4-2 Content of the server termination command sample file.

LOGNAME
Specifies a server log file name. Specify a unique file name for each server.
We recommend that you match the server log file name to the server's server alias name. In the provided sample file, orasrv is assumed as the server alias name.
For the server log output destination, see 5.4 Checking the server logs.

LOGSIZE
Specify an integer for the server log file size (in bytes). The default is 1048576. Under normal conditions, use this default file size. Increase the size only if the volume of output is large and may cause the log to run out of space. For details about how to calculate the server log file size, see 5.4(2) File size.
For servers in which you specified the same value for the LOGNAME environment variable, specify the same value for the LOGSIZE environment variable of the server start, termination, and monitoring commands.

(2) Setup method

This subsection explains how to set up the termination processes 2 and 3 in Figure 4-2 Content of the server termination command sample file.

Setting up Oracle instance termination process (2 in the figure)

The provided sample file uses a setting that terminates a single Oracle instance (instance identifier: orcl1) that is associated with a server. If there is no Oracle instance that is associated with a server, comment out the code indicated by 2 in the figure, or delete it. To terminate multiple Oracle instances on a single server, duplicate the code indicated by 2 in the figure.

In the provided sample file, orcl1 is assumed as the instance identifier. Change the underlined argument orcl1 to the instance identifier of the Oracle instance you want to terminate.

Setting up Oracle listener termination process (3 in the figure)

The provided sample file uses a setting that terminates a single Oracle listener (listener name: LISTENER) that is associated with a server. If there is no Oracle listener that is associated with a server, comment out the code indicated by 3 in the figure, or delete it. To terminate multiple Oracle listeners on a single server, duplicate the code indicated by 3 in the figure.

In the provided sample file, LISTENER is assumed as the listener name. Change the underlined argument LISTENER to the listener name of the Oracle listener you want to terminate.

To change the order for terminating Oracle instances and Oracle listeners, change the termination order in the file.

(3) Server termination command customization

You can add non-Oracle application processes necessary for a job, and user-specific processes, to the server termination command. When customizing the command, note the following:

Before adding a process, see the description under Creating a server termination command in the manual HA Monitor (for Linux(R)).

Add the process in the term_proc() shell function inside the server termination command. Do not edit any items other than the term_proc() shell function and the shell variables.
The process described here must be executed with superuser permissions. If you have changed your permissions, change them back to superuser permissions.

Do not use the exit command inside the term_proc() shell function. To terminate the process, use the return command.

The return code of the term_proc() shell function becomes the return code of the server termination command.

You can check the argument that HA Monitor transfers to the server termination command by referencing the first argument ($1) of the term_proc() shell function.

For the output destination for the messages issued by the term_proc() shell function, see 5.4 Checking the server logs.

Specify absolute paths for the commands and files to be input/output inside the term_proc() shell function. (That is, commands other than the ones provided by HA Monitor Kit.)

Do not modify the names of the following shell variables:

LOGNAME

LOGSIZE

Before adding a shell variable, search through the command to make sure it is not already being used.

4.4 Server monitoring command setup

This section explains the server monitoring command and the server monitoring command runtime shell.

Organization of this section

4.4.1 Server monitoring command

4.4.2 Server monitoring command runtime shell

4.4.1 Server monitoring command

HA Monitor Kit provides a sample file of the server monitoring command. Copy the sample file and set it up as explained in this chapter, and customize it as needed. For the storage location and copying destination of the sample file, see 3.2.7(1) Setting up the server start, termination, and monitoring commands.

The server monitoring command sample file executes the patrol_proc() shell function that describes the process to monitor Oracle instances and Oracle listeners at the user-specified time interval.

The following figure shows the content of the server monitoring command sample file.

Figure 4-3 Content of the server monitoring command sample file

Numbers 1 through 4 in the figure are explained below.

This is the environment variable for setting up the operating environment necessary for monitoring programs.
For details about the content to be set up here, see (1) Environment variables.

This is the Oracle instance monitoring process.
The process for executing the haorainsptl command to monitor the Oracle instance specified by the instance identifier is described here. For details about the content to be set up here, see (2) Setup method.
For details about the haorainsptl command, see haorainsptl (Checks the status of an Oracle instance) in 4.5 Commands used for setting up the server start, termination, and monitoring commands.

This is the Oracle listener monitoring process.
The process for executing the haoralsnrptl command to monitor the Oracle listener specified by the listener name is described here. For details about the content to be set up here, see (2) Setup method.
For details about the haoralsnrptl command, see haoralsnrend (Terminates an Oracle listener) in 4.5 Commands used for setting up the server start, termination, and monitoring commands.

This is the process executed by the patrol_proc() shell function.
To add your own process, add it inside this patrol_proc() shell function. For details about the content to be set up here, see (3) Server monitoring command customization

Organization of this subsection

(1) Environment variables

(2) Setup method

(3) Server monitoring command customization

(1) Environment variables

This subsection explains the environment variables indicated by 1 in Figure 4-3 Content of the server monitoring command sample file.

LOGNAME
Specifies a server log file name. Specify a unique file name for each server.
We recommend that you match the server log file name to the server's server alias name. In the provided sample file, orasrv is assumed as the server alias name.
For the server log output destination, see 5.4 Checking the server logs.

LOGSIZE
Specify an integer for the server log file size (in bytes). The default is 1048576. Under normal conditions, use this default file size. Increase the size only if the volume of output is large and may cause the log to run out of space. For details about how to calculate the server log file size, see 5.4(2) File size.
For servers in which you specified the same value for the LOGNAME environment variable, specify the same value for the LOGSIZE environment variable of the server start, termination, and monitoring commands.

PATROL_INTERVAL
Specify a value between 1 and 3,600 (seconds) for the monitoring execution interval (the interval between the end of a monitoring process and the start of the next monitoring process). The default is 5. Normally, there is no need to change the default execution interval.

(2) Setup method

This subsection explains how to set up the monitoring processes 2 and 3 in Figure 4-3 Content of the server monitoring command sample file.

Setting up Oracle instance monitoring process (2 in the figure)

The provided sample file uses a setting that monitors a single Oracle instance (instance identifier: orcl1) that is associated with a server. If there is no Oracle instance that is associated with a server, comment out the code indicated by 2 in the figure, or delete it. To monitor multiple Oracle instances on a single server, duplicate the code indicated by 2 in the figure.

In the provided sample file, orcl1 is assumed as the instance identifier. Change the underlined argument orcl1 to the instance identifier of the Oracle instance you want to monitor.

Setting up Oracle listener monitoring process (3 in the figure)

The provided sample file uses a setting that monitors a single Oracle listener (listener name: LISTENER) that is associated with a server. If there is no Oracle listener that is associated with a server, comment out the code indicated by 3 in the figure, or delete it. To monitor multiple Oracle listeners on a single server, duplicate code indicated by 3 in the figure.

In the provided sample file, LISTENER is assumed as the listener name. Change the underlined argument LISTENER to the listener name of the Oracle listener you want to monitor.

To change the order for monitoring Oracle instances and Oracle listeners, change the monitoring order in the file.

(3) Server monitoring command customization

You can add non-Oracle application processes necessary for a job, and user-specific processes, to the server monitoring command. When customizing the command, note the following:

Before adding a process, see the description under Creating a server monitoring command in the manual HA Monitor (for Linux(R)).

Add the process in the patrol_proc() shell function inside the server monitoring command. Do not edit any items other than the patrol_proc() shell and the shell variables.
The process described here must be executed with superuser permissions. If you have changed your permissions, change them back to superuser permissions.

Do not use the exit command inside the patrol_proc() shell function. To terminate the process, use the return command.

If the return code of the patrol_proc() shell function is not 0, the server monitoring command terminates and HA Monitor detects a server failure. When this occurs, the return code of the patrol_proc() shell function becomes the return code of the server monitoring command.

Be careful not to create a never-ending process, such as an endless loop, in the patrol_proc() shell function.

For the output destination for the messages issued by the patrol_proc() shell function, see 5.4 Checking the server logs.

Specify absolute paths for the commands and files to be input/output inside the patrol_proc() shell function. (That is, commands other than the ones provided by HA Monitor Kit.)

Do not modify the names of the following shell variables:

LOGNAME

LOGSIZE

PATROL_INTERVAL

Before adding a shell variable, search through the command to make sure it is not already being used.

4.4.2 Server monitoring command runtime shell

HA Monitor Kit provides a sample server monitoring command runtime shell, which is a shell for executing the server monitoring command. Copy the sample file and set it up as explained in this chapter, and customize it as needed. For the storage location and copying destination of the sample file, see 3.2.7(1) Setting up the server start, termination, and monitoring commands.

The following figure shows the content of the server monitoring command runtime shell.

Figure 4-4 Content of the sample file of the server monitoring command runtime shell

Number 1 in the figure is explained below.

This is the environment variable for setting up the operating environment necessary for monitoring programs.
For details about the content to be set up here, see (1) Environment variables.

Organization of this subsection

(1) Environment variables

(2) Note

(1) Environment variables

This subsection explains the environment variables indicated by 1 in Figure 4-4 Content of the sample file of the server monitoring command runtime shell.

ORACLE_USER
Specifies an Oracle user account name.

PATROL_SHELL
Use an absolute path to specify the server monitoring command that was set up in 4.4.1 Server monitoring command.

(2) Note

Do not modify the names of the following shell variables:

ORACLE_USER

PATROL_SHELL

4.5 Commands used for setting up the server start, termination, and monitoring commands

The table below lists the commands used for setting up the server start, termination, and monitoring commands.

Table 4-1 Commands used for setting up the server start, termination, and monitoring commands

Command name
Function
Relevant subcommands#

haorainsbgn
Starts an Oracle instance.
haoraenvcheck

haorainsend
Terminates an Oracle instance.
haoraenvcheck
haorainsendsub

haorainsptl
Checks the status of an Oracle instance.
haoraenvcheck
haorainsptlsub

haoralsnrbgn
Starts an Oracle listener.
haoraenvcheck

haoralsnrend
Terminates an Oracle listener.
haoraenvcheck
haoralsnrendsub

haoralsnrptl
Checks the status of an Oracle listener.
haoraenvcheck
haoralsnrptlsub

#

The subcommands are invoked from the command shown in the first column. Therefore, there is no need to set them in the server start, termination, and monitoring commands.

Organization of this section

Headings used for describing the commands

Syntax conventions

haorainsbgn (Starts an Oracle instance)

haorainsend (Terminates an Oracle instance)

haorainsptl (Checks the status of an Oracle instance)

haorainsendsub (Terminates an Oracle instance)

haorainsptlsub (Checks the status of an Oracle instance)

haoralsnrbgn (Starts an Oracle listener)

haoralsnrend (Terminates an Oracle listener)

haoralsnrptl (Checks the status of an Oracle listener)

haoralsnrendsub (Terminates an Oracle listener)

haoralsnrptlsub (Checks the status of an Oracle listener)

haoraenvcheck (Checks a control definition file)

Headings used for describing the commands

The headings used for describing the commands are explained below. In the command sections, only those headings that are necessary out of the following list are given:

Format

Provides the command entry format.

Executable user

Indicates the user account type that has the permissions to execute the command.

Description

Provides a description of the command.

Option

Provides details about the option specifications and any notes about specifying the option.

Return value

Lists and describes the return value returned from the command.

Note

Explains any issues the user must be aware of when executing the command.

Syntax conventions

This section describes the conventions used to describe the syntax of commands. The meanings of the symbols used in the syntax descriptions are shown in the following table.

Table 4-2 Conventions used to describe command syntax

Symbol
Meaning

{ }
Curly brackets indicate that only one of the enclosed items is to be selected.
Example:
haorainsendsub instance-identifier { -n | -a }
This indicates that you can specify either -n or -a in the haorainsendsub subcommand.

|
A vertical bar separates multiple items enclosed in curly brackets ({}).
Example:
haoralsnrendsub listener-name { -n | -a }
This indicates that you can specify either -n or -a in the haoralsnrendsub subcommand.

haorainsbgn (Starts an Oracle instance)

Format

haorainsbgn instance-identifier

Executable user

Superuser, Oracle user

Description

Starts the Oracle instance to be monitored by HA Monitor Kit. You can use this command only inside the server start command. For a description of the server start command, see 4.2 Server start command setup.

This command can start only those Oracle instances that you have set up to be monitored by HA Monitor Kit.

Option

instance-identifier
Specify the instance identifier of the Oracle instance you want to start and have already defined in ORACLE_SID of the Oracle instance control definition file.

Return value

Value | Meaning
0 | Normal termination (The Oracle instance was started successfully.)
Other than 0 | Abnormal termination

haorainsend (Terminates an Oracle instance)

Format

haorainsend instance-identifier

Executable user

Superuser, Oracle user

Description

Terminates an Oracle instance being monitored by HA Monitor Kit. You can use this command only inside the server termination command. For a description of the server termination command, see 4.3 Server termination command setup.

This command can terminate only those Oracle instances being monitored by HA Monitor Kit.

If an Oracle instance cannot be terminated normally, the Oracle instance is forcibly terminated after the wait time specified in ABORT_LIMIT of the Oracle instance control definition file has elapsed.

Option

instance-identifier
Specify the instance identifier of the Oracle instance you want to terminate and have already defined in ORACLE_SID of the Oracle instance control definition file.

Return value

Value | Meaning
0 | Normal termination (The Oracle instance was terminated successfully.)
Other than 0 | Abnormal termination

Note

If an Oracle instance cannot be terminated normally, the forced termination process is suspended until after the wait time specified in ABORT_LIMIT of the Oracle instance control definition file has elapsed. Executing and terminating this command could take at the most twice the wait time that was set.

haorainsptl (Checks the status of an Oracle instance)

Format

haorainsptl instance-identifier

Executable user

Superuser, Oracle user

Description

Checks whether an Oracle instance process exists. If DB_NOACCESS of the Oracle instance control definition file is omitted or no is specified for it, the command also checks the status of the Oracle instance for slowdowns.

You can use this command only inside the server monitoring command. For a description of the server monitoring command, see 4.4 Server monitoring command setup.

This command can check the status of only those Oracle instances that have been set up to be monitored by HA Monitor Kit.

Option

instance-identifier
Specify the instance identifier of the Oracle instance whose status you want to check and that has been defined in ORACLE_SID of the Oracle instance control definition file.

Return value

Value | Meaning
0 | Normal termination (Checking detected no Oracle instance error.)
1 | Normal termination (Checking detected an Oracle instance error.)
Other than the above | Abnormal termination (An error occurred during check processing or processing terminated abnormally.)

haorainsendsub (Terminates an Oracle instance)

Format

haorainsendsub instance-identifier { -n | -a }

Executable user

Superuser, Oracle user

Description

Terminates an Oracle instance being monitored by HA Monitor Kit. This is a subcommand called from haorainsend (terminates an Oracle instance).

Option

instance-identifier
Specify the instance identifier of the Oracle instance you want to terminate and that has been defined in ORACLE_SID of the Oracle instance control definition file.

-n
Terminate the Oracle instance normally.

-a
Forcibly terminate the Oracle instance.

Return value

Value | Meaning
0 | Normal termination (The Oracle instance was terminated successfully.)
Other than 0 | Abnormal termination

haorainsptlsub (Checks the status of an Oracle instance)

Format

haorainsptlsub instance-identifier { -slowdown | -status }

Executable user

Superuser, Oracle user

Description

Executes SQL statements to check an Oracle instance for slowdown. This is a subcommand called from haorainsptl (checks the status of an Oracle instance).

Option

instance-identifier
Specify the instance identifier of the Oracle instance whose status you want to check and that has been defined in ORACLE_SID of the Oracle instance control definition file.

-slowdown
Execute SQL for slowdown monitoring.

-status
Execute SQL for checking the status.

Return value

Value | Meaning
0 | Normal termination (The status of the Oracle instance was checked successfully.)
Other than 0 | Abnormal termination

haoralsnrbgn (Starts an Oracle listener)

Format

haoralsnrbgn listener-name

Executable user

Superuser, Oracle user

Description

Starts the Oracle listener to be monitored by HA Monitor Kit. You can use this command only inside the server start command. For a description of the server start command, see 4.2 Server start command setup.

This command can start only those Oracle listeners that you have set up to be monitored by HA Monitor Kit.

Option

listener-name
Specify the listener name of the Oracle listener you want to start and have already defined in LISTENER_NAME of the Oracle listener control definition file.

Return value

Value | Meaning
0 | Normal termination (The Oracle listener was started successfully.)
Other than 0 | Abnormal termination

haoralsnrend (Terminates an Oracle listener)

Format

haoralsnrend listener-name

Executable user

Superuser, Oracle user

Description

Terminates an Oracle listener being monitored by HA Monitor Kit. You can use this command only inside the server termination command. For a description of the server termination command, see 4.3 Server termination command setup.

This command can terminate only those Oracle listeners being monitored by HA Monitor Kit.

Option

listener-name
Specify the listener name of the Oracle listener you want to terminate and have already defined in LISTENER_NAME of the Oracle listener control definition file.

Return value

Value | Meaning
0 | Normal termination (The Oracle listener was terminated successfully.)
Other than 0 | Abnormal termination

haoralsnrptl (Checks the status of an Oracle listener)

Format

haoralsnrptl listener-name

Executable user

Superuser, Oracle user

Description

Checks whether an Oracle listener process exists. If LISTENER_NOACCESS of the Oracle listener control definition file is omitted or no is specified for it, the command also checks the Oracle listener for slowdown.

You can use this command only inside the server monitoring command. For a description of the server monitoring command, see 4.4 Server monitoring command setup.

This command can check the status of only those Oracle listeners that have been set up to be monitored by HA Monitor Kit.

Option

listener-name
Specify the listener name of the Oracle listener whose status you want to check and which has been defined in LISTENER_NAME of the Oracle listener control definition file.

Return value

Value | Meaning
0 | Normal termination (Checking detected no Oracle listener error.)
1 | Normal termination (Checking detected an Oracle listener error.)
Other than the above | Abnormal termination (An error occurred during check processing or check processing terminated abnormally.)

haoralsnrendsub (Terminates an Oracle listener)

Format

haoralsnrendsub listener-name { -n | -a }

Executable user

Superuser, Oracle user

Description

Terminates an Oracle listener being monitored by HA Monitor Kit. This is a subcommand called from haoralsnrend (terminates an Oracle listener).

Option

listener-name
Specify the listener name of the Oracle listener you want to terminate and have already defined in LISTENER_NAME of the Oracle listener control definition file.

-n
Terminate the Oracle listener normally.

-a
Forcibly terminate the Oracle listener.

Return value

Value | Meaning
0 | Normal termination (The Oracle listener was terminated successfully.)
Other than 0 | Abnormal termination

haoralsnrptlsub (Checks the status of an Oracle listener)

Format

haoralsnrptlsub listener-name

Executable user

Superuser, Oracle user

Description

Executes SQL statements to check the Oracle listener for slowdown. This is a subcommand called from haoralsnrptl (checks the status of an Oracle listener).

Option

listener-name
Specify the listener name of the Oracle listener whose status you want to check and that has been defined in LISTENER_NAME of the Oracle listener control definition file.

Return value

Value | Meaning
0 | Normal termination (The status of the Oracle listener was checked successfully.)
Other than 0 | Abnormal termination

haoraenvcheck (Checks a control definition file)

Format

haoraenvcheck control-definition-file-name { -instance | -listener }

Executable user

Superuser, Oracle user

Description

This is a subcommand that checks a control definition file to determine whether its definitions match the rules. It can be used only in the following commands:

haorainsbgn

haoralsnrbgn

haorainsend

haoralsnrend

haorainsptl

haoralsnrptl

Option

control-definition-file-name
Specifies the name of a control definition file.

-instance
Specify this to check the definitions in a control definition file for an Oracle instance.

-listener
Specify this to check the definitions in a control definition file for an Oracle listener.

Return value

Value | Meaning
0 | Normal termination (The control definition file was checked successfully.)
Other than 0 | Abnormal termination

5. System Operation

This chapter explains how to operate the system that you have built. Specifically, it explains how to start and terminate Oracle instances and Oracle listeners, how to check the server logs, what action the operator must take in case an error occurs, and how to modify the system.

For descriptions about system operations not described in this manual, see System Operation in the manual HA Monitor (for Linux(R)).

Organization of this chapter

5.1 Flow of operations

5.2 Starting, terminating, and monitoring Oracle instances and Oracle listeners

5.3 Starting or terminating an Oracle instance or Oracle listener without involving HA Monitor

5.4 Checking the server logs

5.5 Hot standby operation when a failure occurs

5.6 Error handling

5.7 System modification

5.1 Flow of operations

Oracle instances and Oracle listeners in a hot-standby configuration that uses HA Monitor Kit operate as HA Monitor's monitor-mode servers. The flow of operations in a hot-standby configuration that uses HA Monitor Kit is the same as that in a general hot-standby configuration that uses HA Monitor. For details about the flow of operations in the system you have built, see the description under Operation flow in the manual HA Monitor (for Linux(R))

Note

If, for an Oracle instance, DB_NOACCESS of the Oracle instance control definition file is omitted or no is specified for it, do not stop database input/output by executing the ALTER SYSTEM SUSPEND SQL statement while the system is running.

If you execute the ALTER SYSTEM SUSPEND statement while the system is running, HA Monitor Kit may report this as an Oracle instance error.

5.2 Starting, terminating, and monitoring Oracle instances and Oracle listeners

This section explains how to start, terminate, and monitor Oracle instances or Oracle listeners on a system that uses HA Monitor Kit. Oracle instances and Oracle listeners are managed as monitor-mode servers. Therefore, you use HA Monitor commands to start, terminate, or monitor Oracle instances or Oracle listeners.

To start or terminate an Oracle instance or Oracle listener that is running as a HA Monitor server, you must use an HA Monitor command. If you use a command other than a HA Monitor command for starting or terminating an Oracle instance or Oracle listener in a hot-standby configuration that uses HA Monitor Kit, HA Monitor may report this as an error.

Organization of this section

(1) Starting an Oracle instance or Oracle listener

(2) Terminating an Oracle instance or Oracle listener

(3) Monitoring an Oracle instance or Oracle listener

(1) Starting an Oracle instance or Oracle listener

To start an Oracle instance or Oracle listener, execute the monitor-mode server start command (monbegin command). This command is an HA Monitor operation command.

For details about this command, see the description under monbegin (starts a server in the monitor mode) in the manual HA Monitor (for Linux(R)).

(2) Terminating an Oracle instance or Oracle listener

To terminate an Oracle instance or Oracle listener, execute the monitor-mode server termination command (monend command). This command is an HA Monitor operation command. If HA Monitor Kit fails to terminate an Oracle instance or Oracle listener during a server termination process, or if an Oracle instance or Oracle listener does not terminate within the user-specified time, HA Monitor Kit forcibly terminates the Oracle instance or Oracle listener.

For details about this command, see the description under monend (terminates a server in the monitor mode) in the manual HA Monitor (for Linux(R)).

(3) Monitoring an Oracle instance or Oracle listener

Once an Oracle instance or Oracle listener has started, HA Monitor monitors servers by executing the server monitoring command.

The server monitoring command monitors servers using the following methods:

When the server is an Oracle instance:

Monitors for the operation of an Oracle instance process.

Uses SQL to check the status of an Oracle instance.

Monitors for slowdown of an Oracle instance.

When the server is an Oracle listener:

Monitors for the operation of an Oracle listener process.

Uses the listener control utility to monitor for slowdown based on the response time.

For an overview and the operational details of the server monitoring command, see the description under Monitoring servers in the monitor mode and Controlling the server monitoring command in the manual HA Monitor (for Linux(R)).

5.3 Starting or terminating an Oracle instance or Oracle listener without involving HA Monitor

HA Monitor Kit starts an Oracle instance after a database is mounted and opened. Therefore, if you plan to perform any of the following operations on an Oracle instance, you must start and stop the Oracle instance without involving HA Monitor:

To start an Oracle instance without mounting a database

To start an Oracle instance without opening a database

To start an Oracle instance while limiting access to it

To forcibly start an Oracle instance

To start an Oracle instance and perform media recovery

To execute a switchover in a disaster recovery configuration that uses Oracle Data Guard

To use the primary database as a standby database after a switchover in a disaster recovery configuration that uses Oracle Data Guard

The following subsections explain how to start and terminate an Oracle instance or Oracle listener without involving HA Monitor:

Organization of this section

5.3.1 Not involving HA Monitor when starting an Oracle instance or Oracle listener

5.3.2 Not involving HA Monitor when terminating an Oracle instance or Oracle listener

5.3.1 Not involving HA Monitor when starting an Oracle instance or Oracle listener

To start an Oracle instance or Oracle listener without involving HA Monitor, you must performs all controls that normally are executed by HA Monitor and HA Monitor Kit when starting a server.

When a server is started without involving HA Monitor, monitoring by HA Monitor (hot standby) cannot be performed. Furthermore, while an Oracle instance or Oracle listener started without involving HA Monitor is active, do not start the server that is associated with the active Oracle component.

To start an Oracle instance or Oracle listener without involving HA Monitor:

If the server associated with the Oracle instance or Oracle listener is running, stop the server using the HA Monitor server termination command (monend command).

Manually connect the resources (shared disk and IP address for client connection) that the Oracle instance or Oracle listener will use.

Start the Oracle instance or Oracle listener.
For the startup method and the options necessary for startup, see the Oracle documentation.

5.3.2 Not involving HA Monitor when terminating an Oracle instance or Oracle listener

To terminate an Oracle instance or Oracle listener that was started without involving HA Monitor, you must performs all controls that normally are executed by HA Monitor and HA Monitor Kit when stopping a server.

To terminate an Oracle instance or Oracle listener without involving HA Monitor:

Terminate the Oracle instance or Oracle listener.
For the termination method and the options necessary for termination, see the Oracle documentation.

Manually disconnect the resources (shared disk and IP address for client connection) that the Oracle instance or Oracle listener use.
If you do not disconnect these resources, the hosts in both the primary and secondary system will connect to the shared disk when you resume operation with HA Monitor involvement. This may corrupt the database. Therefore, be sure to disconnect these resources.

5.4 Checking the server logs

HA Monitor Kit outputs the messages issued by the server start, termination, and monitoring commands to server logs. The standard output and standard errors of any Oracle commands executed inside the server start, termination, and monitoring commands are also output to server logs. By checking the server logs, you can review the processing results of the server start, termination, and monitoring commands. While the system is running, the server log content is constantly being updated based on the result of the monitoring process.

Note

When checking the server logs while the server is running, only view them and do not save them. Otherwise, the log information that is output while you are viewing the server log may be lost.

This section explains the details of the server logs.

Organization of this section

(1) File types

(2) File size

(3) Output example

(4) Message language environment

(1) File types

The table below lists the server logs. The value of the LOGNAME environment variable of the server start, termination, or monitoring command provided by HA Monitor Kit is substituted for server-alias-name in the file name.

Table 5-1 List of server logs

File name
Description

/opt/hitachi/HAmon/spool/server-alias-name.oralog
This is the server log file. The content of the messages issued by the server start, termination, and monitoring commands is output to this file.

/opt/hitachi/HAmon/spool/server-alias-name.oralog_old
This is the previous generation of the server log file. It is created by adding the _old suffix to the server log file when it exceeds the specified size at the following times:

When the server start, termination, or monitoring command is executed

Immediately before a monitoring process is executed in the server monitoring command

/opt/hitachi/HAmon/spool/server-alias-name.oralog_err
This is the server log file created during the occurrence of the most recent error. It is created by adding the _err suffix to the server log file when the server start, termination, or monitoring command terminates with a non-zero return value. View this file when investigating an error.

(2) File size

You can change the server log file size in the LOGsize environment variable of the server start, termination, or monitoring command. The default is 1,048,576 bytes (1 megabyte). Under normal conditions, use this default file size. Increase the size only if the volume of output is large and may cause the log to run out of room.

When changing the file size, choose the largest value among the results (in bytes) obtained from the following Formulas 1 through 3:

Formula 1

240 + (number of Oracle instances and Oracle listeners inside server) 160

+ (combined total size of messages output by products other than HA Monitor Kit such as Oracle during startup process)

Formula 2

60 + (number of Oracle instances and Oracle listeners inside server) 170

+ (combined total size of messages output by products other than HA Monitor Kit such as Oracle during single monitoring process)

Formula 3

240 + (number of Oracle instances and Oracle listeners inside server) 160

+ (combined total size of messages output by products other than HA Monitor Kit such as Oracle during termination process)

For details about the LOGSIZE environment variable of the server start, termination, or monitoring command, see 4. Setting Up the Server Start, Termination, and Monitoring Commands.

To estimate a size that is large enough to hold logs for a certain duration or to calculate the total size of the logs that the file can hold, determine the combined total size of messages that are output during the entire period from server start to termination.

Note that formula 1, formula 2, and formula 3 in the following formula mean the results of the above formulas for estimating the server log file size:

formula 1 + formula 2 (server operation time (in seconds) monitoring interval of server monitoring command (in seconds)) + formula 3

(3) Output example

The following figure shows an example of a server log that is output when the startup of an Oracle instance fails:

The numbers in the figure are explained below.

This is a message that is output by a HA Monitor Kit command. Message output date/time and ID are output. In this example, messages KAMK00401-I and other information are output. For details about the messages that are output by HA Monitor Kit, see 6.2 Message details.

These are messages that are output by Oracle after it is initiated by HA Monitor Kit and messages that are output by processes that the user added to the server start, termination, and monitoring commands. For details about the messages that are output in these lines, see the Oracle documentation and other documents.

(4) Message language environment

Oracle messages are output to the server logs in the language corresponding to HA Monitor's language environment. The following table shows the correspondence between message language environments:

Table 5-2 Correspondence between message language environments

HA Monitor's language environment (LANG) | Oracle's language environment (NLS_LANG) | Message language
ja_JP.UTF-8 | JAPANESE_JAPAN.AL32UTF8 | Japanese (UTF-8)
ja_JP.eucJP | JAPANESE_JAPAN.JA16EUC | Japanese (EUC)
ja_JP.SJIS | JAPANESE_JAPAN.JA16SJIS | Japanese (S-JIS)
Other | Not set. | English

5.5 Hot standby operation when a failure occurs

This section explains the error information collected when a failure occurs in a system that uses HA Monitor Kit. For the actions that the operator must take after a failure has been detected, see Hot standby operation when a failure occurs in the manual HA Monitor (for Linux(R)).

Organization of this section

5.5.1 Collecting error information

5.5.1 Collecting error information

When a failure occurs, you need to collect HA Monitor Kit's error information. HA Monitor Kit's error information is collected along with HA Monitor's error information by the HA Monitor troubleshooting information collection command (monts command). For details about collecting HA Monitor's error information, see the description under Collecting error information in the manual HA Monitor (for Linux(R)).

HA Monitor Kit's error information include the following:

Server log
For details, see 5.4 Checking the server logs.

Control definition file
For details, see 3.2.7(2) Setting up control definition files.

Server start, termination, and monitoring commands
Collected when the server start, termination, or monitoring command is located in /opt/hitachi/HAmon/etc. For details, see 3.2.7(1) Setting up the server start, termination, and monitoring commands.

5.6 Error handling

Based on the message that is output when an error occurs, the operator must take the appropriate action. Additionally, as needed, the operator must collect HA Monitor Kit's error information. For details about HA Monitor Kit's error information, see 5.5.1 Collecting error information.

When any of the errors listed below occurs, HA Monitor Kit checks the content of the server log and handles the error. The contents of the messages issued by the server start, termination, and monitoring commands are output to the server log.

Oracle instance startup failure

Oracle instance restart failure

Oracle instance termination failure

Oracle instance error

Oracle listener startup failure

Oracle listener restart failure

Oracle listener termination failure

Oracle listener error

For details about the server log, see 5.4 Checking the server logs; for details about how to handle messages, see 6. Messages or the manual HA Monitor Messages. For examples of the typical errors that occur in an HA Monitor hot-standby system, see the description under Handling errors in the manual HA Monitor (for Linux(R)).

5.7 System modification

This section explains the settings that are necessary and the notes that are applicable when you modify the system.

Organization of this section

5.7.1 Upgrading Oracle

5.7.2 Adding Oracle instances and Oracle listeners

5.7.3 Deleting Oracle instances and Oracle listeners

5.7.4 Modifying the settings related to Oracle instances and Oracle listeners

5.7.1 Upgrading Oracle

When upgrading Oracle, you need to set up the server again. The following is the server setup procedure necessary when upgrading Oracle:

Stop the server by executing the monitor-mode server termination command (monend command).

Manually reference and update the connection for the shared disk used by Oracle.
For details about how to connect a shared disk, see the documentation for your OS.

Upgrade Oracle.
For details about how to upgrade Oracle, see the Oracle documentation.

Correct the definition files for controlling Oracle instances and Oracle listeners.
When Oracle is updated, the Oracle home directory may change. Therefore, as needed, correct ORACLE_HOME in the control definition files to the value that is in effect following the upgrade.

Manually disconnect the shared disk.
For details about how to disconnect the shared disk, see the documentation for your OS.

Start the server by executing the monitor-mode server start command (monbegin command).

5.7.2 Adding Oracle instances and Oracle listeners

This subsection explains how to add an Oracle instance or Oracle listener as a control target of HA Monitor Kit, either to a server that has already been built or as a new server.

Organization of this subsection

(1) Adding to a built server

(2) Adding as a new server

(1) Adding to a built server

To add an Oracle instance or Oracle listener to an existing server:

Stop the server by executing the monitor-mode server termination command (monend command).
If you make a mistake when adding a setting to the server termination command without first stopping the server to which you want to add the Oracle instance or Oracle listener, you may no longer be able to terminate the Oracle instance or Oracle listener. Therefore, be sure to stop the server before adding a setting.

Create the database.
Create the database to be added by referencing the Oracle documentation.

Create the control definition file for the Oracle instance or Oracle listener to be added.
For details about control definition files, see 3.2.7(2) Setting up control definition files.

Add settings to the server start, termination, and monitoring commands.
Add processes for the Oracle instance or Oracle listener that you are adding to the server start, termination, and monitoring commands. For details about the settings in the server start, termination, and monitoring commands, see 4. Setting Up the Server Start, Termination, and Monitoring Commands.

Start the server by executing the monitor-mode server start command (monbegin command).

(2) Adding as a new server

For details about how to add an Oracle instance or Oracle listener as a new server, see 3.2 Building a system. For details about the server start, termination, and monitoring commands in which to set up the Oracle instance and Oracle listeners, see 4. Setting Up the Server Start, Termination, and Monitoring Commands.

5.7.3 Deleting Oracle instances and Oracle listeners

To delete an Oracle instance or Oracle listener from being a control target of HA Monitor Kit:

Stop the server by executing the monitor-mode server termination command (monend command).
If you make a mistake when deleting the setting of the server termination command without first stopping the server from which you want to delete the Oracle instance or Oracle listener, you may no longer be able to terminate the Oracle instance or Oracle listener. Therefore, be sure to stop the server before deleting the setting.

Delete the settings from the server start, termination, and monitoring commands.
Delete the processes for the Oracle instance or Oracle listener that you want to delete from the server start, termination, and monitoring commands or turn them into comments. For details about the settings in the server start, termination, and monitoring commands, see 4. Setting Up the Server Start, Termination, and Monitoring Commands.

Delete the definition files for controlling Oracle instances and Oracle listeners (optional).
Deleting the definition files is optional. Delete them if you are deleting the database. If you are retaining the database, you can retain the definition files for reuse in the future.

Start the server by executing the monitor-mode server start command (monbegin command).

5.7.4 Modifying the settings related to Oracle instances and Oracle listeners

To modify the server start, termination, and monitoring commands and the control definition file that are associated with an Oracle instance or Oracle listener that has already been built:

Stop the server by executing the monitor-mode server termination command (monend command).
If you modify a setting without first stopping the server, a setting error may make it impossible to terminate the Oracle instance or Oracle listener, or the setting content may no longer match the action of the server monitoring command that is running. As a result, you may no longer be able to assess the status of the server monitoring command. Therefore, be sure to stop the server before modifying the setting.

Modify the settings for the server start, termination, and monitoring commands and the control definition file.
For details about the server start, termination, and monitoring commands, see 4. Setting Up the Server Start, Termination, and Monitoring Commands.
For details about control definition files, see 3.2.7(2) Setting up control definition files.

Start the server by executing the monitor-mode server start command (monbegin command).

6. Messages

This chapter explains the messages that HA Monitor Kit outputs.

Organization of this chapter

6.1 Message format

6.2 Message details

6.1 Message format

This section explains the message format.

Organization of this section

6.1.1 Message output format

6.1.2 Message description format

6.1.1 Message output format

The output format for the messages is as follows:

KAMKnnnnn-i xx....xx

KAMKnnnnn-i: Message ID

xx....xx: Message text

For messages that are output to server logs, the output date/time in the form of yyyy/mm/dd HH:MM:SS is added as a prefix to the message ID.

6.1.2 Message description format

Organization of this subsection

(1) Description format

(2) Explanation of the message ID symbol

(1) Description format

In this manual, messages are described as follows:

KAMKnnnnn-i (X)

message-text

Explains the meaning of the message.
aa....aa: Explains variable text that is output in the message.

S:

Indicates the main processing that the system performs after outputting the message.

O:

Indicates the action that the operator must take after checking the message.

Action to take:

Indicates the action that the system administrator must take after checking the message.

A request to contact the customer support center under the heading O (operator action) or Action to take means that the system administrator needs to contact their Hitachi representative stipulated in the purchase agreement.

(2) Explanation of the message ID symbol

The meaning of the message ID symbol follows:

KAMK:

Indicates an HA Monitor Kit message.

nnnnn:

Indicates the message sequence number.

i:

Indicates the message type.

E:

Indicates an error occurred in which a library, command, or server function does not work.

Indicates that the system cannot run because of an invalid definition or invalid command operand specification.

W:

Indicates a warning about command operation status or the like.

I:

Message that does not fall under E or W and simply indicates a condition.

(X):

Indicates a message output destination type.

(P): Server log file

(L): Message log file (syslog)

If a message has multiple output destinations, the output destination types are described by linking them with a plus (+) sign.

6.2 Message details

This section explains the details of HA Monitor Kit messages.

Organization of this section

KAMK00021-E (P + L)

KAMK00120-E (P + L)

KAMK00130-E (P + L)

KAMK00201-I (P)

KAMK00202-I (P)

KAMK00203-I (P)

KAMK00204-I (P)

KAMK00205-I (P)

KAMK00206-I (P)

KAMK00301-I (P)

KAMK00302-I (P)

KAMK00303-E (P + L)

KAMK00304-I (P)

KAMK00305-I (P)

KAMK00306-E (P + L)

KAMK00308-E (P + L)

KAMK00309-E (P + L)

KAMK00310-W (P)

KAMK00311-W (P)

KAMK00312-I (P)

KAMK00321-I (P)

KAMK00322-I (P)

KAMK00323-E (P + L)

KAMK00324-I (P)

KAMK00325-I (P)

KAMK00326-E (P + L)

KAMK00328-E (P + L)

KAMK00329-E (P + L)

KAMK00330-W (P)

KAMK00331-W (P)

KAMK00332-I (P)

KAMK00401-I (P)

KAMK00402-I (P)

KAMK00411-I (P)

KAMK00412-I (P)

KAMK00421-I (P)

KAMK00422-I (P)

KAMK00423-E (P + L)

KAMK00601-E (P + L)

KAMK00021-E (P + L)

The aa....aa command contains syntax errors.

The HA Monitor Kit command syntax is invalid.
aa....aa: Command name

S:

Suspends command processing.

O:

Re-execute the command using the correct syntax.

KAMK00120-E (P + L)

The file aa....aa contains errors. Parameter : bb....bb

The parameter specification method indicated by bb....bb in the control definition file indicated by aa....aa is invalid.
aa....aa: File name
bb....bb: Parameter name

S:

Terminates processing.

Action to take:

See 3.2.7(2) Setting up control definition files to correct the invalid parameter indicated by bb....bb.

KAMK00130-E (P + L)

The file aa....aa does not exist.

The control definition file indicated by aa....aa does not exist.
aa....aa: File name

S:

Terminates processing.

Action to take:

Make sure that the instance identifier of the Oracle instance or the listener name of the Oracle listener specified in the command argument is correct. If it is correct, check the file name. If the file name does not exist, see 3.2.7(2) Setting up control definition files to create it.

KAMK00201-I (P)

Usage : haorainsbgn ORACLE_SID

This message is output following message KAMK00021-E if the execute form of the Oracle instance start command (haorainsbgn command) is invalid.

S:

Continues processing.

KAMK00202-I (P)

Usage : haorainsend ORACLE_SID

This message is output following message KAMK00021-E if the execute form of the Oracle instance termination command (haorainsend command) is invalid.

S:

Continues processing.

KAMK00203-I (P)

Usage : haorainsptl ORACLE_SID

This message is output following message KAMK00021-E if the execute form of the Oracle instance status monitoring command (haorainsptl command) is invalid.

S:

Continues processing.

KAMK00204-I (P)

Usage : haoralsnrbgn LISTENER_NAME

This message is output following message KAMK00021-E if the execute form of the Oracle listener start command (haoralsnrbgn command) is invalid.

S:

Continues processing.

KAMK00205-I (P)

Usage : haoralsnrend LISTENER_NAME

This message is output following message KAMK00021-E if the execute form of the Oracle listener termination command (haoralsnrend command) is invalid.

S:

Continues processing.

KAMK00206-I (P)

Usage : haoralsnrptl LISTENER_NAME

This message is output following message KAMK00021-E if the execute form of the Oracle listener status monitoring command (haoralsnrptl command) is invalid.

S:

Continues processing.

KAMK00301-I (P)

Oracle instance aa....aa will now start.

Starts the Oracle instance indicated by aa....aa.
aa....aa: Instance identifier

S:

Continues processing.

KAMK00302-I (P)

Oracle instance aa....aa has started.

Startup of the Oracle instance indicated by aa....aa was completed.
aa....aa: Instance identifier

S:

Continues processing.

KAMK00303-E (P + L)

An attempt to start Oracle instance aa....aa has failed.

Startup of the Oracle instance indicated by aa....aa has failed.
aa....aa: Instance identifier

S:

Terminates processing.

O:

Investigate the cause of the error based on the information that was output to the server log.

KAMK00304-I (P)

Oracle instance aa....aa will now stop.

The Oracle instance indicated by aa....aa will be terminated.
aa....aa: Instance identifier

S:

Continues processing.

KAMK00305-I (P)

Oracle instance aa....aa has stopped.

The Oracle instance indicated by aa....aa has been terminated.
aa....aa: Instance identifier

S:

Continues processing.

KAMK00306-E (P + L)

An attempt to stop Oracle instance aa....aa has failed.

Termination of the Oracle instance indicated by aa....aa has failed.
aa....aa: Instance identifier

S:

Terminates processing.

O:

Investigate the cause of the error based on the information that was output to the server log.

KAMK00308-E (P + L)

An error was detected in Oracle instance aa....aa.

Oracle instance status monitoring detected an error in the Oracle instance indicated by aa....aa.
aa....aa: Instance identifier

S:

Continues processing.

Action to take:

Check the Oracle log and eliminate the cause of the error that was detected.

KAMK00309-E (P + L)

A slowdown was detected in Oracle instance aa....aa.

Oracle instance status monitoring detected a slowdown in the Oracle instance indicated by aa....aa.
aa....aa: Instance identifier

S:

Continues processing.

Action to take:

Check the Oracle log and eliminate the cause of the slowdown that was detected.

KAMK00310-W (P)

Oracle instance aa....aa could not be terminated.

The Oracle instance indicated by aa....aa could not be terminated normally.
aa....aa: Instance identifier

S:

Continues processing.

KAMK00311-W (P)

The processing to stop Oracle instance aa....aa timed out.

The Oracle instance indicated by aa....aa could not be terminated normally because the process that terminates the Oracle instance normally timed out.
aa....aa: Instance identifier

S:

Continues processing.

KAMK00312-I (P)

Oracle instance aa....aa will be forcibly stopped.

The Oracle instance indicated by aa....aa will be forcibly terminated.
aa....aa: Instance identifier

S:

Continues processing.

KAMK00321-I (P)

Oracle listener aa....aa will now start.

Starts the Oracle listener indicated by aa....aa.
aa....aa: Listener name

S:

Continues processing.

KAMK00322-I (P)

Oracle listener aa....aa has started.

Startup of the Oracle listener indicated by aa....aa was completed.
aa....aa: Listener name

S:

Continues processing.

KAMK00323-E (P + L)

An attempt to start Oracle listener aa....aa has failed.

Startup of the Oracle listener indicated by aa....aa has failed.
aa....aa: Listener name

S:

Terminates processing.

O:

Investigate the cause of the error based on the information that was output to the server log.

KAMK00324-I (P)

Oracle listener aa....aa will now stop.

The Oracle listener indicated by aa....aa will be terminated.
aa....aa: Listener name

S:

Continues processing.

KAMK00325-I (P)

Oracle listener aa....aa has stopped.

The Oracle listener indicated by aa....aa has been terminated.
aa....aa: Listener name

S:

Continues processing.

KAMK00326-E (P + L)

An attempt to stop Oracle listener aa....aa has failed.

Termination of the Oracle listener indicated by aa....aa has failed.
aa....aa: Listener name

S:

Continues processing.

O:

Investigate the cause of the error based on the information that was output to the server log.

KAMK00328-E (P + L)

An error was detected in Oracle listener aa....aa.

Oracle listener status monitoring detected an error in the Oracle listener indicated by aa....aa.
aa....aa: Listener name

S:

Continues processing.

Action to take:

Check the Oracle log and eliminate the cause of the error that was detected.

KAMK00329-E (P + L)

A slowdown was detected in Oracle listener aa....aa.

Oracle listener status monitoring detected a slowdown in the Oracle listener indicated by aa....aa.
aa....aa: Listener name

S:

Continues processing.

Action to take:

Check the Oracle log and eliminate the cause of the slowdown that was detected.

KAMK00330-W (P)

Oracle listener aa....aa could not be terminated.

The Oracle listener indicated by aa....aa could not be terminated normally.
aa....aa: Listener name

S:

Continues processing.

KAMK00331-W (P)

The processing to stop Oracle listener aa....aa timed out.

The Oracle listener indicated by aa....aa could not be terminated normally because the process that terminates the Oracle listener normally timed out.
aa....aa: Listener name

S:

Continues processing.

KAMK00332-I (P)

Oracle listener aa....aa will be forcibly stopped.

The Oracle listener indicated by aa....aa will be forcibly terminated.
aa....aa: Listener name

S:

Continues processing.

KAMK00401-I (P)

********** The actcommand start. **********

Execution of the server start command has begun.

S:

Continues processing.

KAMK00402-I (P)

********** The actcommand end. (aa....aa) **********

Execution of the server start command has ended.
aa....aa: Return value of the server start command

S:

Continues processing.

Action to take:

If the return value is not 0, check the settings in the server start command and the server log.

KAMK00411-I (P)

********** The termcommand start. **********

Execution of the server termination command has begun.

S:

Continues processing.

KAMK00412-I (P)

********** The termcommand end. (aa....aa) **********

Execution of the server termination command has ended.
aa....aa: Return value of the server termination command

S:

Continues processing.

Action to take:

If the return value is not 0, check the settings in the server termination command and the server log.

KAMK00421-I (P)

********** The patrolcommand start. **********

Execution of the server monitoring command has begun.

S:

Continues processing.

KAMK00422-I (P)

********** The patrolcommand end. (aa....aa) **********

Execution of the server monitoring command has ended.
aa....aa: Return value of the server monitoring command

S:

Continues processing.

Action to take:

If the return value is not 0, check the settings in the server monitoring command and the server log.

KAMK00423-E (P + L)

An error was detected during monitoring process. The patrolcommand will now stop.

The monitoring command will be terminated because an error was detected in the monitoring process in the server monitoring command.

S:

Terminates processing.

Action to take:

Investigate the cause of the error based on the information that was output to the server log.

KAMK00601-E (P + L)

An error occurred in the aa....aa system call. Error details : bb....bb Functional name : cc....cc

An error occurred in a system call issued by an HA Monitor Kit function. For details about the codes, see the documentation for your OS.
aa....aa: System call issued
bb....bb: Detail code (1 to 3 digits)
cc....cc: Name of the function in which the error occurred

S:

Continues processing.

O:

Contact the system administrator.

Action to take:

Investigate the cause of the error based on the detail code.

Appendix

Organization of the appendixes

A. Reference Material for This Manual

Appendix A. Reference Material for This Manual

This appendix provides reference information, including various conventions, for this manual.

Organization of this section

A.1 Related publications

A.2 Conventions: Abbreviations for product names

A.3 Conventions: Acronyms

A.4 Conventions: KB, MB, GB, and TB

Appendix A.1 Related publications

This manual is part of a related set of manuals. The manuals in the set are listed below (with the manual numbers):

HA Monitor Cluster Software Guide (3000-9-132(E)), for Linux(R) systems, and HA Monitor Cluster Software Guide (3000-9-140(E)), for Linux(R) (x86) systems
Herein, these manual titles are abbreviated to HA Monitor (for Linux(R)).

HA Monitor Cluster Software Messages (3000-9-134(E))
Herein, this manual title is abbreviated to HA Monitor Messages.

Appendix A.2 Conventions: Abbreviations for product names

This manual uses the following abbreviations for product names:

Abbreviation | Full name or meaning
Linux | Linux(R)
Linux | Linux (x86) | Red Hat Enterprise Linux AS 4 (AMD64 & Intel EM64T)
Red Hat Enterprise Linux AS 4 (x86)
Red Hat Enterprise Linux ES 4 (AMD64 & Intel EM64T)
Red Hat Enterprise Linux ES 4 (x86)
Red Hat Enterprise Linux 5 (x86)
Red Hat Enterprise Linux 5 Advanced Platform (x86)
Red Hat Enterprise Linux 5 (AMD/Intel 64)
Red Hat Enterprise Linux 5 Advanced Platform (AMD/Intel 64)
Red Hat Enterprise Linux server 6 (x86)
Red Hat Enterprise Linux server 6 (x86_64)
Oracle | Oracle Database 10g
Oracle Database 11g
Oracle Database 12c#
UNIX | UNIX(R)

#: Supports only non-CDB databases.

Appendix A.3 Conventions: Acronyms

This manual also uses the following acronyms:

Acronym | Full name or meaning
ASM | Automatic Storage Management
DB | Database
EUC | Extended UNIX Code
FS | File System
HA | High Availability
I/O | Input/Output
LAN | Local Area Network
LVM | Logical Volume Manager
Non-CDB | Non-Multitenant Container Database
OS | Operating System
RAC | Real Application Clusters
UTF-8 | 8-bit UCS Transformation Format

Appendix A.4 Conventions: KB, MB, GB, and TB

This manual uses the following conventions:

1 KB (kilobyte) is 1,024 bytes.

1 MB (megabyte) is 1,024² bytes

1 GB (gigabyte) is 1,024³ bytes.

1 TB (terabyte) is 1,024⁴ bytes.

Index

[A][C][D][E][G][H][I][L][M][N][O][P][R][S][T][W]

(A)

ABORT_LIMIT

acttype

(C)

control definition file

checking (subcommand)

setting up

source files and copy destination directories for

(D)

DB_NOACCESS

definition file

directories for storing sample files for server start, termination, and monitoring commands

directory configuration

(E)

environment definition example

error

collecting information about

error detection framework

host failure

network error

Oracle instance error

Oracle instance slowdown

Oracle listener error

Oracle listener slowdown

that HA Monitor detects

that HA Monitor Kit detects

errors, types of

(G)

group

(H)

HA Monitor Kit

configuration

purpose of

required hardware

required software

setting up

haoraenvcheck

haorainsbgn

haorainsend

haorainsendsub

haorainsptl

haorainsptlsub

haoralsnrbgn

haoralsnrend

haoralsnrendsub

haoralsnrptl

haoralsnrptlsub

hot-standby configuration

hot-standby test

(I)

instance identifier

(L)

LAN

listener name

LISTENER_NAME

LISTENER_NOACCESS

local disk

LOGNAME

server monitoring command

server start command

server termination command

LOGSIZE

server monitoring command

server start command

server termination command

(M)

message language environment, correspondence between

monbegin

monend

monts

(N)

name

(O)

Oracle

building database

component configuration

component configuration and hot standby switchable unit consideration

installing

setting up shared resource to be used by

upgrading

Oracle instance

adding

checking status of (command)

checking status of (subcommand)

control definition file

deleting

monitoring

not involving HA Monitor when starting

not involving HA Monitor when terminating

starting

starting (command)

terminating

terminating (command)

terminating (subcommand)

Oracle listener

adding

building

checking status of (command)

checking status of (subcommand)

control definition file

deleting

monitoring

not involving HA Monitor when starting

not involving HA Monitor when terminating

starting

starting (command)

terminating

terminating (command)

terminating (subcommand)

ORACLE_BASE

ORACLE_HOME

ORACLE_SID

ORACLE_USER

server monitoring command runtime shell

(P)

PATROL

PATROL_INTERVAL (server monitoring command)

PATROL_SHELL (server monitoring command runtime shell)

patrolcommand

processing flow

for hot standby when server failure occurs in Oracle instance

for hot standby when server failure occurs in Oracle listener

for monitoring Oracle instance

for monitoring Oracle listener

for starting Oracle instance

for starting Oracle listener

for stopping Oracle instance

for stopping Oracle listener

when startup of Oracle instance fails

when startup of Oracle instance times out

when startup of Oracle listener fails

when startup of Oracle listener times out

(R)

resource

configuration consideration

server correspondence

resource server, consideration

(S)

server

checking operation of

server log, checking

server monitoring command

customization

setup

server start command

customization

setup

server termination command

customization

setup

servexec_retry

setting item, necessary for building system and setting overview

shared disk

shared resource to be used by Oracle, setting up

start_timeout

stop_timeout

system

checking operation of

general procedure for building

(T)

termcommand

(W)

waitserv_exec

Trademarks

AMD, AMD Opteron, and combinations thereof, are trademarks of Advanced Micro Devices, Inc.
Linux(R) is the registered trademark of Linus Torvalds in the U.S. and other countries.
Oracle and Java are registered trademarks of Oracle and/or its affiliates.
Red Hat is a trademark or a registered trademark of Red Hat Inc. in the United States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Other company and product names mentioned in this document may be the trademarks of their respective owners.
Throughout this document Hitachi has attempted to distinguish trademarks from descriptive terms by writing the name with the capitalization used by the manufacturer, or by writing the name with initial capital letters. Hitachi cannot attest to the accuracy of this information. Use of a trademark in this document should not be regarded as affecting the validity of the trademark.