2.3.23 Monitoring processor performance based on results output by the sar command in an AIX environment
If you are using simultaneous multithreading (SMT) or micro-partitioning in an AIX environment, the amount of physical CPU allocated is included in the calculation of CPU usage displayed by the sar command, but is not included in the calculation of CPU usage displayed by PFM - Agent for Platform. For this reason, the CPU usage figure displayed by PFM - Agent for Platform might be lower than that output by the sar command.
PFM - Agent for Platform enables settings to be changed so that processor performance data can be collected from the results output by the sar command of AIX.
This section explains how to collect and monitor processor performance data from the results output by the sar command of AIX when PFM - Agent for Platform is used.
- Organization of this subsection
(1) Settings
This section describes how to set the system to collect processor performance data from the results output by AIX's sar command.
-
From the monitoring console browser, log in to PFM - Web Console.
The Main window appears.
-
On the navigation frame of the Main window, click the Services tab.
The Services window appears.
-
From the navigation frame, expand the contents of the Machines folder.
A folder that has the name of the host on which the Performance Management service is installed appears. Expand the folder that has the host name to display the services installed on that host.
The names of the services are displayed, in order of the service IDs. For details about service IDs, see C. List of Identifiers and the sections that explain the service naming conventions in the appendixes of the JP1/Performance Management Planning and Configuration Guide.
The format of the service ID depends on whether the product name display functionality is enabled. For details about the product name display functionality, see the chapter that describes the Performance Management functionality in the JP1/Performance Management Planning and Configuration Guide.
-
Expand the contents of the monitoring agent host folder, and select host-name<UNIX> (Agent Collector service).
A check mark is displayed next to the selected Agent Collector service.
-
Select the Properties method in the method frame.
The Service Properties window appears.
-
Select Agent Configuration.
-
In sar Command Monitoring at the bottom of the information frame, specify whether to collect processor performance data from the results output by the sar command.
Specify one of the following:
-
Yes: Collect from the sar command's output results.
-
No: Do not collect from the sar command's output results (default).
-
-
In sar Command Interval at the bottom of the information frame, specify the interval (in seconds) to be specified as the sar command argument.
Specify the numeric value that corresponds to the INTERVAL argument of the following AIX sar command:
/usr/sbin/sar -P ALL INTERVAL 1
Specify a value between 1 and 50 (the default is 5).
(2) Correspondence between sar command output results and PFM - Agent for Platform records
If you are using simultaneous multithreading (SMT) or micro-partitioning in an AIX environment, PFM - Agent for Platform enables you to collect the results output by the following AIX sar command:
/usr/sbin/sar -P ALL INTERVAL 1
An example of the results output by the AIX sar command is shown below.
With PFM - Agent for Platform, the results output by the sar command are stored in the fields that monitor processor performance data of the System Summary Overview (PI) and CPU - Per Processor Detail (PI_CPUP) records.
The following table describes the correspondence between the fields of the System Summary Overview (PI) record and the results output by the sar command.
Record |
Field |
Field description |
Results output by the sar command |
---|---|---|---|
System Summary Overview (PI) |
Active CPUs (NUMBER_OF_ACTIVE_CPUS) |
Number of processors |
Number of rows in which a number is displayed in the cpu column |
CPU % (KERNELMODE_USERMODE_PERCENT) |
CPU usage (%). This also indicates the average percentage for each processor. |
%sys column + %usr column |
|
Idle % (IDLE_TIME_PERCENT) |
Percentage of time spent in idle state (%). This is also the average of the percentages for all processors. |
%idle column |
|
Kernel CPU % (KERNELMODE_PERCENT) |
Percentage of time running in kernel mode (%). This is also the average of the percentages for all processors. |
%sys column |
|
User CPU % (USERMODE_PERCENT) |
Percentage of time running in user mode (%). This is also the average of the percentages for all processors. |
%usr column |
|
Wait % (WAIT_TIME_PERCENT) |
Percentage of time spent in I/O standby (%). This is also the average of the percentages for each processor. |
%wio column |
The following table describes the correspondence between the fields of the CPU - Per Processor Detail (PI_CPUP) record and the results output by the sar command.
Record |
Field |
Field description |
Results output by the sar command |
---|---|---|---|
CPU - Per Processor Detail (PI_CPUP) |
CPU % (KERNELMODE_USERMODE_PERCENT) |
CPU usage per processor (%) |
%sys column + %usr column |
Idle % (PROCESSOR_IDLE_PERCENT) |
Percentage of time spent in idle state (%) |
%idle |
|
Processor ID (LOGICAL_PROCESSOR_ID) |
Processor identifier |
cpu column |
|
System % (PROCESSOR_SYSTEM_PERCENT) |
Percentage of time running in kernel mode (%) |
%sys column |
|
User % (PROCESSOR_USER_PERCENT) |
Percentage of time running in user mode (%) |
%usr column |
|
Wait % (PROCESSOR_WAIT_PERCENT) |
Percentage of time spent in I/O standby (%) |
%wio column |
When the setting is made to collect processor performance data from the results output by AIX's sar command, 0 is set as the value for the fields of the System Summary Overview (PI) record shown in the table below.
Record |
Field |
Field description |
Alarm used |
Report used |
---|---|---|---|---|
System Summary Overview (PI) |
Total Idle Time (TOTAL_IDLE_TIME) |
Total time spent in idle state by all processors (sec) |
None |
None |
Total Kernel-Mode Time (TOTAL_KERNELMODE_TIME) |
Total time spent running in kernel mode by all processors (sec) |
None |
None |
|
Total User-Mode Time (TOTAL_USERMODE_TIME) |
Total time spent running in user mode by all processors (sec) |
None |
None |
|
Total Wait Time (TOTAL_WAIT_TIME) |
Total time spent in I/O standby by all processors (sec) |
None |
None |
When the setting is made to collect processor performance data from the results output by AIX's sar command, a blank is set as the value for the fields of the CPU - Per Processor Detail (PI_CPUP) record shown in the table below.
Record |
Field |
Field description |
Alarm used |
Report used |
---|---|---|---|---|
CPU - Per Processor Detail (PI_CPUP) |
Context Switches (PROCESSOR_CONTEXT_SWITCHES) |
Number of context switch executions |
None |
None |
Context Switches/sec (PROCESSOR_CONTEXT_SWITCHES_PER_SECOND) |
Frequency of context switch executions (executions per sec) |
None |
None |
|
Idle Time (PROCESSOR_IDLE_TIME) |
Time in idle state (sec) |
None |
None |
|
Sys Calls/sec (PROCESSOR_SYSTEM_CALLS_PER_SECOND) |
Frequency of system calls (calls per sec) |
None |
None |
|
System Calls (PROCESSOR_SYSTEM_CALLS) |
Number of system calls |
None |
None |
|
System Time (PROCESSOR_SYSTEM_TIME) |
Time running in kernel mode (sec) |
None |
None |
|
Type (PROCESSOR_TYPE) |
Description of processor |
None |
None |
|
User Time (PROCESSOR_USER_TIME) |
Time running in user mode (sec) |
None |
None |
|
Wait Time (PROCESSOR_WAIT_TIME) |
Time spent in I/O standby (sec) |
None |
None |
(3) Records supported in global environment and WPAR environments
When processor performance data is collected from the results output by the sar command, the records supported and the range of data collected for each record will depend on whether PFM - Agent for Platform is running in the global environment or in the System WPAR environment.
The following table describes the fields of System Summary Overview (PI) records supported in the global environment and in the System WPAR environment.
Record |
Field |
Global environment |
System WPAR environment |
||
---|---|---|---|---|---|
Collected? |
Range collected |
Collected? |
Range collected |
||
System Summary Overview (PI) |
15-Minute Run Queue Avg (FIFTEEN_MINUTE_RUN_QUEUE_AVG) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
1-Minute Run Queue Avg (ONE_MINUTE_RUN_QUEUE_AVG) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
5-Minute Run Queue Avg (FIVE_MINUTE_RUN_QUEUE_AVG) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Active CPUs (NUMBER_OF_ACTIVE_CPUS) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Block Ops (BLOCKIO_IO_OPS) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Block Reads (BLOCKIO_READ_OPS) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Block Reads/sec (BLOCKIO_READ_OPS_PER_SECOND) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Block Writes (BLOCKIO_WRITE_OPS) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Block Writes/sec (BLOCKIO_WRITE_OPS_PER_SECOND) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Cache Read % (CACHE_READ_PERCENT) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Cache Write % (CACHE_WRITE_PERCENT) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Context Switches (CONTEXT_SWITCHES) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Context Switches/sec (CONTEXT_SWITCHES_PER_SECOND) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
CPU % (KERNELMODE_USERMODE_PERCENT) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Idle % (IDLE_TIME_PERCENT) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Kernel CPU % (KERNELMODE_PERCENT) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Logical I/O Ops (LOGICAL_IO_OPS) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Logical Read Mbytes (LOGICAL_READ_MBYTES) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Logical Reads (LOGICAL_READ_OPS) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Logical Reads/sec (LOGICAL_READ_MBYTES_PER_SECOND) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Logical Write Mbytes (LOGICAL_WRITE_MBYTES) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Logical Writes (LOGICAL_WRITE_OPS) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Logical Writes/sec (LOGICAL_WRITE_MBYTES_PER_SECOND) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Physical I/O Ops (PHYSICAL_IO_OPS) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Physical Reads (PHYSICAL_READ_OPS) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
Physical Writes (PHYSICAL_WRITE_OPS) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
System Calls (SYSTEM_CALLS) |
Yes |
Global environment and all WPAR environments |
Yes |
Global environment and all WPAR environments |
|
User CPU % (USERMODE_PERCENT) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Wait % (WAIT_TIME_PERCENT) |
Yes |
Global environment and all WPAR environments |
No |
No |
The following table describes the fields of CPU - Per Processor Detail (PI_CPUP) records supported in the global environment and in the System WPAR environment.
Record |
Field |
Global environment |
System WPAR environment |
||
---|---|---|---|---|---|
Collected? |
Range collected |
Collected? |
Range collected |
||
CPU - Per Processor Detail (PI_CPUP) |
Boot Time (SYSTEM_BOOT_TIME) |
Yes |
Global environment |
No |
No |
CPU % (KERNELMODE_USERMODE_PERCENT) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Idle % (PROCESSOR_IDLE_PERCENT) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Interval (INTERVAL) |
Yes |
Global environment |
No |
No |
|
Processor ID (LOGICAL_PROCESSOR_ID) |
Yes |
Global environment |
No |
No |
|
Record Time (RECORD_TIME) |
Yes |
Global environment |
No |
No |
|
Record Type (INPUT_RECORD_TYPE) |
Yes |
Fixed value |
No |
No |
|
Status (PROCESSOR_STATUS) |
Yes |
Global environment |
No |
No |
|
System % (PROCESSOR_SYSTEM_PERCENT) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Up Time (SYSTEM_UP_TIME) |
Yes |
Global environment |
No |
No |
|
User % (PROCESSOR_USER_PERCENT) |
Yes |
Global environment and all WPAR environments |
No |
No |
|
Wait % (PROCESSOR_WAIT_PERCENT) |
Yes |
Global environment and all WPAR environments |
No |
No |
(4) Notes
This section provides additional information about collecting and monitoring processor performance data from the results output by AIX's sar command when PFM - Agent for Platform is used.
-
More time is needed to collect processor performance data from results output by a sar command than when data is not collected from sar command results. The difference in the amount of time used is the interval set on the PFM - Web Console's Service Properties window. If a value greater than the record collection time is specified for the interval on the PFM - Web Console's Service Properties window, record collection will not be performed.
-
If performance data could not be collected because execution of the sar command failed or for some other reason, the values in the corresponding fields in the System Summary Overview (PI) record will be 0, and performance data will not be gathered in the CPU - Per Processor Detail (PI_CPUP) record.
-
The values set in the fields of the System Summary Overview (PI) and CPU - Per Processor Detail (PI_CPUP) records depend on whether processor performance data is collected from results output by the sar command. The following tables describe the differences between the values stored in the various fields depending on whether processor performance data is collected from the results output by the sar command.
-
When processor performance is monitored by using the output results of the sar command, do not use functions (1) and (2) below simultaneously, or functions (1) and (3) below simultaneously. If you use them simultaneously, log data collection or alarm monitoring might be skipped.
(1) Real-time display of the PI record or PI_CPUP record.
(2) Log data collection of the PI record or PI_CPUP record.
(3) Alarm monitoring of the PI record or PI_CPUP record.
Table 2‒51: Differences in values stored in fields depending on whether processor performance data is collected from the results output by the sar command (when Indicate delta value is selected on real-time reports) Record
Field
Data source
Not from results output by the sar command
From results output by the sar command
System Summary Overview (PI)
Active CPUs (NUMBER_OF_ACTIVE_CPUS)
Results calculated based on value at time of collection
Value at time of collection
CPU % (KERNELMODE_USERMODE_PERCENT)
Results calculated based on amount of change
Value at time of collection
Idle % (IDLE_TIME_PERCENT)
Results calculated based on amount of change
Value at time of collection
Kernel CPU % (KERNELMODE_PERCENT)
Results calculated based on amount of change
Value at time of collection
User CPU % (USERMODE_PERCENT)
Results calculated based on amount of change
Value at time of collection
Wait % (WAIT_TIME_PERCENT)
Results calculated based on amount of change
Value at time of collection
CPU - Per Processor Detail (PI_CPUP)
CPU % (KERNELMODE_USERMODE_PERCENT)
Results calculated based on amount of change
Value at time of collection
Idle % (PROCESSOR_IDLE_PERCENT)
Results calculated based on amount of change
Value at time of collection
Processor ID (LOGICAL_PROCESSOR_ID)
Value at time of collection
Value at time of collection
System % (PROCESSOR_SYSTEM_PERCENT)
Results calculated based on amount of change
Value at time of collection
User % (PROCESSOR_USER_PERCENT)
Results calculated based on amount of change
Value at time of collection
Wait % (PROCESSOR_WAIT_PERCENT)
Results calculated based on amount of change
Value at time of collection
Table 2‒52: Differences in values stored in fields depending on whether processor performance data is collected from the results output by the sar command (when Indicate delta value is not selected on real-time reports) Record
Field
Data source
Not from results output by the sar command
From results output by the sar command
System Summary Overview (PI)
Active CPUs (NUMBER_OF_ACTIVE_CPUS)
Results calculated based on value at time of collection
Value at time of collection
CPU % (KERNELMODE_USERMODE_PERCENT)
Results calculated based on cumulative value
Value at time of collection
Idle % (IDLE_TIME_PERCENT)
Results calculated based on cumulative value
Value at time of collection
Kernel CPU % (KERNELMODE_PERCENT)
Results calculated based on cumulative value
Value at time of collection
User CPU % (USERMODE_PERCENT)
Results calculated based on cumulative value
Value at time of collection
Wait % (WAIT_TIME_PERCENT)
Results calculated based on cumulative value
Value at time of collection
CPU - Per Processor Detail (PI_CPUP)
CPU % (KERNELMODE_USERMODE_PERCENT)
Results calculated based on cumulative value
Value at time of collection
Idle % (PROCESSOR_IDLE_PERCENT)
Results calculated based on cumulative value
Value at time of collection
Processor ID (LOGICAL_PROCESSOR_ID)
Value at time of collection
Value at time of collection
System % (PROCESSOR_SYSTEM_PERCENT)
Results calculated based on cumulative value
Value at time of collection
User % (PROCESSOR_USER_PERCENT)
Results calculated based on cumulative value
Value at time of collection
Wait % (PROCESSOR_WAIT_PERCENT)
Results calculated based on cumulative value
Value at time of collection
-
Performance data is collected only for instances whose CPU - Per Processor Detail (PI_CPUP) record's Status field is On Line.
-
If the setting for collecting processor performance data from the results output by the sar command is changed from not collecting performance data to collecting performance data, the performance data from before the setting was changed will not be consistent with the data collected after the change. In such a case, treat performance data before and after the change as separate sets of performance data. If necessary, take steps such as backing up the Store database prior to changing system resources, and then clearing the Store database and starting collection of new performance data. For details about backing up the Store database, see the chapter on backing up and restoring in the JP1/Performance Management User's Guide.
-
Executing the sar command when PFM - Agent for Platform is running in an AIX System WPAR environment will result in an error. In such a case, the fields of the System Summary Overview (PI) record will have a value of 0, and no performance data will be collected in the fields of the CPU - Per Processor Detail (PI_CPUP) record.
-
When you monitor processor performance according to the output results of the sar command, do not use functions 1 and 2 or functions 1 and 3 (below) at the same time. Using these functions at the same time might cause historical data collection or alarm monitoring to be skipped.
-
Displaying a real-time report for the PI or PI_CPUP record
-
Collecting historical data for the PI or PI_CPUP record
-
Monitoring alarms for the PI or PI_CPUP record
-
-
You cannot connect to the Agent Collector service from the Service Properties window while the Agent Collector service is collecting performance data from the sar command. Connect to the Agent Collector service after the time specified in sar Command Interval has elapsed.
-
If you change the value of sar Command Monitoring in the Service Properties window for PFM - Web Console, performance data might become extremely large or negative at the initial collection. Therefore, if you have bound alarms that monitor performance for the PI_CPUP and PI records, unbind them before updating the properties. Normal values will be collected for performance data at the second collection and thereafter.
-
If the sar Command Monitoring property (displayed by selecting Agent Configuration in PFM - Agent for Platform) is set to Yes, performance information might not be collected correctly if either of the following conditions are met:
-
The iostat kernel parameter is set to false.
-
The sar command has been executed.
-
On two or more hosts within the same machine, Yes is set for the sar Command Monitoring property (displayed by selecting Agent Configuration in PFM - Agent for Platform).
In addition to the above conditions, performance information might not be collected correctly if the PFM - Remote Monitor for Platform instance settings match one of the following conditions:
-
A host that runs PFM - Agent for Platform is specified as a host to be monitored.
-
The iostat kernel parameter is set to false.
-
The Std_Category property or Disk_Category property of the instance is set to Y.#
#
- If the Std_Category property is set to Y, CPU performance information might not be collected.
- If the Disk_Category property is set to Y, disk performance information might not be collected.
-