Administering DMP Using vxdmpadm

The vxdmpadm utility is a command line administrative interface to the DMP feature of VxVM. You can use the vxdmpadm utility to perform the following tasks.

• Retrieve the name of the DMP device corresponding to a particular path.
• Display the members of a LUN group.
• List all paths under a DMP device.
• List all controllers connected to disks attached to the host.
• List all the paths connected to a particular controller.
• Set the attributes of the paths to an enclosure.
• Set the I/O policy that is used for the paths to an enclosure.
• Enable or disable a host controller on the system.
• Rename an enclosure.
• Control the operation of the DMP restore daemon.

The following sections cover these tasks in detail along with sample output. For more information, see the vxdmpadm(1M) manual page.

Retrieving Information About a DMP Node

The following command displays the DMP node that controls a particular physical path:

The physical path is specified by argument to the nodename attribute, which must be a valid path listed in the /dev/rdsk directory.

The above command displays output such as the following:

Use the enclosure attribute with getdmpnode to obtain a list of all DMP nodes for the specified enclosure.

Displaying the Members of a LUN Group

The following command displays the DMP nodes that are in the same LUN group as a specified DMP node:

The above command displays output such as the following:

Displaying All Paths Controlled by a DMP Node

The following command displays the paths controlled by the specified DMP node:

The specified DMP node must be a valid node in the /dev/vx/rdmp directory.

The state of a path that is currently enabled and available for I/O is shown as ENABLED(A):

For A/A arrays, all enabled paths that are available for I/O are shown as ENABLED(A).

For A/P arrays in which the I/O policy is set to singleactive, only one path is shown as ENABLED(A). The other paths are enabled but not available for I/O. If the I/O policy is not set to singleactive, DMP can use a group of paths (all primary or all secondary) for I/O, which are shown as ENABLED(A). See Specifying the I/O Policy for more information.

You can also use getsubpaths to obtain all paths through a particular host disk controller:

Listing Information About Host I/O Controllers

The following command lists attributes of all host I/O controllers on the system:

This form of the command lists controllers belonging to a specified enclosure and enclosure type:

Listing Information About Enclosures

To display the attributes of a specified enclosure, use the following command:

The following command lists attributes for all enclosures in a system:

The following is example output from this command:

Displaying Information About TPD-Controlled Devices

The third-party driver (TPD) coexistence feature allows I/O that is controlled by third-party multipathing drivers to bypass DMP while retaining the monitoring capabilities of DMP. The following commands allow you to display the paths that DMP has discovered for a given TPD device, and the TPD device that corresponds to a given TPD-controlled node discovered by DMP:

See Changing Device Naming for TPD-Controlled Enclosures for information on how to select whether OS or TPD-based device names are displayed.

For example, consider the following disks in an EMC Symmetrix array controlled by PowerPath, which are known to DMP:

The following command displays the paths that DMP has discovered, and which correspond to the PowerPath-controlled node, emcpower10:

Conversely, the next command displays information about the PowerPath node that corresponds to the path, c7t0d10, discovered by DMP:

Gathering and Displaying I/O Statistics

You can use the vxdmpadm iostat command to gather and display I/O statistics for a specified DMP node, enclosure or path.

To enable the gathering of statistics, enter this command:

To reset the I/O counters to zero, use this command:

The memory attribute can be used to limit the maximum amount of memory that is used to record I/O statistics for each CPU. The default limit is 32k (32 kilobytes) per CPU.

To display the accumulated statistics at regular intervals, use the following command:

This command displays I/O statistics for all controllers (all), or for a specified DMP node, enclosure or path. The statistics displayed are the CPU usage and amount of memory per CPU used to accumulate statistics, the number of read and write operations, the number of blocks read and written, and the average time in milliseconds per read and write operation.

The interval and count attributes may be used to specify the interval in seconds between displaying the I/O statistics, and the number of lines to be displayed. The actual interval may be smaller than the value specified if insufficient memory is available to record the statistics.

To disable the gathering of statistics, enter this command:

Examples of Using the vxdmpadm iostat Command

The follow is an example session using the vxdmpadm iostat command. The first command enables the gathering of I/O statistics:

The next command displays the current statistics including the accumulated total numbers of read and write operations and kilobytes read and written, on all paths:

The following command changes the amount of memory that vxdmpadm can use to accumulate the statistics:

The displayed statistics can be filtered by path name, DMP node name, and enclosure name (note that the per-CPU memory has changed following the previous command):

You can also specify the number of times to display the statistics and the time interval. Here the incremental statistics for a path are displayed twice with a 2-second interval:

Setting the Attributes of the Paths to an Enclosure

You can use the vxdmpadm setattr command to set the following attributes of the paths to an enclosure or disk array:

• active

Changes a standby (failover) path to an active path. The example below specifies an active path for an A/P-C disk array:


# vxdmpadm setattr path c2t10d0 pathtype=active
• nomanual

Restores the original primary or secondary attributes of a path. This example restores the attributes for a path to an A/P disk array:


# vxdmpadm setattr path c3t10d0 pathtype=nomanual
• nopreferred

Restores the normal priority of a path. The following example restores the default priority to a path:


# vxdmpadm setattr path c1t20d0 pathtype=nopreferred
• preferred [priority=N]

Specifies a path as preferred, and optionally assigns a priority number to it. If specified, the priority number must be an integer that is greater than or equal to one. Higher priority numbers indicate that a path is able to carry a greater I/O load.

---

  Note    Setting a priority for path does not change the I/O policy. The I/O policy must be set independently as described in Specifying the I/O Policy.

---

This example first sets the I/O policy to priority for an Active/Active disk array, and then specifies a preferred path with an assigned priority of 2:


# vxdmpadm setattr enclosure enc0 iopolicy=priority
# vxdmpadm setattr path c1t20d0 pathtype=preferred \
priority=2
• primary

Defines a path as being the primary path for an Active/Passive disk array. The following example specifies a primary path for an A/P disk array:


# vxdmpadm setattr path c3t10d0 pathtype=primary
• secondary

Defines a path as being the secondary path for an Active/Passive disk array. This example specifies a secondary path for an A/P disk array:


# vxdmpadm setattr path c4t10d0 pathtype=secondary
• standby

Marks a standby (failover) path that it is not used for normal I/O scheduling. This path is used if there are no active paths available for I/O. The next example specifies a standby path for an A/P-C disk array:


# vxdmpadm setattr path c2t10d0 pathtype=standby

Displaying the I/O Policy

To display the current and default settings of the I/O policy for an enclosure, array or array type, use the vxdmpadm getattr command.

The following example displays the default and current setting of iopolicy for JBOD disks:

The next example displays the setting of partitionsize for the enclosure enc0, on which the balanced I/O policy with a partition size of 2MB has been set:

Specifying the I/O Policy

You can use the vxdmpadm setattr command to change the I/O policy for distributing I/O load across multiple paths to a disk array or enclosure. You can set policies for an enclosure (for example, HDS01), for all enclosures of a particular type (such as HDS), or for all enclosures of a particular array type (A/A for Active/Active, or A/P for Active/Passive).

The following policies may be set:

• adaptive

This policy attempts to maximize overall I/O throughput from/to the disks by dynamically scheduling I/O on the paths. It is suggested for use where I/O loads can vary over time. For example, I/O from/to a database may exhibit both long transfers (table scans) and short transfers (random look ups). The policy is also useful for a SAN environment where different paths may have different number of hops. No further configuration is possible as this policy is automatically managed by DMP.

In this example, the adaptive I/O policy is set for the enclosure enc1:


# vxdmpadm setattr enclosure enc1 iopolicy=adaptive
• balanced [partitionsize=size]

This policy is designed to optimize the use of caching in disk drives and RAID controllers, and is the default policy for A/A arrays. The size of the cache typically ranges from 120KB to 500KB or more, depending on the characteristics of the particular hardware. During normal operation, the disks (or LUNs) are logically divided into a number of regions (or partitions), and I/O from/to a given region is sent on only one of the active paths. Should that path fail, the workload is automatically redistributed across the remaining paths.

You can use the size argument to the partitionsize attribute to specify the partition size. The partition size in blocks is adjustable in powers of 2 from 2 up to 2^31 as illustrated in the table below:

Partition Size in Blocks	Equivalent Size in Bytes
2	2,048
4	4,096
8	8,192
16	16,384
32	32,768
64	65,536
128	131,072
256	262,144
512	524,288
1024 (default)	1,048,576
2048	2,097,152
4096	4,194,304

The default value for the partition size is 1024 blocks (1MB). A value that is not a power of 2 is silently rounded down to the nearest acceptable value. Specifying a partition size of 0 is equivalent to the default partition size of 1024 blocks (1MB). For example, the suggested partition size for an Hitachi HDS 9960 A/A array is from 16,384 to 65,536 blocks (16MB to 64MB) for an I/O activity pattern that consists mostly of sequential reads or writes.

---

  Note    The benefit of this policy is lost if the value is set larger than the cache size.

---

The default value can be changed by adjusting the value of a tunable parameter (see dmp_pathswitch_blks_shift) and rebooting the system.

The next example sets the balanced I/O policy with a partition size of 2048 blocks (2MB) on the enclosure enc0:


# vxdmpadm setattr enclosure enc0 iopolicy=balanced \
  partitionsize=2048
• minimumq

This policy sends I/O on paths that have the minimum number of outstanding I/O requests in the queue for a LUN. This is suitable for low-end disks or JBODs where a significant track cache does not exist. No further configuration is possible as DMP automatically determines the path with the shortest queue.

The following example sets the I/O policy to minimumq for a JBOD:


# vxdmpadm setattr enclosure Disk iopolicy=minimumq
• priority

This policy is useful when the paths in a SAN have unequal performance, and you want to enforce load balancing manually. You can assign priorities to each path based on your knowledge of the configuration and performance characteristics of the available paths, and of other aspects of your system. See Setting the Attributes of the Paths to an Enclosure for details of how to assign priority values to individual paths.

In this example, the I/O policy is set to priority for all SENA arrays:


# vxdmpadm setattr arrayname SENA iopolicy=priority
• round-robin

This policy shares I/O equally between the paths in a round-robin sequence. For example, if there are three paths, the first I/O request would use one path, the second would use a different path, the third would be sent down the remaining path, the fourth would go down the first path, and so on. No further configuration is possible as this policy is automatically managed by DMP. This is the default policy for A/PC configurations with multiple active paths per controller.

The next example sets the I/O policy to round-robin for all Active/Active arrays:


# vxdmpadm setattr arraytype A/A iopolicy=round-robin
• singleactive

This policy routes I/O down one single active path. This is the default policy for A/P arrays with one active path per controller, where the other paths are used in case of failover. If configured for A/A arrays, there is no load balancing across the paths, and the alternate paths are only used to provide high availability (HA). If the currently active path fails, I/O is switched to an alternate active path. No further configuration is possible as the single active path is selected by DMP.

The following example sets the I/O policy to singleactive for JBOD disks:


# vxdmpadm setattr arrayname DISK iopolicy=singleactive

Example of Applying Load Balancing in a SAN

This example describes how to configure load balancing in a SAN environment where there are multiple primary paths to an Active/Passive device through several SAN switches. As can be seen in this sample output from the vxdisk list command, the device c3t2d15 has eight primary paths:

In addition, the device is in the enclosure ENC0, belongs to the disk group mydg, and contains a simple concatenated volume myvol1.

The first step is to enable the gathering of DMP statistics:

Next the dd command is used to apply an input workload from the volume:

By running the vxdmpadm iostat command to display the DMP statistics for the device, it can be seen that all I/O is being directed to one path, c5t4d15:

The vxdmpadm command is used to display the I/O policy for the enclosure that contains the device:

This shows that the policy for the enclosure is set to singleactive, which explains why all the I/O is taking place on one path.

To balance the I/O load across the multiple primary paths, the policy is set to round-robin as shown here:

The DMP statistics are now reset:

With the workload still running, the effect of changing the I/O policy to balance the load across the primary paths can now be seen.

The enclosure can be returned to the single active I/O policy by entering the following command:

Disabling a Controller

Disabling I/O to a host disk controller prevents DMP from issuing I/O through the specified controller. The command blocks until all pending I/O issued through the specified disk controller are completed.

To disable a controller, use the following command:

The disable operation fails if it is issued to a controller connected to the root disk through a single path. If there is a single path connected to a disk, the disable command fails with an error message. Use the -f option to forcibly disable the controller.

Enabling a Controller

Enabling a controller allows a previously disabled host disk controller to accept I/O. This operation succeeds only if the controller is accessible to the host and I/O can be performed on it. When connecting Active/Passive disk arrays in a non-clustered environment, the enable operation results in failback of I/O to the primary path. The enable operation can also be used to allow I/O to the controllers on a system board that was previously detached.

To enable a controller, use the following command:

Renaming an Enclosure

The vxdmpadm setattr command can be used to assign a meaningful name to an existing enclosure, for example:

This example changes the name of an enclosure from enc0 to GRP1.

The following command shows the changed name:

Starting the DMP Restore Daemon

The DMP restore daemon re-examines the condition of paths at a specified interval. The type of analysis it performs on the paths depends on the specified checking policy.

Use the start restore command to start the restore daemon and specify one of the following policies:

• check_all

The restore daemon analyzes all paths in the system and revives the paths that are back online, as well as disabling the paths that are inaccessible. The command to start the restore daemon with this policy is:


# vxdmpadm start restore policy=check_all [interval=seconds]
• check_alternate

The restore daemon checks that at least one alternate path is healthy. It generates a notification if this condition is not met. This policy avoids inquiry commands on all healthy paths, and is less costly than check_all in cases where a large number of paths are available. This policy is the same as check_all if there are only two paths per DMP node. The command to start the restore daemon with this policy is:


# vxdmpadm start restore policy=check_alternate [interval=seconds]
• check_disabled

This is the default policy. The restore daemon checks the condition of paths that were previously disabled due to hardware failures, and revives them if they are back online. The command to start the restore daemon with this policy is:


# vxdmpadm start restore policy=check_disabled [interval=seconds]
• check_periodic

The restore daemon performs check_all once in a given number of cycles, and check_disabled in the remainder of the cycles. This policy may lead to periodic slowing down (due to check_all) if there are a large number of paths available. The command to start the restore daemon with this policy is:


# vxdmpadm start restore policy=check_periodic interval=seconds \
  [period=number]

The interval attribute must be specified for this policy. The default number of cycles between running the check_all policy is 10.

The interval attribute specifies how often the restore daemon examines the paths. For example, after stopping the restore daemon, the polling interval can be set to 400 seconds using the following command:

See the vxdmpadm(1M) manual page for more information about DMP restore policies.

Stopping the DMP Restore Daemon

Use the following command to stop the DMP restore daemon:

Displaying the Status of the DMP Restore Daemon

Use the following command to display the status of the automatic path restoration daemon, its polling interval, and the policy that it uses to check the condition of paths:

This produces output such as the following:

Displaying Information About the DMP Error Daemons

To display the number of error daemons that are running, use the following command:

Configuring Array Policy Modules

An array policy module (APM) is a dynamically loadable kernel module that may be provided by some vendors for use in conjunction with an array. An APM defines procedures to:

• Select an I/O path when multiple paths to a disk within the array are available.
• Select the path failover mechanism.
• Select the alternate path in the case of a path failure.
• Put a path change into effect.
• Respond to SCSI reservation or release requests.

DMP supplies default procedures for these functions when an array is registered. An APM may modify some or all of the existing procedures that are provided by DMP or by another version of the APM.

You can use the following command to display all the APMs that are configured for a system:

The output from this command includes the file name of each module, the supported array type, the APM name, the APM version, and whether the module is currently in use (loaded). To see detailed information for an individual module, specify the module name as the argument to the command:

To add and configure an APM, use the following command:

The optional configuration attributes and their values are specific to the APM for an array. Consult the documentation that is provided by the array vendor for details.

Specifying the -r option allows you to remove an APM that is not currently loaded:

For more information about configuring APMs, see the vxdmpadm(1M) manual page.