4 Administering ASM Disk Groups

Creating Disk Groups

The CREATE DISKGROUP SQL statement is used to create disk groups. When creating a disk group, you need to:

• Assign a unique name to the disk group.

• Specify the redundancy level of the disk group.

  If you want ASM to mirror files, you specify the redundancy level as NORMAL REDUNDANCY (2-way mirroring by default for most file types) or HIGH REDUNDANCY (3-way mirroring for all files). You specify EXTERNAL REDUNDANCY if you do not want mirroring by ASM. For example, you might choose EXTERNAL REDUNDANCY if you want to use storage array protection features.

  For more information about redundancy levels, refer to "ASM Mirroring and Failure Groups".

• Specify the disks as belonging to specific failure groups.

  For information about failure groups, refer to "Understanding ASM Concepts" and "ASM Mirroring and Failure Groups".

• Specify the disks that are to be formatted as ASM disks belonging to the disk group.

  The disks can be specified using operating system dependent wildcard characters in search strings that ASM then uses to find the disks. You can specify names for the disks with the NAME clause or use the system-generated names.

• Optionally specify disk group attributes, such software compatibility or allocation unit size.

ASM programmatically determines the size of each disk. If for some reason this is not possible, or if you want to restrict the amount of space used on a disk, you are able to specify a SIZE clause for each disk. ASM creates operating system–independent names for the disks in a disk group that you can use to reference the disks in other SQL statements. Optionally, you can provide your own name for a disk using the NAME clause. Disk names are available in the V$ASM_DISK view.

The ASM instance ensures that any disk in a newly created disk group is addressable and is not currently a member of another disk group. You must use FORCE only when adding a disk that was dropped FORCE. If a disk is dropped NOFORCE, then use can add it NOFORCE. For example, a disk might have failed and was dropped from its disk group. After the disk is repaired, it is no longer part of any disk group, but ASM still recognizes that the disk had been a member of a disk group. You must use the FORCE flag to include the disk in a new disk group. In addition, the disk must be addressable, and the original disk group must not be mounted. Otherwise, the operation fails.

The CREATE DISKGROUP statement mounts the disk group for the first time, and adds the disk group name to the ASM_DISKGROUPS initialization parameter if a server parameter file is being used. If a text initialization parameter file is being used and you want the disk group to be automatically mounted at instance startup, then you must remember to add the disk group name to the ASM_DISKGROUPS initialization parameter before the next time that you shut down and restart the ASM instance. You can also create disk groups with Oracle Enterprise Manager. Refer to "Creating Disk Groups".

% SQLPLUS /NOLOG
SQL> CONNECT / AS SYSASM
Connected to an idle instance.
SQL> STARTUP NOMOUNT

CREATE DISKGROUP dgroup1 NORMAL REDUNDANCY
FAILGROUP controller1 DISK
'/devices/diska1',
'/devices/diska2',
'/devices/diska3',
'/devices/diska4'
FAILGROUP controller2 DISK
'/devices/diskb1',
'/devices/diskb2',
'/devices/diskb3',
'/devices/diskb4'
ATTRIBUTE 'AU_SIZE'='4M';

Altering Disk Groups

You can use the ALTER DISKGROUP statement to alter a disk group configuration. You can add, resize, or drop disks while the database remains online. Whenever possible, multiple operations in a single ALTER DISKGROUP statement are recommended.

ASM automatically rebalances when the configuration of a disk group changes. By default, the ALTER DISKGROUP statement does not wait until the operation is complete before returning. Query the V$ASM_OPERATION view to monitor the status of this operation.

You can use the REBALANCE WAIT clause if you want the ALTER DISKGROUP statement processing to wait until the rebalance operation is complete before returning. This is especially useful in scripts. The statement also accepts a REBALANCE NOWAIT clause that invokes the default behavior of conducting the rebalance operation asynchronously in the background.

You can interrupt a rebalance running in wait mode by typing CTRL-C on most platforms. This causes the statement to return immediately with the message ORA-01013: user requested cancel of current operation, and to continue the operation asynchronously. Typing CTRL-C does not cancel the rebalance operation or any disk add, drop, or resize operations.

To control the speed and resource consumption of the rebalance operation, you can include the REBALANCE POWER clause in statements that add, drop, or resize disks. Refer to "Manually Rebalancing Disk Groups" for more information about this clause.

ALTER DISKGROUP dgroup1 ADD DISK
     '/devices/diska*';

ALTER DISKGROUP dgroup1 ADD DISK
     '/devices/diska5' NAME diska5,
     '/devices/diska6' NAME diska6,
     '/devices/diska7' NAME diska7,
     '/devices/diska8' NAME diska8,

ALTER DISKGROUP dgroup1 ADD DISK
     '/devices/disk*4';

ALTER DISKGROUP dgroup1 ADD DISK
      '/devices/diskd*'
       REBALANCE POWER 5 WAIT;

ALTER DISKGROUP dgroup2 ADD DISK
     '/devices/diskc3' FORCE;

ALTER DISKGROUP dgroup1 DROP DISK diska5;

ALTER DISKGROUP dgroup1 DROP DISK diska5
     ADD FAILGROUP failgrp1 DISK '/devices/diska9' NAME diska9;

ALTER DISKGROUP dgroup1 
     RESIZE DISKS IN FAILGROUP failgrp1 SIZE 100G;

ALTER DISKGROUP dgroup1 UNDROP DISKS;

ALTER DISKGROUP dgroup2 REBALANCE POWER 5 WAIT;

How A Disk is Discovered

While an ASM instance is initialized, ASM discovers and examines the contents of all of the disks that are in the paths that you designated with values in the ASM_DISKSTRING initialization parameter. Disk discovery also occurs when you:

• Run the ALTER DISKGROUP...ADD DISK and ALTER DISKGROUP...RESIZE DISK commands

• Query the V$ASM_DISKGROUP and V$ASM_DISK views

After ASM successfully discovers a disk, the disk appears in the V$ASM_DISK view. Disks that belong to a disk group, that is, disks that have a disk group name in the disk header, show a status of MEMBER. Disks that were discovered, but that have not yet been assigned to a disk group, have a status of either CANDIDATE or PROVISIONED.

The PROVISIONED status implies that an additional platform-specific action has been taken by an administrator to make the disk available for ASM. For example, on Windows computers, the administrator might have used asmtool or asmtoolg to stamp the disk with a header. On Linux computers, the administrator might have used ASMLIB to prepare the disk for ASM.

The following SQL query shows one candidate and six member disks:

SELECT name, header_status, path FROM V$ASM_DISK;

Disk Discovery Rules

The rules for discovering ASM disks are as follows:

• ASM can discover up to 10,000 disks. That is, if more than 10,000 disks match the ASM_DISKSTRING initialization parameter, then ASM discovers only the first 10,000.

• ASM only discovers disks that contain a partition table.

  Note:

  ASM does not discover a disk that contains an operating system partition table, even if the disk is in an ASM disk string search path and ASM has read and write permission for the disk.

• When adding a disk, the FORCE option must be used if ASM recognizes that the disk was managed by Oracle. Such a disk appears in the V$ASM_DISK view with a status of FOREIGN. In this case, you can only add the disk to a disk group by using the FORCE keyword.

In addition, ASM identifies the following configuration errors during discovery:

• Multiple paths to the same disk

  In this case, if the disk is part of a disk group, then disk group mount fails. If the disk is being added to a disk group with the ADD DISK or CREATE DISKGROUP command, then the command fails. To correct the error, adjust the ASM_DISKSTRING value so that ASM will not discover multiple paths to the same disk. Or if you are using multipathing software, then ensure that you include only the pseudo-device name in the ASM_DISKSTRING value. See "ASM and Multipathing".

• Multiple ASM disks with the same disk header

  This can be caused by having copied one disk onto another. In this case, the disk group mount operation fails.

Improving Disk Discovery Time

The value for the ASM_DISKSTRING initialization parameter is an operating system–dependent value that ASM uses to limit the set of paths that the discovery process uses to search for disks. When a new disk is added to a disk group, each ASM instance that has the disk group mounted must be able to discover the new disk using its ASM_DISKSTRING.

In many cases, the default value (NULL) is sufficient. Using a more restrictive value might reduce the time required for ASM to perform discovery, and thus improve disk group mount time or the time for adding a disk to a disk group. Oracle might need to dynamically change the ASM_DISKSTRING before adding a disk so that the new disk will be discovered through this parameter.

Note that the default value of ASM_DISKSTRING might not find all disks in all situations. If your site is using a third-party vendor ASMLIB, then the vendor might have discovery string conventions that you must use for ASM_DISKSTRING. In addition, if your installation uses multipathing software, then the software might place pseudo-devices in a path that is different from the operating system default. See "ASM and Multipathing" and consult the multipathing vendor documentation for details.

Negative Values of USABLE_FILE_MB

Due to the relationship between FREE_MB, REQUIRED_MIRROR_FREE_MB, and USABLE_FILE_MB, USABLE_FILE_MB can become negative. Although this is not necessarily a critical situation, it does mean that:

• Depending on the value of FREE_MB, you may not be able to create new files.

• The next failure might result in files with reduced redundancy.

If USABLE_FILE_MB becomes negative, it is strongly recommended that you add more space to the disk group as soon as possible.

ASM Mirroring and Failure Groups

If you specify mirroring for a file, then ASM automatically stores redundant copies of the file extents in separate failure groups. Failure groups apply only to normal and high redundancy disk groups. You can define the failure groups for each disk group when you create or alter the disk group.

There are three types of disk groups based on the ASM redundancy level. Table 4-1 lists the types with their supported and default mirroring levels. The default mirroring levels indicate the mirroring level with which each file is created unless a different mirroring level is designated.

The redundancy level controls how many disk failures are tolerated without dismounting the disk group or losing data. Each file is allocated based on its own redundancy, but the default comes from the disk group. The redundancy levels are:

• External redundancy

  ASM does not provide mirroring redundancy and relies on the storage system to provide RAID functionality. Any write error cause a forced dismount of the disk group. All disks must be located to successfully mount the disk group.

• Normal redundancy

  ASM provides two-way mirroring. By default all files are mirrored so that there are two copies of every data extent. A loss of one ASM disk is tolerated.

• High redundancy

  ASM provides triple mirroring by default. A loss of two ASM disks in different failure groups is tolerated.

Failure groups enable the mirroring of metadata and user data. System reliability can diminish if your environment has an insufficient number of failure groups.

This section contains these topics:

• ASM Failure Groups

• How ASM Manages Disk Failures

• Guidelines for Using Failure Groups

• Failure Group Frequently Asked Questions

ASM Fast Mirror Resync

Restoring the redundancy of an ASM disk group after a transient disk path failure can be time consuming. This is especially true if the recovery process requires rebuilding an entire ASM failure group. ASM fast mirror resync significantly reduces the time to resynchronize a failed disk in such situations. When you replace the failed disk, ASM can quickly resynchronize the ASM disk extents.

Any problems that make a failure group temporarily unavailable are considered transient failures that can be recovered by the ASM fast mirror resync feature. Disk path malfunctions; such as cable failures, host bus adapter failures, controller failures, or disk power supply interruptions; can cause transient failures.

ASM fast resync keeps track of pending changes to extents on an OFFLINE disk during an outage. The extents are resynced when the disk is brought back online.

By default, ASM drops a disk in 3.6 hours after it is taken offline. You can set the DISK_REPAIR_TIME attribute to prevent this operation by specifying a time interval to repair the disk and bring it back online. The time can be specified in units of minutes (m or M) or hours (h or H). If you omit the unit, then the default unit is hours. If the attribute is not set explicitly, then the default value is 3.6h. Note that this default applies to disks that have been set to OFFLINE mode without an explicit DROP AFTER clause. The default DISK_REPAIR_TIME attribute value of 3.6h should be adequate for most environments.

The elapsed time (since the disk was set to OFFLINE mode) is incremented only when the disk group containing the offline disks is mounted. The REPAIR_TIME column of V$ASM_DISK shows the amount of time left (in seconds) before an offline disk is dropped. After the specified time has elapsed, ASM drops the disk. You can override this attribute with an ALTER DISKGROUP DISK OFFLINE statement and the DROP AFTER clause.

If an offline disk is taken offline for a second time, then the elapsed time is reset and restarted. If another time is specified with the DROP AFTER clause for this disk, the first value is overridden and the new value applies. A disk that is in OFFLINE mode cannot be dropped with an ALTER DISKGROUP DROP DISK statement; an error is returned if attempted. If for some reason the disk needs to be dropped (such as the disk cannot be repaired) before the repair time has expired, a disk can be dropped immediately by issuing a second OFFLINE statement with a DROP AFTER clause specifying 0h or 0m.

You can use ALTER DISKGROUP to set the DISK_REPAIR_TIME attribute to a specified hour or minute value, such as 4.5 hours or 270 minutes. For example:

ALTER DISKGROUP dg01 SET ATTRIBUTE 'disk_repair_time' = '4.5h'
ALTER DISKGROUP dg01 SET ATTRIBUTE 'disk_repair_time' = '270m'

After you repair the disk, run the SQL statement ALTER DISKGROUP DISK ONLINE. This statement brings a repaired disk group back online to enable writes so that no new writes are missed. This statement also starts a procedure to copy of all of the extents that are marked as stale on their redundant copies.

If a disk goes offline when the ASM instance is in rolling upgrade mode, the disk remains offline until the rolling upgrade has ended and the timer for dropping the disk is stopped until the ASM cluster is out of rolling upgrade mode. See "Using ASM Rolling Upgrades". Examples of taking disks offline and bringing them online follow.

The following example takes disk D3_0001 offline and drops it after five minutes.

ALTER DISKGROUP D3 OFFLINE DISK D3_0001 DROP AFTER 5m;

The next example takes the disk D3_0001 offline and drops it after the time period designated by DISK_REPAIR_TIME elapses:

ALTER DISKGROUP D3 OFFLINE DISK D3_0001;

This example takes all of the disk in failure group F2 offline and drops them after the time period designated by DISK_REPAIR_TIME elapses. IF you used a DROP AFTER clause, then the disks would be dropped after the specified time:

ALTER DISKGROUP D3 OFFLINE DISKS IN FAILGROUP F2;

The next example brings all of the disks in failure group F2 online:

ALTER DISKGROUP D3 ONLINE DISKS IN FAILGROUP F2;

This example brings only disk D3_0001 online:

ALTER DISKGROUP D3 ONLINE DISK D3_0001;

This example brings all of the disks in disk group D3 online:

ALTER DISKGROUP D3 ONLINE ALL;

Querying the V$ASM_OPERATION view after you run any of these types of online and offline statements displays the current operation that you are performing. For example, the query:

SELECT GROUP_NUMBER, OPERATION, STATE FROM V$ASM_OPERATION;

Displays output similar to the following:

GROUP_NUMBER OPERA STAT 
------------ ----- ---- 
           1 ONLIN RUN  
           1 ONLIN RUN 

Preferred Read Failure Groups

When you configure ASM failure groups, it might be more efficient for a node to read from an extent that is closest to the node, even if that extent is a secondary extent. In other words, you can configure ASM to read from a secondary extent if that extent is closer to the node instead of ASM reading from the primary copy which might be farther from the node. Using preferred read failure groups is most useful in extended clusters.

To use this feature, Oracle recommends that you configure at least one mirrored extent copy from a disk that is local to a node in an extended cluster. However, a failure group that is preferred for one instance might be remote to another instance in the same Oracle RAC database. The parameter setting for preferred read failure groups is instance specific.

Determining the Number of Disk Groups

Use the following criteria to determine the number of disk groups to create:

• Disks in a given disk group should have similar size and performance characteristics. If you have several different types of disks in terms of size and performance, then create several disk groups that contain similar characteristics.

• Create separate disk groups for your database files and flash recovery area for backup files. This configuration allows fast recovery in case of a disk group failure.

Performance Characteristics When Grouping Disks

ASM load balances the file activity by uniformly distributing file extents across all of the disks in a disk group. For this technique to be effective it is important that disks in a disk group be of similar performance characteristics. For example, the newest and fastest disks might reside in a disk group reserved for the database work area, and slower drives could reside in a disk group reserved for the flash recovery area.

There might be situations where it is acceptable to temporarily have disks of different sizes and performance co-existing in a disk group. This would be the case when migrating from an old set of disks to a new set of disks. The new disks would be added and the old disks dropped. As the old disks are dropped, their storage is migrated to the new disks while the disk group is online.

ASM Storage Limits

ASM has the following limits:

• 63 disk groups in a storage system

• 10,000 ASM disks in a storage system

• 4 PB maximum storage for each ASM disk

• 40 exabyte maximum storage for each storage system

• 1 million files for each disk group

Oracle Database supports datafile sizes up to 128 TB. ASM supports file sizes greater than 128 TB in any redundancy mode. This provides near unlimited capacity for future growth. The ASM file size limits are as follows:

• External redundancy - 140 PB

• Normal redundancy - 42 PB

• High redundancy - 15 PB

Overview of Disk Group Compatibility

The disk group compatibility feature enables environments to interoperate when they use disk groups from both Oracle Database 10g and Oracle Database 11g. Compatibility settings that are set to previous releases enable database clients to access disk groups of higher releases. For example, Oracle Database 10g clients could access an Oracle Database 11g ASM disk group.

The ASM and Oracle Database disk group attribute settings for compatibility determine the minimum ASM and Oracle Database software version numbers that a system can use. For instance, if ASM compatibility is 11.1, and Oracle Database compatibility is 10.1, then the ASM software version must be at least 11.1, and the Oracle Database client software version must be at least 10.1. The two attribute settings are COMPATIBLE.ASM and COMPATIBLE.RDBMS.

The software version of ASM determines the default compatibility of newly created disk groups. You can override the disk group compatibility default setting when you create disk groups with the CREATE DISKGROUP SQL statement. The ALTER DISKGROUP SQL statement can update the compatibility settings for existing disk groups. The compatibility settings for a disk group can only be advanced; you cannot revert to a lower compatibility setting.

Advancing the disk group Oracle Database and ASM compatibility settings enables you to use the new ASM features that are available in latest release. For example, a disk group with the RDBMS and ASM compatibility set to 11.1 can take advantage of new Oracle Database 11g features, such as variable extent sizes and fast mirror resync.

Disk Group Compatibility Attributes

The COMPATIBLE.ASM and COMPATIBLE.RDBMS disk group attributes specify the compatibility settings for Oracle Database and the ASM instance respectively. To enable disk compatibility, you must first set the COMPATIBLE.ASM attribute and then set the COMPATIBLE.RDBMS attribute. These attributes are described under the following topics:

• COMPATIBLE.ASM

• COMPATIBLE.RDBMS

Setting Disk Group Compatibility

You can set disk group compatibility as shown in Table 4-2 with the CREATE DISKGROUP or ALTER DISKGROUP SQL statement.

Using CREATE DISKGROUP with Compatibility Attributes

You can specify the compatibility settings for a disk group with the CREATE DISKGROUP statement when creating the disk group.

asmdskgrp1 with ASM compatibility set to 10.2 and database compatibility set to the default (assuming that the COMPATIBLE.RDBMS default is less than or equal to 10.2):

CREATE DISKGROUP asmdskgrp1 DISK '/dev/raw/*' 
       SET ATTRIBUTE 'compatible.asm' = '10.2';

The following example creates a normal redundancy disk group asmdskgrp2 with the ASM compatibility set to 11.1 and the database compatibility set to the default (assuming that the COMPATIBLE.RDBMS default is less than or equal to 11.1):

CREATE DISKGROUP asmdskgrp2 DISK '/dev/raw/*' 
       SET ATTRIBUTE 'compatible.asm' = '11.1';

The following example creates a normal redundancy disk group asmdskgrp3 with both the ASM and the database compatibility set to 11.1:

CREATE DISKGROUP asmdskgrp3 DISK '/dev/raw/*' 
       SET ATTRIBUTE 'compatible.rdbms' = '11.1', 'compatible.asm' = '11.1';

Using ALTER DISKGROUP with Compatibility Attributes

After a disk group has been created, you can use the ALTER DISKGROUP SQL statement to change the compatibility attributes. Using the ALTER DISKGROUP SQL statement ensures that Oracle can advance the compatibility of the specified disk group before committing the change. All of the affected databases and file systems should be online when running ALTER DISKGROUP to ensure that advancing compatibility does not reduce the database and file system functionality.

The following example advances the database compatibility of the disk group asmdskgrp4 to 11.1. This example assumes that the ASM compatibility is already advanced to 11.1.

ALTER DISKGROUP asmdskgrp4 SET ATTRIBUTE 'compatible.rdbms' = '11.1'

The following example advances the ASM compatibility for disk group asmdskgrp5 to 11.1. An ASM instance must be at release 11.1 and higher to access the asmdskgrp5 disk group.

ALTER DISKGROUP asmdskgrp5 SET ATTRIBUTE 'compatible.asm' = '11.1'

Considerations When Setting Disk Group Compatibility

When changing the disk group compatibility settings, there are some considerations that you should be aware of.

• If a backup of a disk group was made with the ASMCMD md_backup command prior to changing the compatibility settings, that full backup file would be incorrect for the updated disk group. Restoring that the full previous backup would set the disk group to the previous compatibility settings.

  You can still use the previous backup to restore some metadata. For example, you could create a new disk group and use the backup file to restore templates and alias directories metadata. For information about using md_backup and md_restore, refer to "md_backup Command" and "md_restore Command".

• The disk group compatibility settings should be the same for all replication environments.

• The compatibility settings can only be advanced and the settings are irreversible.

• Not all combinations of the disk group compatibility settings are valid and some features are not enabled in various combinations. Table 4-2 lists the possible COMPATIBLE.ASM and COMPATIBLE.RDBMS disk group settings and the features enabled.

Considerations When Setting Disk Group Compatibility in Replicated Environments

If you advance disk group compatibility, then you could enable the creation of files that are too large to be managed by a previous Oracle database release. You need to be aware of the file size limits because replicated sites cannot continue using the software from a previous release to manage these large files.

Table 4-3 show the maximum ASM file sizes. Note that memory consumption is identical in all cases: 280 MB. For Oracle Database 11g, the number of extent pointers for each size is 16800, with the largest size using the remainder.

Table 4-3 shows that Oracle Database 10g can only support a file size of up to 35 TB for external redundancy. If you advance the compatibility to 11.1, then a file can grow beyond 35 TB making the file unusable in replicated and disaster recovery sites.

Example: Dismounting Disk Groups

The following statement dismounts all disk groups that are currently mounted to the ASM instance:

ALTER DISKGROUP ALL DISMOUNT;

Example: Mounting Disk Groups

The following statement mounts disk group dgroup1:

ALTER DISKGROUP dgroup1 MOUNT;

Mounting Disk Groups Using the FORCE Option

For normal and high redundancy disk groups, you can use the FORCE option of the ALTER DISKGROUP statement's MOUNT clause to mount disk groups if there are sufficient ASM disks available. The disk group mount succeeds if ASM finds at least one complete set of extents in a disk group. If ASM determines that one or more disks are not available, then ASM moves those disks off line and drops the disks after the DISK_REPAIR_TIME expires.

In clustered ASM environments, if an ASM instance is not the first instance to mount the disk group, then using the MOUNT FORCE statement fails. This is because the disks have been accessed by another instance and the disks are not locally accessible.

Use the FORCE option as in the following example where disk_group_name is the name of the disk group that you want to force mount:

ALTER DISKGROUP disk_group_name MOUNT FORCE