6 System and Network Configuration

6.2.1 Ensure That All Hardware Components Are Fully Redundant and Fault-Tolerant

All hardware components of a storage array must be fully redundant, from physical interfaces to physical disks, including redundant power supplies and connectivity to the array itself.

The storage array should contain one or more spare disks (often called hot spares). When a physical disk starts to report errors to the monitoring infrastructure, or fails suddenly, the firmware should immediately restore fault tolerance by mirroring the contents of the failed disk onto a spare disk.

Connectivity to the storage array must be fully redundant (referred to as multipathing) so that the failure of any single component in the data path from any node to the shared disk array (such as controllers, interface cards, cables, and switches) is transparent and keeps the array fully accessible. This achieves addressing the same logical device through multiple physical paths. A host-based device driver reissues the I/O to one of the surviving physical interfaces.

If the storage array includes a write cache, then it must be protected to guard against memory board failures. The write cache should be protected by multiple battery backups to guard against a failure of all external power supplies. The cache must be able to survive either by using the battery backup long enough for the external power to return or until all the dirty blocks in cache are guaranteed to be fully flushed to a physical disk.

6.2.2 Use an Array That Can Be Serviced Online

If a physical component fails, then the array must allow the failed device to be repaired or replaced without requiring the array to be shut down or taken offline. Also, the storage array must allow the firmware to be upgraded and patched without shutting down the storage array.

6.2.3 Mirror and Stripe for Protection and Performance

Data should be mirrored to protect against disk and other component failures and should be striped over a large number of disks to achieve optimal performance. This method of storage configuration is known as Stripe and Mirror Everything (SAME). The SAME methodology provides a simple, efficient, and highly available storage configuration.

Oracle's automatic storage management (ASM) feature always evenly stripes data across all drives within a disk troup with the added benefit of automatically rebalancing files across new disks when disks are added, or across existing disks if disks are removed. In addition, ASM can provide redundancy protection to protect against component failures or enable mirroring to be provided by the underlying storage array.

6.2.4 Load-Balance Across All Physical Interfaces

Load balancing of I/O operations across physical interfaces is usually provided by a software package that is installed on the host. The purpose of this load balancing software is to redirect I/O operations to a less busy physical interface if a single host bus adapter (HBA) is overloaded by the current workload.

6.2.5 Create Independent Storage Areas

Create separate, independent storage areas for software, active database files, and recovery-related files.

The following storage areas are needed:

• Software area: A location where software is installed and log and trace files are created

• Database area: A location where active database files such as datafiles, control files, online redo logs, and change tracking files used in incremental backups are stored

• Flash recovery area: A location where recovery-related files are created, such as multiplexed copies of the current control file and online redo logs, archived redo logs, backup sets, and flashback log files

The storage containing the database area should be as physically distinct as possible from the flash recovery area. At a minimum, the database area and flash recovery area should not share the same physical disk drives or controllers. This practice ensures that the failure of a component that provides access to a datafile in the database area does not also cause the loss of the backups or redo logs in the flash recovery area needed to recover that datafile.

Storage options are defined as follows:

• Regular file system: A local file system that is not shared by any other system

• Cluster file system (CFS): A file system that is equally addressable and accessible by all nodes in a cluster (unlike a shared file system where access is funneled through a master node)

• Automatic storage management (ASM): A tool designed specifically for Oracle database files that integrates the capabilities of a file system and a volume manager

• Raw device or logical volume: A piece of storage typically managed by a logical volume manager that does not contain a file system

The rest of this section includes these topics:

• Storage Recommendations for Specific HA Architectures

• Define ASM Disk and Failure Groups Properly

6.2.6 Define ASM Disk and Failure Groups Properly

ASM uses a concept called failure groups to protect data against disk or component failures. When using failure groups, ASM optimizes file layout to reduce the unavailability of data due to the failure of a shared resource. Failure groups define disks that share components, so that if one disk fails, then other disks sharing the component might also fail. An example of what might be defined as a failure group is a string of SCSI disks on the same SCSI controller. Failure groups are used to determine which ASM disks to use for storing redundant data. For example, if 2-way mirroring is specified for a file, then redundant copies of file extents will be stored in separate failure groups.

Failure groups are used with storage that does not provide its own redundancy capability, such as disks that have not been configured according to RAID. The manner in which failure groups are defined for an ASM disk group is site-specific because the definition depends on the configuration of the storage and how it is connected to the database systems. ASM attempts to maintain three copies of its metadata, requiring a minimum of three failure groups for proper protection.

When using ASM with intelligent storage arrays, the storage array's protection features (such as hardware RAID-1 mirroring) are typically used instead of ASM's redundancy capabilities, which are implemented by using the EXTERNAL REDUNDANCY clause of the CREATE DISKGROUP statement. When using external redundancy, ASM failure groups are not used, because disks in an external redundancy disk group are presumed to be highly available.

Disks, as presented by the storage array to the operating system, should not be aggregated or subdivided by the array because it can hide the physical disk boundaries needed for proper data placement and protection. If, however, physical disk boundaries are always hidden by the array, and if each logical device, as presented to the operating system, has the same size and performance characteristics, then simply place each logical device in the same ASM disk group (with redundancy defined as EXTERNAL REDUNDANCY),and place the database and flash recovery areas in that one ASM disk group. An alternative approach is to create two disk groups, each concsisting of a single logical device, and place the database area in one disk group and the flash recovery in the other disk group. This method provides additional protection against disk group metadata failure and corruption.

For example, suppose a storage array takes eight physical disks of size 36GB and configures them in a RAID 0+1 manner for performance and protection, giving a total of 144GB of mirrored storage. Futhermore, this 144GB of storage is presented to the operating system as two 72GB logical devices with each logical device being striped and mirrored across all eight drives. When configuring ASM, place each 72GB logical device in the same ASM disk group, and place the database area and flash recovery areas on that disk group.

If the two logical devices have different performance characteristics,( for example, one corresponds to the inner half and the other to the outer half of the underlying physical drives) then the logical devices should be placed in separate disk groups. Because the outer portion of a disk has a higher transfer rate, the outer half disk group should be used for the database area; the inner half disk group should be used for the flash recovery area.

If multiple, distinct storage arrays are used with a database under ASM control, then multiple options are available. One option is to create multiple ASM disk groups that do not share storage across multiple arrays and place the database and flash recovery areas in separate disk groups, thus physically separating the database area from the flash recovery area. Another option is to create a single disk group across arrays that consists of failure groups, where each failure group contains disks from just one array.These options provide protection against the failure of an entire storage array.

6.2.7 Use HARD-Compliant Storage for the Greatest Protection Against Data Corruption

The Hardware Assisted Resilient Data (HARD) initiative is a program designed to prevent data corruptions before they happen. Data corruptions are very rare, but when they do occur, they can have a catastrophic effect on business. Under the HARD initiative, Oracle's storage partners implement Oracle's data validation algorithms inside storage devices. This makes it possible to prevent corrupted data from being written to permanent storage. The goal of HARD is to eliminate a class of failures that the computer industry has so far been powerless to prevent. RAID has gained a wide following in the storage industry by ensuring the physical protection of data; HARD takes data protection to the next level by going beyond protecting physical data to protecting business data.

In order to prevent corruptions before they happen, Oracle tightly integrates with advanced storage devices to create a system that detects and eliminates corruptions before they happen. Oracle has worked with leading storage vendors to implement Oracle's data validation and checking algorithms in the storage devices themselves. The classes of data corruptions that Oracle addresses with HARD include:

• Writes that physically and logically corrupt data file, control file and log file blocks

• Writes of Oracle blocks to incorrect locations

• Erroneous writes to Oracle data by programs other than Oracle

• Writes of partial or incomplete blocks

End-to-end block validation is the technology employed by the operating system or storage subsystem to validate the Oracle data block contents. By validating Oracle data in the storage devices, corruptions will be detected and eliminated before they can be written to permanent storage. This goes beyond the current Oracle block validation features that do not detect a stray, lost, or corrupted write until the next physical read.

Oracle vendors are given the opportunity to implement validation checks based on a specification. A vendors' implementation may offer features specific to their storage technology. Oracle maintains a Web site that will show a comparison of each vendor's solution by product and Oracle version. For the most recent information, see http://otn.oracle.com/deploy/availability/htdocs/HARD.html.

6.2.8 Storage Recommendation for RAC

The following recommendation applies to the "RAC only" architecture and MAA:

• Protect the Oracle Cluster Registry and Voting Disk From Media Failure

6.3.1 Server Hardware Recommendations for All Architectures

The following recommendations apply to all architectures:

• Use Fewer, Faster, and Denser Components

• Use Redundant Hardware Components

• Use Systems That Can Detect and Isolate Failures

• Protect the Boot Disk With a Backup Copy

6.3.2 Server Hardware Recommendations for RAC

The following recommendations apply to the "RAC only" and MAA environments:

• Use a Supported Cluster System to Run RAC

• Choose the Proper Cluster Interconnect

6.3.3 Server Hardware Recommendations for Data Guard

The following recommendation applies to both the primary and secondary sites in "Data Guard only" and MAA environments.

6.4.1 Server Software Recommendations for All Architectures

The following recommendations apply to all architectures:

• Use the Same OS Version, Patch Level, Single Patches, and Driver Versions

• Use an Operating System That is Fault-Tolerant to Hardware Failures

• Configure Swap Partititions Appropriately

• Set Operating System Parameters to Enable Future Growth

• Use Logging or Journal File Systems

• Mirror Disks That Contain Oracle and Application Software

6.4.2 Server Software Recommendations for RAC

The following recommendations apply to a "RAC only" environment:

• Use Supported Clustering Software

• Use Network Time Protocol (NTP) On All Cluster Nodes

6.5.1 Network Configuration Best Practices for All Architectures

The following recommendations apply to all architectures:

• Ensure That All Network Components Are Redundant

• Use Load Balancers to Distribute Incoming Requests

6.5.1.1 Ensure That All Network Components Are Redundant

Table 6-2 describes the necessary redundant network components that are illustrated in Figure 6-1.

Table 6-2 Redundant Network Components

Component	Fault Tolerance
Exterior connectivity (including remote IT staff)	Multiple internet service providers (ISP) ensure that an ISP failure does not make your site unavailable. Note: The cables should not be housed within the same trunk into your facility. Otherwise, a saw or shovel can cut connections to multiple ISPs simultaneously.
WAN traffic manager	Use a primary and a backup WAN traffic manager. (Not used with "Database only" or "RAC only" environments) Environments using Data Guard contain a primary and a backup WAN traffic manager at each site.
Application load balancer	Implement redundant application load balancers for directing client requests to the application server tier. The secondary load balancer should be identically configured with a heartbeat to the primary load balancer. If the primary load balancer fails, then the secondary load balancer initiates a takeover function.
Firewall	Implement redundant firewalls and load-balance across the pair. Because the connections through a firewall tend to be longer (socket connections versus HTTP), the load-balanceing decision needs to be made when the session is initiated and maintained during the session. Check with your firewall vendor for available functionality.
Middle-tier application server	Implement application server farms. This provides fault tolerance and scalability.

Figure 6-1 depicts a single site in an MAA environment, emphasizing the redundant network components.

Figure 6-1 Network Configuration

Description of the illustration maxav010.gif

6.5.2 Network Configuration Best Practices for RAC

This recommendation applies to RAC environments.

6.5.3 Network Configuration Best Practices for Data Guard

The following recommendations apply to Data Guard:

• Configure System TCP Parameters Appropriately

• Use WAN Traffic Managers to Provide Site Failover Capabilities