2 Administering Oracle Clusterware

Backing Up Voting Disks

Oracle recommends that you back up your voting disk after the initial cluster creation and after you complete any node addition or deletion procedures.

First, as root user, stop Oracle Clusterware (with the crsctl stop crs command) on all nodes. Then, determine the current voting disk by issuing the following command:

crsctl query votedisk css 

Then, issue the dd or ocopy command to back up a voting disk, as appropriate. In the following examples, voting_disk_name is the name of the active voting disk and backup_file_name is the name of the file to which you want to back up the voting disk contents:

• On Linux or UNIX systems:

  dd if=voting_disk_name of=backup_file_name
  

  Oracle recommends you use the dd command to backup the voting disk with a minimum block size of 4KB.

• On Windows systems, use the ocopy command:

  ocopy voting_disk_name backup_file_name
  

  To display online documentation for OCOPY, enter OCOPY by itself at the Windows prompt.

Recovering Voting Disks

To restore the backup of your voting disk, issue the dd or ocopy command.

In the following examples, backup_file_name is the name of the voting disk backup file and voting_disk_name is the name of the active voting disk:

• On Linux or UNIX systems:

  dd if=backup_file_name of=voting_disk_name
  

  Oracle recommends you use the dd command to backup the voting disk with a minimum block size of 4KB.

• On Windows systems, use the ocopy command:

  ocopy backup_file_name voting_disk_name
  

If you have multiple voting disks, then you can remove the voting disks and add them back into your environment using the following commands (where path is the complete path of the location where the voting disk resides):

crsctl delete votedisk css path
crsctl add votedisk css path

Adding, Moving, or Deleting Voting Disks

You can add, move, and remove voting disks after Oracle Clusterware has been installed. Before making any modification to the voting disk, as root user, stop Oracle Clusterware using the crsctl stop crs command on all nodes. The following commands describe how to add, move, or remove voting disks:

• To retrieve the current voting disk, issue the following command:

  crsctl query css votedisk
  

  This command lists the voting disks used by Cluster Synchronization Services (CSS).

• To add a voting disk, issue the following command as the root user, replacing the path variable with the fully qualified path name for the voting disk you want to add:

  crsctl add votedisk css path -force
  

• To move a voting disk, issue the following commands as the root user, replacing the path variable with the fully qualified path name for the voting disk you want to move:

  crsctl delete votedisk css path -force
  crsctl add votedisk css path -force
  

• To remove a voting disk, issue the following command as the root user, replacing the path variable with the fully qualified path name for the voting disk you want to remove:

  crsctl delete votedisk css path -force
  

  Note:

  If your cluster is down, then you can include the -force option to modify the voting disk configuration, without interacting with active Oracle Clusterware daemons. However, using the -force option while any cluster node is active may corrupt your configuration.

After modifying the voting disk, restart Oracle Clusterware using the crsctl start crs command on all nodes, and verify the voting disk location using the following command:

crsctl query css votedisk css

Adding, Replacing, Repairing, and Removing the Oracle Cluster Registry

The Oracle installation process for Oracle Clusterware gives you the option of automatically mirroring the OCR. You can put the mirrored OCR on an Oracle cluster file system disk, on a shared raw device, or on a shared raw logical volume. In addition, Oracle supports block devices for the OCR on Linux.

You can manually mirror the OCR, as described in the "Adding an Oracle Cluster Registry" section, if you:

• Upgraded to release 11.1 but did not choose to mirror the OCR during the upgrade

• Created only one OCR during the Oracle Clusterware installation

In addition to mirroring the OCR, you can also:

• Replace the OCR if Oracle displays an OCR failure alert in Enterprise Manager or in the Oracle Clusterware alert log file, as described in the "Replacing an Oracle Cluster Registry" section.

• Repair an OCR location if there is a misconfiguration or other type of OCR error, as described in the "Repairing an Oracle Cluster Registry Configuration on a Local Node" section.

• Remove an OCR location if, for example, your system experiences a performance degradation due to OCR processing or if you transfer your OCR to RAID storage devices and chose to no longer use multiple OCRs. This is described in the "Removing an Oracle Cluster Registry" section.

ocrconfig -replace ocr destination_file or disk

ocrconfig -replace ocrmirror destination_file or disk

ocrconfig –repair ocrmirror device_name 

Managing Backups and Recovering the Oracle Cluster Registry

This section describes how to backup OCR content and use it for recovery. The first method uses automatically generated OCR file copies and the second method enables you to issue a backup command on demand:

• Automatic backups—Oracle Clusterware automatically creates OCR backups every four hours. At any one time, Oracle always retains the last three backup copies of the OCR. The CRSD process that creates the backups also creates and retains an OCR backup for each full day and at the end of each week. You cannot customize the backup frequencies or the number of files that Oracle retains.

• Manual backups—Use the ocrconfig -manualbackup command to force Oracle Clusterware to perform a backup of the OCR at any time, rather than wait for the automatic backup that occurs at 4-hour intervals. The -manualbackup option is especially useful when you to need to obtain a binary backup on demand, such as before you make changes to the OCR.

You can use any backup software to copy the automatically generated backup files at least once daily to a different device from where the primary OCR resides.

The default location for generating backups on Linux or UNIX systems is CRS_home/cdata/cluster_name where cluster_name is the name of your cluster. The Windows default location for generating backups uses the same path structure. Because the default backup is on a local file system, Oracle recommends that you include the backup file created with the ocrconfig command as part of your operating system backup using standard operating system or third-party tools.

You can issue the ocrconfig -showbackup command to display the backup location, timestamp, and the originating node name of the backup files that Oracle created. By default, the -showbackup option displays information for both automatic and manual backups but you can include the auto or manual flag to display only the automatic backup information or only the manual backup information, respectively.

Diagnosing Oracle Cluster Registry Problems

You can use the OCRDUMP and OCRCHECK utilities to diagnose OCR problems as described under the following topics:

• Using the OCRDUMP Utility

• Using the OCRCHECK Utility

Administering the Oracle Cluster Registry with OCR Export and Import Commands

In addition to using the automatically created OCR backup files, you should also export the OCR contents before and after making significant configuration changes, such as adding or deleting nodes from your environment, modifying Oracle Clusterware resources, or creating a database. Do this by using the ocrconfig -export command. This exports the OCR content to a file format.

Using the ocrconfig -export command enables you to restore the OCR using the -import option if your configuration changes cause errors. For example, if you have unresolvable configuration problems, or if you are unable to restart your clusterware after such changes, then restore your configuration using one of the following platform-specific procedures:

• Importing Oracle Cluster Registry Content on Linux or UNIX Systems

• Importing Oracle Cluster Registry Content on Windows Systems

Implementing the Oracle HARD Initiative for the Oracle Cluster Registry

The Oracle Hardware Assisted Resilient Data (HARD) initiative prevents data corruptions from being written to permanent storage. If you enable HARD, then the OCR writes HARD-compatible blocks. To determine whether the device used by the OCR supports HARD and then enable it, review the Oracle HARD white paper at:

http://www.oracle.com/technology/deploy/availability/htdocs/HARD.html

Upgrading and Downgrading the Oracle Cluster Registry Configuration

When you install Oracle Clusterware, Oracle automatically runs the ocrconfig -upgrade command. To downgrade, follow the downgrade instructions for each component and also downgrade the OCR using the ocrconfig -downgrade command. If you are upgrading the OCR, then you can use the ocrcheck command to verify the integrity of the OCR.

Changing Public Network Addresses

Clients use the Virtual Internet Protocol (VIP) address to connect to the Oracle RAC database instances. The VIP address resolves all timeouts entering the TCP/IP stack, resulting in higher application availability. The VIP address is a static IP address that defines and resolves to a hostname through the Domain Name System (DNS).

During the installation of Oracle Clusterware, you are prompted to enter a VIP address and a virtual hostname for each node in the cluster. These items are stored in the OCR and different components in the high availability framework depend on the VIP addresses.

Changing the VIP address requires modification of the node applications, including the VIP address, Global Services Daemon (GSD), the listener, and Oracle Notification Services (ONS). You can modify the VIP address while the node applications are running. However, the changes do not take effect until you restart the VIP address (hence, restarting the node applications). Other resources on a node, such as database instances and ASM instances, are dependent on the VIP address. Therefore, stopping the node applications takes the VIP address offline only if you stop the other resources.

Perform the following steps to change a VIP address:

1. Stop all database and ASM instances. Stop all resources that are dependent on the VIP address on a given node. This includes all Oracle RAC database instances on that node, and the ASM instance on that node, if it exists:

   1. Stop database instances:

      srvctl stop instance -d grid -i grid1
      

      This example specifies the database name (grid) using the -d option and specifies the instance (grid1) on the appropriate node using the -i option.

      Alternatively, to stop the entire database, on all nodes, you can issue the stop database command, as follows:

      srvctl stop database -d grid
      

   2. To stop the ASM instance on the node, issue the following command:

      srvctl stop asm -n mynode
      

      Note:

      Alternatively, you can stop these resources using SQL*Plus statements, or on Windows systems you can stop the associated services.

2. Confirm the current IP address for the VIP by issuing the ifconfig -a command. This command displays the current VIP bound to one of the network interfaces. The following example displays the current VIP address that is bound to one of the network interfaces (eth0:1):

   eth0:1    Link encap:Ethernet  HWaddr 00:01:03:2C:69:BB  
             inet addr:192.168.1.125  Bcast:192.168.1.255  Mask:255.255.255.0
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1   
             ...
             ...
   

   Note:

   On Windows systems, issue the ipconfig /all command, to list all IP addresses bound to a given adapter, including the VIP address.

3. Stop node applications using the srvctl stop nodeapps command. For example:

   srvctl stop nodeapps -n mynode
   

   This command stops the VIP address, the GSD, the listener, and the ONS daemon currently running on the specified node.

4. Verify that the VIP address is no longer running by issuing the ifconfig -a command on Linux and UNIX systems (or issue the ipconfig /all command on Windows systems), and confirm that the interface (in the example it was eth0:1) is no longer listed in the output.

   If the interface is still online, it indicates that a resource, which is dependent on the VIP address, is still running. Use the crs_stat command to show resources that are still online.

5. Make any changes necessary to the /etc/hosts files on all nodes on Linux and Unix systems, or the %SystemRoot%\system32\drivers\etc\hosts file on Windows systems, and make any necessary DNS changes to associate the new IP address with the old host name.

6. Modify the node applications and provide the new VIP address using the following srvctl modify nodeapps command syntax:

   srvctl modify nodeapps -n node_name; [-o oracle_home] [-A new_vip_address]
   

   In the syntax statement include the following options:

   • -n node_name is the node name

   • -o Oracle_home is the Oracle home for the cluster database.

   • -A new_vip_address is the node-level VIP address: name|ip/netmask[/if1[|if2|...]])

     For example, issue the following command as the root user:

     srvctl modify nodeapps -n mynode -A 192.168.2.125/255.255.255.0/eth0
     

     Attempting to issue this command as the oracle user results in the PRKO-2117: This command should be executed as the system privilege user error

7. Start node-level applications by issuing the srvctl start nodeapps command:

   srvctl start nodeapps -n node_name
   

   The following command example starts applications on the node named mynode:

   srvctl start nodeapps -n mynode
   

8. Repeat the steps for each node in the cluster.

   Because the SRVCTL utility is a clusterwide management tool, you can accomplish these tasks for any specific node from any node in the cluster, without the need to log in to each of the cluster nodes.

9. Run the following command to verify node connectivity between all of the nodes for which your cluster is configured. This command discovers all of the network interfaces available on the cluster nodes and verifies the connectivity between all of the nodes by way of the discovered interfaces. This command also lists all of the interfaces available on the nodes which are suitable for use as VIP addresses.

   cluvfy comp nodecon -n node_list all [-verbose]
   

Changing Private Network Addresses

Oracle Clusterware requires that each node is connected through a private network (in addition to the public network). The private network connection is referred to as the cluster interconnect. During the installation of Oracle Clusterware, you are prompted to enter a private network IP address for each node in the cluster. This address—unlike the VIP addresses—must be assigned already to a certain network interface on each node.

In addition, you must provide a private node name, which resolves to the IP address given, and is usually stored in the hosts file of the operating system. This private node name is stored automatically in the OCR. Oracle does not recommend using Domain Name Services (DNS) to resolve private network IP address names.

Oracle currently only supports clusters in which all nodes use the same network interface connected to the same subnet (defined as a global interface with the oifcfg command). You cannot use different network interfaces for each node (node-specific interfaces). See Appendix C, " Oracle Interface Configuration (OIFCFG) Command Reference" for more information about global and node-specific interfaces.

Table 2-1 describes how the Network Interface Card (NIC), the private IP address, and the private node name are stored.

The following topics describe how to change a private network address, the network interface, or both.

To change a private network IP address:

If you want to change a private network IP address on one node and the new IP address is on the same subnet as the former IP address, perform the following steps:

1. Stop Oracle Clusterware by issuing the following command as root user on the node on which you want to change the IP address:

   crsctl stop crs
   

2. Assign a new network address to the Network Interface Card (NIC).^Foot 1 

   As root user, assign a new network address to the NIC using the ifconfig operating system command. Ensure the new private IP address is in the same subnet as the private IP addresses of the remaining nodes in the cluster.

3. Update the hosts file.

   The private network address is a static IP address. Any changes that you make to the private network address must be reflected in the hosts file of the operating system.

   Note:

   Oracle does not recommend that you use the Domain Name System (DNS) for the private node name resolution. However, if you use DNS, you must change the DNS entries accordingly.

4. Restart Oracle Clusterware by issuing the following command as root user

   crsctl start crs
   

   The changes take effect when Oracle Clusterware restarts. To change more than one private IP address on more than one node in the cluster, repeat steps 1 through 3 in a rolling manner for each of those nodes.

To change a private network IP address so that a new subnet is used:

If you want to change a private network IP address so that a new subnet is used, you must change all IP addresses on all cluster nodes to reflect the change of the IP subnet. Perform the following steps to change the subnet:

1. On all nodes, stop Oracle Clusterware by issuing the following command as root user:

   crsctl stop crs
   

2. Assign a new network address from the new subnet to the NIC.^Footref 1

   As root user, assign a new network address to the NIC using the ifconfig operating system command. Ensure all private IP addresses are taken from the same subnet.

3. Restart Oracle Clusterware by issuing the following command as root user

   crsctl start crs
   

   The changes take effect when Oracle Clusterware restarts.

4. Issue the oifcfg command to change and store the new private IP address subnet in the OCR. See Appendix C for more information about using the OIFCFG command. The IP address changes take effect when Oracle Clusterware restarts.

If you use the CLUSTER_INTERCONNECTS initialization parameter, you must update it to reflect the changes. If you do not use the CLUSTER_INTERCONNECT parameter, then skip this step.

The private network address is a static IP address. Any changes that you make to the private network address must be reflected in the hosts file of the operating system.

To change the Network Interface Card (NIC) for the interconnect:

If you want to change the network interface for the private interconnect (for example, eth1), you must perform the change on all nodes (globally). This is because Oracle currently does not support the use of different network interface cards in the same subnet for the cluster interconnect.

Perform the following steps:

1. On all nodes, stop Oracle Clusterware by issuing the following command as root user:

   crsctl stop crs
   

2. Assign the current network address to the new NIC using the ifconfig command.

   As root user, issue the ifconfig operating system command to assign the currently used private network address to the NIC intended to be used for the interconnect. This usually requires some downtime for the current interface and the new interface. See your platform-specific operating system documentation for more information about issuing the ifconfig command.

3. Restart Oracle Clusterware by issuing the following command as root user on all nodes:

   crsctl start crs
   

4. Issue the oifcfg command to change and store the new private network/private IP address subnet assignment in the OCR. See Appendix C for more information about using the OIFCFG command. The changes take effect when Oracle Clusterware restarts.

If you want to change both the private IP address and the network interface card in one step, assign the new IP address to the new NIC and then perform the remaining steps accordingly.