5 Adding and Deleting Nodes and Instances

Overview of Node Addition Procedures

This section explains how to add nodes to clusters. Do this by setting up the new nodes to be part of your cluster at the network level. Then extend the Cluster Ready Services (CRS) home from an existing CRS home to the new nodes and then extend the Oracle database software with RAC components to the new nodes. Finally, make the new nodes members of the existing cluster database.

If the nodes that you are adding to your cluster do not have clusterware or Oracle software, then you must complete the following five steps. The procedures in these steps assume that you already have an operative UNIX-based or Windows-based RAC environment. The details of these steps appear in the following sections.

• Step 1: Connecting New Nodes to the Cluster

• Step 2: Extending Clusterware and Oracle Software to New Nodes

• Step 3: Preparing Storage for RAC on New Nodes

• Step 4: Adding Nodes at the Oracle RAC Database Layer

• Step 5: Adding Database Instances to New Nodes

To add a node to your cluster when the node is already configured with clusterware and Oracle software, follow the procedure described in "Adding Nodes that Already Have Clusterware and Oracle Software to a Cluster".

Step 1: Connecting New Nodes to the Cluster

Complete the following procedures to connect the new nodes to the cluster and to prepare them to support your cluster database:

• Making Physical Connections

• Installing Operating System

• Creating Oracle Users

• Checking the Installation

Step 2: Extending Clusterware and Oracle Software to New Nodes

The following topics describe how to add new nodes to the clusterware and to the Oracle database software layers using the OUI:

• Adding Nodes at the Vendor Clusterware Layer (UNIX only)

• Adding Nodes at the Oracle Clusterware Layer (UNIX and Windows)

Step 3: Preparing Storage for RAC on New Nodes

To extend an existing RAC database to your new nodes, configure storage for the new nodes so that the storage is the same as on the existing nodes. For example, the Oracle Cluster Registry (OCR) and the voting disk must be accessible by the new nodes using the same path as the other nodes use. In addition, the OCR and voting disk devices must have the same permissions as on the existing nodes. Prepare the same type of storage on the new nodes as you are using on the other nodes in the RAC environment that you want to extend as follows:

• Automatic Storage Management (ASM)

  If you are using ASM, then make sure that the new nodes can access the ASM disks with the same permissions as the existing nodes.

• Oracle Cluster File System (OCFS)

  If you are using Oracle Cluster File Systems, then make sure that the new nodes can access the cluster file systems in the same way that the other nodes access them.

• Vendor Cluster File Systems

  If your cluster database uses vendor cluster file systems, then configure the new nodes to use the vendor cluster file systems. Refer to the vendor clusterware documentation for the pre-installation steps for your platform.

• Raw Device Storage

  If your cluster database uses raw devices, then prepare the new raw devices by following the procedures described in the following section.

Step 4: Adding Nodes at the Oracle RAC Database Layer

To add nodes at the Oracle RAC database later, run the OUI in add node mode to configure your new nodes. If you have multiple Oracle homes, then perform the following steps for each Oracle home that you want to include on the new nodes:

1. On an existing node from the $ORACLE_HOME/oui/bin directory on UNIX-based systems, run the addNode.sh script. From the %ORACLE_HOME%\oui\bin on Windows-based systems, run the addNode.bat script. This starts the OUI in the add node mode and displays the OUI Welcome page. Click Next on the Welcome page and the OUI displays the Specify Cluster Nodes for Node Addition page.

2. The Specify Cluster Nodes for Node Addition page has a table showing the existing nodes associated with the Oracle home from which you launched the OUI. A node selection table appears on the bottom of this page showing the nodes that are available for addition. Select the nodes that you want to add and click Next.

3. The OUI verifies connectivity and performs availability checks on both the existing nodes and on the nodes that you want to add. Some of checks performed determine whether:

4. • The nodes are up

   • The nodes are accessible by way of the network

   • The user has write permission to create the Oracle home on the new nodes

   • The user has write permission to the OUI inventory in the oraInventory directory on UNIX or the Inventory directory on Windows on the existing nodes and on the new nodes

5. If the new nodes do not have an inventory set up, then on UNIX-based systems the OUI displays a dialog asking you to run the oraInstRoot.sh script on the new nodes. On Windows-based systems the OUI automatically updates the Registry entries for the inventory location. If any of the other checks fail, then fix the problem and proceed or deselect the node that has the error and proceed. You cannot deselect existing nodes; you must correct problems on the existing nodes before proceeding with node addition. If all of the checks succeed then the OUI displays the Node Addition Summary page.

6. The Node Addition Summary page has the following information about the products that are installed in the Oracle home that you are going to extend to the new nodes:

   • The source for the add node process, which in this case is the Oracle home

   • The existing nodes and new nodes

   • The new nodes that you selected

   • The required and available space on the new nodes

   • The installed products listing all the products that are already installed in the existing Oracle home

   Click Finish and the OUI displays the Cluster Node Addition Progress page.

7. The Cluster Node Addition Progress page shows the status of the cluster node addition process. The table on this page has two columns showing the phase of the node addition process and the phase's status according to the following platform-specific content:

   On UNIX-based systems, the Cluster Node Addition Progress page shows the following four OUI phases:

   • Instantiate Root Scripts—Instantiates the root.sh script in the Oracle home by copying it from the local node

   • Copy the Oracle Home to the New Nodes—Copies the entire Oracle home from the local node to the new nodes unless the Oracle home is on a cluster file system

   • Run root.sh—Displays the dialog prompting you to run root.sh on the new nodes

   • Save Cluster Inventory—Updates the node list associated with the Oracle home and its inventory

   On Windows-based systems, the Cluster Node Addition Progress shows the following three OUI phases:

   • Copy the Oracle Home To New Nodes—Copies the entire Oracle home to the new nodes unless the Oracle home is on a cluster file system

   • Performs Oracle Home Setup—Updates the Registry entries for the new nodes, creates the services, and creates folder entries

   • Save Cluster Inventory—Updates the node list associated with the Oracle home and its inventory

   For all platforms, the Cluster Node Addition Progress page's Status column displays Succeeded if the phase completes, In Progress if the phase is in progress, and Suspended when the phase is pending execution. After the OUI displays the End of Node Addition page, click Exit to end the OUI session.

8. On UNIX-based systems only, run the root.sh script.

9. Execute the VIPCA utility from the bin subdirectory of the Oracle home using the -nodelist option with the following syntax that identifies the complete set of nodes that are now part of your RAC database beginning with Node1 and ending with NodeN:

   vipca -nodelist Node1,Node2,Node3,...NodeN
   
   

   
   

   Note: You must run the VIPCA utility as root user on UNIX-based systems, and as a user with Administrative privileges on Windows-based systems.

10. If the private interconnect interface names on the new nodes are not the same as the interconnect names that are on the existing nodes, then change the private interconnect configuration for the new nodes as described in this step, otherwise, proceed to step 10. Change the configuration by executing the oifcfg utility with the setif option from the bin directory of the Oracle home using the following syntax where subnet is the subnet for the private interconnect of the RAC databases to which you are adding nodes. Specify the -n nodename option to enter node-specific configuration information for the new nodes. The syntax for oifcfg commands are as follows:

    oifcfg iflist
    
    oifcfg setif {-node nodename | -global} {if_name/subnet:if_type}...
    
    oifcfg getif [-node nodename | -global] [ -if if_name[/subnet] [-type if_type] ]
    oifcfg delif [-node nodename | -global] [if_name[/subnet]]
    
    oifcfg [-help]
    

    A Cluster Ready Services (CRS) installation issues the oifcfg command as in the following example:

    oifcfg setif -global  eth0/146.56.76.0:public eth1/192.0.0.0:cluster_interconnect 
    
    

    This sets both networks to global. Therefore, you do not need to run the oifcfg command manually after you add a node unless the network interfaces differ.

11. Add a listener to the new node by running the Net Configuration Assistant (NetCA).

After completing the procedures in the previous section, you have defined the new nodes at the cluster database layer. You can now add database instances to the new nodes as described in Step 5.

Step 5: Adding Database Instances to New Nodes

Execute the following procedures for each new node to add instances:

1. Start the Database Configuration Assistant (DBCA) by entering dbca at the system prompt from the bin directory in the $ORACLE_HOME on UNIX. On Windows-based systems, choose Start > Programs > Oracle - HOME_NAME > Configuration and Migration Tools > Database Configuration Assistant.

   The DBCA displays the Welcome page for RAC. Click Help on any DBCA page for additional information.

2. Select Real Application Clusters database, click Next, and the DBCA displays the Operations page.

3. Select Instance Management, click Next, and the DBCA displays the Instance Management page.

4. Select Add Instance and click Next. The DBCA displays the List of Cluster Databases page that shows the databases and their current status, such as ACTIVE, or INACTIVE.

5. From the List of Cluster Databases page, select the active RAC database to which you want to add an instance. If your user ID is not operating-system authenticated, then the DBCA prompts you for a user ID and password for a database user that has SYSDBA privileges. If the DBCA prompts you, then enter a valid user ID and password and click Next. The DBCA displays the List of Cluster Database Instances page showing the names of the existing instances for the RAC database that you selected.

6. Click Next to add a new instance and the DBCA displays the Adding an Instance page.

7. On the Adding an Instance page, enter the instance name in the field at the top of this page if the instance name that the DBCA provides does not match your existing instance name sequence. Then select the new node name from the list, click Next, and the DBCA displays the Services Page.

8. Enter the services information for the new node's instance, click Next, and the DBCA displays the Instance Storage page.

9. If you are using raw devices or raw partitions, then on the Instance Storage page select the Tablespaces folder and expand it. Then select the undo tablespace storage object and a dialog appears on the right-hand side. Change the default datafile name to the raw device name for the tablespace.

10. If you are using raw devices or raw partitions or if you want to change the default redo log group file name, then on the Instance Storage page select and expand the Redo Log Groups folder. For each redo log group number that you select, the DBCA displays another dialog box.

    • • For UNIX-based systems, enter the raw device name that you created in the section "Configure Raw Storage on UNIX-Based Systems" in the File Name field.

      • On Windows-based systems, enter the symbolic link name that you created in the section, "Configure Raw Partitions on Windows-Based Systems".

11. If you are using a cluster file system, then click Finish on the Instance Storage page. If you are using raw devices on UNIX-based systems or disk partitions on Windows-based systems, then repeat step 10 for all of the other redo log groups, click Finish, and the DBCA displays a Summary dialog.

12. Review the information on the Summary dialog and click OK. Or click Cancel to end the instance addition operation. The DBCA displays a progress dialog showing the DBCA performing the instance addition operation. When the DBCA completes the instance addition operation, the DBCA displays a dialog asking whether you want to perform another operation.

13. Click No and exit the DBCA, or click Yes to perform another operation. If you click Yes, then the DBCA displays the Operations page.

    After you have completed the procedures in this section, the DBCA has successfully added the new instance to the new node and completed the following steps:

    • Created and started an ASM instance on each new node if the existing instances were using ASM

    • Created a new database instance on each new node

    • For Windows-based systems, created and started the required services

    • Created and configured high availability components

    • Configured and started node applications for the GSD, Oracle Net Services listener, and Enterprise Manager agent

    • Created the Oracle Net configuration

    • Started the new instance

    • Created and started services if you entered services information on the Services Configuration page

    After adding the instances to the new nodes using the steps described in this section, perform any needed service configuration procedures as described in Chapter 4, "Administering Services".

Adding Nodes that Already Have Clusterware and Oracle Software to a Cluster

To add nodes to a cluster that already have clusterware and Oracle software installed on them, you must configure the new nodes with the Oracle software that is on the existing nodes of the cluster. To do this, you must run two versions of an OUI process: one for the clusterware and one for the database layer as described in the following procedures:

1. Add new nodes at the Oracle clusterware layer by running the OUI from the CRS home on an existing node according to the following platform-specific procedures:

   • On UNIX execute the following command:

     CRS home/oui/bin/addNode.sh -noCopy 
     

   • On Windows execute the following command:

     CRS home\oui\bin\addNode.bat -noCopy 
     
     

2. Add new nodes at the Oracle software layer by running the OUI from the Oracle home as follows:

   • On UNIX, execute:

     $ORACLE_HOME/oui/bin/addNode.sh -noCopy 
     
     

     On Windows, execute

     %ORACLE_HOME%\oui\bin\addNode.bat -noCopy 
     
     

In the -noCopy mode, the OUI performs all add node operations except for the copying of software to the new nodes.

Alert: The following file(s) have been modified on the disk:
y:\oracle\rac\inventory\ContentsXML\comps.xml y:\oracle\rac\inventory
\ContentsXML\libs.xml
Proceeding with the installation may corrupt some important data. You should 
stop this session and restart OUI. Do you want to stop this session now?

Deleting Instances from Real Application Clusters Databases

The procedures in this section explain how to use the DBCA to delete an instance from a RAC database. To delete an instance:

1. Start the DBCA on a node other than the node that hosts the instance that you want to delete. On the DBCA Welcome page select Oracle Real Application Clusters Database, click Next, and the DBCA displays the Operations page.

2. On the DBCA Operations page, select Instance Management, click Next, and the DBCA displays the Instance Management page.

3. On the Instance Management page, Select Delete Instance, click Next, and the DBCA displays the List of Cluster Databases page.

4. Select a RAC database from which to delete an instance. If your user ID is not operating-system authenticated, then the DBCA also prompts you for a user ID and password for a database user that has SYSDBA privileges. If the DBCA prompts you for this, then enter a valid user ID and password. Click Next and the DBCA displays the List of Cluster Database Instances page. The List of Cluster Database Instances page shows the instances associated with the RAC database that you selected and the status of each instance.

5. Select a remote instance to delete and click Finish.

6. If you have services assigned to this instance, then the DBCA Services Management page appears. Use this feature to reassign services from this instance to other instances in the cluster database.

7. Review the information about the instance deletion operation on the Summary page and click OK. Otherwise, click Cancel to cancel the instance deletion operation. If you click OK, then the DBCA displays a Confirmation dialog.

8. Click OK on the Confirmation dialog to proceed with the instance deletion operation and the DBCA displays a progress dialog showing that the DBCA is performing the instance deletion operation. During this operation, the DBCA removes the instance and the instance's Oracle Net configuration. When the DBCA completes this operation, the DBCA displays a dialog asking whether you want to perform another operation.

9. Click No and exit the DBCA or click Yes to perform another operation. If you click Yes, then the DBCA displays the Operations page.

At this point, you have accomplished the following:

• De-registered the selected instance from its associated Oracle Net Services listeners

• Deleted the selected database instance from the instance's configured node

• For Windows-based systems, deleted the selected instance's services

• Removed the Oracle Net configuration

• Deleted the Oracle Flexible Architecture directory structure from the instance's configured node.

Deleting Nodes from Oracle Clusters on UNIX-Based Systems

Use the following procedures to delete nodes from Oracle clusters on UNIX-based systems:

1. If there are instances on the node that you want to delete, then execute the procedures in the section titled Deleting Instances from Real Application Clusters Databases before executing these procedures. If you are deleting more than one node, then delete the instances from all the nodes that you are going to delete.

2. If you use ASM, then perform the procedures in the following section, "ASM Instance Clean-Up Procedures for Node Deletion".

3. If this is the Oracle home from which the node-specific listener named LISTENER_nodename runs, then use NetCA to remove this listener and its CRS resources. If necessary, re-create this listener in another home.

4. After you have deleted the instances from the nodes that you want to delete, delete the node applications for each node by running the following command from the Oracle home (not the CRS home) where node1, node2 are the nodes that you are removing from your cluster:

   rootdeletenode.sh node1,node2
   
   

5. On the same node that you are deleting, execute the command runInstaller -updateNodeList ORACLE_HOME=Home location CLUSTER_NODES=node1,node2,...nodeN where node1 through nodeN is a comma-delimited list of nodes that are remaining in the cluster. This list must exclude the nodes that you are deleting. The runInstaller command is located in the directory $ORACLE_HOME/oui/bin. Executing this command does not launch an installer GUI.

6. If you are not using a cluster file system for the Oracle home, then on the node that you are deleting, remove the Oracle database software by executing the rm command. Make sure that you are in the correct Oracle home of the node that you are deleting when you execute the rm command. Execute this command on all the nodes that you are deleting.

7. On the node that you are deleting, run the command CRS Home/install/rootdelete.sh to disable the CRS applications that are on the node. Only run this command once and use the nosharedhome argument if you are using a local file system. The default for this command is sharedhome which prevents you from updating the permissions of local files such that they can be removed by the oracle user.

   If the ocr.loc file is on a shared file system, then execute the command CRS home/install/rootdelete.sh remote sharedvar. If the ocr.loc file is not on a shared file system, then execute the CRS home/install/rootdelete.sh remote nosharedvar command.

   If you are deleting more than one node from your cluster, then repeat this step on each node that you are deleting.

8. Run CRS Home/install/rootdeletenode.sh on any remaining node in the cluster to delete the nodes from the Oracle cluster and to update the Oracle Cluster Registry (OCR). If you are deleting multiple nodes, then execute the command CRS Home/install/rootdeletenode.sh node1,node1-number,node2,node2-number,... nodeN,nodeN-number where node1 through nodeN is a list of the nodes that you want to delete, and node1-number through nodeN-number represents the node number. To determine the node number of any node, execute the command CRS Home/bin/olsnodes -n. To delete only one node, enter the node name and number of the node that you want to delete. <CRS home>/install/rootdeletenode.sh node1,node1-number

9. On the same node and as the oracle user, run the command CRS home/oui/bin/runInstaller -updateNodeList ORACLE_HOME=CRS home CLUSTER_NODES=node1,node2,... nodeN where node1 through nodeN is a comma-delimited list of nodes that are remaining in the cluster.

10. If you are not using a cluster file system, then on the node that you are deleting, remove the Oracle CRS software by executing the rm command. Make sure that you execute the rm command from the correct Oracle CRS home. Execute the rm command on every node that you are deleting.

Deleting Nodes from Oracle Clusters on Windows-Based Platforms

Execute the following procedures to delete nodes from Oracle clusters on Windows-based platforms.

Perform the following steps on a node other than the node you want to delete:

1. Use the Database Configuration Assistant (DBCA) to delete the instance.

2. Use the Net Configuration Assistant (NetCA) to delete the listener.

3. If the node that you are deleting has an ASM instance, then delete the ASM instance using the srvctl stop asm and srvctl remove asm commands.

4. Run the command srvctl stop nodeapps -n nodename of the node to be deleted to stop the node applications.

5. Run the command srvctl remove nodeapps -n nodename of the node to be deleted to remove the node applications.

6. Stop isqlplus if it is running.

7. Run the command setup.exe -updateNodeList ORACLE_HOME=Oracle_home ORACLE_HOME_NAME=Oracle_home_name CLUSTER_NODES=remaining nodes where remaining nodes is a list of the nodes that are to remain part of the cluster.

Perform the following steps on the deleted RAC node:

1. Run the command setup.exe -updateNodeList -local -noClusterEnabled ORACLE_HOME=Oracle_home ORACLE_HOME_NAME=Oracle_home_name CLUSTER_NODES="". Note that you do not need a value for "" after the CLUSTER_NODES= entry in this command. If you delete more than one node, then you must run this command on every deleted node to remove the Oracle home if you have a non-shared Oracle home (non-cluster file system) installation.

2. On the same node, delete the Windows Registry entries and ASM services using Oradim.

3. From the deleted RAC node, run the command Oracle_home\oui\bin\setup.exe to start the Oracle Universal Installer (OUI). Select Deinstall Products and select the Oracle home that you want to de-install.

4. Then to delete the CRS node, from a remaining node run the command crssetup del -nn node_name of the deleted node, node number

5. Then run the command setup.exe -updateNodeList ORACLE_HOME=CRS home ORACLE_HOME_NAME=CRS home name CLUSTER_NODES=remaining nodes where remaining nodes is a list of the nodes that are to remain in the cluster.

6. Then on the deleted CRS node, run the command setup.exe -updateNodeList -local -noClusterEnabled ORACLE_HOME=CRS home ORACLE_HOME_NAME=CRS home name CLUSTER_NODES=""

7. Remove the Oracle home manually from the new node if the home is not shared and then manually remove the HKLM/software/Oracle registry keys and the Oracle services. 7

8. After adding or deleting nodes from your Oracle Database 10g with RAC environment, and after you are sure that your system is functioning properly, make a backup of the contents of the voting disk using the dd.exe utility. The dd.exe utility is part of the MKS toolkit.