Configuring CVM and Oracle Service Groups

The CVM and Oracle service groups can be configured using the following two methods:

• By editing the VCS configuration file, main.cf, to define the service groups; see Configuring CVM and Oracle Service Groups Manually.
• By using a configuration wizard for Oracle RAC; see Configuring the Service Groups Using the Wizard.

Configuring CVM and Oracle Service Groups Manually

You can edit the main.cf file directly to configure the CVM and Oracle service groups. If you would prefer to use the VCS RAC wizard to perform the configuration, see Configuring the Service Groups Using the Wizard.

1. Log in to one system as root.
2. Save your existing configuration to prevent any changes while you modify main.cf:
     # haconf -dump -makero
3. Ensure VCS is not running while you edit main.cf by using the hastop command to stop the VCS engine on all systems and leave the resources available:
     # hastop -all -force
4. Make a backup copy of the main.cf file:
   # cd /etc/VRTSvcs/conf/config 
   # cp main.cf main.orig

Using vi or another text editor, edit the main.cf file and modify the cvm service group and create Oracle service groups using the guidelines in the following sections.

Two example configuration are shown for modifying the CVM service group: Example 1: CVM Group with Oracle Binaries on CFS, shown below, and Example 2: CVM Group with Oracle Binaries Installed Locally. Adding the Oracle database service group is the same for either configuration. See Creating Oracle Service Group in the main.cf File

Example 1: CVM Group with Oracle Binaries on CFS

For this example, you can refer to the illustration shown in CVM Group with Oracle Binaries and SRVM on CFS; Virtual IP.

Modifying the CVM Service Group in the main.cf File

The cvm service group is created during the installation of Storage Foundation for Oracle RAC. After installation, the main.cf file resembles the example shown in Sample main.cf: After SFRAC Installation and Before Oracle9i Installation. Because Oracle had not been installed, the cvm service group includes only resources for the CFSfsckd daemon and the CVMCluster resource.

You must modify the cvm service group to add the Netlsnr, NIC, IP, CVMVolDg, and CFSMount resources to the configuration. You can refer to Sample Configuration Files to see a complete example of how a cvm group is configured.

1. Make sure the cvm group has the group Parallel attribute set to 1. Typically this is already done during installation.
     .
     group cvm (
           SystemList = { galaxy = 0, nebula = 1 }
           AutoFailOver = 0
           Parallel = 1
           AutoStartList = { galaxy, nebula }
           )
     .
2. Define the NIC and IP resources. The VCS bundled NIC and IP agents are described in VERITAS Cluster Server Bundled Agents Reference Guide. The device name and the IP addresses are required by the listener for public network communication. Note that for the IP resource, the Address attribute is localized for each node (see Attributes of CVM, Oracle Groups Defined as Local).
     .
           NIC listener_NIC (
                 Device = lan0
                 NetworkType = ether
          )
   
          IP listener_ip (
                 Device = lan0
                 Address @galaxy = "192.2.40.21"
                 Address @nebula = "192.2.40.22"
          )
    . 
3. Define the Netlsnr resource. The Netlsnr listener agent is described in detail in the VERITAS Cluster Server Enterprise Agent for Oracle, Installation and Configuration Guide. Note that the listener attribute is localized for each node.
     .
           Netlsnr LISTENER (
                   Owner = oracle
                   Home = "/oracle/orahome"
                   TnsAdmin = "/oracle/orahome/network/admin"
                   MonScript = "./bin/Netlsnr/LsnrTest.pl"
                   Listener @galaxy = LISTENER_a
                   Listener @nebula = LISTENER_b
                   EnvFile = "/opt/VRTSvcs/bin/Netlsnr/envfile"
           )
     .
4. You must configure the CVMVolDg and CFSMount resources in the cvm group for the Oracle binaries installed on shared storage. Refer to the appendix CVMCluster, CVMVxconfigd, CVMVolDg, and CFSMount Agents for description of CVMVolDg and CFSMount agents.
     .
           CVMVolDg orabin_voldg (
                   CVMDiskGroup = orabindg
                   CVMVolume = { "orabinvol", "srvmvol" }
                   CVMActivation = sw
           )
   
           CFSMount orabin_mnt (
                   Critical = 0
                   MountPoint = "/oracle"
                   BlockDevice = "/dev/vx/dsk/orabindg/orabinvol"
                   Primary =  galaxy
           )
     .
     .
5. Define the dependencies of resources in the group. The dependencies are specified such that the Netlsnr resource requires the IP resource that, in turn, depends on the NIC resource. The Netlsnr resource also requires the CFSMount resource. The CFSMount resource requires the daemons and vxfsckd used by the cluster file system. The CFSMount resource also depends on the CVMVolDg resource that, in turn, requires the CVMCluster resource, which autoimports the shared storage and establishes GAB communications within the cluster. The CVMCluster resource depends on the CVMVxconfigd daemon to maintain the disk and disk group configuration.
     .
     .
           cvm_clus require cvm_vxconfigd
           orabin_voldg requires cvm_clus
           orabin_mnt requires vxfsckd
           orabin_mnt requires orabin_voldg
           listener_ip requires listener_NIC
           LISTENER requires listener_ip
           LISTENER requires orabin_mnt
     .
     .

Creating the Oracle Service Group and Adding Resources

The procedure for adding Oracle database resources is identical for the two example configurations shown. See Creating Oracle Service Group in the main.cf File.

Example 2: CVM Group with Oracle Binaries Installed Locally

For this example, you can refer to the illustration shown in CVM Group with Oracle Installed Locally; SRVM on Shared File System; Host IP Used.

Modifying the CVM Service Group in the main.cf File

The cvm service group is created during the installation of Storage Foundation for Oracle RAC. After installation, the main.cf file resembles the example shown in Sample main.cf: After SFRAC Installation and Before Oracle9i Installation. Because Oracle had not been installed, the cvm service group includes only resources for the CFSfsckd daemon and CVMCluster resource.

You must modify the cvm service group to add the Netlsnr, Mount, Disk Group, CVMVolDg, and CFSMount resources to the configuration. You can refer to Sample Configuration Files, Sample Configuration Files, to see a complete example of how a cvm group is configured.

1. Make sure the cvm group has the group Parallel attribute set to 1. Typically this is already done during installation.
     .
     group cvm (
           SystemList = { galaxy = 0, nebula = 1 }
           AutoFailOver = 0
           Parallel = 1
           AutoStartList = { galaxy, nebula }
           )
     .
2. Define the Mount and DiskGroup resources for use by the Oracle binaries. The VCS bundled Mount and DiskGroup agents are described in VERITAS Cluster Server Bundled Agents Reference Guide.
    .
           Mount orabin_mnt (
                   MountPoint = "/prod/u01" 
                   BlockDevice = "/dev/vx/dsk/ora_bin_dg/ora_bin"
                   FSType = vxfs FsckOpt = "-n" 
           )
   
           DiskGroup orabin_dg (
                   DiskGroup = ora_bin_dg 
                   StartVolumes = 0 
                   StopVolumes = 0 
           )
     .
     .
3. Define the Netlsnr resource. The Netlsnr listener agent is described in detail in the VERITAS Cluster Server Enterprise Agent for Oracle, Installation and Configuration Guide. Note that the Listener attribute is localized for each node.
     .
           Netlsnr LISTENER (
                   Owner = oracle
                   Home = "/oracle/orahome"
                   TnsAdmin = "/oracle/orahome/network/admin"
                   MonScript = "./bin/Netlsnr/LsnrTest.pl"
                   Listener @galaxy = LISTENER_a
                   Listener @nebula = LISTENER_b
                   EnvFile = "/opt/VRTSvcs/bin/Netlsnr/envfile"
           )
     .
4. You must configure the CVMVolDg and CFSMount resources in the cvm group for the SRVM disk group configured on shared storage. Refer to the appendix CVMCluster, CVMVxconfigd, CVMVolDg, and CFSMount Agents for description of CVMVolDg and CFSMount agents.
     .
     .
           CFSMount ora_srv_mnt (
                   Critical = 0
                   MountPoint = "/orasrv"
                   BlockDevice = "/dev/vx/dsk/orasrv_dg/srvm_vol"
                   Primary =  galaxy
           )
   
           CVMVolDg srvm_vol_dg (
                   CVMDiskGroup = orasrv_dg
                   CVMVolume = { "srvmvol" }
                   CVMActivation = sw
           )
    .
    .
5. Define the dependencies of resources in the group. The Netlsnr resource requires the Mount resource. The Mount resource requires DiskGroup resource. The CFSMount resource depends on the CVMVolDg resource, which, in turn, requires the CVMCluster resource, which autoimports the shared storage and establishes GAB communications within the cluster. The CFSMount resource also requires the CFSfsckd resource daemon, vxfsckd, used by the cluster file system. The CVMCluster resource depends on the CVMVxconfigd daemon to maintain the disk and disk group configuration.
     .
     .
           LISTENER requires orabin_mnt
           orabin_mnt requires orabin_dg
           srvm_mnt requires srvm_vol_dg
           srvm_mnt requires vxfsckd
           srvm_vol_dg requires cvm_clus
           cvm_clus requires cvm_vxconfigd
    .
    .

Creating the Oracle Service Group and Adding Resources

The procedure for adding Oracle database resources is identical for the two example configurations shown. See the next section, Creating Oracle Service Group in the main.cf File.

Creating Oracle Service Group in the main.cf File

For a complete description of the VCS Oracle enterprise agent, refer to the document, VERITAS Cluster Server Enterprise Agent for Oracle, Installation and Configuration Guide. That document includes instructions for configuring the Oracle and Netlsnr agents.

The VCS Enterprise Agent for Oracle version 4.1 is installed when you run the installer script. When you refer to the VERITAS Cluster Server Enterprise Agent for Oracle Installation and Configuration Guide, ignore the steps described in the section "Installing the Agent Software."

1. Using the Sample Configuration Files as an example, add a service group to contain the resources for an Oracle database. For example, add the group oradb1_grp. Make sure you assign the Parallel attribute a value of 1.
     .
     group oradb1_grp (
           SystemList = { galaxy = 0, nebula = 1 }
           AutoFailOver = 1
           Parallel = 1
           AutoStartList = { galaxy, nebula }
           )
     .
     .
2. Create the CVMVolDg and CFSMount resource definitions. See CVMCluster, CVMVxconfigd, CVMVolDg, and CFSMount Agents for a description of these agents and their attributes.
     .
     .
            CVMVolDg ora_db1_voldg (
                   CVMDiskGroup = oradb1dg
                   CVMVolume = { "oradb1vol" }
                   CVMActivation = sw
            )
   
           CFSMount ora_db1_mnt (
                   MountPoint = "/oradb1"
                   BlockDevice = "/dev/vx/dsk/oradb1dg/oradb1vol"
                   Primary =  galaxy
   
           )
     .
     .
3. Define the Oracle database resource. Refer to the VERITAS Cluster Server Enterprise Agent for Oracle, Installation and Configuration Guide for information on the VCS enterprise agent for Oracle. Note that the Oracle attributes Sid, Pfile, and Table attributes must be set locally, that is, they must be defined for each cluster system.
    .       
    .       
           Oracle VRT (
                Sid @galaxy = VRT1
                Sid @nebula = VRT2
                Owner = oracle
                Home = "/oracle/orahome"
                Pfile @galaxy = "/oracle/orahome/dbs/initVRT1.ora"
                Pfile @nebula = "/oracle/orahome/dbs/initVRT2.ora"
                User = scott
                Pword = tiger
                Table @galaxy = vcstable_galaxy
                Table @nebula = vcstable_nebula
                MonScript = "./bin/Oracle/SqlTest.pl"
                AutoEndBkup = 1
                EnvFile = "/opt/VRTSvcs/bin/Oracle/envfile"
           )
    .       
    .
4. Define the dependencies for the Oracle service group. Note that the Oracle database group is specified to require the cvm group, and that the required dependency is defined as "online local firm," meaning that the cvm group must be online and remain online on a system before the Oracle group can come online on the same system. Refer to the VERITAS Cluster Server User's Guide for a description of group dependencies.

   Refer to the illustration in Understanding VCS Service Group Dependency Capabilities and Limitations.

   
     .
     .
           requires group cvm online local firm
           oradb1_mnt requires oradb1_voldg
           VRT requires oradb1_mnt
     .
     .

See the Sample Configuration Files for a complete example. You can also find the complete file in /etc/VRTSvcs/conf/sample_rac/main.cf.

When you complete editing the main.cf file, see the next section, Saving and Checking the Configuration.

Saving and Checking the Configuration

When you complete editing the main.cf file to configure the CVM and Oracle service groups, verify the new configuration.

1. Save and close the main.cf file.
2. Verify the syntax of the file /etc/VRTSvcs/conf/config/main.cf:
     # hacf -verify /etc/VRTSvcs/conf/config
3. Start the VCS engine on one system:
     # hastart
4. Type the command hastatus:
     # hastatus
5. When "LOCAL_BUILD" is listed in the message column, start VCS on the other system:
     # hastart
6. Verify that the service group resources are brought online. On one system, enter:
     # hagrp -display

Additional RAC Processes Monitored by the VCS Oracle Agent

For shallow monitoring, the VCS Oracle agent monitors the Oracle9i RAC processes ora_lmon and ora_lmd, in addition to the processes ora_dbw, ora_smon, ora_pmon, and ora_lgwr.

Attributes of CVM, Oracle Groups Defined as Local

The following table lists attributes that must be defined as local for the CVM and Oracle service groups (note that each attribute has string-scalar as the type and dimension).

If other attributes for the Oracle resource differ for various RAC instances, define them locally as well. These other attributes may include the Oracle resource attributes User, PWord, the CVMVolDg resource attribute CVMActivation, and others.

Modifying the VCS Configuration

For additional information and instructions on modifying the VCS configuration, refer to the VERITAS Cluster Server User's Guide.

Location of VCS and Oracle Agent Log Files

On all cluster nodes, look at the following log files for any errors or status messages:

When large amounts of data are written, multiple log files may be required. For example, engine_B.log, engine_C.log, and so on, may be required. The engine_A.log contains the most recent data.

Configuring the Service Groups Using the Wizard

You can use a configuration wizard to configure the VCS service groups for Storage Foundation for Oracle RAC environment. The wizard enables you to create the service group for Oracle and modify the CVM service group.

Creating Oracle Service Group - Overview

The Oracle9i RAC configuration wizard guides you through the creation of an Oracle service group and the definition of the Oracle, CFSMount, and CVMVolDg resources. It adds the Netlsnr resources to the existing CVM group. If the listeners use the virtual IP, the wizard also adds the IP and NIC resources to the CVM group.

The wizard configures the Oracle service group to depend on CVM group with an online-local-firm dependency.

Before Starting the Wizard

Before starting the Wizard, you can verify that your Oracle installation can be configured. Review the requirements listed below. Also, you need to provide the wizard information as it proceeds. Make sure you have that information at hand.

Prerequisites

  Oracle RAC instances and listeners must be running on all cluster nodes.

  The database files of all instances must be on a cluster file system.

---

  Note    The Wizard does not support using the same file system for the Oracle binary and Oracle datafiles.

---

  The SRVM location must be on a raw volume or a cluster file system.

  Each Oracle instance must be associated with a listener. The listener may be configured to listen to either the base IP or a virtual IP.

---

  Note    The RAC configuration wizard requires that for the default listener, the listener parameter file, listener.ora, must reside in$ORACLE_HOME/network/admin. No such restriction applies for non-default listeners.

---

  The IP addresses and host names specified in the files listener.ora and tnsnames.ora must be the same.

  If detail monitoring is to be used for a database instance, the table used for detail monitoring must be set up, with user and password assigned.

Information Required From the User

  The names of the database instances to be configured

  The information to required for the detail monitoring configuration

  The location of the Pfile for each instance

Establishing Graphical Access for the Wizard

The configuration wizard requires graphical access to the VCS systems where you want to configure service groups. If your VCS systems do not have monitors, or if you want to run the wizards from a remote HP system, do the following:

Establishing Graphical Access from a Remote System

1. From the remote system, (jupiter, for example), run xhost +
       # xhost +
2. Complete one of the following operations (depending on your shell):
   • If you are running ksh, run this step on one of the systems where the wizard will run (for example, jupiter):
   
        # export DISPLAY=jupiter:0.0
   • If you are running csh, run this step
   
        # setenv DISPLAY jupiter:0.0

3. Verify the DISPLAY environment variable is updated:
       # echo $DISPLAY
       jupiter:0.0
4. Make sure to set the JRE_HOME variable to /opt/VRTSjre/jre1.4. If VRTSjre1.4 is not installed, the hawizard exits after displaying an error message.

Starting the Configuration Wizard

The configuration wizard for Oracle 9i RAC is started at the command line.

1. Log on to one of your VCS systems as root.
2. Start the configuration wizard.
      # /opt/VRTSvcs/bin/hawizard rac

   

   Click the thumbnail above to view full-sized image.

The Welcome Window

The RAC wizard starts with a Welcome window that highlights the prerequisites for configuration and the information required of the user to complete the configuration.

If your configuration does not meet the configuration requirements, you can stop the wizard by pressing Cancel. Take the necessary steps to meet the requirements and start the wizard again (see Starting the Configuration Wizard above).

The Wizard Discovers the RAC Configuration

If you are ready to configure Oracle service group, press Next on the Welcome screen. The wizard begins discovering the current Oracle RAC information before proceeding with the next screen.

Click the thumbnail above to view full-sized image.

If the wizard does not find all databases and listeners running on all systems in the cluster, it halts with an error, indicating the problem. Press Cancel, and start the wizard again after you correct the problem.

The Wizard Options Screen

The first configuration screen presents options to "Create RAC service group" or "Modify RAC service group." If you are using the Wizard to modify an existing service group, see Modifying the Oracle 9i RAC Service Group Configuration.

Creating a RAC Service Group

To create an Oracle RAC service group, click the corresponding button and provide a name for the Oracle service group. Guidelines for naming an Oracle service group are available by clicking Help ("?"). After entering a service group name, click Next.

Click the thumbnail above to view full-sized image.

Database Selection Screen

The databases and their instances running on the cluster are listed on this screen. Highlight only one of the databases if more than one are listed. Click Next.

Click the thumbnail above to view full-sized image.

Instance Configuration Screen

Configure the basic database instance information on the Instance Configuration screen.

Click the thumbnail above to view full-sized image.

Confirming Basic Database Instance Information

For each database instance discovered, basic configuration information is displayed. If necessary, double click in a field to select and edit its contents.

• Instance name: Each instance is listed in the left hand column.
• The Oracle Parameter File (Pfile): The file that is used to start Oracle. The default location for a given instance is listed. Edit the information if necessary.
• Startup Option:

Accept the displayed STARTUP option, or select an option from the drop-down menu. The startup options include starting in RESTRICTED, RECOVERDB, SRVCTLSTART, or CUSTOM modes.

• Stop Option:

Accept the displayed IMMEDIATE option, or select an option from the drop-down menu. The stop options also include TRANSACTIONAL, SRVCTLSTOP, and CUSTOM.

Check box for Detailed Monitoring

On the Instance Configuration Screen, you can choose to enable Detail Monitoring. If you check Enable Detail Monitoring, be sure you have previously set up the database table, user, and password for the agent to use during monitoring. See Detail Monitoring Screen - Oracle Resource.If you are not set up for detail monitoring, do not select it.

Check box for Advanced Options

On the Instance Configuration Screen, you can choose to Specify Advanced Options. The advanced options include setting up an EnvFile (to define environment variables), Encoding, and an AutoEndBkup parameter. See Oracle Advanced Configuration Screen.

Detail Monitoring Screen - Oracle Resource

This screen is displayed if you have checked Detail Monitoring at the bottom of the Instance Configuration Screen.

Click the thumbnail above to view full-sized image.

For each database instance identified by its Sid, this screen displays fields for defining the attributes that enable detail monitoring of the Oracle database resource. You do not have to enable detail monitoring on all instances, but for each instance you check, all of the fields are required:

• User: Oracle user, which the Oracle agent uses to log on to monitor the health of the database.
• Password: Password for the Oracle user.
• Table: Name of the database table to be used by the Oracle agent monitor.

Oracle Advanced Configuration Screen

This screen is displayed if you have checked Specify Advance Options at the bottom of the Instance Configuration Screen.

Click the thumbnail above to view full-sized image.

For each database instance identified by its Sid, this screen displays fields for configuring the advanced attributes of the Oracle service group. You may select which database instance you want to configure advance attributes for, and which attributes you want to define.

The advance attributes include:

• EnvFile: the source file used by the agent entry point scripts
• Encoding: the operating system encoding that corresponds to Oracle encoding for the displayed Oracle output; the encoding value must match the encoding value used by the Netlsnr configuration.
• AutoEndBkup: specifies that datafile in the database are taken out of the backup mode when instance is onlined.

See the VERITAS Cluster Server Enterprise Agent for Oracle Installation and Configuration Guide for a complete description of the EnvFile, Encoding, and AutoEndBkup attributes.

Database Configuration Screen

If you have installed the database on a cluster file system, the wizard discovers the mount point and displays it on the Database Configuration screen. You can confirm the mount options displayed, or you can modify them. If the database exists on raw volumes, the wizard discovers the volumes.

Click the thumbnail above to view full-sized image.

Listener Configuration Screen

The Listener Configuration screen displays the name of the listener corresponding to each database instance, as well as the IP address and device name used by each listener. Typically, you cannot change this information, only verify it.

Click the thumbnail above to view full-sized image.

Check boxes: Detail Monitoring, Advanced Listener Options

You can choose to configure detail monitoring for the Netlsnr agent by clicking the "Enable detail monitoring" check box. The wizard uses the monitor script /opt/VRTSvcs/bin/Netlsnr/LsnrTest.p1 to monitor the listeners in detail.

You can also choose to "Specify Advanced options," which include setting up an EnvFile (to define environment variables), Encoding, and LsnrPwd parameters. See Listener Advanced Configuration Screen that follows.

Using the Host IP Address

If you have set up the listener to use the base, or host, IP address, the wizard displays the following message when you press Next on the Listener Configuration screen:

Click the thumbnail above to view full-sized image.

Listener Advanced Configuration Screen

This screen displays if you have checked "Specify Advanced options at the bottom of the Listener Configuration Screen.

Click the thumbnail above to view full-sized image.

For each listener identified by name, this screen displays fields for defining the advanced attributes of the Netlsnr resource.

• The Netlsnr EnvFile: the name of the source file used by the agent entry point scripts; this file must exist
• Netlsnr Encoding: the operating system encoding that corresponds to Oracle encoding for the displayed Oracle output; the encoding value must match the encoding value used by the Oracle configuration (see Oracle Advanced Configuration Screen).
• Listener Password: the password used for Netlsnr; must specify the password as it appears in the listener.ora file.

Service Group Summary Screens

After you have configured the database and listener resources, the wizard displays the configuration on a "Summary" screen.

Click the thumbnail above to view full-sized image.

You can click on a resource within the service group to highlight it and display its attributes and their values. For example, if you click on the name of the Oracle resource, Ora-racj1, the wizard displays details of the Oracle resource, as the following illustration shows.

Click the thumbnail above to view full-sized image.

The next illustration shows the attributes for the CFSMount resource. Note the dependencies listed at the bottom of the Attributes screen.

Click the thumbnail above to view full-sized image.

The NetLsnr resource is configured as part of the CVM service group. The CVM service group also contains other resources, but the wizard does not show them because the wizard does not control them.

Click the thumbnail above to view full-sized image.

Implementing the Configuration

The wizard implements the configuration changes when you click Finish. The wizard creates the Oracle service group, adds the Netlsnr resource to the CVM the configuration.

Modifying Oracle RAC Service Groups Using the Wizard

Once an Oracle RAC service group is created on a system, the configuration wizard can be used to modify the service group's Oracle, Netlsnr, and CVM components.

Prerequisites

  To modify some resources, you must make the changes in the Oracle database before you start the RAC configuration wizard. When you start the wizard, it discovers the the new information. This applies to:

• Adding or removing the database mount points
• Adding or removing shared volumes

  To modify network resources, make sure that the service group is offline.

  To add or remove database files from your configuration, make sure that the service group is online.

Modifying the Oracle 9i RAC Service Group Configuration

1. Start the RAC configuration wizard as root on the VCS system:
     # /opt/VRTSvcs/bin/hawizard rac
2. On the Welcome window, click Next.
3. In the Wizard Options window, select the "Modify RAC service group" option, select the service group to be modified, and click Next.
4. Follow the wizard instructions and make modifications as per your configuration. See Configuring the Service Groups Using the Wizard for more information about the configuration wizard.