Taking Over from an Original Primary

The takeover procedure involves transferring the Primary role from an original Primary to a Secondary. When the original Primary fails or is destroyed because of a disaster, the takeover procedure enables you to convert a consistent Secondary to a Primary. The takeover of a Primary role by a Secondary is useful when the Primary experiences unscheduled downtimes or is destroyed because of a disaster.

In the following illustration, the Primary seattle is replicating to the Secondary hosts london and tokyo when disaster strikes the Primary seattle. The Secondary london has been identified as the Secondary for takeover and will become the New Primary after the takeover is complete.

Click the thumbnail above to view full-sized image.

After the takeover is complete, the Secondary london becomes the new Primary. If you had already created RLINKs between london and tokyo when setting up replication, you do not need to manually reconfigure the additional Secondary tokyo as a part of the RDS. It will automatically be added as a Secondary of the new Primary london.

You must then synchronize tokyo with the new Primary london and start replication.

Click the thumbnail above to view full-sized image.

VVR provides the vradmin takeover command to transfer the Primary role from an original Primary to a Secondary. This command can be run from the Secondary host only when the Primary is not reachable from the Secondary on which the takeover command is to be run. Upon successful completion of the takeover, the Secondary becomes the Primary. VVR displays an error message if you enter the vradmin takeover command on the Primary.

For configurations with multiple Secondaries, you can use the vxrlink updates command to find the most suitable replacement for the failed Primary. For more information, see Identifying the Most Up-to-Date Secondary.

Important Notes About Taking Over from an Original Primary

• The Secondary that is to become the new Primary must be consistent before taking over the Primary role. Any consistent Secondary can be selected to be the new Primary. A Secondary that is undergoing DCM resynchronization or autosync is inconsistent and cannot be used as a target of a takeover operation. Use the vxprint -l on the Secondary RLINK to check whether the consistent flag is set.
• Writes that were not replicated to the Secondary before the Primary became unusable are lost.

To preserve the data that was not replicated to the Secondary before the Primary became unusable, we recommend that you take a snapshot of the Primary data volumes before starting takeover with fast failback synchronization. The applications can later be started from the snapshot and you can apply the transactions or files that did not get replicated, to the active data.

• The Primary role takeover must be based on Recovery Point Objective (RPO) or the Recovery Time Objective (RTO) depending on the specific business requirement. For example, consider the scenario where the Secondary was behind by 100MB at the time of Primary crash and if the Primary is not expected to be available for another four hours, you will need to decide between having the application up and available on the Secondary or waiting for the Primary to become available after four hours. If it is required that the application must be available immediately, then the data on the Primary that was not yet replicated will be lost. Thus, takeover can result in data loss.
• The Primary role takeover is intended to support disaster recovery applications. Only a limited number of error scenarios prior to loss of the Primary node can prevent a successful takeover. These error scenarios leave the Secondary RVG in an inconsistent state and prevent takeover. All such scenarios involve a hardware failure of a Secondary data volume or Secondary SRL, and as a result, the Secondary RVG will be in an inconsistent state. The chances of such an occurrence are reduced if these volumes are configured (locally) as mirrored volumes.

---

  Note    The takeover procedure does not guarantee that the new Primary and any additional Secondary RVGs have identical contents. The remaining Secondaries must be completely synchronized with the new Primary.

---

• We recommend that you set the size of the SRL the same on the Primary and Secondary nodes because any of the Secondary nodes could be later converted to a Primary by using a migrate or takeover command.
• Each data volume on the new Primary must have a Data Change Map (DCM) associated with it.

The vradmin takeover command performs the following functions on the RDS to which the original Primary belongs:

• Converts the Secondary RVG to Primary RVG.
• Enables the fast failback feature on the new Primary, which speeds up failback to the original Primary when it recovers. The fast failback feature is described in the next section.

About Fast Failback

The applications are started on the new Primary after the takeover is complete. The fast failback feature uses failback logging for incremental synchronization of the original Primary with the new Primary. Fast failback works by keeping track of the incoming writes to the new Primary and the writes to the original Primary that did not reach the Secondary before the original Primary failed. Based on the tracked writes, data can be transferred to the original Primary after it recovers. This eliminates the need to completely resynchronize the original Primary with the new Primary after the original Primary recovers; only the changed blocks are resynchronized. Fast failback uses DCMs on the new Primary to track the changed blocks.

To enable fast failback, each data volume on the Secondary must have an associated DCM. The takeover command enables fast failback on the new Primary by activating the DCM. The DCM is later used to synchronize the data volumes on the original Primary with the data volumes on the new Primary.

When the original Primary recovers, it needs to be synchronized with the new Primary by playing back the DCM on the new Primary. To receive the missing updates, the original Primary must first be converted to a Secondary. The new Primary can then begin DCM playback to the original Primary. This process can be initiated automatically upon recovery of the original Primary by using the -autofb option to the takeover command, or it can be initiated manually at some later time by using the vradmin fbsync command.

When you use the -autofb option with the vradmin takeover command, it automatically synchronizes the original Primary when it becomes available. The -autofb option converts the original Primary to a Secondary after it comes up and also uses the DCM to synchronize the data volumes on the original Primary using fast failback. The -autofb option can be used only if each Secondary data volume has an associated DCM.

If you prefer not to have the original Primary automatically converted to a secondary upon reboot, the process can be performed manually using the vradmin fbsync command.

To change the role from Secondary to Primary without enabling fast failback, use the -N option with the vradmin takeover command. Use the -N option if you are sure that the original Primary will not recover or if most of the data on the Primary is going to change while the Primary is down. When performing vradmin takeover with the -N option the command automatically detaches the RLINKs from the old Primary to the new Primary. This requires either difference-based synchronization (vradmin syncrvg) or full synchronization of the data volumes on the original Primary.

Example:

To take over from the Primary RVG hr_rvg on host seattle to the Secondary RVG hr_rvg on host london, make sure that the data volumes on the Secondary have associated DCMs. Use the following command to check that the LOGONLY attribute is set for the data volumes:

We recommend that you use the fast failback synchronization method to synchronize the original Primary with the new Primary. For details, see Failing Back Using Fast Failback Synchronization.

For an RDS with multiple Secondaries, after take over, VVR automatically reconfigures any additional Secondaries as Secondaries of the new Primary, if RLINKs had been created between the Secondaries. Otherwise, you must manually reconfigure them. For more information, see Example 2---Taking Over from an Original Primary in a Setup with Multiple Secondaries.

To take over an original Primary with fast failback enabled (default), issue the following command on the Secondary that is to take over the Primary role:

The argument diskgroup is the disk group on the local host.

The argument local_rvgname is the name of the RVG on the local host.

Example 1---Taking Over from an Original Primary

In this example the Primary host seattle has failed. This example explains how to take over from the original Primary host seattle to the Secondary host london. The Secondary host london is converted to the new Primary. The disk group name is hrdg.

To take over from the Primary seattle to Secondary RVG hr_rvg on host london:

1. Make sure that the data volumes on the Secondary have associated DCMs.
     # vxprint -g hrdg -ht hr_rvg
2. Make the Secondary RVG hr_rvg the new Primary RVG by typing the following command on the Secondary london:
     # vradmin -g hrdg takeover hr_rvg

   The vradmin takeover command enables fast failback.

3. Verify whether fast failback is enabled by typing the following command on the Secondary london:
     # vxprint -l rlink_name

   If fast failback is enabled, the dcm_logging flag is set.

4. Start the application on the new Primary london. Starting the applications on the new Primary after a takeover may require an application recovery to be run.

Example 2---Taking Over from an Original Primary in a Setup with Multiple Secondaries

We recommend that you create RLINKs between the hosts london and tokyo when setting up the RDS.

In this example the Primary host seattle has failed. The example explains how to take over from the original Primary host seattle to the Secondary host london. This example also explains how to start replication from the new Primary london to the additional Secondary tokyo.

  To take over from the Primary seattle to Secondary RVG on host london

1. Make sure that the data volumes on the Secondary have associated DCMs.
     # vxprint -g hrdg -ht hr_rvg
2. Make the Secondary RVG hr_rvg the new Primary RVG by typing the following command on the Secondary london:
     # vradmin -g hrdg takeover hr_rvg

   The vradmin takeover command enables fast failback.

3. Verify whether fast failback is enabled by typing the following command on the Secondary london:
     # vxprint -l rlink_name

   If fast failback is enabled, the dcm_logging flag is set.

4. If you had created RLINKs between the Secondary london and the additional Secondary tokyo, host tokyo is automatically added to the new configuration.

   Otherwise, you must manually add tokyo as a Secondary to the new Primary london. To do this, create RLINKs between london and tokyo and associate them to the respective RVGs using the following commands.

   On host london:

   
   # vxmake -g hrdg rlink rlk_tokyo_hr_rvg local_host=london \
       remote_host=tokyo remote_rlink=rlk_london_hr_rvg \
        remote_dg=hrdg
   # vxrlink -g hrdg assoc hr_rvg rlk_tokyo_hr_rvg

   On host tokyo:

   
   # vxmake -g hrdg rlink rlk_london_hr_rvg local_host=tokyo \
       remote_host=london remote_rlink=rlk_tokyo_hr_rvg \
        remote_dg=hrdg
   
   # vxrlink -g hrdg assoc hr_rvg rlk_london_hr_rvg

   ---

     Note    By default, the vxmake rlink command creates the RLINK with the protocol set to UDP/IP. To change the protocol to TCP/IP, see Setting the Network Transport Protocol.

   ---

5. Even after takeover, the RLINK from tokyo to the original primary seattle still remains attached. Detach this RLINK using the following command on the new Primary london or on the Secondary tokyo:
      # vradmin -g hrdg stoprep hr_rvg tokyo
6. On the new Primary london:
   • Synchronize the data volumes in the Secondary RVG hr_rvg on tokyo with the data volumes in the original Primary RVG hr_rvg using the difference-based synchronization and checkpoint. To do this, use the following command on any host in the RDS:
   
       # vradmin -g hrdg -c checkpt syncrvg hr_rvg tokyo

   The -c option when used with the vradmin syncrvg command automatically starts a checkpoint with the specified name, checkpt, in this example. After the data volumes are synchronized the checkpoint is ended.

   After the data volumes are synchronized, the checkpoint is ended.

   • Start replication to tokyo using the checkpoint created above:
   
      # vradmin -g hrdg -c checkpt startrep hr_rvg tokyo

7. Start the application on the new Primary london. Starting the applications on the new Primary after a takeover may require an application recovery to be run.