| Oracle® Data Guard Broker 10g Release 2 (10.2) Part Number B14230-01 |
|
|
View PDF |
| Oracle® Data Guard Broker 10g Release 2 (10.2) Part Number B14230-01 |
|
|
View PDF |
This chapter describes various errors and how to solve them. This chapter contains the following topics:
Section 10.3, "Troubleshooting Problems During a Switchover Operation"
Section 10.4, "Troubleshooting Problems During a Failover Operation"
The Data Guard broker provides information about its activities in several forms:
Database status information (see Section 4.7)
Oracle alert log files
The broker records key information in the alert log file for each database. You can check the alert log files for such information when troubleshooting Data Guard.
For each database in a broker configuration, the broker DMON process records important behavior and status information in a "broker log" file, useful in diagnosing Data Guard failures.
For example, the broker log files for all instances in a RAC are named as follows in a UNIX environment:
$ORACLE_HOME/rdbms/log/drc*.log
This section describes general problems and solutions. This section contains the following topics:
ORA-16596: Object Not Part of the Data Guard Broker Configuration
Redo Accumulating on the Primary Is Not Sent to Some Standby Databases
Many Log Files Are Received on a Standby Database But Not Applied
A request was issued to the broker, but the database instance through which you have connected is no longer a part of the broker configuration. You may see this error when the broker monitor (DMON) fails to locate a broker configuration profile for the database upon which it is running.
Solution Reconnect to the configuration through another database instance. Confirm that the value of the DB_UNIQUE_NAME initialization parameter for the database and the database object name in the broker configuration are identical.
This problem can also occur if you attempt to enable a configuration, but the broker configuration file for one of its databases was accidentally removed or is outdated. In this case, remove the database from the broker configuration, manually delete the configuration file for that standby database (not for the primary database), and try again to enable the configuration. After the configuration is enabled, run the Add Standby Database wizard and choose the Add existing standby database option to create a new database profile for the previously deleted standby database.
By viewing the Log File Details page in Enterprise Manager, you have determined that log files are accumulating on the primary database and are not being archived to some of the standby databases in the broker configuration.
Solution To narrow down the problem, do the following:
Verify that the state of the primary database is ONLINE (not LOG-TRANSPORT-OFF).
Verify that the value of the LogShipping property of the standby database in question is ON.
Check the status of the redo transport services on the primary database using the LogXptStatus monitorable property. If redo transport services have an error, then use the error message to determine further checking and resolution action. For example:
If the error indicates the standby database is not available, you need to restart the standby database.
If the error indicates no listener, you need to restart the listener.
If the error indicates the standby database has no local destination, you need to set up a standby location to store archived redo log files from the primary database.
By viewing the Performance page or Log File Details page in Enterprise Manager, you have determined that the standby database accumulates too many log files without applying them.
Solution There are many possible reasons why archived redo log files might not be applied to the standby database. Investigate why the log files are building up and rule out the valid reasons.
If the current status of the standby database is not normal:
Determine whether or not the log apply services might be unexpectedly offline. See the ORA-16766 (for physical standby databases) or ORA-16768 (for logical standby databases) problem and solution statement for more help.
If this is a logical standby database, check to see if a failed transaction has occurred.
If you want to suppress the error while you investigate the problem, you can temporarily disable broker management of the database.
|
See Also: Chapter 8 for additional information about disabling the database using the DGMGRL command-line interface |
If the current status of the standby database is normal:
Verify the state of the standby database is online (not in the LOG-APPLY-OFF, OFFLINE, or READ-ONLY states).
Verify the state of the primary database is online (not in the LOG-TRANSPORT-OFF state).
Check to see if log files are building up because the value of the DelayMins property is set too large. (Log apply services will delay applying the archived redo log files on the standby database for the number of minutes specified.)
If you cannot see any errors, compare the archive rate to the apply rate on the Performance page in Enterprise Manager to see if the apply rate is lower than the archive rate.
If the broker requests are not completing within the normal timeout parameters, try the following actions to solve the problem:
Verify the network is operating appropriately.
Try to ping all of the nodes in the configuration.
Try reconnecting through another database to retry the operation.
Run the VERIFY command from the configuration to see which broker is busy. The output from the VERIFY command should show why (or at least which) broker is not able to process the requests.
If the primary database is flashed back, the standby databases in the configuration must be also be flashed back or reenabled to be viable targets for switchovers or failovers. The broker will report errors for the standby databases if the primary database has been flashed back.
For more information about restoring the viability of a standby database that was disabled by the broker, see Section 5.4.2.
If the switchover fails due to problems with the configuration, the broker reports any problems it encounters in the alert log files or in the broker log files (see Section 10.1). In general, you can choose another database for the switchover or restore the configuration to its pre-switchover state and then retry the switchover. The following subsections describe how to recover from the most common problems.
Problems Transitioning the Primary Database to the Standby Role
If the error status indicates a problem when transitioning the original primary database to the standby role (including stopping redo transport services and starting log apply services), use these general guidelines to recover:
Disable the configuration using DGMGRL.
|
Note: You can enable or disable the configuration using DGMGRL. You cannot disable the configuration using Enterprise Manager. You can enable the configuration using Enterprise Manager if it was previously disabled using DGMGRL. |
Investigate the error message returned by the broker to find the source of the problem on the primary database and correct it. Oracle Enterprise Manager provides an Alert Log Content link for viewing alert log file information. You may also examine the contents of the broker log file for the primary database.
Reenable the configuration to refresh and restore the databases to their original roles and states.
Perform the switchover again.
Problems Transitioning the Standby Database to the Primary Role
If the error status indicates that a problem occurred when transitioning the target standby database to the primary role (including stopping log apply services and starting redo transport services), use these general guidelines to restore to the pre-switchover state.
If fast-start failover is enabled, the broker does not allow failover to any standby database except to the target standby database. In addition, failover to the target standby database is allowed only under certain conditions:
The observer is running
The value of the FS_FAILOVER_STATUS column in the V$DATABASE view on the target standby database is either READY or SUSPENDED.
Step 1 Analyze and correct the detected failure.
Disable the configuration using DGMGRL.
|
Note: You can enable or disable the configuration using DGMGRL. You cannot disable the configuration using the Enterprise Manager. You can enable the configuration using Enterprise Manager if it was previously disabled using DGMGRL. |
Investigate the error messages returned by the broker to find the source of the problem on the standby database and correct it. Examine the alert log file information and the contents of the broker log file for the target standby database.
Step 2 Convert back to a primary database.
The original primary database was already converted to a standby database by the time this stage of the switchover is reached. The database must be converted back to being a primary database as follows:
Ensure the original primary database is at least mounted. A restart may be necessary:
SQL> SHUTDOWN IMMEDIATE; SQL> STARTUP MOUNT
Execute the following SQL*Plus command to convert the database back to running in the primary database role:
SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;
Restart the database if necessary. The database is now in its former role as a primary database.
Additional steps may be needed to allow any logical standby databases in the configuration to accept and apply redo logs from this primary database. See Oracle Data Guard Concepts and Administration for these steps.
Step 3 Finish the recovery.
Reenable the configuration.
Perform the switchover operation again.
Additional Problems that May Occur During a Switchover Operation
Problem: The switchover fails due to problems with redo transport services.
Solution:
Verify the state and status of the standby database by viewing its information on the Data Guard Overview page.
Run the Verify operation after the switchover completes to examine the alert log file of the new primary database and to verify the status of the configuration.
Problem: The switchover may fail during verification checks done by Data Guard broker (for example, redo transport services might return errors on a database that is involved in the switchover).
Solution: Choose another database for the switchover or fix the problem by transporting the archived redo log files.
Although it is possible for a failover to stop, it is unlikely. If an error occurs, it is likely to happen when the standby database is transitioning to the primary role. If the error status indicates that this problem occurred, use these general guidelines to fix the problem:
Investigate the error information provided by the broker to find the source of the problem and correct it.
Perform the failover again.
The following problems may occur during a failover operation:
Problem: The primary database is still running. Enterprise Manager tries to detect if the primary database is still running. However, if the primary database is still running and the Enterprise Manager cannot detect that it is running, the failover will not be successful.
Solution: If the primary database is still running but can no longer function as a primary database, shut it down and retry the failover.
The observer continuously monitors the fast-start failover environment to ensure the primary database is available. Installing and starting the observer is an integral part of enabling fast-start failover. The following sections describe techniques for troubleshooting the observer:
Only one observer can be associated with the broker configuration at any given time. If you attempt to start a second observer, one of the following errors is returned:
ORA-16647: could not start more than one observer DGM-16954: Unable to open and lock the Observer configuration file
Use the DGMGRL SHOW CONFIGURATION VERBOSE command to determine the location of the observer that is currently associated with the broker configuration.
DGMGRL> SHOW CONFIGURATION VERBOSE;
Configuration
Name: DRSolution
Enabled: YES
Protection Mode: MaxAvailability
Fast-Start Failover: ENABLED
Databases:
North_Sales - Primary database
DR_Sales - Physical standby database
- Fast-Start Failover target
Fast-Start Failover
Threshold: 45 seconds
Observer: observer.foo.com
Current status for "DRSolution":
SUCCESS
If the observer host machine crashes, the broker configuration is no longer observed and fast-start failover is no longer possible. In this case, you may have to move the observer to a new host if the original host machine cannot be repaired in a timely fashion. To move the observer, you must stop the first observer and then start a new observer on another host.
Issue the DGMGRL STOP OBSERVER command to sever the link between the original observer and the broker configuration:
DGMGRL> STOP OBSERVER; Done.
Issue the DGMGRL SHOW CONFIGURATION VERBOSE and the SHOW DATABASE command to verify that the configuration is no longer being observed:
DGMGRL> SHOW CONFIGURATION VERBOSE;
Configuration
Name: DRSolution
Enabled: YES
Protection Mode: MaxAvailability
Fast-Start Failover: ENABLED
Databases:
North_Sales - Primary database
DRSales - Physical standby database
- Fast-Start Failover target
Fast-Start Failover
Threshold: 45 seconds
Observer: (none)
Current status for "The SUPER cluster":
Warning: ORA-16608: one or more databases have warnings
DGMGRL> SHOW DATABASE 'North_Sales';
Database
Name: North_Sales
Role: PRIMARY
Enabled: YES
Intended State: ONLINE
Instance(s):
sales1
Current status for "North_Sales":
Error: ORA-16658: unobserved Fast-Start Failover configuration
Note that you do not need to issue the DGMGRL SHOW commands to verify that the observer has actually stopped. Successful completion of the DGMGRL STOP OBSERVER command will allow a new observer to become associated with the configuration.
Now, you may start the new observer. After the observer is started, it prints its failover and reinstatement actions to the console in which it was started. For example:
DGMGRL> START OBSERVER; Observer started 14:41:23.95 Thursday, May 26, 2005 Initiating fast-start failover to database "DRSales"... Performing failover NOW, please wait... Failover succeeded, new primary is "DRSales" 14:41:38.62 Thursday, May 26, 2005 14:42:15.43 Thursday, May 26, 2005 Initiating reinstatement for database "North_Sales"... Reinstating database "North_Sales", please wait... Operation requires shutdown of instance "sales1" on database "North_Sales" Shutting down instance "sales1"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires startup of instance "sales1" on database "North_Sales" Starting instance "sales1"... ORACLE instance started. Database mounted. Continuing to reinstate database "North_Sales" ... Reinstatement of database "North_Sales" succeeded 14:43:23.71 Thursday, May 26, 2005
You can use the DGMGRL -logfile option to start the observer, so that all of the troubleshooting actions performed in Section 10.5.1 can be captured in a file. For example:
% dgmgrl -logfile observer.log / "start observer"
All the observer output is then recorded in a file named observer.log in the current working directory where you issued the DGMGRL command.
