RMAN Troubleshooting Scenarios

After Installation of Media Manager, RMAN Channel Allocation Fails: Diagnosis

The ORA-27211 error indicates that the channel allocation is failing because the database is not loading the media management library. If the channel allocation fails, then the database generates a trace file in the USER_DUMP_DEST location that contains the error that caused the channel allocation to fail. The trace file should have the complete path name of the media management library loaded by the database as well as any other media manager errors or operating system errors. For example, the trace file on UNIX may be called something like /oracle/rdbms/log/prod1_ora_16226.trc, and may contain information such as the following:

*** 2001-08-29 17:16:54.385
SKGFQ OSD: Error in function sbtinit on line 2396
SKGFQ OSD: Look for SBT Trace messages in file /oracle/rdbms/log/sbtio.log
SBT Initialize failed for oracle.static 

The last line of this output indicates that Oracle is loading the default static library instead of the media management library that you installed.

You may find more detailed information in the file sbtio.log, as described in the error message. Note, however, that writing SBT trace messages is the responsibility of the media management software, not the Oracle database or RMAN. The media management vendor may not have implemented the writing of trace messages in a particular situation. Contact the media management vendor for details about the trace messages written to sbtio.log.

To test the loading of the media management library, try allocating a channel by using the PARMS parameter SBT_LIBRARY to force the loading of the media management library. For example, if your library is called /vendor/lib/some_mm_lib.so, then run a command such as the following, making sure to specify whatever PARMS settings are required by your media manager:

RUN
{
  ALLOCATE CHANNEL c1 DEVICE TYPE sbt 
    PARMS='SBT_LIBRARY=/vendor/lib/some_mm_lib.so',
    'ENV=(NSR_SERVER=tape_svr,NSR_CLIENT=oracleclnt,NSR_GROUP=oracle_tapes)';
}

If the channel allocation fails, then check the trace file again to see whether you can learn anything new. If the channel allocation with SBT_LIBRARY succeeds, but an ordinary sbt channel allocation fails, then the database is probably trying to load a library other than the one you installed. By default, the database expects to find the media management library at $ORACLE_HOME/lib/libobk.so on UNIX, or %ORACLE_HOME%/bin/orasbt.dll on NT. You may have more than one library in the operating system path, and the database is loading the wrong one.

After Installation of Media Manager, RMAN Channel Allocation Fails: Solution

If the problem is that the database is not loading the correct library, then make sure that the library is named correctly in the SBT_LIBRARY parameter.

Backup Job Is Hanging: Diagnosis

If a backup job is hanging, that is, not proceeding, then several scenarios are possible:

• A server-side or media management error occurred.

• RMAN is waiting for an event such as the insertion of a new cassette into the tape device.

Query sbt wait events to gain more information. For example, run the following query on the target instance:

COLUMN EVENT FORMAT a10
COLUMN SECONDS_IN_WAIT FORMAT 999
COLUMN STATE FORMAT a20
COLUMN CLIENT_INFO FORMAT a30

SELECT p.SPID, EVENT, SECONDS_IN_WAIT AS SEC_WAIT, 
       STATE, CLIENT_INFO
FROM V$SESSION_WAIT sw, V$SESSION s, V$PROCESS p
WHERE sw.EVENT LIKE 'sbt%'
      AND s.SID=sw.SID
      AND s.PADDR=p.ADDR
;

Examine the SQL output to determine which sbt functions are waiting. For example, the output may be as follows:

SPID EVENT        SEC_WAIT STATE                CLIENT_INFO
---- ---------- ---------- -------------------- ------------------------------
8642 sbtbackup        1500 WAITING              rman channel=ORA_SBT_TAPE_1

Backup Job Is Hanging: Solution

Because the causes of a hung backup job can be varied, so are the solutions. For example, backup jobs often hang simply because the tape device has completely filled the current cassette and is waiting for a new tape to be inserted. Ideally, the query of the sbt wait events should indicate the problem.

In this example, a single sbtbackup has taken 1500 seconds, so RMAN is waiting on the media manager to finish its write operation. Check that the media manager is functioning normally, and contact the media management vendor's technical support for assistance.

If the sbt wait event query is unhelpful, then examine media manager process, log, and trace files for signs of abnormal termination or other errors (refer to the description of message files in "Identifying Types of Message Output").

RMAN Fails to Start RPC Call: Diagnosis

The RPC call appears to have failed message does not usually indicate a problem. The message indicates one of the following:

• The target database instance is slow.

• A timing problem occurred.

Timing problems occur in this way. When RMAN begins an RPC, it checks the V$SESSION performance view. The RPC updates the information in the view to indicate when it starts and finishes. Sometimes RMAN checks V$SESSION before the RPC has indicated it has started, which in turn generates the following message:

RPC call appears to have failed

If a message stating "RPC call ok" does not appear in the output immediately following the message stating "RPC call appears to have failed", then the backup job encountered an internal problem. Contact Oracle Support for further assistance.

Backup Fails with Invalid RECID Error: Diagnosis

In one common scenario, you restore a backup control file created through a non-Oracle mechanism, and then open the database without the RESETLOGS option. If you had created the backup control file through the RMAN BACKUP command or the SQL ALTER DATABASE BACKUP CONTROLFILE statement, then the database would have required you to reset the online logs.

The control file and the recovery catalog are now not synchronized. The database control file is older than the recovery catalog, because at one time the recovery catalog resynchronized with the old current control file, and now the database is using a backup control file. RMAN detects that the control file currently in use is older than the control file previously used to resynchronize.

Another common scenario occurs when you attempt to copy the target database to a new machine as follows:

1. On machine 1, you shut down the database and make a copy of the control file with an operating system utility. You do not use CATALOG to add this control file copy to the repository.

2. You transfer the control file copy to machine 2.

3. On machine 2, you create a new initialization parameter file and new database instance.

4. You mount the control file copy on machine 2. The database does not recognize the control file as a backup control file: to the database it looks like the current control file.

5. You start RMAN and connect to the new target database and the recovery catalog on machine 2. Because the control file was not created with RMAN and was not cataloged as a control file copy, RMAN sees the database on machine 2 as the database on machine 1.

6. You restore and recover database the new database on machine 2 and then open it. As a consequence, various records are added to the recovery catalog during the restore and recovery. For example, the highest RECID in the recovery catalog moves from 90 to 100.

7. On machine 1, you start RMAN and connect to the original target database and recovery catalog. The recovery catalog indicates that the highest RECID is 100, but the control file indicates that the highest RECID is 90. The control file RECID should always be greater than or equal to the recovery catalog RECID, so RMAN issues RMAN-20035.

Backup Fails with Invalid RECID Error: Solution 1

This solution is safest and is strongly recommended. It preserves the control file, so that the historical information about the database stored in the control file continues to be available after the procedure.

To reset the database with RMAN:

1. Connect to the target database with SQL*Plus. For example, enter:

   % sqlplus '/ AS SYSDBA'
   
   

2. Mount the database if it is not already mounted. For example, enter:

   ALTER DATABASE MOUNT;
   
   

3. Start cancel-based recovery by using the backup control file, then cancel it. The reason for canceling is that the USING BACKUP CONTROLFILE clause stamps the control file as a backup, which then permits OPEN RESETLOGS. For example, enter:

   ALTER DATABASE RECOVER DATABASE UNTIL CANCEL USING BACKUP CONTROLFILE;
   ALTER DATABASE RECOVER CANCEL;
   
   

4. Use RMAN to connect to the target database and recovery catalog. For example, enter:

   % rman TARGET SYS/oracle@trgt CATALOG rman/cat@catdb
   
   

5. Open the database with the RESETLOGS option. For example, enter:

   RMAN> ALTER DATABASE OPEN RESETLOGS;
   
   

6. Take new backups so that you can recover the database if necessary. For example, enter:

   BACKUP DATABASE PLUS ARCHIVELOG;
   

Backup Fails with Invalid RECID Error: Solution 2

This solution is similar to the previous one, but does require that you re-create your control file. It is better-suited for the case in which you are copying your database to a second system, where you may not want to keep the history from the control file for the copy of the database on the second system, or where you might drop a few datafiles or change the online logs by editing your control file.

To create the control file with SQL*Plus:

1. Connect to the target database with SQL*Plus. For example, enter:

   % sqlplus 'SYS/oracle@trgt AS SYSDBA'
   
   

2. Mount the database if it is not already mounted:

   SQL> ALTER DATABASE MOUNT;
   
   

3. Back up the control file to a trace file:

   SQL> ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
   
   

4. Edit the trace file as necessary. The relevant section of the trace file looks something like the following:

   # The following commands will create a new control file and use it
   # to open the database.
   # Data used by the recovery manager will be lost. Additional logs may
   # be required for media recovery of offline data files. Use this
   # only if the current version of all online logs are available.
   STARTUP NOMOUNT
   CREATE CONTROLFILE REUSE DATABASE "TRGT" NORESETLOGS  ARCHIVELOG
   --  STANDBY DATABASE CLUSTER CONSISTENT AND UNPROTECTED
       MAXLOGFILES 32
       MAXLOGMEMBERS 2
       MAXDATAFILES 32
       MAXINSTANCES 1
       MAXLOGHISTORY 226
   LOGFILE
     GROUP 1 '/oracle/oradata/trgt/redo01.log'  SIZE 25M,
     GROUP 2 '/oracle/oradata/trgt/redo02.log'  SIZE 25M,
     GROUP 3 '/oracle/oradata/trgt/redo03.log'  SIZE 500K
   -- STANDBY LOGFILE
   DATAFILE
     '/oracle/oradata/trgt/system01.dbf',
     '/oracle/oradata/trgt/undotbs01.dbf',
     '/oracle/oradata/trgt/cwmlite01.dbf',
     '/oracle/oradata/trgt/drsys01.dbf',
     '/oracle/oradata/trgt/example01.dbf',
     '/oracle/oradata/trgt/indx01.dbf',
     '/oracle/oradata/trgt/tools01.dbf',
     '/oracle/oradata/trgt/users01.dbf'
   CHARACTER SET WE8DEC
   ;
   # Take files offline to match current control file.
   ALTER DATABASE DATAFILE '/oracle/oradata/trgt/tools01.dbf' OFFLINE;
   ALTER DATABASE DATAFILE '/oracle/oradata/trgt/users01.dbf' OFFLINE;
   # Configure RMAN configuration record 1
   VARIABLE RECNO NUMBER;
   EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('CHANNEL','DEVICE TYPE DISK DEBUG 255');
   # Recovery is required if any of the datafiles are restored backups,
   # or if the last shutdown was not normal or immediate.
   RECOVER DATABASE
   # All logs need archiving and a log switch is needed.
   ALTER SYSTEM ARCHIVE LOG ALL;
   # Database can now be opened normally.
   ALTER DATABASE OPEN;
   # Commands to add tempfiles to temporary tablespaces.
   # Online tempfiles have complete space information.
   # Other tempfiles may require adjustment.
   ALTER TABLESPACE TEMP ADD TEMPFILE '/oracle/oradata/trgt/temp01.dbf' REUSE;
   # End of tempfile additions.
   
   

5. Shut down the database:

   SHUTDOWN IMMEDIATE
   
   

6. Execute the script to create the control file, recover (if necessary), archive the logs, and open the database:

   STARTUP NOMOUNT
   CREATE CONTROLFILE ...;
   EXECUTE ...;
   RECOVER DATABASE
   ALTER SYSTEM ARCHIVE LOG CURRENT;
   ALTER DATABASE OPEN ...;
   

7. If you intend to keep and continue using this copy of the database, use the DBNEWID utility to change the name and DBID of the new database as needed.

   
   

   Caution: If you do not open with the RESETLOGS option, then two copies of an archived redo log for a given log sequence number may exist—even though these two copies have completely different contents. For example, one log may have been created on the original host and the other on the new host. If you accidentally confuse the logs during a media recovery, then the database will be corrupted but Oracle and RMAN cannot detect the problem.

   
   

Backup Fails Because of Control File Enqueue: Diagnosis

When RMAN needs to back up or resynchronize from the control file, it first creates a snapshot or consistent image of the control file. If one RMAN job is already backing up the control file while another needs to create a new snapshot control file, then you may see the following message:

waiting for snapshot control file enqueue

Under normal circumstances, a job that must wait for the control file enqueue waits for a brief interval and then successfully obtains the enqueue. RMAN makes up to five attempts to get the enqueue and then fails the job. The conflict is usually caused when two jobs are both backing up the control file, and the job that first starts backing up the control file waits for service from the media manager.

To determine which job is holding the conflicting enqueue:

1. After you see the first message stating "RMAN-08512: waiting for snapshot control file enqueue", start a new SQL*Plus session on the target database:

   % sqlplus 'SYS/oracle@trgt AS SYSDBA'
   
   

2. Execute the following query to determine which job is causing the wait:

   SELECT s.SID, USERNAME AS "User", PROGRAM, MODULE, 
          ACTION, LOGON_TIME "Logon", l.* 
   FROM V$SESSION s, V$ENQUEUE_LOCK l
   WHERE l.SID = s.SID
   AND l.TYPE = 'CF'
   AND l.ID1 = 0
   AND l.ID2 = 2;
   
   

   You should see output similar to the following (the output in this example has been truncated):

   SID User Program              Module                    Action           Logon
   --- ---- -------------------- ------------------- ---------------- ---------
   9 SYS rman@h13 (TNS V1-V3) backup full datafile: c10000210 STARTED 21-JUN-01
   

Backup Fails Because of Control File Enqueue: Solution

Commonly, enqueue situations occur when a job is writing to a tape drive, but the tape drive is waiting for new tape to be inserted. If you start a new job in this situation, then you will probably receive the enqueue message because the first job cannot complete until the new tape is loaded.

After you have determined which job is creating the enqueue, you can do one of the following:

• Wait until the job holding the enqueue completes

• Cancel the current job and restart it after the job holding the enqueue completes

• Cancel the job creating the enqueue

RMAN Fails to Delete All Archived Logs: Diagnosis

This problem is not an error. When you specify DELETE INPUT without the ALL keyword, RMAN deletes only one copy of each input log. Even if you archive to five destinations, RMAN deletes logs from only one directory.

RMAN Fails to Delete All Archived Logs: Solution

To force RMAN to delete all existing archived redo logs, use the DELETE ALL INPUT clause of the BACKUP command. For example, enter:

BACKUP ARCHIVELOG ALL DELETE ALL INPUT;

Backup Fails Because RMAN Cannot Locate an Archived Log: Diagnosis

This problem occurs when the archived log that RMAN is looking for cannot be accessed by RMAN, or the recovery catalog needs to be resynchronized. Often, this error occurs when you delete archived logs with an operating system command, which means that RMAN is unaware of the deletion. The RMAN-6089 error occurs because RMAN attempts to back up a log that the repository indicates still exists.

Backup Fails Because RMAN Cannot Locate an Archived Log: Solution

Make sure that the archived logs exists in the specified directory and that the RMAN catalog is synchronized. Check the following:

1. Make sure the archived log file that is specified by the RMAN-6089 error exists in the correct directory.

2. Check that the operating system permissions are correct for the archived log (owner = oracle, group = DBA) to make sure that RMAN can access the file.

3. If the file appears to be correct, then try synchronizing the catalog by running the following command from the RMAN prompt:

   RESYNC CATALOG;
   
   

If you know that the logs are unavailable because you deleted them by using an operating system utility, then run the following command at the RMAN prompt to update RMAN metadata:

CROSSCHECK ARCHIVELOG ALL;

It is always better to use RMAN to delete logs than to use an operating system utility. The easiest method to remove unwanted logs is to specify the DELETE INPUT option when backing up archived logs. For example, enter:

BACKUP DEVICE TYPE sbt 
  ARCHIVELOG ALL 
  DELETE ALL INPUT;

RMAN Does Not Recognize Character Set Name: Diagnosis

Typically, this message means that the character set in the client environment, that is, the environment in which you are running the RMAN client, is different from the character set in the target database environment.

RMAN Does Not Recognize Character Set Name: Solution

1. Query the target database to determine the value of the NLS_CHARACTERSET parameter. For example, run this query:

   SQL>      SELECT VALUE FROM V$NLS_PARAMETERS WHERE PARAMETER='NLS_CHARACTERSET';
   
   

2. Set the character set environment variable in the client to the same value as the variable in the server. For example, you can set the NLS_LANG environment variable on a UNIX system as follows:

   % setenv NLS_LANG american_america.we8dec
   % setenv NLS_DATE_FORMAT "MON DD YYYY HH24:MI:SS"
   
   

If the connection is made througfh a listener, then the listener must be started with the correct Globalization Support settings. Otherwise, the spawned connections inherit the incorrect Globalization Support settings from the listener.

RMAN Denies Logon to Target Database: Diagnosis

RMAN automatically requests a connection to the target database as SYSDBA. In order to connect to the target as SYSDBA, you must do one of the following:

• Be part of the operating system DBA group with respect to the target database (that is, have the ability to connect with SYSDBA privileges to the target database without a password).

• Create a password file with the orapwd command and the initialization parameter REMOTE_LOGIN_PASSWORDFILE.

• Make sure you are connecting with the correct username and password.

If the target database does not have a password file, then the user you are logged in as must be validated with operating system authentication.

RMAN Denies Logon to Target Database: Solution

Either create a password file for the target database or add yourself to the administrator list in the operating system.

Database Duplication Fails Because of Missing Log: Diagnosis

The problem is that RMAN is not able to apply all the archived logs needed for complete recovery. For example, if you only backed up logs through sequence 15, but the most recent archived log is sequence 16, then DUPLICATE fails.

Database Duplication Fails Because of Missing Log: Solution

When creating the duplication script, use the SET UNTIL command to specify a log sequence number for incomplete recovery. For example, to terminate recovery after applying log sequence 15, enter:

RUN
{
  SET UNTIL SEQUENCE 16 THREAD 1;  # recovers up to but not including log 16
  DUPLICATE TARGET DATABASE TO 'dupdb';

}

Duplication Fails with Multiple RMAN-06023 Errors: Diagnosis

The DUPLICATE command recovers to archived redo logs, but cannot recover into online redo logs. Thus, if the restored backup cannot be made consistent without applying the online redo logs, then duplication fails with RMAN-06023 errors because RMAN is looking for backups created before the most recent archived log.

Duplication Fails with Multiple RMAN-06023 Errors: Solution

After backing up the source database, archive and back up the current redo log:

RMAN> SQL 'ALTER SYSTEM ARCHIVE LOG CURRENT';
RMAN> BACKUP ARCHIVELOG ALL;

This archives all records in the online redo logs so that RMAN can now recover the backup by applying the most recent archived redo log.

UNKNOWN Database Name Appears in Recovery Catalog: Diagnosis

One way you get the DB_NAME of UNKNOWN is when you register a database that was once opened with the RESETLOGS option. The DB_NAME can be changed during a RESETLOGS operation, so RMAN does not know what the DB_NAME was for those old incarnations of the database because it was not registered in the recovery catalog at the time. Consequently, RMAN sets the DB_NAME column to UNKNOWN when creating the DBINC record.

UNKNOWN Database Name Appears in Recovery Catalog: Solution

The UNKNOWN name entry is expected behavior after a RESETLOGS operation. You should not attempt to remove UNKNOWN entries from the recovery catalog.