1.问题分析问题描述windows环境oracle 11.2.0.1版本数据库异常关闭之后无法启动无备份、未打开归档。故障分析1.直观查看数据库的数据文件、REDO文件均在查看数据文件的最后修改时间除SYSAUX02.DBF外其它数据文件时间接近2.基于数据库检查点机制分析假定REDO文件未损坏则数据库的检查点在全部REDO日志切换轮循使用过程中必然发生一次因此所有REDO文件均未损坏时可以正常OPEN数据库此种情景如下两种情况均不会导致数据丢失1.实例恢复时数据库会自动使用REDO文件来进行crash recovery并OPEN数据库2.控制文件损坏使用备份或重建控制文件时此时recover database时人工指定日志文件路径让数据库进行恢复。因此基于此分析可以尝试数据库异常OPEN时的处理思路一步步处理。处理过程1.分析数据库中数据文件头及控制文件中记录的数据文件检查点SCN信息判断数据文件的一致性及可能的数据丢失情况FILE# ctl-CHECKPOINT_CHANGE# ctl-LAST_CHANGE# dbf-CHECKPOINT_CHANGE# STATUS FUZ---------- ---------------------- ---------------- ---------------------- ------- ---2 1176659225 1036813104 ONLINE NO 观察此文件的数据文件头检查点SCN与控制文件中记录的差距巨大10 1176708807 1176708807 OFFLINE NO 观察此文件数据文件头检查点SCN与控制文件中记录值一致但是OFFLINE状态怀疑是人为故障处理时操作。2.ONLINE 10号数据文件后正常OPEN数据库查看信息并进一步处理3.正常OPEN时提示需要以RESETLOGS方式打开数据库怀疑前期重建过控制文件。此时直接重建控制文件并进行RECOVER DATABASE操作4.ALTER DATABASE RECOVER database using backup controlfile 恢复时出现报错ORA-00310: archived log contains sequence 29326; sequence 26024 required从此报错可以发现RECOVER数据库时需要用到很久远之前的序号26024号归档日志而当前的REDO LOG日志序号是29326结合前期观察到的数据文件最后修改日期、数据文件头SCN等对比信息判断为2号数据文件的问题。分析(1)、2号文件为SYSAUX结合SCN数据来看此文件状态与最新差异较多如果使用BBED修改数据文件头方式虽然可以骗过OPEN时的检查点校验但是在应用REDO LOG进行实例恢复阶段较大可能会出现数据不一致的错误同时在OPEN后事务回滚阶段还会有UNDO回滚问题因此一般采用此方法。(2)、如不管数据文件头SCN不一致的情况使用隐含参数_allow_resetlogs_corruptiontrue强制打开数据库会有报错ORA-01248: file 4 was created in the future of incomplete recovery这是由于重建控制文件不进去恢复此时认为SCN是异常的2号文件的时间(几个月前)而此时间点之后有新增加的数据文件所以报错由于新增加的文件有SYSTEM表空间的因此也无法将这种后续添加的文件OFFLINE状态后打开数据库。(3)基于对数据库启动机制的了解SYSTEM/UNDO损坏时不能打开SYSAUX的文件损坏情况下可以强制打开数据库经过前期分析、尝试确认以OFFLINE 2号文件(SYSAUX01.DBF)方式尝试打开数据库。5.OFFLINE 2号文件(SYSAUX01.DBF)并进行RECOVER通过输入文件名方式将3个REDO文件放在尝试可以正常恢复。6.OPEN数据库成功临时表空间需要处理。后续导出数据时由于SYSAUX表空间文件OFFLINE无法使用EXPDP以及EXP导出USERS模式可以用EXP导出表模式批量导表。1.分析数据库中数据文件头及控制文件中记录的数据文件检查点SCN信息判断数据文件的一致性及可能的数据丢失情况FILE# ctl-CHECKPOINT_CHANGE# ctl-LAST_CHANGE# dbf-CHECKPOINT_CHANGE# STATUS FUZ---------- ---------------------- ---------------- ---------------------- ------- ---1 1176708807 1176708807 ONLINE NO2 1176659225 1036813104 ONLINE NO 观察此文件的数据文件头检查点SCN与控制文件中记录的差距巨大3 1176659225 1176618204 ONLINE YES4 1176659225 1176618204 ONLINE YES5 1176659225 1176618204 ONLINE YES6 1176659225 1176618204 ONLINE YES7 1176659225 1176618204 ONLINE YES8 1176659225 1176618204 ONLINE YES9 1176659225 1176618204 ONLINE YES10 1176708807 1176708807 OFFLINE NO 观察此文件数据文件头检查点SCN与控制文件中记录值一致但是OFFLINE状态怀疑是人为故障处理时操作。11 1176659225 1176618204 ONLINE YES2.ONLINE 10号数据文件后正常OPEN数据库查看信息并进一步处理Sun Oct 10 20:24:55 2021alter database datafile 10 onlineCompleted: alter database datafile 10 onlineSun Oct 10 20:25:19 2021alter database openErrors in file d:\app\admin\diag\rdbms\orcl\orcl\trace\orcl_ora_6860.trc:ORA-01589: 要打开数据库则必须使用 RESETLOGS 或 NORESETLOGS 选项ORA-1589 signalled during: alter database open...3.正常OPEN时提示需要以RESETLOGS方式打开数据库怀疑前期重建过控制文件。此时直接重建控制文件并进行RECOVER DATABASE操作Sun Oct 10 20:29:23 2021Successful mount of redo thread 1, with mount id 1613376419Completed: CREATE CONTROLFILE REUSE DATABASE ORCL RESETLOGS NOARCHIVELOGMAXLOGFILES 16MAXLOGMEMBERS 3MAXDATAFILES 100MAXINSTANCES 8MAXLOGHISTORY 292LOGFILEGROUP 1 D:\app\Administrator\oradata\orcl\REDO01.log SIZE 50M BLOCKSIZE 512,GROUP 2 D:\app\Administrator\oradata\orcl\REDO02.log SIZE 50M BLOCKSIZE 512,GROUP 3 D:\app\Administrator\oradata\orcl\REDO03.log SIZE 50M BLOCKSIZE 512DATAFILED:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF,D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSAUX01.DBF,D:\APP\ADMINISTRATOR\ORADATA\ORCL\UNDOTBS01.DBF,D:\APP\ADMINISTRATOR\ORADATA\ORCL\USERS01.DBF,D:\APP\ADMINISTRATOR\ORADATA\ORCL\FDC.DBF,D:\APP\ADMINISTRATOR\ORADATA\ORCL\ZCGL.DBF,D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM.DBF,D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM02.DBF,D:\APP\ADMINISTRATOR\ORADATA\ORCL\ZCGL2.DBF,D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSAUX02.DBF,D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM03.DBFCHARACTER SET AL32UTF8ALTER DATABASE RECOVER database using backup controlfileMedia Recovery Startstarted logmerger processSun Oct 10 20:33:35 2021……Parallel Media Recovery started with 4 slavesORA-279 signalled during: ALTER DATABASE RECOVER database using backup controlfile ...Sun Oct 10 20:34:05 2021ALTER DATABASE RECOVER LOGFILE d:\app\administrator\oradata\orcl\redo01.logMedia Recovery Log d:\app\administrator\oradata\orcl\redo01.logSun Oct 10 20:34:05 2021Errors with log d:\app\administrator\oradata\orcl\redo01.logErrors in file d:\app\admin\diag\rdbms\orcl\orcl\trace\orcl_pr00_9952.trc:ORA-00310: archived log contains sequence 29326; sequence 26024 requiredORA-00334: archived log: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOGORA-310 signalled during: ALTER DATABASE RECOVER LOGFILE d:\app\administrator\oradata\orcl\redo01.log ...ALTER DATABASE RECOVER CANCELMedia Recovery CanceledCompleted: ALTER DATABASE RECOVER CANCELALTER DATABASE RECOVER database using backup controlfileMedia Recovery Startstarted logmerger process4.ALTER DATABASE RECOVER database using backup controlfile 恢复时出现报错ORA-00310: archived log contains sequence 29326; sequence 26024 required从此报错可以发现RECOVER数据库时需要用到很久远之前的序号26024号归档日志而当前的REDO LOG日志序号是29326结合前期观察到的数据文件最后修改日期、数据文件头SCN等对比信息判断为2号数据文件的问题。分析(1)、2号文件为SYSAUX结合SCN数据来看此文件状态与最新差异较多如果使用BBED修改数据文件头方式虽然可以骗过OPEN时的检查点校验但是在应用REDO LOG进行实例恢复阶段较大可能会出现数据不一致的错误同时在OPEN后事务回滚阶段还会有UNDO回滚问题因此一般采用此方法。(2)、如不管数据文件头SCN不一致的情况使用隐含参数_allow_resetlogs_corruptiontrue强制打开数据库会有报错ORA-01248: file 4 was created in the future of incomplete recovery这是由于重建控制文件不进去恢复此时认为SCN是异常的2号文件的时间(几个月前)而此时间点之后有新增加的数据文件所以报错由于新增加的文件有SYSTEM表空间的因此也无法将这种后续添加的文件OFFLINE状态后打开数据库。(3)基于对数据库启动机制的了解SYSTEM/UNDO损坏时不能打开SYSAUX的文件损坏情况下可以强制打开数据库经过前期分析、尝试确认以OFFLINE 2号文件(SYSAUX01.DBF)方式尝试打开数据库。FILE# ctl-CHECKPOINT_CHANGE# dbf-CHECKPOINT_CHANGE# STATUS FUZ---------- ---------------------- ---------------------- ------- ---1 1176708807 1176708807 ONLINE NO2 1176659225 1036813104 ONLINE NO 观察此文件的数据文件头检查点SCN与控制文件中记录的差距巨大SQL select * from v$dbfile order by 1;FILE# NAME---------- --------------------------------------------------1 D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF2 D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSAUX01.DBF5.OFFLINE 2号文件(SYSAUX01.DBF)并进行RECOVER通过文件名方式将3个REDO文件放在尝试可以正常恢复。Sun Oct 10 20:38:59 2021alter database datafile 2 offlineCompleted: alter database datafile 2 offlineSun Oct 10 20:39:22 2021ALTER DATABASE RECOVER database using backup controlfileMedia Recovery Startstarted logmerger processSun Oct 10 20:39:22 2021…………Parallel Media Recovery started with 4 slavesWarning: Datafile 2 (D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSAUX01.DBF) is offline during full database recovery and will not be recoveredORA-279 signalled during: ALTER DATABASE RECOVER database using backup controlfile ...ALTER DATABASE RECOVER LOGFILE d:\app\administrator\oradata\orcl\redo01.logMedia Recovery Log d:\app\administrator\oradata\orcl\redo01.logErrors with log d:\app\administrator\oradata\orcl\redo01.logErrors in file d:\app\admin\diag\rdbms\orcl\orcl\trace\orcl_pr00_1364.trc:ORA-00310: archived log contains sequence 29326; sequence 29327 required ORA-00334: archived log: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOGORA-310 signalled during: ALTER DATABASE RECOVER LOGFILE d:\app\administrator\oradata\orcl\redo01.log ...ALTER DATABASE RECOVER CANCELMedia Recovery CanceledCompleted: ALTER DATABASE RECOVER CANCELSun Oct 10 20:39:34 2021ALTER DATABASE RECOVER database using backup controlfileMedia Recovery Startstarted logmerger processSun Oct 10 20:39:34 2021………………Parallel Media Recovery started with 4 slavesWarning: Datafile 2 (D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSAUX01.DBF) is offline during full database recovery and will not be recoveredORA-279 signalled during: ALTER DATABASE RECOVER database using backup controlfile ...ALTER DATABASE RECOVER LOGFILE d:\app\administrator\oradata\orcl\redo02.logMedia Recovery Log d:\app\administrator\oradata\orcl\redo02.logSun Oct 10 20:39:41 2021Hex dump of (file 11, block 1796922) in trace file d:\app\admin\diag\rdbms\orcl\orcl\trace\orcl_pr04_8636.trcCorrupt block relative dba: 0x02db6b3a (file 11, block 1796922)Fractured block found during media recoveryData in bad block:type: 6 format: 2 rdba: 0x02db6b3alast change scn: 0x0000.46225752 seq: 0x2 flg: 0x04spare1: 0x0 spare2: 0x0 spare3: 0x0consistency value in tail: 0x00000001check value in block header: 0xe22fcomputed block checksum: 0xd561Reading datafile D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM03.DBF for corruption at rdba: 0x02db6b3a (file 11, block 1796922)Reread (file 11, block 1796922) found same corrupt data 有坏块此文件为SYSTEM03基表数据不在其中ORA-279 signalled during: ALTER DATABASE RECOVER LOGFILE d:\app\administrator\oradata\orcl\redo02.log ...Sun Oct 10 20:39:54 2021ALTER DATABASE RECOVER LOGFILE d:\app\administrator\oradata\orcl\redo03.logMedia Recovery Log d:\app\administrator\oradata\orcl\redo03.logCompleted: ALTER DATABASE RECOVER LOGFILE d:\app\administrator\oradata\orcl\redo03.log6.OPEN数据库成功临时表空间需要处理。后续导出数据时由于SYSAUX表空间文件OFFLINE无法使用EXPDP以及EXP导出USERS模式可以用EXP导出表模式批量导表。Sun Oct 10 20:40:09 2021alter database open resetlogsRESETLOGS is being done without consistancy checks. This may resultin a corrupted database. The database should be recreated.RESETLOGS after incomplete recovery UNTIL CHANGE 1176708807Resetting resetlogs activation ID 1548501587 (0x5c4c4253)Sun Oct 10 20:40:09 2021Setting recovery target incarnation to 2Sun Oct 10 20:40:09 2021Assigning activation ID 1613415321 (0x602ac399)Thread 1 opened at log sequence 1Current log# 1 seq# 1 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOGSuccessful open of redo thread 1Sun Oct 10 20:40:09 2021SMON: enabling cache recoverySuccessfully onlined Undo Tablespace 2.Dictionary check beginningTablespace TEMP #3 found in data dictionary,but not in the controlfile. Adding to controlfile.File #2 is offline, but is part of an online tablespace.data file 2: D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSAUX01.DBFDictionary check completeVerifying file header compatibility for 11g tablespace encryption..Verifying 11g file header compatibility for tablespace encryption completedSMON: enabling tx recovery…………Updating character set in controlfile to ZHS16GBKNo Resource Manager plan active…………replication_dependency_tracking turned off (no async multimaster replication found)Errors in file d:\app\admin\diag\rdbms\orcl\orcl\trace\orcl_ora_4692.trc (incident140969):ORA-25319: 队列表重新分区已中止Incident details in: d:\app\admin\diag\rdbms\orcl\orcl\incident\incdir_140969\orcl_ora_4692_i140969.trcerror 25319 happened during Queue table repartitioningStarting background process QMNCSun Oct 10 20:40:11 2021QMNC started with pid22, OS id8456XDB UNINITIALIZED: XDB$SCHEMA not accessibleLOGSTDBY: Validating controlfile with logical metadataLOGSTDBY: Validation completeSun Oct 10 20:40:11 2021Sweep [inc][140969]: completedSweep [inc2][140969]: completedSun Oct 10 20:40:11 2021ORA-376 encountered when generating server alert SMG-4120Sun Oct 10 20:40:11 2021Checker run found 1 new persistent data failuresSun Oct 10 20:40:11 2021Trace dumping is performing id[cdmp_20211010204011]Completed: alter database open resetlogs