今天在做恢复测试的时候遇到ORA-27121的错误,具体的表现为用sqlplus登录oracle的时候提示Connected to an idle instance,但是通过ps -ef仍然能够看到oracle的进程存在。
一、故障现象
oracle@bfdx-dsg:[/oracle]rman target / nocatalog
Recovery Manager: Release 10.2.0.2.0 - Production on Fri May 16 09:52:17 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
connected to target database (not started)
RMAN> run {
:@P\&Kn02> allocate channel t1 type 'sbt_tape';ITPUB个人空间 |#\l){6CtSe D,J
3> send 'NB_ORA_SERV=backsrv,NB_ORA_CLIENT=sx_kf_svc';
+P$dP1})T;l04> allocate channel t2 type 'sbt_tape';
$z[)s9mum ~05> send 'NB_ORA_SERV=backsrv,NB_ORA_CLIENT=sx_kf_svc';
?\ u P+u"a)V06> restore database;
Y4JwJ.Z1|/M_i07> release channel t1;ITPUB个人空间0iGq:QM~n#m&R
8> release channel t2;ITPUB个人空间7@ ?'T En1Eu!@[kh(`
9> }
RMAN-00571: ===========================================================
fI#E+zH0RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============ITPUB个人空间'A3CUS"C#z4D+M
RMAN-00571: ===========================================================ITPUB个人空间m.X.L;^d5mT{\
RMAN-03002: failure of allocate command at 05/16/2008 09:52:31ITPUB个人空间X bp|1w`Ef
RMAN-06403: could not obtain a fully authorized sessionITPUB个人空间#O {&F$u'XL`ew
ORA-01034: ORACLE not available
Ty#^X2mU.y#A;dw0ORA-27121: unable to determine size of shared memory segmentITPUB个人空间+V:]$?-a"m5\
IBM AIX RISC System/6000 Error: 13: Permission denied
RMAN> exitITPUB个人空间B&zQ"sr a V|
Recovery Manager complete.
oracle@bfdx-dsg:[/oracle]idITPUB个人空间cP2D#H:bgLi%DNh
uid=300(oracle) gid=300(dba) groups=1(staff)
oracle@bfdx-dsg:[/oracle]sqlplus /nolog
SQL*Plus: Release 10.2.0.2.0 - Production on Fri May 16 09:53:13 2008
}#~yT&[Y5v0Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
SQL> conn /as sysdba
A;ZO4Rv2] hT[0Connected to an idle instance.
SQL> startup nomountITPUB个人空间t1Z)r3t6Z2d
ORA-27121: unable to determine size of shared memory segmentITPUB个人空间~qI)@WJ
IBM AIX RISC System/6000 Error: 13: Permission denied
SQL> exit
6Mq`a]9R#I i0Disconnected
oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]ps -ef|grep oraITPUB个人空间8BKA#G w
oracle 290896 1 0 May 14 - 0:30 ora_ckpt_sibss
)@/Ho.k3k,iDI0 oracle 430180 639092 0 09:53:27 pts/1 0:00 sqlplusITPUB个人空间'KTc0{,z/JGgI's
oracle 450618 1 0 May 14 - 0:02 ora_lgwr_sibss
~z ^Ccs3]@1M"hH0 oracle 503824 671794 0 09:55:05 pts/0 0:00 ps -efITPUB个人空间s1ChR?sd
oracle 512110 1 0 May 14 - 0:03 ora_dbw0_sibssITPUB个人空间a0Ja\0?z
oracle 520252 1 0 May 14 - 0:04 ora_mman_sibss
#FLy4y$aW%D#iv U4m0 oracle 532528 1 0 May 14 - 0:02 ora_psp0_sibss
{+dt/Nz2F0 oracle 561394 1 0 May 14 - 0:28 ora_pmon_sibssITPUB个人空间"^2IP!t l?/v+B:A
oracle 565438 1 0 May 14 - 0:02 ora_smon_sibssITPUB个人空间%rf'xaY#X
oracle 610504 430180 0 09:53:27 - 0:00 oraclesibss (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))ITPUB个人空间)E)NWfO7G:Bi.j
oracle 622730 1 0 May 14 - 0:00 ora_reco_sibss
~-U2RT"g!dz9W0 oracle 639092 635114 0 09:53:21 pts/1 0:00 -kshITPUB个人空间rLH\3dc+\
oracle 647392 1 0 May 14 - 0:04 ora_mmon_sibss
6h!|K j{ Yz&g#X @P0 oracle 651310 671794 0 09:55:05 pts/0 0:00 grep oraITPUB个人空间9H,_q$t0E+O
oracle 671794 663674 0 09:52:00 pts/0 0:00 -ksh
y(?iM{*W!w4eS0 oracle 680056 1 0 May 14 - 0:01 ora_mmnl_sibss
&UeFkt3w:nu7a0 oracle 692432 1 0 May 14 - 0:01 ora_cjq0_sibss
oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]sqlplus '/as sysdba'
SQL*Plus: Release 10.2.0.2.0 - Production on Fri May 16 09:55:15 2008
qY5t4vn3FoJ%t0o{0Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
;C8Owy#u0Connected to an idle instance.
SQL> shutdown immediate
8e;|1mgzW*`b/d0ORA-01034: ORACLE not available
+M5Vr&fMkJO0ORA-27121: unable to determine size of shared memory segment
"b l$}8H5c-C;Lx0IBM AIX RISC System/6000 Error: 13: Permission denied
SQL> exit
-}-N hT'G:^a0Disconnected
二、查看日志文件,没有发现报错
oracle@bfdx-dsg:[/oracle]echo $ORACLE_BASE
(R\6YZ*N:Y0/oracle/app/oracle
oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]tail -f *.logITPUB个人空间f0H2v6bY'w
Setting recovery target incarnation to 2
-Ua'F6JI[ F0Wed May 14 15:21:21 2008ITPUB个人空间:[n,Z8y*@
Successful mount of redo thread 1, with mount id 1604448301ITPUB个人空间&pdcX.Pt
Wed May 14 15:21:21 2008ITPUB个人空间)]c&r@s d:r"{-s
Database mounted in Exclusive Mode
&^.Pa:L_W,N0Completed: ALTER DATABASE MOUNTITPUB个人空间,hw7S5B8S_9}DB(b
Fri May 16 09:53:33 2008ITPUB个人空间LE7M1rF [-M
Starting ORACLE instance (normal)ITPUB个人空间|[ Dw`#f
Fri May 16 09:54:01 2008ITPUB个人空间J }x-XR+bnw8K
Starting ORACLE instance (normal)
三、初步判断是共享内存段出现了问题
oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]ipcs -pmb
MqW0YO(qEd)oi0IPC status from /dev/mem as of Fri May 16 09:56:15 BEIDT 2008
U\M&h? P`~*f0T ID KEY MODE OWNER GROUP SEGSZ CPID LPID
7}"r sS2^tY+iL0Shared Memory:ITPUB个人空间 Z#Ogd4o6C"e
m 0 0x58002034 --rw-rw-rw- root system 134217728 61596 278664
i*\e$lO/Sn0m 1048577 0x52e74b4f --rw-rw-rw- imnadm imnadm 36028 422096 422096ITPUB个人空间5gb/[!i Q{7?Jr
m 1048578 0x9308e451 --rw-rw-rw- imnadm imnadm 97948 422096 422096ITPUB个人空间?^7t-W \F:B
m 1048579 0xe4663d62 --rw-rw-rw- imnadm imnadm 96 422096 422096
#r d+_#y|#Z-B{I0m 4 0xc76283cc --rw-rw-rw- imnadm imnadm 42268 422096 422096ITPUB个人空间k\!Lw*Du
m 5 0x298ee665 --rw-rw-rw- imnadm imnadm 2844 422096 422096
k r1N)h,T_2F0m 6 0xffffffff --rw-rw---- root system 4096 381128 381128
s9F1ZO4E`#k z!a0m 7 0x78000019 --rw-rw-rw- root system 16777216 483570 483570
6d|&S2sd*XX1wcg0m 8 0x0d003a64 --rw-rw---- root system 1440 344086 635114ITPUB个人空间] u9LN3n-@-Y$@3Tn#B_
m 8388617 0xc4d142b0 --rw-r----- 205 204 6442467328 618702 638976ITPUB个人空间2}Z K$t%Y
m 1048586 0x00005643 --rw-rw-rw- root system 1024 659660 688262ITPUB个人空间7~z2N^2?!go
m 1048587 0x00005654 --rw-rw-rw- root system 1024 659660 688262
在做测试的时候,由于测试机的oracle用户和dba组的id和生产库不一致,所以删除后重建了。ITPUB个人空间"YL8r2b!}"G
m 8388617 0xc4d142b0 --rw-r----- 205 204 6442467328 618702 638976
%u8Hiht `'m0这个共享段的205和204分别是删除之前的oracle用户和dba组id。
四、解决步骤
--杀掉oracle相关的进程
6v]4F}*Van0oracle@bfdx-dsg:[/oracle]ps -ef|grep $ORACLE_SID| \
.R`9{"SV?r S0> grep -v grep|awk '{print $2}'|xargs -i kill -9 {}
oracle@bfdx-dsg:[/oracle]ps -ef|grep ora
4E}#z:Y'rp-d0 oracle 430180 639092 0 09:53:27 pts/1 0:00 sqlplus
%@O O!O1os9U0 oracle 639092 635114 0 09:53:21 pts/1 0:00 -kshITPUB个人空间!^4m4eeD'z0xP\4DZ
oracle 671800 663674 0 10:00:38 pts/0 0:00 -kshITPUB个人空间U&Jt!w T$@F![
oracle 680058 671800 0 10:01:05 pts/0 0:00 grep oraITPUB个人空间aa'Y6B6^)\$oY
oracle 692434 671800 0 10:01:05 pts/0 0:00 ps -ef
--清除数据库占用的内存段
}nXc;R0oracle@bfdx-dsg:[/oracle]ipcrm -m 8388617
{+JtT/g["R/D0ipcrm: 0515-021 shmid(8388617): The user does not have the appropriate privileges to perform. this operation.
oracle@bfdx-dsg:[/oracle]exit
bfdx-dsg:root:/acct>ipcs -pmb
5Y)Z@v%zbjY7}0IPC status from /dev/mem as of Fri May 16 10:03:14 BEIDT 2008ITPUB个人空间 `|,L1k5]M ZE
T ID KEY MODE OWNER GROUP SEGSZ CPID LPIDITPUB个人空间W2Bei)M.}5M ?
Shared Memory:
"Y ^\3{2PdIs/k0m 0 0x58002034 --rw-rw-rw- root system 134217728 61596 278664
vU1A kmfo0m 1048577 0x52e74b4f --rw-rw-rw- imnadm imnadm 36028 422096 422096ITPUB个人空间u.H p Q#~-E
m 1048578 0x9308e451 --rw-rw-rw- imnadm imnadm 97948 422096 422096
| m2Dm};V0m 1048579 0xe4663d62 --rw-rw-rw- imnadm imnadm 96 422096 422096
7E9_7W;]/h};o2FF9j0m 4 0xc76283cc --rw-rw-rw- imnadm imnadm 42268 422096 422096ITPUB个人空间Hk7Pse#^
m 5 0x298ee665 --rw-rw-rw- imnadm imnadm 2844 422096 422096ITPUB个人空间3H7U$A%_!p D
m 6 0xffffffff --rw-rw---- root system 4096 381128 381128ITPUB个人空间m;geo+E| p:E"Ub
m 7 0x78000019 --rw-rw-rw- root system 16777216 483570 483570
k;C5W*Ovo0m 8 0x0d003a64 --rw-rw---- root system 1440 344086 635114
ogZ_mY0m 8388617 0xc4d142b0 --rw-r----- 205 204 6442467328 618702 692432
n.B{RR Iy4^0m 1048586 0x00005643 --rw-rw-rw- root system 1024 659660 688262ITPUB个人空间/T(m op]
m 1048587 0x00005654 --rw-rw-rw- root system 1024 659660 688262
bfdx-dsg:root:/acct>ipcrm -m 8388617ITPUB个人空间1JtuU&[ yC8^o,z2]3C
bfdx-dsg:root:/acct>su - oracleITPUB个人空间I`Gq8RMT%L;a:I
oracle@bfdx-dsg:[/oracle]sqlplus '/as sysdba'
SQL*Plus: Release 10.2.0.2.0 - Production on Fri May 16 10:03:35 2008
fLf2h[4vJ&z0Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
0v6RLN"P@P[M0Connected to an idle instance.
SQL> startup nomount
kAjb"Pe0ORACLE instance started.
Total System Global Area 6442450944 bytesITPUB个人空间_W"~#n$km8fj2f:[ f5\
Fixed Size 2080904 bytes
b;N)?'Vif6e6FIK0Variable Size 2013269880 bytes
1q [%R{Z7yLU B0Database Buffers 4412407808 bytesITPUB个人空间+~m+^2j0B;g
Redo Buffers 14692352 bytes
OK,问题解决。