$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.dppc.db application ONLINE ONLINE rac02
ora....c1.inst application ONLINE ONLINE rac01
ora....c2.inst application ONLINE ONLINE rac02
ora....01.lsnr application ONLINE ONLINE rac01
ora.rac01.gsd application ONLINE ONLINE rac01
ora.rac01.ons application ONLINE ONLINE rac01
ora.rac01.vip application ONLINE ONLINE rac01
ora....02.lsnr application ONLINE OFFLINE
ora.rac02.gsd application ONLINE ONLINE rac02
ora.rac02.ons application ONLINE ONLINE rac02
ora.rac02.vip application ONLINE ONLINE rac02

$ crs_start ora.rac02.LISTENER_RAC02.lsnr
Attempting to start `ora.rac02.LISTENER_RAC02.lsnr` on member `rac02`
Start of `ora.rac02.LISTENER_RAC02.lsnr` on member `rac02` failed.
rac01 : CRS-1018: Resource ora.rac02.vip (application) is already running on rac02
CRS-0215: Could not start resource 'ora.rac02.LISTENER_RAC02.lsnr'.

手动启动这个监听

$ lsnrctl start LISTENER_RAC02

LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 26-MAY-2008 15:39:41

Copyright (c) 1991, 2005, Oracle. All rights reserved.

Starting /opt/oracle/product/10.2/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.2.0.1.0 - Production
System parameter file is /opt/oracle/product/10.2/network/admin/listener.ora
Log messages written to /opt/oracle/product/10.2/network/log/listener_rac02.log
TNS-01150: The address of the specified listener name is incorrect

Listener failed to start. See the error message(s) above...

出现了TNS－01150错误，metalink上对此的解释是：

It is possible for the listener.ora file to become corrupted, and
hence, unusable. There maybe hidden misc characters within the file,
that prevents the listener from starting.
By creating a new file, rules out the possibility of dead characters
or potential corruption.

可见对listerner.ora写法比较敏感，于是检查2个节点的listerner.ora，发现故障节点的
(ADDRESS = (PROTOCOL = TCP)(HOST = xx.xx.xx.02)(PORT = 1521)(IP = FIRST)
)
结尾括号换行了，而正常节点的括号没有换行
(ADDRESS = (PROTOCOL = TCP)(HOST = xx.xx.xx.1)(PORT = 1521)(IP = FIRST))

将故障节点的括号上移，问题即得到了解决。

这个故事告诉我们，不要随便手动修改listerner.ora文件，需要作变动的时候应该使用netmgr这个工具

Related posts:

1. crs_stat中UNKNOWN状态的处理方法
2. 打包安装oracle
3. TNS-12602: TNS: Connection Pooling limit reached 的解决