偶然一次停电,导致机房机器重启,于是很多机器静态IP冲突失效。好不容易把网络调整好,Hbase集群却无法正常启动,Hadoop却可以正常使用。大致情况如下: 正常启动Hadoop和zookeeper, 然后启动Hbase,jps查看一切正常。 当打开web页面(60010)查看的时候却出现500错误,错误大致如下: HTTP ERROR: 500 RequestURI=/master.jsp Caused by: at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1002) 2011-12-08 14:34:54,305 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up. 2011-12-08 14:34:54,305 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping. 2011-12-08 14:34:55,305 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region. 2011-12-08 14:34:55,307 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.10.11.184:60020 2011-12-08 14:34:55,308 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up. 2011-12-08 14:35:15,404 INFO org.apache.hadoop.hbase.master.ServerManager: 4 region servers, 0 dead, average load 0.3333333333333333 2011-12-08 14:35:15,484 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.10.11.184:60020, regionname: -ROOT-,,0, startKey: <>} 2011-12-08 14:35:15,485 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up. 2011-12-08 14:35:15,485 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1 at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:429) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:918) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:934) at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:173) at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54) at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79) at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153) at org.apache.hadoop.hbase.Chore.run(Chore.java:68) 于是查看regionServer的日志,初次一看感觉没什么错误,就没怎么注意,然后继续纠结在其他地方……后来多次失败后重新审视日志的时候才发现了一些端倪,如下 2011-12-08 22:45:02,365 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Created ZNode /hbase/rs/1323355502342 with data 0:0:0:0:0:0:0:1:60020 2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.regionserver.address=10.10.11.184 2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: fs.default.name=hdfs://hadoop5-virtual-machine:9000/hbase 2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.rootdir=hdfs://hadoop5-virtual-machine:9000/hbase 2011-12-08 22:45:02,395 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Master passed us address to use. Was=0:0:0:0:0:0:0:1:60020, Now=10.10.11.184 2011-12-08 22:45:02,510 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Log dir hdfs://hadoop5-virtual-machine:9000/hbase/.logs/hadoop4-virtual-machine,60020,1323355502342 2011-12-08 22:45:02,521 INFO org.apache.hadoop.hbase.regionserver.HLog: HLog configuration: blocksize=67108864, rollsize=63753420, enabled=true, flushlogentries=100, optionallogflushinternal=10000ms 2011-12-08 22:45:02,561 INFO org.apache.hadoop.hbase.regionserver.HLog: New hlog /hbase/.logs/hadoop4-virtual-machine,60020,1323355502342/hlog.dat.1323355502522 2011-12-08 22:45:02,566 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=RegionServer, sessionId=regionserver/0:0:0:0:0:0:0:1:60020 然后我在slave节点下jps查看,发现hbase相关的regionServer的进程挂掉了,而master的进程还在。 由上蓝色标记字体看出:为什么会出现IP6的地址呢?检查配置文件也没有出错,因为所有的配置文件都是用主机名替代了IP地址。于是就想到了是不是hosts文件的映射出错了。 打开/etc/hosts文件,果不其然,regionServer的主机名映射的是 ::1, 我记得这个主机名和当前IP的映射应该是网卡初始化的时候由NetworkManager自动加上去的, 回顾整个hadoop集群启动失败这个过程,先是断电,然后是静态IP失效,问题应该就是在这个时候产生的,在实效的同试hosts文件被修改。 其实HBASE集群配置相当简单,但由于一些小小的失误,往往是我们自以为不会出错的地方,查看log的时候也不细心,然后以至于浪费我们很多的时间。至于IP与主机名映射的问题是我们配置集群的时候最容易忽视的地方,大家在下次碰到这类相关问题的时候不妨去看看hosts文件。 |
|