详解hbase搭建遇到问题及解决方法

详解hbase搭建遇到问题及解决方法

1.zookeeper启动报错

错误日志  启动zookeeper报错信息如下：

java.net.NoRouteToHostException: No route to host        at java.net.PlainSocketImpl.socketConnect(Native Method)        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)        at java.net.Socket.connect(Socket.java:579)        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)        at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)        at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)2015-05-19 10:26:26,983 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 800

java.net.NoRouteToHostException: No route to host at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)2015-05-19 10:26:26,983 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 800

解决方法

此问题产生的主要原因是因为zookeeper集群未关闭防火墙。     执行下面命令后仍然报上面的错误：     systemctl start iptables.service     经过仔细查找后发现，CentOS 7.0默认使用的是firewall作为防火墙，需要执行如下命令关闭防火墙:         systemctl stop firewalld.service #停止firewall         systemctl disable firewalld.service #禁止firewall开机启动     关闭各个节点防火墙后，重启zookeeper进程，就可以解决上述问题了。

2.RegionServer进程挂掉

错误日志

Caused by: java.io.IOException: Couldn't set up IO streams    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:786)    at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)    at org.apache.hadoop.ipc.Client.call(Client.java:1438)    ... 60 moreCaused by: java.lang.OutOfMemoryError: unable to create new native thread    at java.lang.Thread.start0(Native Method)    at java.lang.Thread.start(Thread.java:713)    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:779)

Caused by: java.io.IOException: Couldn't set up IO streams at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:786) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521) at org.apache.hadoop.ipc.Client.call(Client.java:1438) ... 60 moreCaused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:713) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:779)

解决方法

/etc/security/limits.conf

/etc/security/limits.d/90-nproc.conf

*       hard    nproc   65536*       soft    nproc   65536*       hard    nofile  65536*       soft    nofile  65536

* hard nproc 65536* soft nproc 65536* hard nofile 65536* soft nofile 65536

3.增加thrift server的线程数

日志信息

2015-06-05 12:46:37,756 INFO  [thrift-worker-6] client.AsyncProcess: #79, waiting for 72000  actions to finish2015-06-05 12:46:37,756 INFO  [thrift-worker-9] client.AsyncProcess: #79, waiting for 48908  actions to finish2015-06-05 12:46:37,855 INFO  [thrift-worker-8] client.AsyncProcess: #79, waiting for 72000  actions to finish2015-06-05 12:46:38,198 INFO  [thrift-worker-2] client.AsyncProcess: #1, waiting for 78000  actions to finish2015-06-05 12:46:38,762 INFO  [thrift-worker-13] client.AsyncProcess: #79, waiting for 72000  actions to finish2015-06-05 12:46:39,547 INFO  [thrift-worker-0] client.AsyncProcess: #17, waiting for 78000  actions to finish2015-06-05 12:47:55,612 INFO  [thrift-worker-9] client.AsyncProcess: #79, waiting for 108000  actions to finish2015-06-05 12:47:55,912 INFO  [thrift-worker-6] client.AsyncProcess: #79, waiting for 114000  actions to finish

2015-06-05 12:46:37,756 INFO [thrift-worker-6] client.AsyncProcess: #79, waiting for 72000 actions to finish2015-06-05 12:46:37,756 INFO [thrift-worker-9] client.AsyncProcess: #79, waiting for 48908 actions to finish2015-06-05 12:46:37,855 INFO [thrift-worker-8] client.AsyncProcess: #79, waiting for 72000 actions to finish2015-06-05 12:46:38,198 INFO [thrift-worker-2] client.AsyncProcess: #1, waiting for 78000 actions to finish2015-06-05 12:46:38,762 INFO [thrift-worker-13] client.AsyncProcess: #79, waiting for 72000 actions to finish2015-06-05 12:46:39,547 INFO [thrift-worker-0] client.AsyncProcess: #17, waiting for 78000 actions to finish2015-06-05 12:47:55,612 INFO [thrift-worker-9] client.AsyncProcess: #79, waiting for 108000 actions to finish2015-06-05 12:47:55,912 INFO [thrift-worker-6] client.AsyncProcess: #79, waiting for 114000 actions to finish

解决方法

增加thriftServer线程数

hbase-daemon.sh start thrift --threadpool -m 200 -w 500

在hbase_home目录下的logs目录中可以看到启动日志信息如下：

INFO  [main] thrift.ThriftServerRunner: starting TBoundedThreadPoolServer on /0.0.0.0:9090; min worker threads=200, max worker threads=500, max queued requests=1000

INFO [main] thrift.ThriftServerRunner: starting TBoundedThreadPoolServer on /0.0.0.0:9090; min worker threads=200, max worker threads=500, max queued requests=1000

hbase-daemon.sh start thrift –threadpool -m

hbase-daemon.sh start thrift –threadpool -m

4. zookeeper使用内存大的问题

日志信息

jps 查看 QuorumPeerMain 进程IP jmap -heap PID 查看进程使用内存情况,具体情况如下：

解决方法

方法一：

Heap Configuration配置中我们可以看到，配置的heap内存很大，现在我们修改zkServer.sh脚本减小MaxHeapSize，具体步骤如下：     * 打开zookeeper安装目录下bin文件夹中的zkServer.sh     * 在 zkServer.sh文件的49行处加入 JVMPARAM="-Xms1000M -Xmx1000M -Xmn512M"     * 然后修改zkServer.sh 109~110行出的内容。

修改前如下：

nohup "$JAVA" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \ -cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

nohup "$JAVA" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \ -cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

修改后如下(将在49上添加的JVMPARAM参数项添加在JVMFLAGS后面)：

nohup "$JAVA" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \-cp "$CLASSPATH" $JVMFLAGS $JVMPARAM $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

nohup "$JAVA" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \-cp "$CLASSPATH" $JVMFLAGS $JVMPARAM $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

重启zookeeper进程 jmap -heap PID 查看修改后进程使用内存情况：

方法二：

打开 zookeeper/bin/zkEnv.sh 文件,在zkEvn.sh中49~52行处有如下内容：

if [ -f "$ZOOCFGDIR/java.env" ]then    . "$ZOOCFGDIR/java.env"fi

if [ -f "$ZOOCFGDIR/java.env" ]then . "$ZOOCFGDIR/java.env"fi

该文件已经明确说明有独立JVM内存的设置文件，路径是zookeeper/conf/java.env     安装的时候这个路径下没有有java.env文件，需要自己新建一个：         * vim java.env         * java.env文件内容如下：

#!/bin/sh# heap size MUST be modified according to cluster environmentexport JVMFLAGS="-Xms512m -Xmx1024m $JVMFLAGS"

#!/bin/sh# heap size MUST be modified according to cluster environmentexport JVMFLAGS="-Xms512m -Xmx1024m $JVMFLAGS"

重启zookeeper进程 jmap -heap PID 查看修改后进程使用内存情况：