Spark 集群出现 standby 的问题

重启 hdfs 集群后,出现的standy 错误的原因:

是因为没有启动zookeeper, zookeeper不会自动重启, zook的启动命令是zkServer.sh.

以后启动集群时,先启动那slave上安装了 zookeeper 的 zookeeper , 然后再启动 hdfs。

Master 上 的 zoomkeeper, vim /opt/zookeeper-3.4.9/conf/zoom.cfg:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/data/zkdata
dataLogDir=/data/zklog
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=slave1:2888:3888
server.2=slave2:2888:3888
server.3=slave3:2888:3888

Salve2 上: ll /opt:

1
2
3
4
5
6
drwxr-xr-x  9 root root 4096 Apr 13 18:02 apache-hive-2.1.1-bin/
drwxrwxr-x 10 root root 4096 Apr 13 01:35 hadoop-2.6.5/
drwxr-xr-x 7 root root 4096 May 24 22:40 kafka_2.11-0.10.1.0/
drwxrwxr-x 6 root root 4096 Apr 13 01:36 scala-2.11.8/
drwxr-xr-x 14 root root 4096 Apr 13 03:45 spark-2.1.0-bin-hadoop2.6/
drwxr-xr-x 10 root root 4096 Apr 13 02:06 zookeeper-3.4.9/

cd /opt/zookeeper-3.4.9 && vim conf/zoo.cfg :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/data/zkdata
dataLogDir=/data/zklog
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=slave1:2888:3888
server.2=slave2:2888:3888
server.3=slave3:2888:3888