dr.who
— 焉知非鱼提交了 Spark Streaming 程序后，看到 Spark Streaming 的 UI 界面， Executors 下面只有一个 driver 在运行，Streaming 界面一直在排队，像卡住了一样！点进去 driver 的 logs -> stdout 界面，看到很多这样的日志：
18/09/04 19:10:45 org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66) WARN YarnClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
18/09/04 19:10:51 org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66) WARN YarnAllocator: Container marked as failed: container_e13_1534402443030_0354_02_000125 on host: WMBigdata6. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_e13_1534402443030_0354_02_000125
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
at org.apache.hadoop.util.Shell.run(Shell.java:507)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
YARN调度报错Stack trace: ExitCodeException exitCode=1解决方式, 这篇日志说是权限问题， 于是找到 CDH 集群下 RESOURCEMANAGER 的日志：
vi /var/log/hadoop-yarn/hadoop-cmf-yarn-RESOURCEMANAGER-WMBigdata0.log.out
看到类似的日志：
2018-08-30 12:58:36,062 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:appattempt_1534402443030_0230_000001 (auth:TOKEN) cause:org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt appattempt_1534402443030_0230_000001 doesn't exist in ApplicationMasterService cache.
2018-08-30 12:58:36,062 INFO org.apache.hadoop.ipc.Server: IPC Server handler 40 on 8030, call org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 10.0.201.124:56400 Call#2491 Retry#0
org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt appattempt_1534402443030_0230_000001 doesn't exist in ApplicationMasterService cache.
at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:442)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2222)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2220)
还发现一个 dr.who:
2018-08-31 10:52:34,563 INFO org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is accessing unchecked http://WMBigdata3:42229/api/v1/applications/application_1534402443030_0277/allexecutors which is the app master GUI of application_1534402443030_0277 owned by root
各种搜索一无所获，此时还注意到一个异常：
Slow ReadProcessor read fields took 30001ms (threshold=30000ms);
reduce100%卡死故障排除 这里让我查 DataNode 的日志：
vi /var/log/hadoop-hdfs/hadoop-cmf-hdfs-DATANODE-bigdata3.log.out
搜索一下 error 关键字：
2018-08-31 16:14:45,695 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: bigdata3:50010:DataXceiver error processing WRITE_BLOCK operation src: /10.0.166.172:45462 dst: /10.0.166.172:50010
java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:203)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:501)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:901)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:808)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
at java.lang.Thread.run(Thread.java:748)
谷歌到了这个解决方法
/etc/security/limits.conf
# End of file
* - nofile 1000000
* - nproc 1000000
在 CDH UI 界面中修改 dfs.datanode.max.transfer.threads 的值 4096, 改为 8192， 然后重启 hdfs 服务：
<property>
<name>dfs.datanode.max.transfer.threads</name>
<value>8192</value>
<description>
Specifies the maximum number of threads to use for transferring data
in and out of the DN.
</description>
</property>
过了一会儿， Spark Streamig 的界面好了， 不卡了。大西瓜啊， 之前忘记修改最大文件数了。