NoClassDefFound

升级了一下 IDEA, 运行 spark 程序的时候给我报错了,

1
NoClassDefFound : Scala/xml/metadata

解决方法, 在pom 文件里面添加如下依赖:

1
2
3
4
5
6
7
8
9
10
11
12
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.8</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.scala-lang.modules/scala-xml -->
<dependency>
<groupId>org.scala-lang.modules</groupId>
<artifactId>scala-xml_2.11</artifactId>
<version>1.1.0</version>
</dependency>

CDH 集群,Spark 升级到 2.3 后,运行任务报错:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

运行 spark-shell 也会报错,原因是,分配 spark2 的角色时, 没有为所有机器分配该角色。

解决方法: 给每一台机器都添加上 gateway 角色。

1
Service 'sparkDriver' failed after 16 retries!

解决办法

https://stackoverflow.com/questions/30085779/apache-spark-error-while-start/30952312#30952312

1
2
3
4
5
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.3.0</version>
</dependency>

artifact => 工件

mvn 命令行打包:

1
mvn clean package -DskipTests=true