build.sbt:
name := “test3”
version := “0.1”
scalaVersion := “2.12.10”
libraryDependencies ++= Seq(
“org.apache.spark” %% “spark-core” % “2.4.7”,
“org.apache.spark” %% “spark-sql” % “2.4.7”,
“org.apache.spark” %% “spark-mllib” % “2.4.7”,
“org.apache.spark” %% “spark-streaming” % “2.4.7”,
“org.twitter4j” % “twitter4j-core” % “4.0.4”,
“org.twitter4j” % “twitter4j-stream” % “4.0.4”,
“org.deeplearning4j” % “deeplearning4j-core” % “1.0.0-beta7”,
“org.deeplearning4j” % “deeplearning4j-ui” % “1.0.0-beta7”,
“org.nd4j” % “nd4j-native-platform” % “1.0.0-beta7”,
“org.scala-lang.modules” %% “scala-parser-combinators” % “1.0.4”
)
JDK: 15
IDE: Intellij, Language: Scala
I’m building the files from build.sbt using package by command line, then running them by spark-submit on the command line or on EMR (which leads to the same errors).
The problem with not compiling due to IUpdater vanished once I implemented it. However, the IDE and package command compile the program even when all the interfaces IUpdater uses aren’t imported. Once I use the spark-submit command errors appear, sometimes even when I import the interfaces. So far I’ve had errors with GradientUpdater, INDarray, and ISchedule.