Variable Length Output Layer for FeedForward Network

DL4J
1 
class CosineSimilarityVertex extends SameDiffLambdaVertex {
     override def defineVertex(sameDiff: SameDiff, inputs:  SameDiffLambdaVertex#VertexInputs): SDVariable = {
      val x = inputs.getInput(0)
      val y = inputs.getInput(1)
      //calculate cosine similarity
      val x_norm = x.norm2(-1)
      val y_norm = y.norm2(-1)
      val dot = sameDiff.sum(x.mul(y), -1)
      dot.div(x_norm.mul(y_norm))
    }
  }

  def createModel(numOutput: Int, numInput: Int, numActions: Int) = {
    val conf = new NeuralNetConfiguration.Builder()
      .seed(123)
      .updater(new Nadam(0.0001))
      .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
      .graphBuilder()
      .addInputs("Features1", "Features2")
      .addLayer("FeatureLayer1", new DenseLayer.Builder().activation(Activation.TANH).nIn(numInput).nOut(numInput*2).weightInit(WeightInit.XAVIER).build(), "Features1")
      .addLayer("FeatureLayer2", new DenseLayer.Builder().activation(Activation.TANH).nIn(numInput*2).nOut(numOutput).weightInit(WeightInit.XAVIER).build(), "FeatureLayer1")
      .addLayer("Embedding", new EmbeddingLayer.Builder().nIn(numActions).nOut(numOutput).build(), "Features2")
      .addVertex("CosineSimilarity", new CosineSimilarityVertex(), "FeatureLayer2", "Embedding")
      .setOutputs("CosineSimilarity")
      .build()

    val model = new ComputationGraph(conf)
    model.init()
    model
  }

    val feature1 = Nd4j.create(Array(1, 0.5, 1.0, 1.2, 0.2, 0.4, 0.5), Array(1, 7))
    val feature2 = Nd4j.create(Array(1.0, 2.0, 0.0, 3.0), Array(4, 1)) //arbitrary size from 1->n

    val model = createModel(2, 7, 4)
    val prediction = model.output(false, feature1, feature2)
    println(prediction(0))
    [0.4629,    0.1703,    0.1670,    0.2561]

If I try to fit the network however, I get the following error:

 println("Fitting...")
 val combined: Array[INDArray] = Array(feature1, feature2)
 val error = Array(Nd4j.create(Array(0.5, 0.2, 1.0, 0), Array(4, 1)))
 model.fit(new MultiDataSet(combined, error))

Exception in thread "main" java.lang.NullPointerException
	at org.deeplearning4j.nn.graph.ComputationGraph.calcBackpropGradients(ComputationGraph.java:2681)
	at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1382)
	at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1342)
	at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:170)
	at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:63)
	at org.deeplearning4j.optimize.Solver.optimize(Solver.java:52)
	at org.deeplearning4j.nn.graph.ComputationGraph.fitHelper(ComputationGraph.java:1166)
	at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1116)
	at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1026)
	at ai.decisive.nntesting.NNTesting$.main(NNTesting.scala:67)
	at ai.decisive.nntesting.NNTesting.main(NNTesting.scala)

I took a look at the stacktrace and it seems to be because there is no Output layer defined. The issue is that I cannot have an output layer of a fixed size because the feature2 input is of variable size. So if there are 3 values in feature2, the output should be [1, 3]. Is there a way to have a variable size output layer in dl4j so that the model can be fitted?

2

The null pointer exception happens while trying to setup the contents for another exception.

Your problem is that the cosine similarity vertex you’ve defined is not a output layer. You can use fit only if you have some kind of output layer defined with your Computation Graph.

If you want to backprop external errors, you can use more or less the same method that you use with a normal MultiLayerNetwork (see examples)

3

Thanks for the quick response! Your suggestion looks like a good way to get around having the variable length output layer.