How can I program this neural network architecture in Deeplearning4j?

I need to implement the following neural network in Deeplearning4j but there are some layers which I don’t know how to use or if deeplearning supports them. Such as the layer to concatenate the results among others. Can someone help me with the most general of this architecture

image (1)

@yuniel-acosta you will want to use the computation graph api for that. Please see the MergeVertex for concat
See here for more:

After many searches on the internet I managed to make a network that looks a bit like the one in the image. Experimenting I found a new problem. This is the code that I have been testing. But after running it with the same dataIterator that I have used in other models with the same data, I get an error and I’m not sure how to solve it

import com.trusty.NewsIterator
import org.apache.commons.io.FilenameUtils
import org.deeplearning4j.core.storage.StatsStorage
import org.deeplearning4j.models.word2vec.Word2Vec
import org.deeplearning4j.nn.conf.ComputationGraphConfiguration
import org.deeplearning4j.nn.conf.ConvolutionMode
import org.deeplearning4j.nn.conf.NeuralNetConfiguration
import org.deeplearning4j.nn.conf.WorkspaceMode
import org.deeplearning4j.nn.conf.graph.MergeVertex
import org.deeplearning4j.nn.conf.layers.*
import org.deeplearning4j.nn.graph.ComputationGraph
import org.deeplearning4j.nn.weights.WeightInit
import org.deeplearning4j.optimize.api.InvocationType
import org.deeplearning4j.optimize.listeners.EvaluativeListener
import org.deeplearning4j.optimize.listeners.ScoreIterationListener
import org.deeplearning4j.ui.api.UIServer
import org.deeplearning4j.ui.model.stats.StatsListener
import org.deeplearning4j.ui.model.storage.InMemoryStatsStorage
import org.nd4j.evaluation.classification.Evaluation
import org.nd4j.linalg.activations.Activation
import org.nd4j.linalg.learning.config.AdaGrad
import org.nd4j.linalg.lossfunctions.LossFunctions
import org.slf4j.LoggerFactory
import java.io.BufferedOutputStream
import java.io.File
import java.io.FileOutputStream

class FNDNet(
    val wordVectors: Word2Vec,
    val vectorSize: Int = wordVectors.getWordVector(wordVectors.vocab().wordAtIndex(0)).size,
    val batchSize: Int = 64,
    val nEpochs: Int = 300,
    val cnnLayerFeatureMaps: Int = 100, //Number of feature maps / channels / depth for each CNN layer
    val iTrain: NewsIterator,
    val iTest: NewsIterator,
    val inputNeurons: Int = wordVectors.getWordVector(wordVectors.vocab().wordAtIndex(0)).size,
    val outputs: Int = iTrain.labels.size,
) {
    private val log = LoggerFactory.getLogger(FNDNet::class.java)
    private fun configuration(): ComputationGraphConfiguration {
        return NeuralNetConfiguration.Builder()
            .trainingWorkspaceMode(WorkspaceMode.ENABLED).inferenceWorkspaceMode(WorkspaceMode.ENABLED)
            .weightInit(WeightInit.RELU)
            .activation(Activation.LEAKYRELU)
            .updater(AdaGrad(0.0018))
            .convolutionMode(ConvolutionMode.Same)      //This is important so we can 'stack' the results later
            .l2(0.0001)
            .graphBuilder()
            .addInputs("input")
            .addLayer(
                "cnn3", Convolution1DLayer.Builder()
                    .kernelSize(3)
                    .stride(1)
                    .nIn(inputNeurons)
                    .nOut(cnnLayerFeatureMaps)
                    .build(), "input"
            )
            .addLayer(
                "cnn4", Convolution1DLayer.Builder()
                    .kernelSize(4)
                    .stride(1)
                    .nIn(inputNeurons)
                    .nOut(cnnLayerFeatureMaps)
                    .build(), "input"
            )
            .addLayer(
                "cnn5", Convolution1DLayer.Builder()
                    .kernelSize(5)
                    .stride(1)
                    .nIn(inputNeurons)
                    .nOut(cnnLayerFeatureMaps)
                    .build(), "input"
            )
            .addVertex("merge", MergeVertex(), "cnn3", "cnn4", "cnn5")      //Perform depth concatenation
            .addLayer(
                "globalPool", GlobalPoolingLayer.Builder()
                    .poolingType(PoolingType.MAX)
                    .dropOut(0.5)
                    .build(), "merge"
            ).addLayer(
                "dense",
                DenseLayer.Builder()
                    .nIn(3 * cnnLayerFeatureMaps)
                    .nOut(128)
                    .activation(Activation.RELU)
                    .weightInit(WeightInit.XAVIER)
                    .dropOut(0.5)
                    .build(), "globalPool"
            )
            .addLayer(
                "out", OutputLayer.Builder()
                    .lossFunction(LossFunctions.LossFunction.MSE)
                    .activation(Activation.SOFTMAX)
                    .nIn(128)
                    .nOut(outputs)    //2 classes: positive or negative
                    .build(), "dense"
            )
            .setOutputs("out")
            .build()
    }

    fun runModel(pathnameToSaveNetwork: String = "FakesModelFNDNet.net") {
       // Obtain configuration
        val configuration = configuration()
        // Initialize NeuronalNetwork
        val net: ComputationGraph = ComputationGraph(configuration)
        //Initialize the user interface backend
        val uiServer: UIServer = UIServer.getInstance()
        //Configure where the network information (gradients, score vs. time etc) is to be stored. Here: store in memory.
        val statsStorage: StatsStorage =
            InMemoryStatsStorage() // FileStatsStorage(File("statsStorage"))  //Alternative:  FileStatsStorage(File), for saving and loading later

        //Attach the StatsStorage instance to the UI: this allows the contents of the StatsStorage to be visualized
        uiServer.attach(statsStorage)
        //Then add the StatsListener to collect this information from the network, as it trains
        net.setListeners(StatsListener(statsStorage))
        net.init()
        log.info("Number of parameters by layer:")
        for (layer in net.layers) {
            log.info("\t ${layer.conf().layer.layerName}  \t ${layer.numParams()}")
        }
        log.info("Network summary : ${net.summary()}")

        log.info("Starting training...")
        net.setListeners(ScoreIterationListener(1), EvaluativeListener(iTest, 1, InvocationType.EPOCH_END))
        net.fit(iTrain, nEpochs)
        log.info("Evaluating...")
        val eval: Evaluation = net.evaluate(iTest)
        log.info(eval.stats())
        net.save(File(pathnameToSaveNetwork), true)
        BufferedOutputStream(FileOutputStream(File("${FilenameUtils.removeExtension(pathnameToSaveNetwork)}.stats"))).use { stream ->
            stream.write(eval.stats().toByteArray())
            stream.close()
        }
        log.info("----- Example complete -----")
    }
}

The data iterator class

class NewsIterator private constructor(
    private val dataDirectory: String,
    wordVectors: WordVectors,
    private val batchSize: Int,
    truncateLength: Int,
    tokenizerFactory: TokenizerFactory,
    private var train: Boolean,
) : DataSetIterator {
    private val wordVectors: WordVectors
    private val vectorSize: Int = wordVectors.getWordVector(wordVectors.vocab().wordAtIndex(0)).size
    private val truncateLength: Int
    var maxLength = 0
        private set
    private var cursor = 0
    private var totalNews = 0
    private val tokenizerFactory: TokenizerFactory
    private val labels: MutableList<String>
    private var trainArticles: MutableList<Article> = ArrayList()
    private var testArticles: MutableList<Article> = ArrayList()

    override fun next(num: Int): DataSet {
        if (cursor >= totalNews) throw NoSuchElementException()
        return nextDataSet(num)
    }

    private fun nextDataSet(num: Int): DataSet {
        val newsArticle: MutableList<Article> = ArrayList(num)
        val veracity = IntArray(num)
        var i = 0
        while (i < num && cursor < this.totalNews) {
            val article = if (train) trainArticles[cursor] else testArticles[cursor]
            newsArticle.add(article)
            veracity[i] = when (article.veracity) {
                Veracity.MOSTLY_TRUE -> 0
                Veracity.MOSTLY_FALSE -> 1
                Veracity.MIXTURE_OF_TRUE_AND_FALSE -> 2
                Veracity.NO_FACTUAL_CONTENT -> 3
                else -> 0
            }
            i++
            cursor++
        }
        //Second: tokenize news and filter out unknown words
        val allTokens: MutableList<List<String>> = ArrayList<List<String>>(newsArticle.size)
        maxLength = 0
        for (article in newsArticle) {
            val tokens = tokenizerFactory.create(article.mainText).tokens
            val tokensFiltered: MutableList<String> = ArrayList()
            for (t in tokens) {
                if (wordVectors.hasWord(t)) tokensFiltered.add(t)
            }
            allTokens.add(tokensFiltered)
            maxLength = maxLength.coerceAtLeast(tokensFiltered.size) // the max length
        }

        //If longest news exceeds 'truncateLength': only take the first 'truncateLength' words
        if (maxLength > truncateLength) maxLength = truncateLength

        //Create data for training
        //Here: we have newsArticle.size() examples of varying lengths
        val features: INDArray = Nd4j.create(newsArticle.size, vectorSize, maxLength)
        //Three labels for veracity: mostly true, mixture of true and false, mostly false, no factual content
        val labels: INDArray = Nd4j.create(newsArticle.size, 4, maxLength)


//        //Because we are dealing with news of different lengths and only one output at the final time step: use padding arrays
//        //Mask arrays contain 1 if data is present at that time step for that example, or 0 if data is just padding
        val featuresMask = Nd4j.zeros(newsArticle.size, maxLength)
        val labelsMask = Nd4j.zeros(newsArticle.size, maxLength)


        val temp = IntArray(2)
        for (i in newsArticle.indices) {
            val tokens = allTokens[i]
            temp[0] = i
            //Get word vectors for each word in news, and put them in the training data
            var j = 0
            while (j < tokens.size && j < maxLength) {
                val token = tokens[j]
                val vector = wordVectors.getWordVectorMatrix(token)
                features.put(
                    arrayOf(
                        NDArrayIndex.point(i.toLong()),
                        NDArrayIndex.all(),
                        NDArrayIndex.point(j.toLong())
                    ), vector
                )
                temp[1] = j
                featuresMask.putScalar(temp, 1.0)
                j++
            }
            val idx: Int = veracity[i]
            val lastIdx = tokens.size.coerceAtMost(maxLength)
            labels.putScalar(intArrayOf(i, idx, lastIdx - 1), 1.0)
            labelsMask.putScalar(intArrayOf(i, lastIdx - 1), 1.0)
        }
        return DataSet(features, labels, featuresMask, labelsMask)
    }

    /* This function is for load data from xmls files*/
    private fun populateData() {
        val articleFile = File("${dataDirectory}${File.separator}")
        val articles: List<Article> = articleFile.walk()
            .maxDepth(1) //The directory hierarchy to be traversed is 1, ie no need to check subdirectories
            .filter { it.isFile } //Select only files, do not process folders
            .filter { it.extension in listOf("xml") } //Select a file with the extension xml
            .map { xmlFile ->
                //give package name as argument
                val context = JAXBContext.newInstance("com.trusty.document")
                val unmarshaller = context.createUnmarshaller()

                val xmlData = unmarshaller.unmarshal(xmlFile) as JAXBElement<*>
                xmlData.value as Article
            }.toList()
        println("Start to split data in train and test")
        val percentage = articles.size * 80 / 100
        articles.groupBy { it.veracity }.forEach { group ->
            val chunked = group.value.chunked(percentage)
            val splitArticles = when (group.key) {
                Veracity.MOSTLY_TRUE -> Pair(chunked.first(), chunked.last())
                Veracity.MOSTLY_FALSE -> Pair(chunked.first(), chunked.last())
                Veracity.MIXTURE_OF_TRUE_AND_FALSE -> Pair(chunked.first(), chunked.last())
                Veracity.NO_FACTUAL_CONTENT -> Pair(chunked.first(), chunked.last())
                else -> Pair(chunked.first(), chunked.last())
            }
            trainArticles.addAll(splitArticles.first)
            testArticles.addAll(splitArticles.second)
        }
        totalNews = if (train) trainArticles.size else testArticles.size
    }

    override fun inputColumns(): Int {
        return vectorSize
    }

    override fun totalOutcomes(): Int {
        return 4 // number of labels for dataset
    }

    override fun reset() {
        cursor = 0
    }

    override fun resetSupported(): Boolean {
        return true
    }

    override fun asyncSupported(): Boolean {
        return true
    }

    override fun batch(): Int {
        return batchSize
    }

    override fun setPreProcessor(preProcessor: DataSetPreProcessor) {
        throw UnsupportedOperationException()
    }

    override fun getLabels(): List<String> {
        return labels
    }

    override fun hasNext(): Boolean {
        return cursor < totalNews
    }

    override fun next(): DataSet {
        return next(batchSize)
    }

    override fun remove() {}
    override fun getPreProcessor(): DataSetPreProcessor {
        throw UnsupportedOperationException("Not implemented")
    }

    class Builder internal constructor() {
        private lateinit var dataDirectory: String
        private lateinit var wordVectors: WordVectors
        private var batchSize = 0
        private var truncateLength = 0
        private lateinit var tokenizerFactory: TokenizerFactory
        private var train = false
        fun dataDirectory(dataDirectory: String): Builder {
            this.dataDirectory = dataDirectory
            return this
        }

        fun wordVectors(wordVectors: WordVectors): Builder {
            this.wordVectors = wordVectors
            return this
        }

        fun batchSize(batchSize: Int): Builder {
            this.batchSize = batchSize
            return this
        }

        fun truncateLength(truncateLength: Int): Builder {
            this.truncateLength = truncateLength
            return this
        }

        fun train(train: Boolean): Builder {
            this.train = train
            return this
        }

        fun tokenizerFactory(tokenizerFactory: TokenizerFactory): Builder {
            this.tokenizerFactory = tokenizerFactory
            return this
        }

        fun build(): NewsIterator {
            return NewsIterator(
                dataDirectory,
                wordVectors,
                batchSize,
                truncateLength,
                tokenizerFactory,
                train
            )
        }

        override fun toString(): String {
            return ("com.trusty.NewsIterator.Builder(dataDirectory= $dataDirectory , wordVectors= $wordVectors , batchSize= $batchSize , truncateLength= $truncateLength, train=$train )")
        }
    }

    companion object {
        fun Builder(): Builder {
            return NewsIterator.Builder()
        }
    }

    /**
     * - initialize various class variables
     * - calls populateData function to load news data in categoryData vector
     * - also populates labels (i.e. category related information) in labels class variable
     */
    init {
        this.wordVectors = wordVectors
        this.truncateLength = truncateLength
        this.tokenizerFactory = tokenizerFactory
        populateData()
        labels = arrayListOf("mostly true", "mixture of true and false", "mostly false", "no factual content")
    }
}

The summary of the network

==========================================================================================================
VertexName (VertexType)           nIn,nOut   TotalParams   ParamsShape                  Vertex Inputs     
==========================================================================================================
input (InputVertex)               -,-        -             -                            -                 
cnn3 (Convolution1DLayer)         300,100    90,100        W:{100,300,3,1}, b:{1,100}   [input]           
cnn4 (Convolution1DLayer)         300,100    120,100       W:{100,300,4,1}, b:{1,100}   [input]           
cnn5 (Convolution1DLayer)         300,100    150,100       W:{100,300,5,1}, b:{1,100}   [input]           
merge (MergeVertex)               -,-        -             -                            [cnn3, cnn4, cnn5]
globalPool (GlobalPoolingLayer)   -,-        0             -                            [merge]           
dense (DenseLayer)                300,128    38,528        W:{300,128}, b:{1,128}       [globalPool]      
out (OutputLayer)                 128,4      516           W:{128,4}, b:{1,4}           [dense]           
----------------------------------------------------------------------------------------------------------
            Total Parameters:  399,344
        Trainable Parameters:  399,344
           Frozen Parameters:  0
==========================================================================================================

The error that throws me is the following:

Exception in thread "main" java.lang.IllegalStateException: Invalid mask array: per-example masking should be a column vector, per output masking arrays should be the same shape as the output/labels arrays. Mask shape: [50, 300], output shape: [50, 4](layer name: out, layer index: 7, layer type: OutputLayer)
	at org.deeplearning4j.nn.layers.BaseOutputLayer.applyMask(BaseOutputLayer.java:342)
	at org.deeplearning4j.nn.layers.BaseLayer.preOutputWithPreNorm(BaseLayer.java:331)
	at org.deeplearning4j.nn.layers.BaseLayer.preOutput(BaseLayer.java:291)
	at org.deeplearning4j.nn.layers.BaseOutputLayer.preOutput2d(BaseOutputLayer.java:328)
	at org.deeplearning4j.nn.layers.BaseOutputLayer.backpropGradient(BaseOutputLayer.java:147)
	at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.doBackward(LayerVertex.java:149)
	at org.deeplearning4j.nn.graph.ComputationGraph.calcBackpropGradients(ComputationGraph.java:2713)
	at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1382)
	at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1342)
	at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:170)
	at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:63)
	at org.deeplearning4j.optimize.Solver.optimize(Solver.java:52)
	at org.deeplearning4j.nn.graph.ComputationGraph.fitHelper(ComputationGraph.java:1166)
	at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1116)
	at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1083)
	at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1019)
	at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1007)
	at com.trusty.models.FNDNet.runModel(FNDNet.kt:128)
	at com.trusty.models.FNDNet.runModel$default(FNDNet.kt:104)
	at com.trusty.TrainNews.main(TrainNews.kt:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.intellij.rt.execution.application.AppMainV2.main(AppMainV2.java:128)

What can I do to solve my problem. The code is written in kotlin. I hope you can help me.

Did you manage to solve it?
I’m looking for this net too

No i have the problem, i need help…