How to config a multi-LSTM net with attentions?

DL4J
1

I want config a multi-LSTM net with attentions like:

       int nIn = featuresCount;
        int layerIndex=0;
        for (int i = 0; i < this.rnnNeurons.length; ++i) {
            listBuilder.layer(layerIndex, new LSTM.Builder().activation(Activation.SOFTSIGN).nIn(nIn).nOut(rnnNeurons[i]).build());            
            nIn = rnnNeurons[i];
            int nhead=8;
            ++layerIndex;
            listBuilder.layer(layerIndex, new RecurrentAttentionLayer.Builder().activation(Activation.SOFTSIGN).nIn(nIn).nOut(nIn).nHeads(nhead).projectInput(true).build());
            ++layerIndex;
        }

        listBuilder.layer(layerIndex, new RnnOutputLayer.Builder(new MSELoss())//
                    .activation(Activation.IDENTITY).nIn(nIn).nOut(1).build());
        
//        listBuilder.setInputType(InputType.recurrent(featuresCount));
        MultiLayerConfiguration conf = listBuilder.build();

There is no exception after the program runs, but there is no display on the Training UI. How to configure it correctly?

2

It can work, just a little slow.

3

@cqiaoYc feel free to browse the examples. We cover all of that in there including the UI.
If something is “slow” did you measure it relative to a known peak performance so we know what “fast” is? Otherwise it’s kind of hard to tell if this is subjective or a real problem.

If we can first identify what “fast” is we can either fix your problem there or optimize your training pipeline a bit.