# DL4J Classification Speed

**URL:** <https://community.konduit.ai/t/dl4j-classification-speed/187>\
**Category:** DL4J\
**Created:** [February 27, 2020, 3:56pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187 "2020-02-27T15:56:39Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![torstenbm](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/torstenbm/32/101_2.png) [@torstenbm](https://community.konduit.ai/u/torstenbm)\
**Post date:** [February 27, 2020, 3:56pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/1 "2020-02-27T15:56:40Z")

</div>

Greetings!

Does anybody have any idea of what sort of classification speed one can expect using DL4J?

I have a single-hidden-LSTM-layer RNN doing sentiment analysis (heavily inspired by [this example](https://github.com/eclipse/deeplearning4j-examples/tree/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/recurrent/word2vecsentiment)) of tweets with the Cuda-10.1-backend (Without cudNN, working on getting that installed but I have limited privileges on the machine) and two Tesla P100-16GB GPU’s. Classifying using `net.output();` I get a throughput of about 100 tweets processed per second. This is way lower than I was hoping for, as I achieved a throughput of 15k tweets per second using a CPU-based implementation of Naive Bayes last semester.

Why am I using `net.output();` instead of a DataSetIterator you might ask. I am using the network in a streaming context and not on a static dataset.

Does anybody have any experience trying to make NN’s faster and more scalable? Would greatly appreciate any nudge in the right direction.

Best wishes,  
Torsten

---

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [February 27, 2020, 4:10pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/2 "2020-02-27T16:10:44Z")

</div>

It depends entirely on what your usecase is.

The worst possible case for inference speed is to run a very small NN on a GPU. This is because there is a certain latency involved with moving your input to the gpu, running the calculation and moving the output back. If you have this one-at-a-time kind of usecase it is usually better to just run it on the CPU directly.

GPUs are faster than CPUs in two cases:

1. You can batch your requests to run many calculations simultaneously
2. You have a NN that is widely complex and requires a lot of computation for even a single example

As you have a very small neural network, the first option might be interesting for you. If you just want to process lots of tweets, and they aren’t coming in one at a time, then you can just pass them all in as a single batch to `net.output()`. Since you are running inference, the batchsize itself will not influence the output for each single example, so you can easily create batches that are as big as your memory can handle. I’d expect that you should be able to get into the 20k tweets/s range easily.

---

<div class="post-metadata">

**Author:** ![torstenbm](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/torstenbm/32/101_2.png) [@torstenbm](https://community.konduit.ai/u/torstenbm)\
**Post date:** [February 27, 2020, 4:27pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/3 "2020-02-27T16:27:25Z")

</div>

Thank you very much, that makes a lot of sense.

Just so I understand you clearly, my best shot would be create some sort of mechanism that bundles up tweets into micro-batches of some size (optimally as big as the memory can handle) before the batch is put through `net.output()`? The stream I’m working on is having tweets arrive one-at-a-time (often at speeds of 5-20k tweets per second), so it would switch up the processing semantics a bit as the tweets would arrive one-at-a-time but be processed in micro-batches, however, I like the idea and it’s definitely worth a shot.

Thanks again for your help.

---

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [February 27, 2020, 4:31pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/4 "2020-02-27T16:31:29Z")

</div>

Take a look at [ParallelInference](https://deeplearning4j.org/api/latest/org/deeplearning4j/parallelism/ParallelInference.html). It does all of that for you. You can tell it to either queue until it is full, or to wait for a specific time to fill up.

---

<div class="post-metadata">

**Author:** ![torstenbm](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/torstenbm/32/101_2.png) [@torstenbm](https://community.konduit.ai/u/torstenbm)\
**Post date:** [February 29, 2020, 6:55pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/5 "2020-02-29T18:55:24Z")

</div>

Thank you so much for your help. I am trying some stuff out now, would you mind giving me some pointers on how to proceed ParalllelInference? I’ve taken a look at [this example](https://github.com/eclipse/deeplearning4j-examples/blob/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/inference/ParallelInferenceExample.java), but I am not sure exactly how to run the inference batch style.

Right now I am reading tweets from a CSV-file, converting them to word2vec-features (INDArray) of size 300 and putting them through the `pi.output()`. To process them batch style, would I have to batch my feature-vectors together in a `[]INDArray` and put that through `pi.output()`? If that’s the case I would assume the output to come back as a `[]INDArray` that I would have to loop through to get the predictions for the individual tweets (keeping track of the indices going in to get the tweet-sentiment pairs), is this correct?

---

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [March 2, 2020, 1:35pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/6 "2020-03-02T13:35:05Z")

</div>

> [@torstenbm](#):
>
> Right now I am reading tweets from a CSV-file

When you are doing it like that, you don’t have to go through ParallelInference, as you can just read all of them at once and have just a single output request.

> [@torstenbm](#):
>
> converting them to word2vec-features (INDArray) of size 300 and putting them through the `pi.output()`

I guess you are converting to w2v features manually. Have you taken a look at your application with a profiler to see what is actually slow? Vectorizing the data inefficiently is often the main bottleneck in this kinds of setups, especially when you are using a manual w2v conversion

---

<div class="post-metadata">

**Author:** ![torstenbm](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/torstenbm/32/101_2.png) [@torstenbm](https://community.konduit.ai/u/torstenbm)\
**Post date:** [March 3, 2020, 4:03pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/7 "2020-03-03T16:03:11Z")

</div>

> When you are doing it like that, you don’t have to go through ParallelInference, as you can just read all of them at once and have just a single output request.

Yes I am aware, I’m just simulating the streaming conditions by reading/processing them one-at-a-time or now appending them to an array of a certain size and processing them batch-wise as you suggested. The final usecase for the NN will be a User Defined Function for [AsterixDB](http://asterixdb.apache.org/) that will process tweets as they are streamed into the system with the Twitter Streaming API as a mean of continual data ingestion.

> Have you taken a look at your application with a profiler to see what is actually slow? Vectorizing the data inefficiently is often the main bottleneck in this kinds of setups.

You are absolutely right, I did some testing processing arrays of 1000, 5000, and 10’000 tweets, and couldn’t achieve a throughput higher than 1000 tweets per second for either of them. Turns out most of the processing time was going to vectorization.

I am thinking that instead of using these slow word-vectors an idea might be to use home-made char-vectors. Given that a tweet has a maximum length of 280 characters, 280 is an obvious choice for the vector length. Each Character could be mapped to a corresponding Integer, and tweets shorter than 280 could be padded out. The vectorization-process then would be a series of HashMap lookups to get the Integers corresponding to the Characters, which is a series of O(1) lookups and should be pretty fast.

@treo does this seem like a sound way to proceed? As long as the accuracy is above 70% or so I only care about throughput for this project, that is, maximizing the amount of tweets processed per second in a streaming context.

---

<div class="post-metadata">

**Author:** ![ethiel](https://avatars.discourse-cdn.com/v4/letter/e/df788c/32.png) [@ethiel](https://community.konduit.ai/u/ethiel)\
**Post date:** [March 3, 2020, 5:20pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/8 "2020-03-03T17:20:33Z")

</div>

Are you training or only classifying the results?

---

<div class="post-metadata">

**Author:** ![torstenbm](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/torstenbm/32/101_2.png) [@torstenbm](https://community.konduit.ai/u/torstenbm)\
**Post date:** [March 3, 2020, 5:33pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/9 "2020-03-03T17:33:46Z")

</div>

Only classifying. The training happens offline, then the model is bundled into a User Defined Function and deployed to a BDMS to classify tweets that are being continually ingested into the system.

---

<div class="post-metadata">

**Author:** ![ethiel](https://avatars.discourse-cdn.com/v4/letter/e/df788c/32.png) [@ethiel](https://community.konduit.ai/u/ethiel)\
**Post date:** [March 3, 2020, 5:36pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/10 "2020-03-03T17:36:06Z")

</div>

Understood. Are you using Vector model?

---

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [March 3, 2020, 5:41pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/11 "2020-03-03T17:41:57Z")

</div>

How exactly are you running your vectorization? Usually our W2V is quite fast with its lookups.

> [@torstenbm](#):
>
> I am thinking that instead of using these slow word-vectors an idea might be to use home-made char-vectors.

That can work, with something like a CharCNN based setup, but usually using something with pretrained word vectors is easier to work with.

---

<div class="post-metadata">

**Author:** ![torstenbm](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/torstenbm/32/101_2.png) [@torstenbm](https://community.konduit.ai/u/torstenbm)\
**Post date:** [March 3, 2020, 5:44pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/12 "2020-03-03T17:44:59Z")

</div>

I think I am using the exact same code as in the example I am drawing inspiration from.

```
public INDArray loadFeaturesFromString(String reviewContents, int maxLength){
    List<String> tokens = this.tokenizerFactory.create(reviewContents).getTokens();
    List<String> tokensFiltered = new ArrayList<>();
    for(String t : tokens ){
        if(wordVectors.hasWord(t)) tokensFiltered.add(t);
    }
    int outputLength = Math.min(maxLength,tokensFiltered.size());

    INDArray features = Nd4j.create(1, vectorSize, outputLength);

    int count = 0;
    for( int j=0; j<tokensFiltered.size() && count<maxLength; j++ ){
        String token = tokensFiltered.get(j);
        INDArray vector = wordVectors.getWordVectorMatrix(token);
        if(vector == null){
            continue; //Word not in word vectors
        }
        features.put(new INDArrayIndex[]{NDArrayIndex.point(0), NDArrayIndex.all(), NDArrayIndex.point(j)}, vector);
        count++;
    }
    return features;
}

```

---

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [March 3, 2020, 5:49pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/13 "2020-03-03T17:49:18Z")

</div>

What you might want to try instead is to set your embedding matrix as the weights of an embedding layer, and then feed it with word indexes instead.

That way you don’t create unnecessarily large inputs and the lookup should be a lot faster.

---

<div class="post-metadata">

**Author:** ![torstenbm](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/torstenbm/32/101_2.png) [@torstenbm](https://community.konduit.ai/u/torstenbm)\
**Post date:** [March 3, 2020, 5:56pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/14 "2020-03-03T17:56:45Z")

</div>

Sound good.

Right now the model is loaded by

```
WordVectors wordVectors = WordVectorSerializer.loadStaticModel(new File(WORD_VECTORS_PATH));

```

And the Network Config looks like

```
        MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
        .seed(seed)
        .updater(new Adam(5e-3))
        .l2(1e-5)
        .weightInit(WeightInit.XAVIER)
        .gradientNormalization(GradientNormalization.ClipElementWiseAbsoluteValue).gradientNormalizationThreshold(1.0)
        .list()
        .layer(new LSTM.Builder().nIn(vectorSize).nOut(256)
            .activation(Activation.TANH).build())
        .layer(new RnnOutputLayer.Builder().activation(Activation.SOFTMAX)
            .lossFunction(LossFunctions.LossFunction.MCXENT).nIn(256).nOut(2).build())
        .build();

```

So I guess I would be adding an embedding layer before the LSTM layer, but how would I go about setting the weights?

---

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [March 3, 2020, 6:06pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/15 "2020-03-03T18:06:03Z")

</div>

Using `wordVectors.lookupTable().getWeights()` you can get the actual weights, and `wordVectors.vocab()` and `vocab.indexOf(word)` will give you a way to get the word index.

If you want to use static weights for your case, you should also set the learningrate for the Embedding layer to 0, and after initialization, you can set the weights for it with `model.setParam("1_W", weights)`. The name of the correct parameter view might be slightly different, you can look it up in the keys of `model.paramTable()`.

---

<div class="post-metadata">

**Author:** ![ethiel](https://avatars.discourse-cdn.com/v4/letter/e/df788c/32.png) [@ethiel](https://community.konduit.ai/u/ethiel)\
**Post date:** [March 4, 2020, 7:58am UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/16 "2020-03-04T07:58:41Z")

</div>

> [@torstenbm](#):
>
> loadStaticModel

I think you can improve the speed by using a custom implementation with ChronicleMap to load the vectors.

---

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [March 4, 2020, 8:43am UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/17 "2020-03-04T08:43:40Z")

</div>

> [@ethiel](#):
>
> using a custom implementation with ChronicleMap to load the vectors.

Let’s not introduce new moving parts until the actual bottleneck is identified.

Unless @torstenbm finds that getting values from a map is the actual bottleneck, adding a different map implementation isn’t going to help at all.

---

<div class="post-metadata">

**Author:** ![torstenbm](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/torstenbm/32/101_2.png) [@torstenbm](https://community.konduit.ai/u/torstenbm)\
**Post date:** [March 9, 2020, 7:29pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/18 "2020-03-09T19:29:26Z")

</div>

Hey @treo so I tried the char-based approach with great success.

Because a tweet is max 280 characters I convert a tweet-string to a `double[280]` and then create batches of 10k tweets with the shape of `double[10000][280] tweetBatch`.

Then I run them through my network using

```auto
INDArray features = Nd4j.create(tweetBatch);
INDArray networkOutput = pi.output(features);

```

and have with this achieved throughputs of up to 230k tweets per second, which is a huge improvement!

Only thing I haven’t been able to do yet is retrieve the sentiment for each tweet from the `networkOutput` object, would you mind helping me construct a loop doing that? My network has an input-layer of 280 nodes and an output-layer of 2 nodes.

---

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [March 9, 2020, 7:32pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/19 "2020-03-09T19:32:48Z")

</div>

> [@torstenbm](#):
>
> I tried the char-based approach with great success.

Great to hear!

> [@torstenbm](#):
>
> Only thing I haven’t been able to do yet is retrieve the sentiment for each tweet from the `networkOutput` object

You are likely looking for the [argMax](https://deeplearning4j.org/api/latest/org/nd4j/linalg/api/ndarray/INDArray.html#argMax-int...-) method. It gives you the index of the label with the highest probability.

---

<div class="post-metadata">

**Author:** ![torstenbm](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/torstenbm/32/101_2.png) [@torstenbm](https://community.konduit.ai/u/torstenbm)\
**Post date:** [March 9, 2020, 8:38pm UTC](https://community.konduit.ai/t/dl4j-classification-speed/187/20 "2020-03-09T20:38:18Z")

</div>

> [@treo](#):
>
> You are likely looking for the [argMax](https://deeplearning4j.org/api/latest/org/nd4j/linalg/api/ndarray/INDArray.html#argMax-int...-) method. It gives you the index of the label with the highest probability.

Am I though? I am classifying 10’000 tweets at once with a 10000x280 INDArray, at output there is one node activating for positive tweets and one node for negative tweets, and I am looking to get the activation values associated with each of these nodes for each tweets in order to see whether the model predicts the tweet to be more positive or more negative.

[Next page](https://community.konduit.ai/t/dl4j-classification-speed/187.md?page=2)