# Most efficient way to do inference?

**URL:** <https://community.konduit.ai/t/most-efficient-way-to-do-inference/263>\
**Category:** Uncategorized\
**Created:** [March 12, 2020, 8:06pm UTC](https://community.konduit.ai/t/most-efficient-way-to-do-inference/263 "2020-03-12T20:06:37Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [March 13, 2020, 12:58pm UTC](https://community.konduit.ai/t/most-efficient-way-to-do-inference/263/4 "2020-03-13T12:58:41Z")

</div>

> [@temerick](#):
>
> So currently, my data is being loaded into a single java queue via multiple threads. Currently, we have a line along the lines of `y = classifierModel.output(false, x)` .

So you already have a preloading implemented, and have a queue of single inputs, right?  
Take as many of those as your hardware can handle, and stack them on top of each other. So you get an input that is like a mini-batch during training. Let your model work on that.

As you can see in [DL4J Classification Speed - #16 by ethiel](https://community.konduit.ai/t/dl4j-classification-speed/187/16) @torstenbm was able to achieve over 200k classifications per second by batching his inputs.

> [@temerick](#):
>
> I see the other signatures for output, and I’m wondering if some of these options allow me to do things like put my data directly into pinned memory, or move the data to the gpu asynchronously.

If you meant the [output(DataSetIterator)](https://javadoc.io/static/org.deeplearning4j/deeplearning4j-nn/1.0.0-beta6/org/deeplearning4j/nn/multilayer/MultiLayerNetwork.html#output-org.nd4j.linalg.dataset.api.iterator.DataSetIterator-boolean-) signature: This more or less just saves you the loop of iterating manually through your iterator at the moment.

But you can still get something close to what you were talking about with Workspaces.  
Take a look at [https://deeplearning4j.konduit.ai/config/config-memory/config-workspaces#iterators](https://deeplearning4j.konduit.ai/config/config-memory/config-workspaces#iterators) and [https://deeplearning4j.konduit.ai/config/config-memory](https://deeplearning4j.konduit.ai/config/config-memory) and the examples for them: [https://github.com/eclipse/deeplearning4j-examples/blob/master/nd4j-examples/src/main/java/org/nd4j/examples/Nd4jEx15\_Workspaces.java](https://github.com/eclipse/deeplearning4j-examples/blob/master/nd4j-examples/src/main/java/org/nd4j/examples/Nd4jEx15_Workspaces.java)

---

_[View the full topic](https://community.konduit.ai/t/most-efficient-way-to-do-inference/263)._