# Why BertInferenceExample can only infer 4 sentence pair at once

**URL:** <https://community.konduit.ai/t/why-bertinferenceexample-can-only-infer-4-sentence-pair-at-once/649>\
**Category:** DL4J\
**Created:** [June 25, 2020, 3:21am UTC](https://community.konduit.ai/t/why-bertinferenceexample-can-only-infer-4-sentence-pair-at-once/649 "2020-06-25T03:21:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maple](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/maple/32/410_2.png) [@Maple](https://community.konduit.ai/u/Maple)\
**Post date:** [June 25, 2020, 3:21am UTC](https://community.konduit.ai/t/why-bertinferenceexample-can-only-infer-4-sentence-pair-at-once/649/1 "2020-06-25T03:21:04Z")

</div>

i wonder how to infer more sentencePair at once.

```java
//Four sentence pairs to run inference on**
        List<Pair<String, String>> sentencePairs = new ArrayList<>();
        sentencePairs.add(new Pair<>("The broader Standard & Poor's 500 Index <.SPX> was 0.46 points lower, or 0.05 percent, at 997.02.", "The technology-laced Nasdaq Composite Index .IXIC was up 7.42 points, or 0.45 percent, at 1,653.44."));
        sentencePairs.add(new Pair<>("Shares in BA were down 1.5 percent at 168 pence by 1420 GMT, off a low of 164p, in a slightly stronger overall London market.", "Shares in BA were down three percent at 165-1/4 pence by 0933 GMT, off a low of 164 pence, in a stronger market."));
        sentencePairs.add(new Pair<>("Last year, Comcast signed 1.5 million new digital cable subscribers.", "Comcast has about 21.3 million cable subscribers, many in the largest U.S. cities."));
        sentencePairs.add(new Pair<>("Revenue rose 3.9 percent, to $1.63 billion from $1.57 billion.", "The McLean, Virginia-based company said newspaper revenue increased 5 percent to $1.46 billion."));
        sentencePairs.add(new Pair<>("Revenue rose 3.9 percent, to $1.63 billion from $1.57 billion.", "The McLean, Virginia-based company said newspaper revenue increased 5 percent to $1.46 billion."));

```

if i add a sentencePair to it, their will be an error

 ![image](https://canada1.discourse-cdn.com/flex035/uploads/konduit/original/1X/9ea644951f4e13d0199378118516bc124564f9f1.png)

---

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [June 26, 2020, 8:38am UTC](https://community.konduit.ai/t/why-bertinferenceexample-can-only-infer-4-sentence-pair-at-once/649/2 "2020-06-26T08:38:46Z")

</div>

The imported Bert model here expects a fixed batch size of 4 - that is why it can only infer 4 sentence pairs at once.

Because of this the bert iterator is configured to always produce this batch size. When you feed more than 4 sentence pairs to `bertIter.featurizeSentencePairs(sentencePairs)` it breaks, because it doesn’t know how to deal with the fifth sentence.

In any case we should throw a more useful exception here.

---

<div class="post-metadata">

**Author:** ![Maple](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/maple/32/410_2.png) [@Maple](https://community.konduit.ai/u/Maple)\
**Post date:** [June 26, 2020, 3:03pm UTC](https://community.konduit.ai/t/why-bertinferenceexample-can-only-infer-4-sentence-pair-at-once/649/3 "2020-06-26T15:03:32Z")

</div>

the batch size is fixed to 4. i tried to change the batch size to 5, it doesn’t work. is there a probability to change the batch size to a dynamic size or manual operation to a larger size to infer more sentence pair at once?

---

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [June 26, 2020, 3:26pm UTC](https://community.konduit.ai/t/why-bertinferenceexample-can-only-infer-4-sentence-pair-at-once/649/4 "2020-06-26T15:26:03Z")

</div>

Unfortunately that depends on the model - the bert model that is imported in that examples is fixed to a batch size of 4.

---

<div class="post-metadata">

**Author:** ![Maple](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/maple/32/410_2.png) [@Maple](https://community.konduit.ai/u/Maple)\
**Post date:** [June 27, 2020, 2:12am UTC](https://community.konduit.ai/t/why-bertinferenceexample-can-only-infer-4-sentence-pair-at-once/649/5 "2020-06-27T02:12:04Z")

</div>

oh, i see, maybe i could try change the batch size at TensorFlow, then export the model by myself.