# How to increase Batch size **URL:** \ **Category:** DL4J\ **Created:** [September 16, 2020, 9:06am UTC](https://community.konduit.ai/t/how-to-increase-batch-size/855 "2020-09-16T09:06:50Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![chaoyongyue](https://avatars.discourse-cdn.com/v4/letter/c/a9a28c/32.png) [@chaoyongyue](https://community.konduit.ai/u/chaoyongyue)\ **Post date:** [September 16, 2020, 9:06am UTC](https://community.konduit.ai/t/how-to-increase-batch-size/855/1 "2020-09-16T09:06:50Z")
When I used multiple GPUs to train VGG, I increased the batch size, but I found that the program would have errors and show insufficient memory. However, these memory is sufficient on tensorflow. The pictures below show the errors ---
**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\ **Post date:** [September 16, 2020, 9:20am UTC](https://community.konduit.ai/t/how-to-increase-batch-size/855/2 "2020-09-16T09:20:40Z")
> [@chaoyongyue](#): > > d the batch size, but I found that the program would have errors and show insufficient memory. However "This happens on tensorflow and “this happens in dl4j” doesn’t really help us in anyway. We don’t know what you were using or what the context was. Could you give us something we can run to compare for ourselves? ---
**Author:** ![chaoyongyue](https://avatars.discourse-cdn.com/v4/letter/c/a9a28c/32.png) [@chaoyongyue](https://community.konduit.ai/u/chaoyongyue)\ **Post date:** [September 16, 2020, 9:21am UTC](https://community.konduit.ai/t/how-to-increase-batch-size/855/3 "2020-09-16T09:21:34Z")
![{X@FRYB7{9XKXWDNRRUDOFT](https://canada1.discourse-cdn.com/flex035/uploads/konduit/original/1X/db7edeaeb7075a854bc5e4168d804f8c4bd3cbd8.png) ---
**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\ **Post date:** [September 16, 2020, 9:22am UTC](https://community.konduit.ai/t/how-to-increase-batch-size/855/4 "2020-09-16T09:22:15Z")
Right I know what your error is. I want your code you were using in tensorflow to compare. Give me 2 side by side runnable scripts you actually ran yourself and I can tell you more. ---
**Author:** ![chaoyongyue](https://avatars.discourse-cdn.com/v4/letter/c/a9a28c/32.png) [@chaoyongyue](https://community.konduit.ai/u/chaoyongyue)\ **Post date:** [September 16, 2020, 9:28am UTC](https://community.konduit.ai/t/how-to-increase-batch-size/855/5 "2020-09-16T09:28:53Z")
![FQIFY_YMW@JCJ}K45HPH62](https://canada1.discourse-cdn.com/flex035/uploads/konduit/original/1X/c64a153e7603095adf68b4393aa9a418ef007c6f.png) Thanks for your reply. This is part of the code。 I think it may be that the memory allocation method is different during the training process, but I don’t know how dl4j allocates memory in the multi-GPU training era. ---
**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\ **Post date:** [September 16, 2020, 10:20am UTC](https://community.konduit.ai/t/how-to-increase-batch-size/855/6 "2020-09-16T10:20:42Z")
Ah, parallelwrapper has mutliple copies of the model in memory. That’s why. ParallelWrapper is mainly for multi GPU. Go ahead and just use singular dl4j and you should be fine. ---
**Author:** ![chaoyongyue](https://avatars.discourse-cdn.com/v4/letter/c/a9a28c/32.png) [@chaoyongyue](https://community.konduit.ai/u/chaoyongyue)\ **Post date:** [September 16, 2020, 10:57am UTC](https://community.konduit.ai/t/how-to-increase-batch-size/855/7 "2020-09-16T10:57:36Z")
Well.I am sorry. I can’t understand where I need to modify the code specifically .I use two GPUs to train vgg network. So if I can’t use the parallelwrapper,how can I train vgg for multiGPU with large batch size? ---
**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\ **Post date:** [September 16, 2020, 1:09pm UTC](https://community.konduit.ai/t/how-to-increase-batch-size/855/8 "2020-09-16T13:09:28Z")
Oh I see that was on purpose. Sorry I missed that in your post. Ensure your number of workers equals number of gpus and adjust your prefetch buffer to be less. That might be what’s taking up your memory. ---
**Author:** ![chaoyongyue](https://avatars.discourse-cdn.com/v4/letter/c/a9a28c/32.png) [@chaoyongyue](https://community.konduit.ai/u/chaoyongyue)\ **Post date:** [September 16, 2020, 2:26pm UTC](https://community.konduit.ai/t/how-to-increase-batch-size/855/9 "2020-09-16T14:26:19Z")
I tried it but failed again. So the parallelwrapper’s methods of allocating memory may be different from others. Any other ways to take up memory?