# Correct resetable DataSetIterator for a Dataset?

**URL:** <https://community.konduit.ai/t/correct-resetable-datasetiterator-for-a-dataset/1451>\
**Category:** DL4J\
**Created:** [June 17, 2021, 3:53pm UTC](https://community.konduit.ai/t/correct-resetable-datasetiterator-for-a-dataset/1451 "2021-06-17T15:53:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![MPdaedalus](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/mpdaedalus/32/454_2.png) [@MPdaedalus](https://community.konduit.ai/u/MPdaedalus)\
**Post date:** [June 17, 2021, 3:53pm UTC](https://community.konduit.ai/t/correct-resetable-datasetiterator-for-a-dataset/1451/1 "2021-06-17T15:53:26Z")

</div>

i’ve been using new IteratorDataSetIterator(trainingData.iterator(),batchSize) with my models up till now which has been fine but now I am using a much larger training set of 1 million examples which is too big to normalize in one go by running normalizer.fit(trainingData) as I run out of GPU memory\>

Exception in thread “main” java.lang.RuntimeException: Memory allocation for tadOffsets failed; Error code: [2]  
at org.nd4j.linalg.jcublas.ops.executioner.CudaExecutioner.tadShapeInfoAndOffsets(CudaExecutioner.java:2179)  
at org.nd4j.jita.allocator.tad.BasicTADManager.getTADOnlyShapeInfo(BasicTADManager.java:49)  
at org.nd4j.jita.allocator.tad.DeviceTADManager.getTADOnlyShapeInfo(DeviceTADManager.java:89)  
at org.nd4j.linalg.jcublas.ops.executioner.CudaExecutioner.exec(CudaExecutioner.java:149)  
at org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner.execAndReturn(DefaultOpExecutioner.java:173)  
at org.nd4j.linalg.dataset.api.preprocessor.StandardizeStrategy.preProcess(StandardizeStrategy.java:56)  
at org.nd4j.linalg.dataset.api.preprocessor.StandardizeStrategy.preProcess(StandardizeStrategy.java:38)  
at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.transform(AbstractDataSetNormalizer.java:163)  
at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.preProcess(AbstractDataSetNormalizer.java:131)  
at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.transform(AbstractDataSetNormalizer.java:142)  
at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.transform(AbstractDataSetNormalizer.java:36)

Unfortunatly neither IteratorDataSetIterator nor trainingData.iterateWithMiniBatches() works with normalizer.fit as they are not resetable

Exception in thread “main” java.lang.NullPointerException: Cannot invoke “org.nd4j.linalg.dataset.api.iterator.DataSetIterator.reset()” because “iterator” is null  
at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.fit(AbstractDataSetNormalizer.java:107)

i’ve been looking at all the DataSetIterators available at [https://deeplearning4j.org/api/latest/org/nd4j/linalg/dataset/api/iterator/DataSetIterator.html](https://deeplearning4j.org/api/latest/org/nd4j/linalg/dataset/api/iterator/DataSetIterator.html)

but can’t seem to find one that is applicable for just a vanilla Dataset.

Any recommendations?

on a side note, I’ve noticed that when I save my normalizer settings after using fit() using \>  
StandardizeSerializerStrategy ns = new StandardizeSerializerStrategy();  
try {  
ns.write(normalizer, new FileOutputStream(new File(“/normalizeData.data”)));  
} catch (IOException e) {  
e.printStackTrace();  
}

the resulting file is only 267 bytes! is this how large its meant to be?

Thanks in advance