# Reproducibility question
**URL:** https://community.konduit.ai/t/reproducibility-question/1402
**Category:** DL4J
**Created:** [May 25, 2021, 4:02pm UTC](https://community.konduit.ai/t/reproducibility-question/1402 "2021-05-25T16:02:28Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@ramarro123](https://community.konduit.ai/u/ramarro123)
#### Post date: [May 25, 2021, 4:02pm UTC](https://community.konduit.ai/t/reproducibility-question/1402/1 "2021-05-25T16:02:28Z")
Hello,
i create a MultiLayerConfiguration using
```auto
new NeuralNetConfiguration.Builder()
.seed(0)
.activation(Activation.LEAKYRELU)
.weightInit(WeightInit.RELU)
.updater(new Adam(0.001))
.l2(0.000000010000)
.... (4 DenseLayer + 1 OutputLayer, config ti's constant between every test)
```
and i expected that since the seed it’s equals, if i re-run the simulation (given the data are the same) the output will be consistent, but they arent.
test and train are exactly the asme, on the first run i got
```auto
=========================Confusion Matrix=========================
0 1 2
----------
92 28 5 | 0 = 0
26 39 1 | 1 = 1
6 0 8 | 2 = 2
```
but if i rerun the number are a bit different, like
```auto
=========================Confusion Matrix=========================
0 1 2
----------
97 24 4 | 0 = 0
27 38 1 | 1 = 1
4 1 9 | 2 = 2
```
of course they sum up to the same value 🙂 but that’s expected.
do you have any idea for why the results are a bit different every time i re-run it? shouldn’t be setting the same seed make them consistent?
---
### Author:  [@treo](https://community.konduit.ai/u/treo)
#### Post date: [May 25, 2021, 4:12pm UTC](https://community.konduit.ai/t/reproducibility-question/1402/2 "2021-05-25T16:12:37Z")
How exactly are you loading your train and test data? The most likely reason for this is that there is randomization in the training data loading, so you don’t get exactly the same order of data in your mini batches.
---
### Author:  [@ramarro123](https://community.konduit.ai/u/ramarro123)
#### Post date: [May 25, 2021, 7:45pm UTC](https://community.konduit.ai/t/reproducibility-question/1402/3 "2021-05-25T19:45:07Z")
i don’t use batch, nor loader, i load them fro memory.
i also checked twice that featureMatrix, labelMatrix are the same
```
double[][] featureMatrix = new double[size][features];
double[][] labelMatrix = new double[size][Config.classVal.size()];
for (int q = 0; q < size; q++) {
for (int attr = 0; attr < features; attr++) {
featureMatrix[q][attr] = calculated.get(q).value(attr);
}
labelMatrix[q] = new double[Config.classVal.size()];
int indexOf = Config.classVal.indexOf(calculated.get(q).stringValue(features));
labelMatrix[q][indexOf] = 1;
}
INDArray trainingIn = Nd4j.create(featureMatrix);
INDArray label = Nd4j.create(labelMatrix);
return new DataSet(trainingIn, label);
```
then the data are fitted in that way
```
DataNormalization normalizer = new NormalizerStandardize();
normalizer.fit(this.train);
normalizer.transform(this.train);
normalizer.transform(this.test);
```
but again, i also checked with a simple this.train.toString().hashCode that the string it’s equals after each run, and so it is.
dataset basically are exactly the same at every run of the code
---
### Author:  [@ramarro123](https://community.konduit.ai/u/ramarro123)
#### Post date: [May 26, 2021, 9:22am UTC](https://community.konduit.ai/t/reproducibility-question/1402/4 "2021-05-26T09:22:12Z")
i created a post on github withmore details and an example
>
>
> i created a little test case to help me investigate on reproducibility issue.
>
> …
> i basically create test/train dataset in memory they are 100% the same, created with some junk data but in a consistent way (junk formula doens't use any random operation, so it's constant)
>
> two iteration, same dataset(s), same MultiLayerNetwork, same iterations (500)
>
> confusion matrix it's different in the 2 iterations, example
>
> \`\`\`
> 0 1 2
> \----------
> 23 6 4 | 0 = 0
> 22 6 5 | 1 = 1
> 23 6 5 | 2 = 2
>
> 0 1 2
> \----------
> 25 1 7 | 0 = 0
> 25 1 7 | 1 = 1
> 24 1 9 | 2 = 2
> \`\`\`
> i am using osx with cpu backend, and beta7 as deeplearning version.
>
> i wasn't expecting to see different data, since the seed it's constant. also restarting the program won't make any difference (in case we wonder if some internal random generator change while jvm it's running)
>
> if i start again the results are
> \`\`\`
> 0 1 2
> \----------
> 17 0 16 | 0 = 0
> 15 1 17 | 1 = 1
> 16 0 18 | 2 = 2
>
> 0 1 2
> \----------
> 17 0 16 | 0 = 0
> 15 1 17 | 1 = 1
> 16 0 18 | 2 = 2
> \`\`\`
> regards
>
> \`\`\`
>
> import org.deeplearning4j.nn.conf.MultiLayerConfiguration;
> import org.deeplearning4j.nn.conf.NeuralNetConfiguration;
> import org.deeplearning4j.nn.conf.layers.DenseLayer;
> import org.deeplearning4j.nn.conf.layers.OutputLayer;
> import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
> import org.deeplearning4j.nn.weights.WeightInit;
> import org.nd4j.evaluation.classification.Evaluation;
> import org.nd4j.linalg.activations.Activation;
> import org.nd4j.linalg.api.ndarray.INDArray;
> import org.nd4j.linalg.dataset.DataSet;
> import org.nd4j.linalg.dataset.api.preprocessor.DataNormalization;
> import org.nd4j.linalg.dataset.api.preprocessor.NormalizerStandardize;
> import org.nd4j.linalg.factory.Nd4j;
> import org.nd4j.linalg.learning.config.Adam;
> import org.nd4j.linalg.lossfunctions.impl.LossMCXENT;
>
> public class Reproducibility {
>
> public static void main(String args\[\]) {
> for (int iter = 0; iter \< 2; iter++) {
> DataSet train = create(1000, 50, 3);
> DataSet test = create(100, 50, 3);
>
> DataNormalization normalizer = new NormalizerStandardize();
>
> normalizer.fit(train);
> normalizer.transform(train);
> normalizer.transform(test);
>
> int layers\[\] = {
> 128, 96, 64, 32
> };
>
> NeuralNetConfiguration.ListBuilder builder = new NeuralNetConfiguration.Builder()
> .seed(0)
> .activation(Activation.LEAKYRELU)
> .weightInit(WeightInit.RELU)
> .updater(new Adam(0.001))
> .l2(0.000000010000)
> .list();
>
> builder.layer(new DenseLayer.Builder().nIn(50).nOut(layers\[0\])
> .build());
>
> for (int i = 1; i \< layers.length; i++) {
> builder.layer(new DenseLayer.Builder().nIn(layers\[i - 1\]).nOut(layers\[i\])
> .build());
> }
>
> MultiLayerConfiguration conf = builder.layer(new OutputLayer.Builder(new LossMCXENT())
> .activation(Activation.SOFTMAX) //Override the global TANH activation with softmax for this layer
> .nIn(layers\[layers.length - 1\]).nOut(Config.classVal.size()).build())
> .build();
>
> //run the model
> MultiLayerNetwork classifier = new MultiLayerNetwork(conf);
> classifier.init();
>
> for (int i = 0; i \< 500; i++)
> classifier.fit(train);
>
> Evaluation testEval = new Evaluation(Config.classVal.size());
> testEval.eval(test.getLabels(), classifier.output(test.getFeatures()));
> classifier.close();
> System.out.println(testEval);
> }
> }
>
> private static DataSet create(int size, int indicatorSize, int classes) {
> double\[\]\[\] featureMatrix = new double\[size\]\[indicatorSize\];
> double\[\]\[\] labelMatrix = new double\[size\]\[classes\];
>
> int pass = 0;
> for (int q = 0; q \< size; q++) {
> for (int attr = 0; attr \< indicatorSize; attr++) {
> pass++;
> //random junk data :D
> featureMatrix\[q\]\[attr\] = (q + attr \* pass) / (double) pass;
> }
> labelMatrix\[q\] = new double\[classes\];
> labelMatrix\[q\]\[Math.abs(pass) % 3\] = 1;
> }
>
>
> INDArray trainingIn = Nd4j.create(featureMatrix);
> INDArray label = Nd4j.create(labelMatrix);
>
> return new DataSet(trainingIn, label);
> }
> }
>
>
>
>
> \`\`\`
---
### Author:  [@treo](https://community.konduit.ai/u/treo)
#### Post date: [May 29, 2021, 4:57pm UTC](https://community.konduit.ai/t/reproducibility-question/1402/5 "2021-05-29T16:57:12Z")
For completeness:
> `.seed(0)`
This is the problem here. I’m not entirely sure why, but for some reason the `0` seed is being handled as if no seed has been set. If you put anything else in there, you will get a reproducible training run (I typically use something like `0xBEEF` , `0xC0FFEE` , `0x5EED` ).