How to data normalization for INDArray/Array

ND4J
1 
    val model = modelLoadHDFS("hdfs://xx/tmp/mutiTimestepsLstm.h5")
    val myArray = new Array[Float](size)
    .....
    val xx = splitSequence(myArray , 5, 1)
    val myINDArray  = xx(0).asInstanceOf[INDArray]

    val pridictOut = model.output(myINDArray)

How to data normalization for myINDArray or my myArray ?

Thanks.

2

How you want to normalize it, depends on your data.

There are multiple Normalizers available for direct normalization:
https://deeplearning4j.org/api/latest/org/nd4j/linalg/dataset/api/preprocessor/Normalizer.html (see “All Known Implementing Classes”)

Once you’ve chosen an appropriate normalizer, you can normalize a given NDArray by calling normalizer.transform(array)

3 
    val model = modelLoadHDFS("hdfs://xx/tmp/mutiTimestepsLstm.h5")
    val myArray = new Array[Float](size)
    .....
    val xx = splitSequence(myArray , 5, 1)
    val myINDArray  = xx(0).asInstanceOf[INDArray]

    val scaler = new NormalizerMinMaxScaler()
    scaler.transform(myINDArray)

    val pridictOut = model.output(myINDArray)

Thanks. I am try it, throw errors

Exception in thread "main" org.nd4j.linalg.exception.ND4JIllegalStateException: Features statistics were not yet calculated. Make sure to run fit() first.
	at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.transform(AbstractDataSetNormalizer.java:160)
	at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.transform(AbstractDataSetNormalizer.java:152)

But run fir() need input Dataset type

shoud I translate INDArray to Dataset ?

Any otherways to fix this.

Thanks.

4

The NormalizerMinMaxScaler needs to learn about your data first. It has to know the range it has to normalize from.

As you are trying to run inference, I guess you already know the range. In that case, you can set them directly on the normalizer without having to run .fit by using .setFeatureStats.
It takes two arrays, one that provides the max value for each feature and one that provides a min value for each feature.

5

Good. Thank you very much.

   ...
    val scaler = new NormalizerMinMaxScaler()
    scaler.setFeatureStats(Nd4j.create(1).add(sequence.min),Nd4j.create(1).add(sequence.max))
    scaler.transform(XXnd)
    scaler.transform(yynd)
    ...

Add the code, and it works.