How To feed model with data from memory

DataVec
1

Hi,

I can see some examples using csv as the source for the data, but I would rather use a database that i already prepared.

I can see that (MultiLayerNetwork)model.fit() accepts INDArray, and Dataset among other things.

given I have a database with data, I can load the data in batches into the program,

for example,
i have 10 inputs(int and double or float) in an object
i am expecting the output to be 2 float numbers

MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
…
.build();

MultiLayerNetwork model = new MultiLayerNetwork(conf);
model.init();
…

model.fit(data);

my questions are,

  • how to use INDArray or Dataset or the iterator versions, with ten values per row/entry? can i see a simple example if you don’t mind

  • can I use model.fit() multiple times in a loop?

for(var data : dataList) {
model.fit(data)
}

2

It isn’t documented very well, but there is a JDBC Record reader.

You can see it being used here: https://github.com/eclipse/deeplearning4j/blob/master/datavec/datavec-jdbc/src/test/java/org/datavec/api/records/reader/impl/JDBCRecordReaderTest.java

Once you have set it up, you can use it like any other record reader (e.g. like a csv record reader).

3

Thanks for the reply.

I don’t want to use the data from the database directly, the data is raw in the database, and the database tables are designed, not a table of input data.

I want to fetch the data from the database, transform it into the input which is integers and double(can change to float) in memory, queue it in, and pass it to the model.

But i am not familiar with the Dataset and INDArray creation,
if the input is 10 values and the output is 2, where or how to map the data to the those objects/classes

is the first parameter expected an input array with a size of the 10 input values? then the output as an array of 2 values? how to add multiple objects? should it be array[sizeOfEntriesToTrainAtOneTime][10][2] for example, or put INDArray into a dataset? as in the dataset is a collection of INDArrays

INDArray nd = Nd4j.create(new float{1,2,3,4,5,6,7,8,9,10},new float{2,2});

4

A Dataset object is at least two INDArrays:

  • one for a batch of features
  • one for a batch of labels

So if you create it like in your example, you will need to Nd4j.vstack them into a batch first before you pass it to the model.

But if you are building it like that you can also directly pass it to model.fit

5

simplifying the question,

if I have a,
List customInput; //3 features
customInput.size() = 1000;

List customExpectedOutput; ////1 output
customExpectedOutput.size() = 1000;

public class CustomInput {
public float val1;
public float val2;
public float val3;
}

public class CustomExpectedOutput {
public float val1;
}

and I want to use model.fit(dataSet),

how to create dataSet of type DataSet?