I wanted to ask if I am on the right path for training my LSTM RNN. I have been reading this forum, had a look at git, all the documentation and there seems to be some outdated information. I could not find a coherent, single place for an LSTM tutorial, or it had not the full information needed. Therefore, I will appreciate any help.
I have read the training notes, advice for choosing different parameters, expected input to the RNN on the official website. However, my network is not performing as expected, which might be a question of tuning the parameters. Regardless, I wanted reassurance that the input data is in the right format and the way I am training the network is correct.
LSTM RNN task: I have a single time series of numbers (x1), and I have a second-time series which is the expected output (x2), basically I have 2 variables (x1 and x2) that correlate and given the first one (x1) I want to be able to predict the second one (x2) at next timestep. From the tutorial on the official website, I have formatted by data in the following way [1- as I have only one time serries][1 - as I have the first variable (x1) as input][ 500 - as this is the length of my time series, basically it is 500 seconds]. This is my features data format, is it correct? My label data format is [1 - storing the second variable]. I combine both to the DataSet. Is it correct?
Lastly, I wanted to enquire about the training. I have seen that the getFeaturesMatrix() method on the data iterator is deprecated and now removed. As a result, I have a for loop, looping through a specified number of epochs and I simply call the fit() method passing the whole data set and then I clear the previous RNN state (in the loop).
As my data is in the same units, I do not normalise it, so I am not expecting it to be a cause of the problem.
Overall, I just wanted to ask if I format the data correctly, as I am suspecting that somehow I am using only 1 sample, not the full time series, could that be the case?