Time Series Prediction - Retraining of Neural Network

DL4J
1

Dear all

I was wondering what you would recommend on how to do retraining/relearning of an already trained network?

So I have implemented a neural network (config doesnt matter for now) to predict time series. And now I am struggling when it comes to retraining/relearning. The neural net predicts data for a whole week and I would like to retrain weekly as soon as new data for the past week is available.

Is it a good idea to just call net.fit(….) with the new data and with same number of epochs as the inital training was done?
Or should I discard the whole training and train the network from scratch with all the old and the new data?

@agibsonccc
@treo

Thanks in advance for answering :slight_smile:

2

It really depends on what exactly your problem is.

Just retraining with the new data may result in “catastrophic forgetting” - but sometimes that is exactly what you want.

For this reason, you will need to at least keep statistics how good your model predictions are, and ideally you would also keep statistics over your existing data.

For example, if the statistics over your old data show that there is a concept drift (i.e. the data distribution changes over time), then you may be better of with retraining regularly.

If you don’t see a concept shift, you may not need any retraining at all.

3

Hi @treo

Thanks for your fast response.

It might be that there are concept shifts.
So a reuglarly retraining would be necessary.

Can you give a short example on how to do the retraining?

Is it just something like net.fit(datasetiteratorWithNewData, sameNumberOfEpochsAsForInitialTraining) Or do I need do consider something else?

Or maybe better discard the old training and start the training from beginning with the old and new data?

Kind regards

4

It depends on how large the concept shift is and how you want to treat your weights.

Usually, when you save your model for deployment, you don’t keep the updater / optimizer state. If you reload just the weights and then train your model, it will essentially treat your weights as the initialization and go from there.

That may result in significantly faster training if the concept shift is small. And that in turn means that you need to ensure that you don’t overfit on your new data because you don’t need as many training iterations as the original training took.

If the concept shift is large enough, you may want to throw away any previous weights because it may keep you in a local minimum that is worse than what you could achieve with retraining from scratch.

Unfortunately there is no single right answer, so that will depend on you actual problem and the data that you get and how it changes over time.

5

Ok - maybe some more info about the current situation.

The neural net is to predict the timesteps of the following week and it was trained on the data from the last 10 weeks.

Example:
We have a backup server which does weekly backups of all other servers on Sunday.
The server needs about 400MB Memory to do these backups in the last 10 weeks. In week 11 the backup server needs to do 10 more backups as usual in week 1 to 10 on which the neural net was trained on. This means that the backup server needs 1GB Memory to do the backups after week 11 and also in week 12,13,14 and so on. So the predictions are „crap“ after week 11 which leads to a retraining

My approach:
Just do retraining after every week since the time series can change every week or even day.
Process would look like that:

  1. Do initial training as soon as data from week 1 to week 10 is available
  2. Do prediction for next week
  3. At end of week do retraining with data from last week (not from scratch)

→ steps 2 and 3 are going to be done every week

Do you agree with my approach?

6

The approach itself should work.

But I’m confused why you even need a neural network in the first place for the prediction. It looks like a simple linear regression or maybe if you need to be fancy a multivariate regression would work a lot better.

1 Like