How to shift a time series?

DataVec
1

I found no easy way to shift a column of a time series of data with Datavec. How can this done easily?

More or less all Transforms work at he record level; I fail to see how to do this with SequenceOffsetTransform for a dataset that is not yet converted into a Sequence

So far, my solution is to write a custom RecordReader that reads records one by one from its source, store them in memory, and when calling .next(), make sure it shifts +n/-n lines for those columns where you need to shift, and then emit the record.

2

@kgoderis could you clarify why you can’t just convert it to a sequence or directly encode things as sequences? We have a separate record reader there for a reason.

3

I am actually using a SequenceRecordReader, but that one constructs a sequence of Records that come from an underlying RecordReader. In that case, the Records, generated by the RecordReader, would need to be modified by the SSR in order to shift Columns over time/Records. In fact, the complexity comes from the fact that Readers do not store data, they are Iterators at the base (e.g. it would involve a lot of peeking to get the offset record)

but image that I have a SequenceRecord, after it came from a Reader, how would you do such a shifting then?

Initially I tried to use Transformers (Transform to Sequence, Transform, Transform back to ordinary Records) but this is a complicated work-around IMO.