Ideally, you would create a custom SequenceRecordReader implementation for your case, and then use the regular datavec methods. (see also this post and the replies to it, for a way that can be used if the data fits in memor, I don’t quite recommend it, but sometimes that way can be faster to get started Training CNN error / CNN text classification - #8 by treo).
You can print out your data after conversion to take another look at it. Take also a look at https://deeplearning4j.konduit.ai/models/recurrent#data-for-rnns for more information on how to structure data for recurrent networks.
It is easy to select hyper parameter values where the network isn’t going to learn anything, so that may be a reason