Migrating RL4J to DL4J Contrib

RL4J
1

Hi all

I have been working on updating RL4J since a few weeks, and things are getting in a state that it can be reviewed, and benefit from your feedback. In short, this is what has been done

  • Generalise the generic classes a bit further with support for Observations and Actions classes. The initial refactoring that was started some time ago was kept as much as possible
  • Add functionality to save/load models
  • Removal of all legacy code
  • Refactoring of all examples (ALE, Gym, Malmo, Doom, CartPole,…) to use the refactored code instead
  • Consolidation of the various modules/repo’s into a single repo

What is left to do is

  • Further code clean up and removal of comments
  • Fix all Unit Tests (I have no experience with test-driven dev, I need help on that)
  • As proof of case, implement a new algorithm, for example PPO

Adam’s proposal is to finally migrate the refactored code to org.deeplearning4j.contrib.rl4j

The code is currently sitting in GitHub - kgoderis/deeplearning4j at rl4j-refactor

2

@kgoderis Confirming I would be fine with continuing to help publish this.

Could you clarify what else you would want to do after this refactoring?
Publishing the new version will be great but there are still docs and other things to consider.
Since this is purely voluntary I’m ok making that clear. I just want core modules to have full time maintainers otherwise it’s hard for users to know what to expect in terms of support.

Great work so far!

3

@agibsonccc I refactored the lib because I intend to use it in a personal project. I would thus add functionality that is the result of what I do in that project. Things that come my mind are additional policy networks, new algo’s (maybe venture into model-based RL). In that respect, arXiv:2201.02135v1 ("Deep Reinforcement Learning by Aske Plaat/Springer Nature) is a nice write-up of the current state of the art in RL, and that could serve as the base of such extensions. It seems that the first few chapters are well covered by RL4J, and since he footnoted a lot of links to implementations on GitHub, a potential work could be to take those python repo’s and implement the java equivalent in RL4J making use of the core DL4J

By no means I am busy with ML in my day profession, so I am not best placed when it comes to giving support wrt core algorithms or ML concepts, but let’s say that the “D” in R&D is where I can be helpful, e.g, the java code base itself

K