Importing a pytorch model

SameDiff
41

@agibsonccc Hi Adam, how has it gone with adding the onnx ops?

42

@mdebeer was checking through forum posts, realized I typed a response and never submitted it. Sorry about that.
I didnā€™t get the detectron2 op done yet but wanted to take a look at that soon. A few more PRs have been merged that improve coverage quite a bit. I still want to get to the detectron 2 op but I wanted to make sure that we covered standard ops first. Weā€™re getting there though!

1 Like
43

Hey Adam, itā€™s been a while! Wishing you a prosperous new year.

Weā€™ve left importing detectron2 to dl4j on the back-burner, but itā€™s growing into an important use case for migrating our python code base to Scala.

Could you please update me on the latest progress? Will it be possible for us to start testing the onnx model import again?

44

There have been quite a lot of improvements on the import side of things, you can probably try the current snapshots and see if it maybe just runs out of the box :slight_smile:

45

@treo @mdebeer I never got around to this specific op. We did however add new model import infra including a new model zoo with pre converted models. It is much easier to add new ops now. I can give that a shot and see. Thanks for checking in.

46

BTW, we can use the full C++ API of PyTorch from Java quite easily, you might want to try that instead:

47

@agibsonccc @treo @saudet Thank you for the prompt feedback!
Glad to hear thereā€™s new improvements + a model zoo to check out. Iā€™ll dive into it throughout the week and will let you know of any findings or questions. Cheers!

48

@mdebeer I took another look at this since release QA is about wrapped up. I implemented 2 of the necessary ops. Thereā€™s only 1 more to do which I can wrap up tomorrow.

I just had to introduce a few new op overrides and re express the generate bounding box proposals op used by pytorch in samediff. Like I thought all the stuff was already there to do it. Iā€™ll put up a PR this week and let you know.

In the mean time take a look at:

This contains all the models already pre converted. Iā€™ll do the same for the detectron model. Then you can just use the new improvements for finetuing your model. Youā€™ll find gpt2 among other graphs already in there.

1 Like
49

Thatā€™s brilliant, thank you!

50

By the way, I tried to pull the latest snapshot again (I presume itā€™s 1.0.0-SNAPSHOT), and Iā€™m again getting the XML parsing error for the "org.nd4j" % "samediff-import-onnx" package (as described earlier, and fixed by @treo in this post: Importing a pytorch model - #34 by treo).

Again, itā€™s only for this package, and only for the snapshot. Could you please verify if the previous XML spacing error has crept in again? SBT / coursier presumably has stricter parsing rules than other tools.

51

@mdebeer Iā€™ll flag you when a PR is merged that solves this. Otherwise for now donā€™t worry about it. I found out there were a few more ops that needed to be implemented there. That graph is huge and I missed some ops.

Iā€™m currently in the middle of doing M2. After M2 Iā€™ll just convert the model and publish it on our zoo. The PR Iā€™d say is about 60% done for that.

Unfortunately during release time that also means dealing with long compile times for cuda (usually cpu/windows) which takes 5-6 hours and can sometimes fail for odd reasons (eg: network failures)

Edit: Hereā€™s the branch in case youā€™re curious. GitHub - eclipse/deeplearning4j at ag_detectron_2

One thing I need to finish implementing was a new masking array function. That will allow me to implement 1 more op. It turns out detectron2 is full of proprietary pytorch ops not just one.

Thanks for flagging though!

1 Like
52

Hey Adam - how is the progress to M2, and the detectron2 imports?

53

Now that M2 is released, Iā€™ve tested the ONNX import of a Detectron 2 model as follows (using the following as a guide):

import org.junit.Assert.assertNotNull
import org.nd4j.autodiff.samediff.SameDiff
import org.nd4j.samediff.frameworkimport.onnx.importer.OnnxFrameworkImporter

import java.io.File
import java.util.Collections

object ImportOnnxModel extends App {
  //create the framework importer
  val onnxImport = new OnnxFrameworkImporter()

  val onnxFile: File  = new File("model.onnx")
  val graph: SameDiff = onnxImport.runImport(onnxFile.getAbsolutePath, Collections.emptyMap(), true)

  assertNotNull(graph)
}

As found previously, the above code errors because the AliasWithName op is not defined. Hopefully itā€™s a trivial operation (seems like a node rename), so is it something that I can work on in a merge request? Please just direct me to relevant implementation.

Hereā€™s a Netron view of the ONNX model beginning:

image
image888Ɨ573 25.7 KB

54

@mdebeer sorry I guess I wasnā€™t clear. Detectron2 didnā€™t make it in to M2. I still have to finish up the PR for it. Other things took priority but Iā€™m working on the next wave of onnx import now that thatā€™s all done.

55

Okay ā€“ please keep me posted if a snapshot becomes available for testing. Best wishes with the work.