Python4J Cannot find the customized Python path in miniconda

Dear Community,

Recently, I’m trying to integrate my Pytorch models in the DL4J environment. Since my Pytorch models require additional modules, I need to create a virtual environment to import all dependencies.

As shown in the section Python Path in DL4J, DL4J enables the users to customize their own python environments. I created the new python virtual environment with miniconda. Unfortunately, I do not understand the following points:

  • org.eclipse.python4j.path: Should I change this path? As I understood, The user should use the following codes to get the path info:

import sys
import os
print(os.pathsep.join(sys.path))

I created a virtual environment for python 3.10.4. However, using the code above, the python version turns to be python 3.10.2. After inspecting the deep4j project, I found that the printed version is the one that is automatically loaded and cached by the deep4j framework (javacpp) within the CPython-3.10.2.xxx.jar.

In such a case, how to let the D4J project accept the newly created virtual environment?

  • org.eclipse.python4j.path.append: How can I use this property concretely? What does the following explanation mean? Further information or a demo would be great. What do the “none, before, after” imply here? How to use these flags?
A user can select none, before, or after.
In order to specify a custom python path, a user should be aware of 3 properties
  • Eager graph (TensorFlow 2.x eager/PyTorch) graph execution is planned. Is there any release plan for this supportive feature available now?

I’m working with Eclipse. Any further information & input about integrating the Pytorch model into DL4J will be appreciated.

Thanks in advance!

@Anakin that just gets you the python path you still have to set the system property. The append type just sets whether we should use any of the pre built in libraries (numpy mainly) or if we should let the user specify.

Hi @agibsonccc thanks a lot for your response and the explanation. I’m still a little bit confused by these preperties. Here is the Java snippet:

public class PythonExecutor {
	public static void main(String[] args) throws InterruptedException {
			System.setProperty("org.eclipse.python4j.path.append", "/home/skywalker/miniconda3/envs/venv_dl4j_3.10.4:/home/skywalker/miniconda3/envs/venv_dl4j_3.10.4/lib/python310.zip:/home/skywalker/miniconda3/envs/venv_dl4j_3.10.4/lib/python3.10:/home/skywalker/miniconda3/envs/venv_dl4j_3.10.4/lib/python3.10/lib-dynload:/home/skywalker/miniconda3/envs/venv_dl4j_3.10.4/lib/python3.10/site-packages");
			System.out.println("org.eclipse.python4j.path.append " + System.getProperty("org.eclipse.python4j.path.append"));
	
			try (PythonGIL pythonGIL = PythonGIL.lock()) {
				
				List<PythonVariable> inputs = new ArrayList<PythonVariable>();
				String code = Files.readString(Paths.get("/home/skywalker/miniconda3/envs/venv_dl4j_3.10.4/hello_python2.py"));
				
				PythonExecutioner.exec(code);
			} catch (Throwable e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			}
		}
    }
}

The python script is shown as follows:

import os
import sys
from platform import python_version

print('path:{}'.format(os.pathsep.join(sys.path)))

print('python_version', python_version())

import torch
import torchvision

After running the Java code, I got the following error:

org.eclipse.python4j.path.append /home/skywalker/miniconda3/envs/venv_dl4j_3.10.4:/home/skywalker/miniconda3/envs/venv_dl4j_3.10.4/lib/python310.zip:/home/skywalker/miniconda3/envs/venv_dl4j_3.10.4/lib/python3.10:/home/skywalker/miniconda3/envs/venv_dl4j_3.10.4/lib/python3.10/lib-dynload:/home/skywalker/miniconda3/envs/venv_dl4j_3.10.4/lib/python3.10/site-packages
12:13:38.371 [main] DEBUG org.nd4j.python4j.PythonGIL - Acquiring GIL on 1
12:13:38.373 [main] INFO org.nd4j.python4j.PythonGIL - Pre Gil State ensure for thread 1
12:13:38.373 [main] INFO org.nd4j.python4j.PythonGIL - Thread 1 acquired GIL
Traceback (most recent call last):
  File "<string>", line 65, in <module>
ModuleNotFoundError: No module named 'torch'
path:/home/skywalker/.javacpp/cache/cpython-3.10.2-1.5.7-linux-x86_64.jar/org/bytedeco/cpython/linux-x86_64/lib:/home/skywalker/.javacpp/cache/cpython-3.10.2-1.5.7-linux-x86_64.jar/org/bytedeco/cpython/linux-x86_64/lib/site-packages:/home/skywalker/.javacpp/cache/cpython-3.10.2-1.5.7-linux-x86_64.jar/org/bytedeco/cpython/linux-x86_64/lib/python3.10:/home/skywalker/.javacpp/cache/cpython-3.10.2-1.5.7-linux-x86_64.jar/org/bytedeco/cpython/linux-x86_64/lib/python3.10/lib-dynload:/home/skywalker/.javacpp/cache/cpython-3.10.2-1.5.7-linux-x86_64.jar/org/bytedeco/cpython/linux-x86_64/lib/python3.10/site-packages:/home/skywalker/.javacpp/cache/numpy-1.22.2-1.5.7-linux-x86_64.jar/org/bytedeco/numpy/linux-x86_64/python
0 /home/skywalker/.javacpp/cache/cpython-3.10.2-1.5.7-linux-x86_64.jar/org/bytedeco/cpython/linux-x86_64/lib
1 /home/skywalker/.javacpp/cache/cpython-3.10.2-1.5.7-linux-x86_64.jar/org/bytedeco/cpython/linux-x86_64/lib/python3.10
2 /home/skywalker/.javacpp/cache/cpython-3.10.2-1.5.7-linux-x86_64.jar/org/bytedeco/cpython/linux-x86_64/lib/python3.10/site-packages
sys.path:/home/skywalker/.javacpp/cache/cpython-3.10.2-1.5.7-linux-x86_64.jar/org/bytedeco/cpython/linux-x86_64/lib/site-packages:/home/skywalker/.javacpp/cache/cpython-3.10.2-1.5.7-linux-x86_64.jar/org/bytedeco/cpython/linux-x86_64/lib/python3.10/lib-dynload:/home/skywalker/.javacpp/cache/numpy-1.22.2-1.5.7-linux-x86_64.jar/org/bytedeco/numpy/linux-x86_64/python
python_version 3.10.2
No module named 'torch'
12:13:38.384 [main] DEBUG org.nd4j.python4j.PythonGIL - Pre gil state release for thread 1
12:13:38.385 [main] INFO org.nd4j.python4j.PythonGIL - Releasing GIL on thread 1
org.nd4j.python4j.PythonException: ModuleNotFoundError: No module named 'torch'
	at org.nd4j.python4j.PythonExecutioner.throwIfExecutionFailed(PythonExecutioner.java:201)
	at org.nd4j.python4j.PythonExecutioner.exec(PythonExecutioner.java:229)
	at com.demo.PythonExecutor.main(PythonExecutor.java:111)

It turns out that although the append property is set (not sure, whether it has been done correctly), the javacpp stills calls the pre-built python, i.e., cpython-3.10.2.xxx.jar, in which the torch is not installed yet.

In my case, there are multiple Pytorch models, converting these models with onnx is much more daunting work since there are different issues during the exporting. As a short term solution, therefore I intent to use all the existing python-based solutions diretly and wait until D4J framework releases the powerful features for (eager/PyTorch) graph. Is it necessary to activate the miniconda environment at first within D4J?

I’m still struggling with this issue to use this customized python environment in D4J ecosystem. Any inputs will be appreciated.

@Anakin python4j doesn’t really understand miniconda concepts. You have to activate the right profile in miniconda then use the python script to get the right python path. All we can do is load a python path you tell us to.
I think you should isolate anything to do with python4j specifically and first just understand what a python path is and how it works. Googling brings this up:

Once you understand what a PYTHONPATH is the properties we list make a bit more sense.

Beyond that…I’m not sure what the issue is with onnx models. I’d at least try it. It’s just a few lines of code. You can see some examples here: deeplearning4j/pytorch.py at master · eclipse/deeplearning4j · GitHub

It’s basically just setting up the data in pytorch and then calling export. That export call traces the call graph and sets up the appropriate ops. I’m happy to support either use case though.

@Anakin I was confused by the documentation too but I have Java now executing Python code using the packages from my miniconda environment.
There are a couple of key things to understand.

Your miniconda Python interpreter will not be used. The CPython interpreter bundled with javacpp will be used. Your miniconda environment is used to simply install python packages the javacpp CPython interpreter can load.
Ideally the miniconda Python versions would be identical so that the packages miniconda installs will be for that exact version, and avoid any issues where the interface to C extension modules or something isn’t backwards compatible, but as long as this hasn’t changed between versions it will run fine.
My installed version of CPython with javacpp is using v3.10.2, I installed v3.10.4 with miniconda.

To let your javacpp interpreter load your packages from your miniconda env you just need to point javacpp to your site-packages /home/skywalker/miniconda3/envs/venv_dl4j_3.10.4/lib/python3.10/site-packages

In the Java code running System.setProperty('org.eclipse.python4j.path', pythonEnvPath) didn’t do anything for me.
You can check if your value got loaded into the path by running print statements in your python script to see if your miniconda sitepackages path is there

import sys
for path in sys.path:
    print(path)
out = str(sys.path)

It should print

../.javacpp/cache/cpython-3.10.2-1.5.7-macosx-x86_64.jar/org/bytedeco/cpython/macosx-x86_64/lib
../.javacpp/cache/cpython-3.10.2-1.5.7-macosx-x86_64.jar/org/bytedeco/cpython/macosx-x86_64/lib/site-packages
../.javacpp/cache/cpython-3.10.2-1.5.7-macosx-x86_64.jar/org/bytedeco/cpython/macosx-x86_64/lib/python3.10
../.javacpp/cache/cpython-3.10.2-1.5.7-macosx-x86_64.jar/org/bytedeco/cpython/macosx-x86_64/lib/python3.10/lib-dynload
../.javacpp/cache/cpython-3.10.2-1.5.7-macosx-x86_64.jar/org/bytedeco/cpython/macosx-x86_64/lib/python3.10/site-packages
../.javacpp/cache/cpython-3.10.2-1.5.7-macosx-x86_64.jar/org/bytedeco/cpython/macosx-x86_64
../.javacpp/cache/cpython-3.10.2-1.5.7-macosx-x86_64.jar/org/bytedeco/cpython/macosx-x86_64/site-packages
../miniconda3/envs/envname/lib/python3.10/site-packages

I think setting the System property this way doesn’t work because org.nd4j.python4j has already been imported and loaded javacpp
To get this to work I had to add the System Property as a “VM option”
If running from the command line this is done like
java -Dcustom_key="custom_value" ...

I’m using IntelliJ and had to alter my run configurations to insert this
Under Run/Debug configurations
Under Modify options select Add VM Options
In VM Options insert
-Dorg.eclipse.python4j.path="/home/skywalker/miniconda3/envs/venv_dl4j_3.10.4/lib/python3.10/site-packages"

@Robbie-Palmer thanks for the updated explanation. The next release is about done and I wanted to do a pass on the docs so feedback is appreciated.

I am stuck with the same problem - i have a timeseries model that uses pytorch. I would like to invoke it from java. I have followed this discussion, but I cannot get javacpp to recognize my conda environment. I am wondering if @Anakin has got it to work. Any pointers much appreciated.

@moberc Unfortunately, I gave up on it because the effort is huge. As a compromise, I used a web service to run the torch models. An additional component is built by a DL4J application that is used to exchange the data.

BTW, I haven’t tried @Robbie-Palmer’s solution yet. Maybe this one will help you more or less.

Hi @Anakin, I am thinking about doing the same (= calling python over http). I have tried @Robbie-Palmer’s solution and it did not work. Thanks!

@moberc Since it was a prototype project, it’s OK that I pack all the Python dependencies in Docker as a server. For a long-term solution, we still need a strong DL4J community.

Just copy the py lib to built-in Python4j py lib

@Anakin @moberc can you guys clarify what your issue was? I haven’t had issues as long as the python versions were the same. There are also different flags for controlling which order python libraries are loaded in. I didn’t see any errors being posted so it’s kinda hard to say what is going on here. I can say that the order can matter depending on what libraries are on your path.

There’s only so many issues here. You either have to control the load order, it could be a native dependency or your python path is missing a dependency somewhere.

I am developing on Windows using python 3.10. javacpp installs 3.10.2.
I have created a miniconda env with python 3.10.11 and torch.
I am trying to run a simple script that loads torch:
private static final String PYTHON_SCRIPT = “import sys\n”
+ “import os\n”
+ “print(os.pathsep.join(sys.path))\n”
+ “import torch\n”;
No matter what I do (using env variables or adding variables as VM arguments), I am getting ‘torch not found’.
I have added folder with site_packages in my miniconda env,
org.nd4j.python4j.PythonException: ModuleNotFoundError: No module named ‘torch’
at org.nd4j.python4j.PythonExecutioner.throwIfExecutionFailed(PythonExecutioner.java:201)

When I did what Booker has suggested (copied lib folder from my miniconda to C:\Users\moberc.SMART-TRADE\AppData\Local\miniconda3\Lib, I am getting
Numpy import failed:
java.util.ServiceConfigurationError: org.nd4j.python4j.PythonType: Provider org.nd4j.python4j.numpy.NumpyArray could not be instantiated
at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:581)

I am wondering it difference in python versions causes that (3.10.11 vs 3.10.2).

I am planning to switch to tensorflow and try keras import.
Any suggestions much appreciated. Thx!

@moberc apologies I didn’t see this. I guess you’ve moved on by now. Please do update this if you made any progress. Sorry please ping me in the future if you dont’ hear fro me.