How to feed a network with data from GPU

HGuillemet · March 8, 2021, 12:43pm

I’d like to:

decode video frame
apply some simple processing (rescaling, channel mix etc…) probably with OpenCV
feed a ComputationGraph

while staying GPU-side.

What’s the best way to do this while limiting memory copy/download/upload ? I can decode frames into OpenCV GpuMat, but then how to view the GpuMat as an INDArray (or the inverse : how to create a INDArray and build a GpuMat view ?)

agibsonccc · March 8, 2021, 12:46pm

Pinging @saudet here

Generally a NativeImageLoader can be used on Mats, but I’m not sure about GPUMat. Many of the ways we convert Mat to NDArray can be found here:

github.com

eclipse/deeplearning4j/blob/master/datavec/datavec-data/datavec-data-image/src/main/java/org/datavec/image/loader/NativeImageLoader.java#L181


 * @see #asMatrix(Object)
 */
public INDArray asRowVector(Object image) throws IOException {
    return asMatrix(image).ravel();
}

public INDArray asRowVector(Frame image) throws IOException {
    return asMatrix(image).ravel();
}

public INDArray asRowVector(Mat image) throws IOException {
    INDArray arr = asMatrix(image);
    return arr.reshape('c', 1, arr.length());
}

public INDArray asRowVector(org.opencv.core.Mat image) throws IOException {
    INDArray arr = asMatrix(image);
    return arr.reshape('c', 1, arr.length());
}

static Mat convert(PIX pix) {

HGuillemet · March 8, 2021, 1:28pm

Thank you.
I’m aware of how to share memory between Mat and NDJ4, but GpuMat/UMat are different. You cannot get a normal pointer to their data.

saudet · March 9, 2021, 1:36am

AFAIK, that’s not possible at the moment because ND4J always requires GPU memory to be backed by a buffer in host memory, so you’ll need to transfer your data to CPU anyway.