Huber Loss Function Not Working

DL4J
#1

Any idea why this hubert loss function would fail? Same network is working correctly for MSE loss function.

public class HuberLoss extends SameDiffLoss{

	@Override
	public SDVariable defineLoss(SameDiff sd, SDVariable yPred, SDVariable yTrue) {
		return sd.loss.huberLoss(yTrue, yPred, null, 1.0);
	}

}

NDArray::applyScalarArr BoolOps: this dtype: [5]; scalar dtype: [6]
Exception in thread "main" 14:24:54.952 [main] ERROR o.n.l.c.n.ops.NativeOpExecutioner - Failed to execute op huber_loss_grad. Attempted to execute with 3 inputs, 3 outputs, 1 targs,0 bargs and 1 iargs. Inputs: [(FLOAT,[1,3],c), (FLOAT,[],c), (FLOAT,[1,3],c)]. Outputs: [(FLOAT,[1,3],c), (FLOAT,[],c), (FLOAT,[1,3],c)]. tArgs: [1.0]. iArgs: [3]. bArgs: -. Input var names: [layerInput, sd_var, labels]. Output var names: [layerInput-grad, sd_var-grad, labels-grad] - Please see above message (printed out from c++) for a possible cause of error.
java.lang.RuntimeException: NDArray::applyScalarArr bool method: this and scalar arrays must have the same type!
	at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:1918)
	at org.nd4j.linalg.factory.Nd4j.exec(Nd4j.java:6575)
	at org.nd4j.autodiff.samediff.internal.InferenceSession.doExec(InferenceSession.java:487)
	at org.nd4j.autodiff.samediff.internal.InferenceSession.getOutputs(InferenceSession.java:214)
	at org.nd4j.autodiff.samediff.internal.InferenceSession.getOutputs(InferenceSession.java:60)
	at org.nd4j.autodiff.samediff.internal.AbstractSession.output(AbstractSession.java:386)
	at org.nd4j.autodiff.samediff.SameDiff.directExecHelper(SameDiff.java:2579)
	at org.nd4j.autodiff.samediff.SameDiff.batchOutputHelper(SameDiff.java:2547)
#2

@DeepLearner first question to look in to…this is a data type error. We always throw an exception when the 2 aren’t the same data type. Could you double check the data types of all the variables being passed to the operation?

#3

@agibsonccc

yPred is dType float, yTrue is dType float. A simple example to demonstrate the error is below. This was tested on 1.0.0-M1

public class Example {
	public static final int INPUT_SIZE=2;
	public static final int OUTPUT_SIZE=2;

	public static void main (String av[]) {
		MultiLayerConfiguration conf =new NeuralNetConfiguration.Builder()
			    .seed(12345)
			    .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
			    .list()
			    .layer(0, new DenseLayer.Builder().nIn(INPUT_SIZE).nOut(OUTPUT_SIZE).weightInit(WeightInit.RELU).activation(Activation.RELU)
			            .build())
			    .layer(1, new OutputLayer.Builder(new HuberLoss()).nIn(OUTPUT_SIZE).nOut(OUTPUT_SIZE).weightInit(WeightInit.IDENTITY).activation(Activation.IDENTITY)
			            .build())
			    .build();
		INDArray input = Nd4j.create(new float[] {-1f,1f}, new int[] {1,2});
		INDArray expected = Nd4j.create(new float[] {-0.5f,0.5f}, new int[] {1,2});


		MultiLayerNetwork network = new MultiLayerNetwork(conf);
		network.init();
		
		network.fit(input, expected);
	}
}
#4

@DeepLearner this is wonderful thanks. Let me get back to you this week. Ping me if you don’t hear anything by early next week.

#5

@DeepLearner so I just tried this… the HuberLoss() actually doesn’t extend ILossFunction that the OutputLayer is expecting, normally you would have to wrap that with samediff loss or something.

Could you actually give me something that runs out of the box and compiles? Thanks!

#6

@agibsonccc

This compile for me.

public class Example {
	public static final int INPUT_SIZE=2;
	public static final int OUTPUT_SIZE=2;

	public static void main (String av[]) {
		MultiLayerConfiguration conf =new NeuralNetConfiguration.Builder()
			    .seed(12345)
			    .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
			    .list()
			    .layer(0, new DenseLayer.Builder().nIn(INPUT_SIZE).nOut(OUTPUT_SIZE).weightInit(WeightInit.RELU).activation(Activation.RELU)
			            .build())
			    .layer(1, new OutputLayer.Builder(new HuberLoss()).nIn(OUTPUT_SIZE).nOut(OUTPUT_SIZE).weightInit(WeightInit.IDENTITY).activation(Activation.IDENTITY)
			            .build())
			    .build();
		INDArray input = Nd4j.create(new float[] {-1f,1f}, new int[] {1,2});
		INDArray expected = Nd4j.create(new float[] {-0.5f,0.5f}, new int[] {1,2});


		MultiLayerNetwork network = new MultiLayerNetwork(conf);
		network.init();
		
		network.fit(input, expected);
	}
	
}

class HuberLoss extends SameDiffLoss{

	@Override
	public SDVariable defineLoss(SameDiff sd, SDVariable yPred, SDVariable yTrue) {
		return sd.loss.huberLoss(yTrue, yPred, null, 1.0);
	}

}
#7

@DeepLearner ah so you did use the SameDiffLoss. Thanks for confirming! Let me take a look then.

#8

@DeepLearner I confirmed the cause is due to a default data type of double defined here:
deeplearning4j/HuberLoss.java at 4766032444de8e0c2c3389270576bb6e7c466211 · eclipse/deeplearning4j · GitHub

The default data type for memory saving is actually double.
This works fine:

import org.deeplearning4j.nn.api.OptimizationAlgorithm;
import org.deeplearning4j.nn.conf.MultiLayerConfiguration;
import org.deeplearning4j.nn.conf.NeuralNetConfiguration;
import org.deeplearning4j.nn.conf.WorkspaceMode;
import org.deeplearning4j.nn.conf.layers.DenseLayer;
import org.deeplearning4j.nn.conf.layers.OutputLayer;
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
import org.deeplearning4j.nn.weights.WeightInit;
import org.nd4j.autodiff.samediff.SDVariable;
import org.nd4j.autodiff.samediff.SameDiff;
import org.nd4j.linalg.activations.Activation;
import org.nd4j.linalg.api.buffer.DataType;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.factory.Nd4j;
import org.nd4j.linalg.lossfunctions.ILossFunction;
import org.nd4j.linalg.lossfunctions.SameDiffLoss;

public class HuberLossTest {

    public static final int INPUT_SIZE= 2;
    public static final int OUTPUT_SIZE=2;

    public static void main (String av[]) {
        MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
                .seed(12345)
                .dataType(DataType.DOUBLE)
                .inferenceWorkspaceMode(WorkspaceMode.NONE)
                .trainingWorkspaceMode(WorkspaceMode.NONE)
                .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
                .list()
                .layer(0, new DenseLayer.Builder().nIn(INPUT_SIZE).nOut(OUTPUT_SIZE).weightInit(WeightInit.RELU).activation(Activation.RELU)
                        .build())
                .layer(1, new OutputLayer.Builder(new HuberLoss()).nIn(OUTPUT_SIZE).nOut(OUTPUT_SIZE).weightInit(WeightInit.IDENTITY).activation(Activation.IDENTITY)
                        .build())
                .build();
        INDArray input = Nd4j.create(new float[] {-1f,1f}, new int[] {1,2});
        INDArray expected = Nd4j.create(new float[] {-0.5f,0.5f}, new int[] {1,2});


        MultiLayerNetwork network = new MultiLayerNetwork(conf);
        network.init();

        network.fit(input, expected);
    }

    public static class HuberLoss extends SameDiffLoss {

        @Override
        public SDVariable defineLoss(SameDiff sd, SDVariable yPred, SDVariable yTrue) {
            return sd.loss.huberLoss(yTrue, yPred, null, 1.0);
        }

    }
}

In the mean time, I’ll get this fixed. Thanks for pointing this out.