Implementation of Dropout

Ugai · July 3, 2020, 2:53am

I would like to use Dropout with SameDiff.

eclipse/deeplearning4j/blob/master/nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/api/ops/random/impl/DropOut.java

/*
 *  ******************************************************************************
 *  *
 *  *
 *  * This program and the accompanying materials are made available under the
 *  * terms of the Apache License, Version 2.0 which is available at
 *  * https://www.apache.org/licenses/LICENSE-2.0.
 *  *
 *  *  See the NOTICE file distributed with this work for additional
 *  *  information regarding copyright ownership.
 *  * Unless required by applicable law or agreed to in writing, software
 *  * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 *  * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 *  * License for the specific language governing permissions and limitations
 *  * under the License.
 *  *
 *  * SPDX-License-Identifier: Apache-2.0
 *  *****************************************************************************
 */

This file has been truncated. show original

public DropOut(SameDiff sameDiff, SDVariable input, double p) {
    super(sameDiff, input);
    this.p = p;
    //https://github.com/deeplearning4j/deeplearning4j/issues/5650
    throw new UnsupportedOperationException("Dropout SameDiff support disabled pending backprop support");
}

github.com/eclipse/deeplearning4j

Libnd4j/SameDiff dropout issues

opened 09:35AM - 20 Jun 18 UTC

closed 06:16AM - 20 Feb 19 UTC

AlexDBlack

SameDiff

Related: https://github.com/deeplearning4j/deeplearning4j/issues/5485 Current…ly, we have DropOut and DropOutInverted ops (the latter is used in DL4J). Also AlphaDropout. There are some issues with using these in SameDiff: 1. We use p = "input retain probability". TF/Keras uses p = "input drop probability". Do we want to align this (at least at the public API) in SameDiff? (downside: it'd be inconsistent with DL4J) 2. We need to know the dropout mask for peforming backprop through inverted dropout 3. We need some sort of support for 'train vs inference' mode I'm not sure how we handle (2). But some options: (a) Decompose dropout into 'generate mask' and 'apply mask' steps. This would work, and would be usable now without anything special; however, it pushes the train vs. inference problem into the general RNG distributions (which isn't ideal - we need to conditionally generate an all 1s array from a binomial distribution at inference time) (b) Add some sort of stateful storage mechanism for ops (this is basically what cudnn does for dropout) - store the binary mask and reuse for backprop (c) (variant of 2) Introduce something along the lines of "op outputs that are not SDVariable outputs" (d) add an extra RNG seed arg to the ops - re-generate the mask on demand, and update the seed only on forward pass (use same seed on backprop) As for (3) - we might want to build a more general solution (unless we have one already?) as this will come up in other cases too (batch norm, for example). (Obviously, dropout is train-only. Looks like TF just has this as an op arg, and no-ops if it's inference mode. Only real question is the API for this, and updating the op args on the fly before execution). Edit: more ops that has the same sort of issue (minus train/test) are Choice op and ProbabilisticMerge op. Need to know the mask for performing backprop.

is closed.
Can I have any crue to implemnt this part on my own?

treo · July 3, 2020, 6:11am

It is available on the native side already, but as you have found it wasn’t integrated to java just yet:

But in principle you can implement that integration on your own, as you’ve seen when we fixed the issue with batchNorm in the other thread.

If you are willing to do that, the starting point would be the native op definition:

github.com

eclipse/deeplearning4j/blob/master/libnd4j/include/ops/declarable/generic/random/dropout.cpp#L64-L86


      
          NDArray* input = INPUT_VARIABLE(0);    // lookup param
          NDArray* gradOut = INPUT_VARIABLE(1);  // lookup param
          
          
NDArray* reduceShape = nullptr;         // this param is optional
          NDArray* output = OUTPUT_NULLIFIED(0);  //
          
          
int seed = INT_ARG(0);
          
          
double probValue = T_ARG(0);
          if (block.width() > 2) reduceShape = INPUT_VARIABLE(2);
          
          
REQUIRE_TRUE((probValue > 0. && probValue <= 1.), 0, "dropout_bp: Probability should be with range 0 to 1.");
          if (probValue == 1.0) {
            output->assign(0.f);  // fill up output with 0
            return sd::Status::OK;
          }
          
          
REQUIRE_TRUE(helpers::dropOutFunctorBP(block, input, gradOut, output, reduceShape, seed, probValue) == sd::Status::OK,
                       0, "dropout_bp: Cannot backprop dropout.");

This file has been truncated. show original

Topic		Replies	Views
NPE when use sameDiff.nn.dropout SameDiff	2	188	May 30, 2022
Using gradient as an intermediate SDVariable SameDiff	11	352	June 14, 2022
Problems using SameDiff? Read this first! SameDiff	4	1071	November 16, 2021
The ND4j model convert to SameDiff SameDiff	1	188	April 19, 2023
SameDiff Normalization SameDiff	1	143	September 14, 2023

Implementation of Dropout

Related Topics