# Data slicing failed

**URL:** <https://community.konduit.ai/t/data-slicing-failed/2506>\
**Category:** ND4J\
**Created:** [April 17, 2023, 2:27am UTC](https://community.konduit.ai/t/data-slicing-failed/2506 "2023-04-17T02:27:47Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![luuu](https://avatars.discourse-cdn.com/v4/letter/l/b4bc9f/32.png) [@luuu](https://community.konduit.ai/u/luuu)\
**Post date:** [April 17, 2023, 2:27am UTC](https://community.konduit.ai/t/data-slicing-failed/2506/1 "2023-04-17T02:27:48Z")

</div>

Why some slices succeed and some slices fail.

```auto
INDArray data = Nd4j.rand(17379, 59);
int casualIndex = 56;
int cntIndex = 58;
INDArray features = data.get(NDArrayIndex.all(), NDArrayIndex.interval(0, casualIndex));
INDArray targets = data.get(NDArrayIndex.all(), NDArrayIndex.interval(casualIndex, cntIndex + 1));

int total_row = (int) data.shape()[0];
int val_size = total_row - 21 * 24;
int train_size = val_size - 60 * 24;
INDArray train_features = features.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.all());
INDArray train_targets = targets.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.all());
INDArray val_features = features.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.all());
INDArray val_targets = targets.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.all());
INDArray test_features = features.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.all());
INDArray test_targets = targets.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.all());

System.out.printf("train_features shape:%s\n", Arrays.toString(train_features.shape()));
System.out.printf("train_targets shape:%s\n", Arrays.toString(train_targets.shape()));
System.out.printf("val_features shape:%s\n", Arrays.toString(val_features.shape()));
System.out.printf("val_targets shape:%s\n", Arrays.toString(val_targets.shape()));
System.out.printf("test_features shape:%s\n", Arrays.toString(test_features.shape()));
System.out.printf("test_targets shape:%s\n", Arrays.toString(test_targets.shape()));

```

Run the results

train\_features shape:[15435, 56]  
train\_targets shape:[15435, 3]  
val\_features shape:[1440, 56]  
val\_targets shape:[]  
test\_features shape:[]  
test\_targets shape:[]

---

<div class="post-metadata">

**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\
**Post date:** [April 17, 2023, 2:29am UTC](https://community.konduit.ai/t/data-slicing-failed/2506/2 "2023-04-17T02:29:35Z")

</div>

@luuu could you clarify what you expect the results to be for each one and if you got any exceptions? Thanks! Otherwise I have to guess what you want the result to be. I might be able to help reword your code to give the correct result.

---

<div class="post-metadata">

**Author:** ![luuu](https://avatars.discourse-cdn.com/v4/letter/l/b4bc9f/32.png) [@luuu](https://community.konduit.ai/u/luuu)\
**Post date:** [April 17, 2023, 2:38am UTC](https://community.konduit.ai/t/data-slicing-failed/2506/3 "2023-04-17T02:38:05Z")

</div>

@agibsonccc The original data is first divided into features and labels. Then the features and labels are divided into training, validation, and testing again.

---

<div class="post-metadata">

**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\
**Post date:** [April 17, 2023, 2:42am UTC](https://community.konduit.ai/t/data-slicing-failed/2506/4 "2023-04-17T02:42:48Z")

</div>

@luuu just to clarify I’m purely talking about the shapes of the data. Try to give me something self contained I can use with the shapes pre defined. Start from the original dataset with an expected full shape (say 1000 rows by whatever columns) and then give me the indexing calls you’re trying to do.  
Try to make it so I don’t have to spend time reverse engineering your specific problem context.

---

<div class="post-metadata">

**Author:** ![luuu](https://avatars.discourse-cdn.com/v4/letter/l/b4bc9f/32.png) [@luuu](https://community.konduit.ai/u/luuu)\
**Post date:** [April 17, 2023, 2:54am UTC](https://community.konduit.ai/t/data-slicing-failed/2506/5 "2023-04-17T02:54:36Z")

</div>

@agibsonccc  
Using dup() before slicing will get the correct result, but it is not a great way to do it.

```auto
INDArray data = Nd4j.rand(17379, 59);
int casualIndex = 56;
int cntIndex = 58;
INDArray features = data.get(NDArrayIndex.all(), NDArrayIndex.interval(0, casualIndex));
INDArray targets = data.get(NDArrayIndex.all(), NDArrayIndex.interval(casualIndex, cntIndex + 1));

int total_row = (int) data.shape()[0];
int val_size = total_row - 21 * 24;
int train_size = val_size - 60 * 24;
INDArray train_features = features.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.all());
INDArray train_targets = targets.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.all());
INDArray val_features = features.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.all());
INDArray val_targets = targets.dup().get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.all());
INDArray test_features = features.dup().get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.all());
INDArray test_targets = targets.dup().get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.all());

System.out.printf("train_features shape:%s\n", Arrays.toString(train_features.shape()));
System.out.printf("train_targets shape:%s\n", Arrays.toString(train_targets.shape()));
System.out.printf("val_features shape:%s\n", Arrays.toString(val_features.shape()));
System.out.printf("val_targets shape:%s\n", Arrays.toString(val_targets.shape()));
System.out.printf("test_features shape:%s\n", Arrays.toString(test_features.shape()));
System.out.printf("test_targets shape:%s\n", Arrays.toString(test_targets.shape()));

```

I expect the effect of wanting to slice as follows:  
train\_features shape:[15435, 56]  
train\_targets shape:[15435, 3]  
val\_features shape:[1440, 56]  
val\_targets shape:[1440, 3]  
test\_features shape:[504, 56]  
test\_targets shape:[504, 3]

---

<div class="post-metadata">

**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\
**Post date:** [April 17, 2023, 3:46am UTC](https://community.konduit.ai/t/data-slicing-failed/2506/6 "2023-04-17T03:46:10Z")

</div>

@luuu almost there. Yeah dup() should not be a requirement. I mainly want to make sure that I have something reproducible. Thanks for working with me here. Let me run your code in a bit and I’ll try to troubleshoot this.

Just to check you’re using M2.1 right?

---

<div class="post-metadata">

**Author:** ![luuu](https://avatars.discourse-cdn.com/v4/letter/l/b4bc9f/32.png) [@luuu](https://community.konduit.ai/u/luuu)\
**Post date:** [April 17, 2023, 3:49am UTC](https://community.konduit.ai/t/data-slicing-failed/2506/7 "2023-04-17T03:49:41Z")

</div>

@agibsonccc Yes, the version I use is 1.0.0-M2.1.

---

<div class="post-metadata">

**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\
**Post date:** [April 17, 2023, 3:56am UTC](https://community.konduit.ai/t/data-slicing-failed/2506/8 "2023-04-17T03:56:15Z")

</div>

@luuu awesome thanks. Let me check if this is already fixed on the latest version first.

---

<div class="post-metadata">

**Author:** ![luuu](https://avatars.discourse-cdn.com/v4/letter/l/b4bc9f/32.png) [@luuu](https://community.konduit.ai/u/luuu)\
**Post date:** [April 17, 2023, 4:22am UTC](https://community.konduit.ai/t/data-slicing-failed/2506/9 "2023-04-17T04:22:27Z")

</div>

@agibsonccc  
I may need to add some information. There are 2 use cases below, one row count is relatively small, and the slices are small number of failures; Another row count is relatively large, and most of the slices fail.

**row count is relatively small:**

```auto
INDArray data = Nd4j.rand(17379 / 3, 59);
int casualIndex = 56;
int cntIndex = 58;
INDArray features = data.get(NDArrayIndex.all(), NDArrayIndex.interval(0, casualIndex));
INDArray targets = data.get(NDArrayIndex.all(), NDArrayIndex.interval(casualIndex, cntIndex + 1));

int total_row = (int) data.shape()[0];
int val_size = total_row - 21 * 24;
int train_size = val_size - 60 * 24;
INDArray train_features = features.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.all());
INDArray train_targets = targets.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.all());
INDArray val_features = features.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.all());
INDArray val_targets = targets.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.all());
INDArray test_features = features.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.all());
INDArray test_targets = targets.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.all());

System.out.printf("train_features shape:%s\n", Arrays.toString(train_features.shape()));
System.out.printf("train_targets shape:%s\n", Arrays.toString(train_targets.shape()));
System.out.printf("val_features shape:%s\n", Arrays.toString(val_features.shape()));
System.out.printf("val_targets shape:%s\n", Arrays.toString(val_targets.shape()));
System.out.printf("test_features shape:%s\n", Arrays.toString(test_features.shape()));
System.out.printf("test_targets shape:%s\n", Arrays.toString(test_targets.shape()));

```

Run the results:  
train\_features shape:[3849, 56]  
train\_targets shape:[3849, 3]  
val\_features shape:[1440, 56]  
val\_targets shape:[]  
test\_features shape:[504, 56]  
test\_targets shape:[]

**row count is relatively large:**

```auto
INDArray data = Nd4j.rand(17379 * 3, 59);
int casualIndex = 56;
int cntIndex = 58;
INDArray features = data.get(NDArrayIndex.all(), NDArrayIndex.interval(0, casualIndex));
INDArray targets = data.get(NDArrayIndex.all(), NDArrayIndex.interval(casualIndex, cntIndex + 1));

int total_row = (int) data.shape()[0];
int val_size = total_row - 21 * 24;
int train_size = val_size - 60 * 24;
INDArray train_features = features.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.all());
INDArray train_targets = targets.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.all());
INDArray val_features = features.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.all());
INDArray val_targets = targets.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.all());
INDArray test_features = features.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.all());
INDArray test_targets = targets.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.all());

System.out.printf("train_features shape:%s\n", Arrays.toString(train_features.shape()));
System.out.printf("train_targets shape:%s\n", Arrays.toString(train_targets.shape()));
System.out.printf("val_features shape:%s\n", Arrays.toString(val_features.shape()));
System.out.printf("val_targets shape:%s\n", Arrays.toString(val_targets.shape()));
System.out.printf("test_features shape:%s\n", Arrays.toString(test_features.shape()));
System.out.printf("test_targets shape:%s\n", Arrays.toString(test_targets.shape()));

```

Run the results:  
train\_features shape:[50193, 56]  
train\_targets shape:[50193, 3]  
val\_features shape:[]  
val\_targets shape:[]  
test\_features shape:[]  
test\_targets shape:[]

If the slice is using raw data, it can succeed despite the increase in the number of rows in the data.

```auto
INDArray data = Nd4j.rand(17379 * 3, 59);
int casualIndex = 56;
int cntIndex = 58;
int total_row = (int) data.shape()[0];
int val_size = total_row - 21 * 24;
int train_size = val_size - 60 * 24;
INDArray train_features = data.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.interval(0, casualIndex));
INDArray train_targets = data.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.interval(casualIndex, cntIndex + 1));
INDArray val_features = data.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.interval(0, casualIndex));
INDArray val_targets = data.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.interval(casualIndex, cntIndex + 1));
INDArray test_features = data.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.interval(0, casualIndex));
INDArray test_targets = data.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.interval(casualIndex, cntIndex + 1));
System.out.printf("train_features shape:%s\n", Arrays.toString(train_features.shape()));
System.out.printf("train_targets shape:%s\n", Arrays.toString(train_targets.shape()));
System.out.printf("val_features shape:%s\n", Arrays.toString(val_features.shape()));
System.out.printf("val_targets shape:%s\n", Arrays.toString(val_targets.shape()));
System.out.printf("test_features shape:%s\n", Arrays.toString(test_features.shape()));
System.out.printf("test_targets shape:%s\n", Arrays.toString(test_targets.shape()));

```

Run the results:  
train\_features shape:[50193, 56]  
train\_targets shape:[50193, 3]  
val\_features shape:[1440, 56]  
val\_targets shape:[1440, 3]  
test\_features shape:[504, 56]  
test\_targets shape:[504, 3]

---

<div class="post-metadata">

**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\
**Post date:** [April 18, 2023, 3:15am UTC](https://community.konduit.ai/t/data-slicing-failed/2506/10 "2023-04-18T03:15:10Z")

</div>

@luuu I took the time to run everything today locally and it appears to work (I checked with numpy indexing). Could you try this and let me know?

> **[Snapshots](https://deeplearning4j.konduit.ai/multi-project/explanation/snapshots)**
>
> Using daily builds for access to latest Eclipse Deeplearning4j features.

---

<div class="post-metadata">

**Author:** ![luuu](https://avatars.discourse-cdn.com/v4/letter/l/b4bc9f/32.png) [@luuu](https://community.konduit.ai/u/luuu)\
**Post date:** [April 19, 2023, 12:14pm UTC](https://community.konduit.ai/t/data-slicing-failed/2506/11 "2023-04-19T12:14:20Z")

</div>

@agibsonccc I tested it using the snapshot version, and when I sliced the data again with the sliced data, there was still a problem, and the sliced out had a dimension that was wrong. The specific test code and test results are as follows

```auto
package org.example;

import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.factory.Nd4j;
import org.nd4j.linalg.indexing.NDArrayIndex;

import java.util.Arrays;

public class ND4JSlice {

    public static void main(String[] args) {
        System.out.println("\nslice from small rows");
        test0();
        System.out.println("\nslice from large rows");
        test1();
        System.out.println("\nslice from raw rows");
        test2();
    }

    private static void test0() {
        test_slice(Nd4j.rand(17379 / 3, 59));
    }

    private static void test1() {
        test_slice(Nd4j.rand(17379 * 3, 59));
    }

    private static void test_slice(INDArray data) {
        int casualIndex = 56;
        int cntIndex = 58;
        INDArray features = data.get(NDArrayIndex.all(), NDArrayIndex.interval(0, casualIndex));
        INDArray targets = data.get(NDArrayIndex.all(), NDArrayIndex.interval(casualIndex, cntIndex + 1));

        int total_row = (int) data.shape()[0];
        int val_size = total_row - 21 * 24;
        int train_size = val_size - 60 * 24;
        INDArray train_features = features.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.all());
        INDArray train_targets = targets.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.all());
        INDArray val_features = features.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.all());
        INDArray val_targets = targets.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.all());
        INDArray test_features = features.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.all());
        INDArray test_targets = targets.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.all());

        System.out.printf("train_features shape:%s\n", Arrays.toString(train_features.shape()));
        System.out.printf("train_targets shape:%s\n", Arrays.toString(train_targets.shape()));
        System.out.printf("val_features shape:%s\n", Arrays.toString(val_features.shape()));
        System.out.printf("val_targets shape:%s\n", Arrays.toString(val_targets.shape()));
        System.out.printf("test_features shape:%s\n", Arrays.toString(test_features.shape()));
        System.out.printf("test_targets shape:%s\n", Arrays.toString(test_targets.shape()));
    }

    private static void test2() {
        INDArray data = Nd4j.rand(17379 * 3, 59);
        int casualIndex = 56;
        int cntIndex = 58;
        int total_row = (int) data.shape()[0];
        int val_size = total_row - 21 * 24;
        int train_size = val_size - 60 * 24;

        INDArray train_features = data.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.interval(0, casualIndex));
        INDArray train_targets = data.get(NDArrayIndex.interval(0, train_size), NDArrayIndex.interval(casualIndex, cntIndex + 1));
        INDArray val_features = data.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.interval(0, casualIndex));
        INDArray val_targets = data.get(NDArrayIndex.interval(train_size, val_size), NDArrayIndex.interval(casualIndex, cntIndex + 1));
        INDArray test_features = data.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.interval(0, casualIndex));
        INDArray test_targets = data.get(NDArrayIndex.interval(val_size, total_row), NDArrayIndex.interval(casualIndex, cntIndex + 1));

        System.out.printf("train_features shape:%s\n", Arrays.toString(train_features.shape()));
        System.out.printf("train_targets shape:%s\n", Arrays.toString(train_targets.shape()));
        System.out.printf("val_features shape:%s\n", Arrays.toString(val_features.shape()));
        System.out.printf("val_targets shape:%s\n", Arrays.toString(val_targets.shape()));
        System.out.printf("test_features shape:%s\n", Arrays.toString(test_features.shape()));
        System.out.printf("test_targets shape:%s\n", Arrays.toString(test_targets.shape()));
    }

}

```

Run the results  
slice from small rows  
train\_features shape:[3849, 56]  
train\_targets shape:[3849, 3]  
val\_features shape:[1440, 56]  
val\_targets shape:[1440, 0]  
test\_features shape:[504, 56]  
test\_targets shape:[504, 0]

slice from large rows  
train\_features shape:[50193, 56]  
train\_targets shape:[50193, 3]  
val\_features shape:[1440, 0]  
val\_targets shape:[1440, 0]  
test\_features shape:[504, 0]  
test\_targets shape:[504, 0]

slice from raw rows  
train\_features shape:[50193, 56]  
train\_targets shape:[50193, 3]  
val\_features shape:[1440, 56]  
val\_targets shape:[1440, 3]  
test\_features shape:[504, 56]  
test\_targets shape:[504, 3]

Process finished with exit code 0

---

<div class="post-metadata">

**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\
**Post date:** [April 19, 2023, 1:44pm UTC](https://community.konduit.ai/t/data-slicing-failed/2506/12 "2023-04-19T13:44:08Z")

</div>

@luuu hmm…let me confirm and maybe update the snapshots. I’ll do that tomorrow. Is a cpu build fine for now?

---

<div class="post-metadata">

**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\
**Post date:** [April 20, 2023, 11:49am UTC](https://community.konduit.ai/t/data-slicing-failed/2506/13 "2023-04-20T11:49:08Z")

</div>

@luuu sorry will publish a build tomorrow. Still in the middle of some other customer related work.