INDArray Shannon Entropy returns negative values

1

Hi all

I have a question about the way entropy is calculated for the NDArrays. I have some negative values for Shannon Entropy and a lot of NaNs for log entropy. Is it expected?

I created a gist with an example of a matrix that will return a negative entropy value. The same matrix in Octave returns a positive value.

2

File an issue please, might be a bug

3

Done. Please have a look at issue #8818.

4

A post was split to a new topic: Compressing CNNs by removing filters with low entropy

5

For completeness, from the issue:

Thanks for the report, but I don’t believe your data is valid for this operation.

Shannon entropy is defined as:

- sum_i( p_i log2(p_i) )

where p_i is the probability of a given symbol/random variable.
See for example: Shannon Entropy

Given that, we are restricted to:

  • 0 <= p_i <= 1 for all i, and
  • sum_i(p_i) = 1

In the case of your test data, neither condition holds - most values are greater than 1.0, and the sum doesn’t equal 1.0.