INDArray Shannon Entropy returns negative values

strolling-coder · March 31, 2020, 5:55pm

Hi all

I have a question about the way entropy is calculated for the NDArrays. I have some negative values for Shannon Entropy and a lot of NaNs for log entropy. Is it expected?

I created a gist with an example of a matrix that will return a negative entropy value. The same matrix in Octave returns a positive value.

gist.github.com

https://gist.github.com/strolling-coder/ae0469b67164627f79d22a5fdc04a0da

INDArrayNegativeEntropyExample

package wld.nd4j.tests;

import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.factory.Nd4j;

public class TestINDArraryEntropy {

	public static void main(String[] args) {
		TestINDArraryEntropy test = new TestINDArraryEntropy();
		test.execute();

This file has been truncated. show original

raver119 · March 31, 2020, 6:04pm

File an issue please, might be a bug

strolling-coder · March 31, 2020, 6:23pm

Done. Please have a look at issue #8818.

treo · April 1, 2020, 6:59am

A post was split to a new topic: Compressing CNNs by removing filters with low entropy

treo · April 1, 2020, 6:58am

For completeness, from the issue:

Thanks for the report, but I don’t believe your data is valid for this operation.

Shannon entropy is defined as:
- sum_i( p_i log2(p_i) )
where p_i is the probability of a given symbol/random variable.
See for example: http://bearcave.com/misl/misl_tech/wavelets/compression/shannon.html

Given that, we are restricted to:

0 <= p_i <= 1 for all i, and

sum_i(p_i) = 1

In the case of your test data, neither condition holds - most values are greater than 1.0, and the sum doesn’t equal 1.0.