# Binarize autoencoder neck layer for semantic hashing

**URL:** <https://community.konduit.ai/t/binarize-autoencoder-neck-layer-for-semantic-hashing/876>\
**Category:** DL4J\
**Created:** [September 22, 2020, 3:43pm UTC](https://community.konduit.ai/t/binarize-autoencoder-neck-layer-for-semantic-hashing/876 "2020-09-22T15:43:51Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mstrap](https://avatars.discourse-cdn.com/v4/letter/m/7ba0ec/32.png) [@mstrap](https://community.konduit.ai/u/mstrap)\
**Post date:** [September 22, 2020, 3:43pm UTC](https://community.konduit.ai/t/binarize-autoencoder-neck-layer-for-semantic-hashing/876/1 "2020-09-22T15:43:51Z")

</div>

Is it possible to binarize the code layer of a plain autoencoder (`DenseLayer`) or the out-layer of a `VariationalAutoencoder`? My naive approach would be trying to regularize the activation [1], however I can’t see an obvious way how to do that with DL4J. Am I missing something?

If activation regularization is not possible, are there any other suggestions on how I could extract a binary code from the autoencoder?

Thanks for your ideas!

[1] [tensorflow - How to force a bottleneck in an autoencoder to produce binary values? - Stack Overflow](https://stackoverflow.com/a/60774980)

---

<div class="post-metadata">

**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\
**Post date:** [September 25, 2020, 12:32pm UTC](https://community.konduit.ai/t/binarize-autoencoder-neck-layer-for-semantic-hashing/876/2 "2020-09-25T12:32:23Z")

</div>

I would use the transferlearning api for this. [https://deeplearning4j.konduit.ai/tuning-and-training/transfer-learning](https://deeplearning4j.konduit.ai/tuning-and-training/transfer-learning).

You can remove whatever layers you want and create neural networks from that.  
For semantic hashing, I would look at using normal autoencoders trained end to end where you do:  
autoencoder, autoencoder, (repeat for how ever many encoding layers you need typically 2 to 3)  
then do the reverse architecture for the encoder.

“Binarizing” just comes down to a sigmoid threshold you set for 0/1. Sometimes semantic hashing just uses the floats with a nearest neighbors look up with the weights as an index of sorts.

---

<div class="post-metadata">

**Author:** ![mstrap](https://avatars.discourse-cdn.com/v4/letter/m/7ba0ec/32.png) [@mstrap](https://community.konduit.ai/u/mstrap)\
**Post date:** [September 26, 2020, 5:46pm UTC](https://community.konduit.ai/t/binarize-autoencoder-neck-layer-for-semantic-hashing/876/3 "2020-09-26T17:46:27Z")

</div>

Thanks for this idea! I got a very very rough idea how this approach might work. On the other hand, I failed with my experiments with `org.deeplearning4j.nn.conf.layers.AutoEncoder`. Also I don’t get where the transfer learning step would occur (I’m quite new to deep learning and to DL4J): given my first trained autoencoder-network (e.g. for the `MNISTAutoencoder` starting with 784 → 784), how could parts of this network be transferred to the next network iteration (e.g. 784 → 250 → 784)? Or did I completely misunderstand you? 😕

Either way, I have done some more experiments with a custom `BaseActivationFunction` which activates only to `-1` or `1` and uses `TanhDerivative` for the backprop-step. Results look quite promising [1]. So I’d consider my problem as solved.

[1] [MNISTSemanticHashing example · eclipse/deeplearning4j-examples@a9efd38 · GitHub](https://github.com/eclipse/deeplearning4j-examples/commit/a9efd3877cbb62a0a1b0cde44f10eaec24ff29e8)

---

<div class="post-metadata">

**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\
**Post date:** [September 29, 2020, 12:47pm UTC](https://community.konduit.ai/t/binarize-autoencoder-neck-layer-for-semantic-hashing/876/4 "2020-09-29T12:47:23Z")

</div>

> [@mstrap](#):
>
> ned autoencoder-network (e.g. for the `MNISTAutoencoder` starting with 784 → 784), how could parts of this network be transferred to the next network iteration (e.g. 784 → 250 → 784)? Or did I complet

Hey if it works for you great! It sounds like you’re on the right track.