Bert Model in DL4j - for text similarity

rajeshetty · December 13, 2023, 8:04pm

I’m trying to find BERT model import and how to get handle to the INDArray of vector embeddings for a given text based one BERT.

When I search google, I get following code

BertModelPretrainedResources resource = BertModelResources.BASE_UNCASED;
ComputationGraph bert = BertModelLoader.loadPretrainedModel(resource);

and these classes just do no exist in DL4j libraries at all

I have code for Word2Vec , Just need a similar code for BERT where I load the model and pass text to get INDArray out.

Any help ?

agibsonccc · December 14, 2023, 4:21am

@rajeshetty could you clarify where you even found those search results? A 3rd party library maybe? We have the building blocks for BERT and some examples but none of what you’re seeing there.

Did you use chatgpt or something? The only BERT we even support is using the lower level samediff api not the older dl4j api.

rajeshetty · December 14, 2023, 5:42pm

@agibsonccc - yes thats what ChatGPT gave me, when I asked that question. I almost think, it has took some of the python APIs , turned snake case to camel case convention on class , variable names and came back with the recommendation. Just wanted to make sure Im not missing something.

So we do not have any java code or implementation within DL4J

Load BERT (any model , cased/uncased , base/large etc) from locally downloaded artifact or remote
Pass given text/sentence/words to get the embeddings (INDArray) from it

Can you confirm ?

agibsonccc · December 15, 2023, 11:40am

@rajeshetty the normal way to do it is import an existing BERT Model and then finetune that one. We have support for tokenization and the like.

Please don’t misinterpret what I said though. “Not old dl4j api with computation graphs that won’t receive updates” isn’t equal to “newer samediff api we insist on using for more complex architectures”

If you want examples with that newer api I’m happy to show that but if you want to use the old api I won’t be able to help you much there.

What chatgpt gave you was by definition fake. If you don’t see something in the examples or the tests just assume chatgpt is wrong.

rajeshetty · December 15, 2023, 5:17pm

@agibsonccc - Please share the newer API examples and I can take a look.

agibsonccc · December 24, 2023, 11:20am

@rajeshetty apologies holidays here. You’ll want to look at our BERT tests: deeplearning4j/platform-tests/src/test/java/org/deeplearning4j/text/tokenization/tokenizer/BertWordPieceTokenizerTests.java at f9c1faaaf968d6f0e5a5add2627908f7a2565f96 · deeplearning4j/deeplearning4j · GitHub

github.com

deeplearning4j/deeplearning4j/blob/f9c1faaaf968d6f0e5a5add2627908f7a2565f96/platform-tests/src/test/java/org/eclipse/deeplearning4j/frameworkimport/tensorflow/TestBERTGraph.java#L63


      
          import java.io.File;
          import java.net.URL;
          import java.util.*;
          
          import static org.junit.jupiter.api.Assertions.assertEquals;
          import static org.junit.jupiter.api.Assertions.assertTrue;
          
          @Slf4j
          @Tag(TagNames.LONG_TEST)
          @Tag(TagNames.LARGE_RESOURCES)
          public class TestBERTGraph extends BaseNd4jTestWithBackends {
          
          
              @Override
              public char ordering(){
                  return 'c';
              }
          
              @ParameterizedTest
              @MethodSource("org.nd4j.linalg.BaseNd4jTestWithBackends#configs")
              @Disabled("Tests old functionality. Needs to be updated.")

Topic		Replies	Views
Fine-tuning Bert in DL4J DL4J	15	4351	March 11, 2024
BERT model to Deeplearning4j DL4J	1	635	March 13, 2020
Chatbot with DL4J DL4J	2	316	August 21, 2022
Using NLP with DL4J DL4J	4	195	February 11, 2023
Transformer/Attention-NLP	4	3188	November 22, 2020

Bert Model in DL4j - for text similarity

Related Topics