Bert Model in DL4j - for text similarity

I’m trying to find BERT model import and how to get handle to the INDArray of vector embeddings for a given text based one BERT.

When I search google, I get following code

BertModelPretrainedResources resource = BertModelResources.BASE_UNCASED;
ComputationGraph bert = BertModelLoader.loadPretrainedModel(resource);

and these classes just do no exist in DL4j libraries at all

I have code for Word2Vec , Just need a similar code for BERT where I load the model and pass text to get INDArray out.

Any help ?

@rajeshetty could you clarify where you even found those search results? A 3rd party library maybe? We have the building blocks for BERT and some examples but none of what you’re seeing there.

Did you use chatgpt or something? The only BERT we even support is using the lower level samediff api not the older dl4j api.

@agibsonccc - yes thats what ChatGPT gave me, when I asked that question. I almost think, it has took some of the python APIs , turned snake case to camel case convention on class , variable names and came back with the recommendation. Just wanted to make sure Im not missing something.

So we do not have any java code or implementation within DL4J

  • Load BERT (any model , cased/uncased , base/large etc) from locally downloaded artifact or remote
  • Pass given text/sentence/words to get the embeddings (INDArray) from it

Can you confirm ?

@rajeshetty the normal way to do it is import an existing BERT Model and then finetune that one. We have support for tokenization and the like.

Please don’t misinterpret what I said though. “Not old dl4j api with computation graphs that won’t receive updates” isn’t equal to “newer samediff api we insist on using for more complex architectures”

If you want examples with that newer api I’m happy to show that but if you want to use the old api I won’t be able to help you much there.

What chatgpt gave you was by definition fake. If you don’t see something in the examples or the tests just assume chatgpt is wrong.

@agibsonccc - Please share the newer API examples and I can take a look.

@rajeshetty apologies holidays here. You’ll want to look at our BERT tests: deeplearning4j/platform-tests/src/test/java/org/deeplearning4j/text/tokenization/tokenizer/ at f9c1faaaf968d6f0e5a5add2627908f7a2565f96 · deeplearning4j/deeplearning4j · GitHub