How generate new sentences based on a set of existing sentences using deeplearning4j?
Where can i see some examples?
@bstrdn don’t expect GPT: deeplearning4j-examples/GenerateTxtCharCompGraphModel.java at 96e51a98dcfb831bdc89e2a44a229554164f19c0 · deeplearning4j/deeplearning4j-examples · GitHub
We’re working on some transformers improvements + examples.
We have to setup the tokenizers and the like and do some more testing but we also have model import + examples in the works for gpt various architectures: omnihub-zoo/samediff at main · KonduitAI/omnihub-zoo · GitHub
1 Like
Hi,
@bstrdn i will try to create a text generator in the next weeks, i have first success with a transformer model. i think if i extend the architecutre with a reccurent attention layer and a new iterator it could also learn like gpt. but like @agibsonccc said don’t expect gpt quality. i have now a 12 % language understanding after 2 days of gpu on my 3 layer transformer with 256 embedding. so it needs a lot of data and training to get good results.
if you would like to work together for an text generator feel free to contact.
Best regards
Thomas
Hi Thomas!
Thank you for the answer! I’m a beginner in this topic, so I can’t help yet. Actually, I was just wondering if there is any simple functionality where I can quickly generate a model by inputting text.
@thomas great to hear! We’ll have an improved transformer for the next release.
I’m in the process of rewriting those with more knobs. I also want to add better kernels for gpus. NVIDIA recently open sourced some very nice transformer kernels I’ll probably integrate.
I also plan on spending some time on some transformers demos. I’d love to learn more about your results and what you found hard. Thanks!
1 Like
@TomHillman1 we’ll be importing gpt and other kinds of transformer models. After we have those weights it’ll be much easier to build demos for that. First I want to do a pass on making the transformers (attention) code more performant.