# BertInferenceExample, fine tune question **URL:** \ **Category:** DL4J\ **Created:** [July 19, 2020, 3:06am UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733 "2020-07-19T03:06:04Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Maple](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/maple/32/410_2.png) [@Maple](https://community.konduit.ai/u/Maple)\ **Post date:** [July 19, 2020, 3:06am UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/1 "2020-07-19T03:06:04Z")
In this example, i wanna fine tune my own bert model, by follow this instruction: > Blockquote > [https://github.com/KonduitAI/dl4j-dev-tools/tree/master/import-tests/model\_zoo/bert](https://github.com/KonduitAI/dl4j-dev-tools/tree/master/import-tests/model_zoo/bert) as google says, the eval\_accuracy should be expected between 84%-88% like this: ![image](https://canada1.discourse-cdn.com/flex035/uploads/konduit/original/1X/52f73171d77033e20c13eb2fc1974ba30a6ff6c8.png) but after my fine tune, my result is: ![image](https://canada1.discourse-cdn.com/flex035/uploads/konduit/original/1X/f77c373bfa0eb9fca2f75517a56930ce60865a96.png) my accuracy have only 68%, but i complish fine tune strictly followed by the instruction, i wonder where the problem is. appreciate for your help. ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [July 20, 2020, 11:13am UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/2 "2020-07-20T11:13:31Z")
I can get ‘eval\_accuracy = 0.8627451’ on linux using cpu to train. The log: > /model.ckpt-2751Saving ‘checkpoint\_path’ summary for global step 2751: /TF\_Graphs/mrpc\_output/ > /model.ckpt-275124854 139784295372608 estimator.py:2109] Saving ‘checkpoint\_path’ summary for global step 2751: /TF\_Graphs/mrpc\_output/ > INFO:tensorflow:evaluation\_loop marked as finished > I0720 10:59:38.225406 139784295372608 error\_handling.py:101] evaluation\_loop marked as finished > INFO:tensorflow:\*\*\*\*\* Eval results \*\*\*\*\* > I0720 10:59:38.225584 139784295372608 run\_classifier.py:923] \*\*\*\*\* Eval results \*\*\*\*\* > INFO:tensorflow: eval\_accuracy = 0.8627451 > I0720 10:59:38.225656 139784295372608 run\_classifier.py:925] eval\_accuracy = 0.8627451 > INFO:tensorflow: eval\_loss = 0.7270211 > I0720 10:59:38.225800 139784295372608 run\_classifier.py:925] eval\_loss = 0.7270211 > INFO:tensorflow: global\_step = 2751 > I0720 10:59:38.225892 139784295372608 run\_classifier.py:925] global\_step = 2751 > INFO:tensorflow: loss = 0.7270211 > I0720 10:59:38.225950 139784295372608 run\_classifier.py:925] loss = 0.7270211 ---
**Author:** ![Maple](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/maple/32/410_2.png) [@Maple](https://community.konduit.ai/u/Maple)\ **Post date:** [July 21, 2020, 1:34am UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/4 "2020-07-21T01:34:36Z")
I’ll try linux.maybe is caused by system. because i fine tuned bert on windows, with tensorflow-gpu. ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [July 23, 2020, 6:38am UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/5 "2020-07-23T06:38:49Z")
I tried to freeze chinese\_L-12\_H-768\_A-12.zip using command like ‘python3 freezeTrainedBert.py --input\_dir=/chinese\_L-12\_H-768\_A-12 --ckpt=bert\_model.ckpt’ and got error: tensorflow.python.framework.errors\_impl.InvalidArgumentError: Input node loss/Softmax not found in graph Should i audit freezeTrainedBert.py? ---
**Author:** ![Maple](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/maple/32/410_2.png) [@Maple](https://community.konduit.ai/u/Maple)\ **Post date:** [July 23, 2020, 7:14am UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/6 "2020-07-23T07:14:53Z")
我还没有做到冻结图这一步，但根据这个网页教程 > **[dl4j-dev-tools/import-tests/model\_zoo/bert at master · KonduitAI/dl4j-dev-tools](https://github.com/KonduitAI/dl4j-dev-tools/tree/master/import-tests/model_zoo/bert)** > > master/import-tests/model\_zoo/bert 的步骤来，应该是没有问题的。 我在Linux下重新fine tune了一下，发现用CPU进行fine tune是没有问题的，得到了 eval\_accuracy = 0.8480392。 ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [July 23, 2020, 8:51am UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/7 "2020-07-23T08:51:26Z")
So it’s fine on cpu. I care how to load chinese\_L-12\_H-768\_A-12 in order to transfer-train using custom data. ---
**Author:** ![Maple](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/maple/32/410_2.png) [@Maple](https://community.konduit.ai/u/Maple)\ **Post date:** [July 23, 2020, 9:48am UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/8 "2020-07-23T09:48:46Z")
About custom data, maybe you need have your own data processor. ---
**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\ **Post date:** [July 24, 2020, 2:09pm UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/9 "2020-07-24T14:09:32Z")
Could you compare results on cpu and gpu? Are you saying the numbers are different? Make sure to make it reproducible (setting a seed, same parameters,…) to see if we have a reproducible issue here. ---
**Author:** ![Maple](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/maple/32/410_2.png) [@Maple](https://community.konduit.ai/u/Maple)\ **Post date:** [July 24, 2020, 2:54pm UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/10 "2020-07-24T14:54:04Z")
you means my fine tune progress? i had compared results both on CPU & GPU and Windows & Ubuntu, i’d like to share my experiment ---
**Author:** ![Maple](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/maple/32/410_2.png) [@Maple](https://community.konduit.ai/u/Maple)\ **Post date:** [July 24, 2020, 3:25pm UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/11 "2020-07-24T15:25:54Z")
I fine tuned bert with MRPC task.experiment is present below. > Environment > python3.6 > tensorflow 1.11.0 # use this for training on CPU > tensorflow-gpu 1.11.0 # use this for traning on GPU > Parameter: > max\_seq\_length=128 > train\_batch\_size=4 > learning\_rate=2e-5 > num\_train\_epochs=3.0 > all these parameter is presented at > [https://github.com/KonduitAI/dl4j-dev-tools/tree/master/import-tests/model\_zoo/bert](https://github.com/KonduitAI/dl4j-dev-tools/tree/master/import-tests/model_zoo/bert) > Experiment > win10 + tensorflow 1.11.0 + CPU. eval\_accuracy = 0.68 > win10 + tensorflow-gpu 1.11.0 + GPU. eval\_accuracy = 0.68 > ubuntu + tensorflow 1.11.0 + CPU. eval\_accuracy = **0.84** > ubuntu + tensorflow-gpu 1.11.0 + GPU. eval\_accuracy = 0.68 obviously, only on ubuntu with CPU, the result is correct. so, we could only fine tuned bert on CPU in ubuntu, but it cost 2-3 hours per training. it only cost a quarter per training on GPU, but the result is wrong. that’s is all my experiment. ---
**Author:** ![Maple](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/maple/32/410_2.png) [@Maple](https://community.konduit.ai/u/Maple)\ **Post date:** [July 25, 2020, 2:46am UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/12 "2020-07-25T02:46:15Z")
oh it’s ridiculous, on my virtual machine with ubuntu using cpu, eval\_accuracy = 0.84, then i install ubuntu to my computer(dual system), using cpu, eval\_accuracy = 0.68, I’m confuse 😂 ---
**Author:** ![Maple](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/maple/32/410_2.png) [@Maple](https://community.konduit.ai/u/Maple)\ **Post date:** [July 26, 2020, 8:56am UTC](https://community.konduit.ai/t/bertinferenceexample-fine-tune-question/733/13 "2020-07-26T08:56:37Z")
solved. I trans the parameter in a wrong way, i see this method at a blog. with google instruction, trans the parameter at command line, all is ok.