BertInferenceExample, fine tune question

In this example, i wanna fine tune my own bert model, by follow this instruction:


as google says, the eval_accuracy should be expected between 84%-88% like this:
but after my fine tune, my result is:

my accuracy have only 68%, but i complish fine tune strictly followed by the instruction, i wonder where the problem is.
appreciate for your help.

I can get ‘eval_accuracy = 0.8627451’ on linux using cpu to train.

The log:

/model.ckpt-2751Saving ‘checkpoint_path’ summary for global step 2751: /TF_Graphs/mrpc_output/
/model.ckpt-275124854 139784295372608] Saving ‘checkpoint_path’ summary for global step 2751: /TF_Graphs/mrpc_output/
INFO:tensorflow:evaluation_loop marked as finished
I0720 10:59:38.225406 139784295372608] evaluation_loop marked as finished
INFO:tensorflow:***** Eval results *****
I0720 10:59:38.225584 139784295372608] ***** Eval results *****
INFO:tensorflow: eval_accuracy = 0.8627451
I0720 10:59:38.225656 139784295372608] eval_accuracy = 0.8627451
INFO:tensorflow: eval_loss = 0.7270211
I0720 10:59:38.225800 139784295372608] eval_loss = 0.7270211
INFO:tensorflow: global_step = 2751
I0720 10:59:38.225892 139784295372608] global_step = 2751
INFO:tensorflow: loss = 0.7270211
I0720 10:59:38.225950 139784295372608] loss = 0.7270211

I’ll try linux.maybe is caused by system. because i fine tuned bert on windows, with tensorflow-gpu.

I tried to freeze using command like ‘python3 --input_dir=/chinese_L-12_H-768_A-12 --ckpt=bert_model.ckpt’ and got error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input node loss/Softmax not found in graph
Should i audit


我在Linux下重新fine tune了一下,发现用CPU进行fine tune是没有问题的,得到了 eval_accuracy = 0.8480392。

So it’s fine on cpu.
I care how to load chinese_L-12_H-768_A-12 in order to transfer-train using custom data.

About custom data, maybe you need have your own data processor.

Could you compare results on cpu and gpu? Are you saying the numbers are different? Make sure to make it reproducible (setting a seed, same parameters,…) to see if we have a reproducible issue here.

you means my fine tune progress? i had compared results both on CPU & GPU and Windows & Ubuntu, i’d like to share my experiment

I fine tuned bert with MRPC task.experiment is present below.

tensorflow 1.11.0 # use this for training on CPU
tensorflow-gpu 1.11.0 # use this for traning on GPU

all these parameter is presented at

win10 + tensorflow 1.11.0 + CPU. eval_accuracy = 0.68
win10 + tensorflow-gpu 1.11.0 + GPU. eval_accuracy = 0.68
ubuntu + tensorflow 1.11.0 + CPU. eval_accuracy =0.84
ubuntu + tensorflow-gpu 1.11.0 + GPU. eval_accuracy = 0.68

obviously, only on ubuntu with CPU, the result is correct. so, we could only fine tuned bert on CPU in ubuntu, but it cost 2-3 hours per training. it only cost a quarter per training on GPU, but the result is wrong.
that’s is all my experiment.

oh it’s ridiculous, on my virtual machine with ubuntu using cpu, eval_accuracy = 0.84, then i install ubuntu to my computer(dual system), using cpu, eval_accuracy = 0.68, I’m confuse :joy:

solved. I trans the parameter in a wrong way, i see this method at a blog. with google instruction, trans the parameter at command line, all is ok.