Train only the recognition branch

[Leonard-GEHIN]

I got an FOTS checkpoint, trained from scratch, which has pretty good results on my database.
I want to finetune the recognition branch of this checkpoint. The finetuning run smoothly but results are a bit weird:

Pretrained results:

Finetuned results:

As you can see, the detection before and after the recognition branch finetuning are different.
I did not see anyone talk about this problem on issues in this project. Do the finetuning run smoothly for you?

[Pay20Y]

Hi, because the recognition branch and detection branch share the backbone, so it's not suitable to only finetune the recognition branch.

[Leonard-GEHIN]

Excuse me for this question, but I thought that the backbone of a neural network was the detection part. Does the backbone, in this case, refer to the shared part of the network before the RoI?
Does a train only on the detection branch is feasible? I don't really know how to test it.

[Pay20Y]

The backbone indicates the FPN structure. I think optimizing the detection branch and the recognition branch simultaneously is suitable when finetuning. However when pretraining the model, it is feasible to optimize one of the branch at first.

[Leonard-GEHIN]

I think I understand why I cannot train the recognition part only. Thank you for your answers and all of your work.

[Leonard-GEHIN]

Do you think it is possible to only train the little "text recognition branch" in green at the end of the graph? This could be exactly what I am looking for.

[Pay20Y]

Sorry, I don't think I understand you. You mean you want only a recognition model not an end-to-end text spotting system? If it is the CRNN model is a choice.

[Leonard-GEHIN]

My bad, I did not explain it properly. In want an end to end solution.
You said that, when you train the recognition part of FOTS, the shared backbone is also trained. So it deteriorate the detection branch.
So I am wondering if, after an end to end training, it is possible to only train the little "text recognition branch" at the end of the graph without modifying the backbone?
If we can do that, a finetuning of the recognition branch would be possible.

Here's why I want to do that:
As the end to end training train the recognition with ground truth bounding boxes instead of detected bounding boxes, the recognition is not fit for the detection branch results during inference. Detected bounding boxes are not perfectly bounding texts in my project. So I think that train the recognition branch with a data augmentation to simulate the erroneous bounding boxes like the detected bounding boxes would be beneficial to FOTS.
I did not find a way to give different ground truth bounding boxes to the detection branch and the recognition branch during an end to end training. But that could solve my problem if you have an idea.

[Pay20Y]

Hi, you can finetune the recognition branch with the tf.stop_gradient(), then the loss of recognition will not affect the backbone. For the bounding boxes, you can shrink or expand some ground-truth boxes for roi_rotate.