Tips on improving inference while ensuring accuracy

[DJay921]

Hi, I'm deploying a small project and wanted some general tips on how can I reduce the time for inference but ensuring the outputs are accurate. Currently I'm using a CPU container instance for deploying my api. I have converted my images to grayscale before passing to the model and currently using the default parameters when initializing. Some of the images are taking quite some time, hence looking for general tips on how can I improve this (for example should I increase the rec_batch_num from the default which is 6?) Please note that I'm using the Rapidocr onnxruntime library for this. Thanks!

[SWHL]

First, it is not recommended to change the rec_batch_num parameter. If it is too large, it will affect the accuracy.

Second, it is recommended to resize the image to a smaller size before passing it into the model.

Third, a large part of the time it takes to infer depends on the amount of text in the image. If the text is dense, it will naturally take longer. You can try rapidocr_openvino.

[DJay921]

Thanks @SWHL, I tried using rapidocr_openvino the response seems to be faster however the issue is its throwing output in random Chinese texts for English documents like 刈陂蜿拳地炎鲶与问蜿陂专地鄙 for the word Travel Charges, I have not changed anything about my code except for using RapidOCR from rapidocr_openvino instead of rapidocr_onnxruntime

[DJay921]

Hi @SWHL ,attaching the image. Although for this image differences are not significant with the online demo results, but I highlighted the ones I found.

[SWHL]

You can try the en_PP-OCRv3_rec_infer.onnx model to rec, which will be better.

[DJay921]

Hi @SWHL, I seem to be facing an issue with the en_PP-OCRv3_rec_infer.onnx model when I use it with rapidocr_openvino, the english document texts are being recogized as chinese somehow, not sure how that is happening. With rapidocr_onnxruntime this works well though. Also note that the images I'm processing are .webp format.

[DJay921]

Also I plan to deploy this using google cloud run, so I guess rapidocr_openvino may not be that performant when api is deployed right, because it would need specifically need Intel cpu only?

[SWHL]

Can you provide a minimum reproducible demo?
I just see the error, and I can't guess what happened.