Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

paddleocr 2.9.1 rec_svtrv2_ch.yml 这个配置文件需要什么系统什么版本的paddle才能训练?需要多大显存?目前训练ResNet34模型没问题,一改这个配置文件就爆显存,或者自动退出训练 #23

Open
nissansz opened this issue Oct 26, 2024 · 26 comments

Comments

@nissansz
Copy link

paddleocr 2.9.1 rec_svtrv2_ch.yml 这个配置文件需要什么系统什么版本的paddle才能训练?需要多大显存?目前训练ResNet34模型没问题,一改这个配置文件就爆显存,或者自动退出训练

有没有办法将部分参数改成ResNet34,然后实现rec_svtrv2_ch架构?

@Topdu
Copy link
Owner

Topdu commented Oct 29, 2024

If out of memory, you can lower first_bs: &bs 192 --> first_bs: &bs 96.

@nissansz
Copy link
Author

Thank you for your reply. Is it possible to train with paddlepaddle 2.4.2, 8G card?

@Topdu
Copy link
Owner

Topdu commented Oct 29, 2024

You can try a smaller batchsize (e.g. first_bs: &bs 48) until it will run.

@nissansz
Copy link
Author

I tried batch size ==1, still cannot train.

@Topdu
Copy link
Owner

Topdu commented Oct 29, 2024

When bs=192, it can run on the 24GB 3090 GPU. Have other settings been changed ?

@nissansz
Copy link
Author

I did not change anything.
Is it possible to adapt parameters based on ResNet34 to use rec_svtrv2_ch.yml? Can you help provide a compatible yml for 8G 1070ti card?

24GB for desktop ?

@nissansz
Copy link
Author

Below setting works, thank you.
first_bs: &bs 48

Is below log normal?

[2024/10/30 09:53:11] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 2000 iterations
[2024/10/30 09:55:20] ppocr INFO: epoch: [1/2], global_step: 10, lr: 0.000000, CTCLoss: 414.616028, NRTRLoss: 11.588322, loss: 426.184021, avg_reader_cost: 0.35949 s, avg_batch_cost: 10.54850 s, avg_samples: 39.2, ips: 3.71617 samples/s, eta: 44 days, 18:30:39, max_mem_reserved: 11567 MB, max_mem_allocated: 9618 MB
[2024/10/30 09:57:46] ppocr INFO: epoch: [1/2], global_step: 20, lr: 0.000000, CTCLoss: 414.420227, NRTRLoss: 11.577787, loss: 425.998169, avg_reader_cost: 0.00190 s, avg_batch_cost: 10.79152 s, avg_samples: 32.8, ips: 3.03942 samples/s, eta: 45 days, 6:51:33, max_mem_reserved: 12032 MB, max_mem_allocated: 9618 MB

@nissansz
Copy link
Author

nissansz commented Oct 30, 2024

The training speed is much slower than ResNet34?

[2024/10/30 10:36:46] ppocr INFO: train dataloader has 1121727 iters
[2024/10/30 10:36:46] ppocr INFO: valid dataloader has 632 iters
[2024/10/30 10:36:46] ppocr INFO: load pretrain successful from C:/Users/Administrator/Desktop/svtr2/latest
[2024/10/30 10:36:46] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 2000 iterations
[2024/10/30 10:37:31] ppocr INFO: epoch: [1/2], global_step: 100, lr: 0.000000, CTCLoss: 406.113434, NRTRLoss: 10.010298, loss: 416.287567, avg_reader_cost: 0.10702 s, avg_batch_cost: 0.37331 s, avg_samples: 5.7, ips: 15.26888 samples/s, eta: 9 days, 16:37:42, max_mem_reserved: 3462 MB, max_mem_allocated: 2746 MB
[2024/10/30 10:38:05] ppocr INFO: epoch: [1/2], global_step: 200, lr: 0.000000, CTCLoss: 407.037354, NRTRLoss: 10.014441, loss: 416.569031, avg_reader_cost: 0.01074 s, avg_batch_cost: 0.26012 s, avg_samples: 5.75, ips: 22.10542 samples/s, eta: 8 days, 5:21:07, max_mem_reserved: 3462 MB, max_mem_allocated: 2746 MB
[2024/10/30 10:38:39] ppocr INFO: epoch: [1/2], global_step: 300, lr: 0.000000, CTCLoss: 406.079590, NRTRLoss: 9.911222, loss: 416.215027, avg_reader_cost: 0.00564 s, avg_batch_cost: 0.26321 s, avg_samples: 5.79, ips: 21.99742 samples/s, eta: 7 days, 18:13:52, max_mem_reserved: 3462 MB, max_mem_allocated: 2746 MB
[2024/10/30 10:39:14] ppocr INFO: epoch: [1/2], global_step: 400, lr: 0.000000, CTCLoss: 405.992401, NRTRLoss: 10.007577, loss: 416.129333, avg_reader_cost: 0.00200 s, avg_batch_cost: 0.26616 s, avg_samples: 5.69, ips: 21.37830 samples/s, eta: 7 days, 13:07:33, max_mem_reserved: 3462 MB, max_mem_allocated: 2746 MB
[2024/10/30 10:39:47] ppocr INFO: epoch: [1/2], global_step: 500, lr: 0.000000, CTCLoss: 406.778412, NRTRLoss: 9.953031, loss: 416.689270, avg_reader_cost: 0.00274 s, avg_batch_cost: 0.25062 s, avg_samples: 5.49, ips: 21.90525 samples/s, eta: 7 days, 8:07:27, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:40:22] ppocr INFO: epoch: [1/2], global_step: 600, lr: 0.000000, CTCLoss: 406.188599, NRTRLoss: 10.091347, loss: 416.248657, avg_reader_cost: 0.00198 s, avg_batch_cost: 0.26253 s, avg_samples: 5.63, ips: 21.44498 samples/s, eta: 7 days, 6:01:26, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:40:57] ppocr INFO: epoch: [1/2], global_step: 700, lr: 0.000000, CTCLoss: 406.254181, NRTRLoss: 9.966766, loss: 416.096924, avg_reader_cost: 0.00615 s, avg_batch_cost: 0.26640 s, avg_samples: 5.95, ips: 22.33496 samples/s, eta: 7 days, 4:51:56, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:41:30] ppocr INFO: epoch: [1/2], global_step: 800, lr: 0.000000, CTCLoss: 405.894531, NRTRLoss: 10.127752, loss: 415.629150, avg_reader_cost: 0.00199 s, avg_batch_cost: 0.25415 s, avg_samples: 5.91, ips: 23.25353 samples/s, eta: 7 days, 3:02:30, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:42:04] ppocr INFO: epoch: [1/2], global_step: 900, lr: 0.000000, CTCLoss: 406.235138, NRTRLoss: 9.916145, loss: 416.167847, avg_reader_cost: 0.00383 s, avg_batch_cost: 0.26179 s, avg_samples: 5.76, ips: 22.00234 samples/s, eta: 7 days, 2:09:00, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:42:39] ppocr INFO: epoch: [1/2], global_step: 1000, lr: 0.000000, CTCLoss: 406.061188, NRTRLoss: 9.864820, loss: 416.161682, avg_reader_cost: 0.00817 s, avg_batch_cost: 0.27304 s, avg_samples: 5.65, ips: 20.69295 samples/s, eta: 7 days, 2:08:09, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:43:13] ppocr INFO: epoch: [1/2], global_step: 1100, lr: 0.000000, CTCLoss: 406.038696, NRTRLoss: 9.906737, loss: 416.162659, avg_reader_cost: 0.00187 s, avg_batch_cost: 0.25589 s, avg_samples: 5.66, ips: 22.11898 samples/s, eta: 7 days, 1:09:06, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:43:48] ppocr INFO: epoch: [1/2], global_step: 1200, lr: 0.000000, CTCLoss: 405.858582, NRTRLoss: 9.972018, loss: 415.892365, avg_reader_cost: 0.00209 s, avg_batch_cost: 0.26372 s, avg_samples: 5.91, ips: 22.40980 samples/s, eta: 7 days, 0:44:14, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:44:22] ppocr INFO: epoch: [1/2], global_step: 1300, lr: 0.000000, CTCLoss: 405.041626, NRTRLoss: 9.983361, loss: 414.910034, avg_reader_cost: 0.00366 s, avg_batch_cost: 0.25955 s, avg_samples: 5.66, ips: 21.80673 samples/s, eta: 7 days, 0:11:08, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:44:56] ppocr INFO: epoch: [1/2], global_step: 1400, lr: 0.000000, CTCLoss: 404.983032, NRTRLoss: 9.942977, loss: 415.109985, avg_reader_cost: 0.00453 s, avg_batch_cost: 0.25915 s, avg_samples: 5.87, ips: 22.65101 samples/s, eta: 6 days, 23:41:37, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:45:29] ppocr INFO: epoch: [1/2], global_step: 1500, lr: 0.000000, CTCLoss: 404.445374, NRTRLoss: 10.062691, loss: 414.394287, avg_reader_cost: 0.00393 s, avg_batch_cost: 0.25078 s, avg_samples: 5.77, ips: 23.00853 samples/s, eta: 6 days, 22:55:07, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:46:05] ppocr INFO: epoch: [1/2], global_step: 1600, lr: 0.000000, CTCLoss: 404.121460, NRTRLoss: 9.951564, loss: 413.924438, avg_reader_cost: 0.00764 s, avg_batch_cost: 0.26466 s, avg_samples: 5.9, ips: 22.29296 samples/s, eta: 6 days, 22:46:48, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:46:39] ppocr INFO: epoch: [1/2], global_step: 1700, lr: 0.000000, CTCLoss: 405.265411, NRTRLoss: 9.788181, loss: 415.131226, avg_reader_cost: 0.00877 s, avg_batch_cost: 0.26352 s, avg_samples: 5.85, ips: 22.19923 samples/s, eta: 6 days, 22:36:55, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:47:13] ppocr INFO: epoch: [1/2], global_step: 1800, lr: 0.000000, CTCLoss: 403.860107, NRTRLoss: 9.935485, loss: 413.863098, avg_reader_cost: 0.00346 s, avg_batch_cost: 0.26153 s, avg_samples: 5.8, ips: 22.17691 samples/s, eta: 6 days, 22:23:57, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:47:48] ppocr INFO: epoch: [1/2], global_step: 1900, lr: 0.000000, CTCLoss: 404.698914, NRTRLoss: 9.907217, loss: 414.335602, avg_reader_cost: 0.00199 s, avg_batch_cost: 0.26656 s, avg_samples: 5.8, ips: 21.75910 samples/s, eta: 6 days, 22:22:11, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:48:24] ppocr INFO: epoch: [1/2], global_step: 2000, lr: 0.000000, CTCLoss: 403.420044, NRTRLoss: 9.855991, loss: 413.276031, avg_reader_cost: 0.00290 s, avg_batch_cost: 0.27319 s, avg_samples: 5.78, ips: 21.15719 samples/s, eta: 6 days, 22:32:56, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:50:20] ppocr INFO: epoch: [1/2], global_step: 2100, lr: 0.000000, CTCLoss: 403.192932, NRTRLoss: 10.050749, loss: 413.049957, avg_reader_cost: 0.00423 s, avg_batch_cost: 0.25838 s, avg_samples: 6.06, ips: 23.45406 samples/s, eta: 6 days, 22:16:16, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:50:54] ppocr INFO: epoch: [1/2], global_step: 2200, lr: 0.000000, CTCLoss: 404.281189, NRTRLoss: 9.671982, loss: 413.857483, avg_reader_cost: 0.00202 s, avg_batch_cost: 0.25638 s, avg_samples: 5.81, ips: 22.66155 samples/s, eta: 6 days, 21:57:42, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:51:29] ppocr INFO: epoch: [1/2], global_step: 2300, lr: 0.000000, CTCLoss: 401.953278, NRTRLoss: 9.729166, loss: 411.901825, avg_reader_cost: 0.00204 s, avg_batch_cost: 0.26224 s, avg_samples: 5.5, ips: 20.97320 samples/s, eta: 6 days, 21:50:13, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:52:03] ppocr INFO: epoch: [1/2], global_step: 2400, lr: 0.000000, CTCLoss: 403.304626, NRTRLoss: 9.679613, loss: 412.912659, avg_reader_cost: 0.00203 s, avg_batch_cost: 0.25845 s, avg_samples: 5.76, ips: 22.28706 samples/s, eta: 6 days, 21:37:24, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:52:36] ppocr INFO: epoch: [1/2], global_step: 2500, lr: 0.000000, CTCLoss: 403.238037, NRTRLoss: 9.842747, loss: 413.008911, avg_reader_cost: 0.00187 s, avg_batch_cost: 0.25434 s, avg_samples: 5.42, ips: 21.30974 samples/s, eta: 6 days, 21:19:28, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:53:11] ppocr INFO: epoch: [1/2], global_step: 2600, lr: 0.000000, CTCLoss: 402.032257, NRTRLoss: 9.940500, loss: 411.935730, avg_reader_cost: 0.00200 s, avg_batch_cost: 0.25773 s, avg_samples: 6.07, ips: 23.55142 samples/s, eta: 6 days, 21:07:45, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:53:46] ppocr INFO: epoch: [1/2], global_step: 2700, lr: 0.000000, CTCLoss: 401.054047, NRTRLoss: 9.943245, loss: 411.183197, avg_reader_cost: 0.00596 s, avg_batch_cost: 0.26255 s, avg_samples: 6.05, ips: 23.04339 samples/s, eta: 6 days, 21:03:31, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:54:20] ppocr INFO: epoch: [1/2], global_step: 2800, lr: 0.000000, CTCLoss: 403.292145, NRTRLoss: 9.825165, loss: 412.814850, avg_reader_cost: 0.00192 s, avg_batch_cost: 0.25515 s, avg_samples: 5.65, ips: 22.14344 samples/s, eta: 6 days, 20:49:42, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:54:55] ppocr INFO: epoch: [1/2], global_step: 2900, lr: 0.000001, CTCLoss: 402.099579, NRTRLoss: 9.877851, loss: 412.230835, avg_reader_cost: 0.00244 s, avg_batch_cost: 0.26537 s, avg_samples: 5.8, ips: 21.85624 samples/s, eta: 6 days, 20:49:58, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:55:29] ppocr INFO: epoch: [1/2], global_step: 3000, lr: 0.000001, CTCLoss: 402.009674, NRTRLoss: 9.640959, loss: 411.604492, avg_reader_cost: 0.00256 s, avg_batch_cost: 0.26128 s, avg_samples: 5.44, ips: 20.82083 samples/s, eta: 6 days, 20:45:05, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:56:02] ppocr INFO: epoch: [1/2], global_step: 3100, lr: 0.000001, CTCLoss: 401.307739, NRTRLoss: 9.560448, loss: 410.975769, avg_reader_cost: 0.00193 s, avg_batch_cost: 0.24797 s, avg_samples: 5.35, ips: 21.57555 samples/s, eta: 6 days, 20:24:27, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:56:36] ppocr INFO: epoch: [1/2], global_step: 3200, lr: 0.000001, CTCLoss: 402.883759, NRTRLoss: 9.599607, loss: 411.953308, avg_reader_cost: 0.00621 s, avg_batch_cost: 0.25958 s, avg_samples: 5.83, ips: 22.45975 samples/s, eta: 6 days, 20:18:38, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:57:10] ppocr INFO: epoch: [1/2], global_step: 3300, lr: 0.000001, CTCLoss: 400.090698, NRTRLoss: 9.704771, loss: 409.665131, avg_reader_cost: 0.00203 s, avg_batch_cost: 0.25804 s, avg_samples: 5.71, ips: 22.12861 samples/s, eta: 6 days, 20:11:24, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:57:44] ppocr INFO: epoch: [1/2], global_step: 3400, lr: 0.000001, CTCLoss: 400.383820, NRTRLoss: 9.531808, loss: 409.876587, avg_reader_cost: 0.00197 s, avg_batch_cost: 0.26092 s, avg_samples: 5.66, ips: 21.69255 samples/s, eta: 6 days, 20:07:44, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:58:17] ppocr INFO: epoch: [1/2], global_step: 3500, lr: 0.000001, CTCLoss: 398.938202, NRTRLoss: 9.562744, loss: 408.564453, avg_reader_cost: 0.00216 s, avg_batch_cost: 0.26237 s, avg_samples: 5.65, ips: 21.53457 samples/s, eta: 6 days, 20:05:48, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:58:51] ppocr INFO: epoch: [1/2], global_step: 3600, lr: 0.000001, CTCLoss: 399.804810, NRTRLoss: 9.723665, loss: 409.550323, avg_reader_cost: 0.00203 s, avg_batch_cost: 0.26309 s, avg_samples: 5.48, ips: 20.82916 samples/s, eta: 6 days, 20:04:41, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 10:59:25] ppocr INFO: epoch: [1/2], global_step: 3700, lr: 0.000001, CTCLoss: 397.357147, NRTRLoss: 9.705851, loss: 407.387512, avg_reader_cost: 0.00205 s, avg_batch_cost: 0.25771 s, avg_samples: 5.94, ips: 23.04925 samples/s, eta: 6 days, 19:58:12, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:00:00] ppocr INFO: epoch: [1/2], global_step: 3800, lr: 0.000001, CTCLoss: 397.367950, NRTRLoss: 9.559217, loss: 406.547424, avg_reader_cost: 0.00281 s, avg_batch_cost: 0.26418 s, avg_samples: 5.74, ips: 21.72764 samples/s, eta: 6 days, 19:58:22, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:00:35] ppocr INFO: epoch: [1/2], global_step: 3900, lr: 0.000001, CTCLoss: 397.470306, NRTRLoss: 9.616619, loss: 407.100708, avg_reader_cost: 0.00190 s, avg_batch_cost: 0.26794 s, avg_samples: 5.9, ips: 22.01999 samples/s, eta: 6 days, 20:02:07, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:01:10] ppocr INFO: epoch: [1/2], global_step: 4000, lr: 0.000001, CTCLoss: 397.468201, NRTRLoss: 9.533884, loss: 406.771667, avg_reader_cost: 0.00201 s, avg_batch_cost: 0.26747 s, avg_samples: 5.71, ips: 21.34823 samples/s, eta: 6 days, 20:05:12, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:03:15] ppocr INFO: epoch: [1/2], global_step: 4100, lr: 0.000001, CTCLoss: 397.076447, NRTRLoss: 9.551342, loss: 407.059235, avg_reader_cost: 0.00864 s, avg_batch_cost: 0.27148 s, avg_samples: 5.55, ips: 20.44372 samples/s, eta: 6 days, 20:11:47, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:03:48] ppocr INFO: epoch: [1/2], global_step: 4200, lr: 0.000001, CTCLoss: 395.360260, NRTRLoss: 9.493979, loss: 404.911377, avg_reader_cost: 0.00206 s, avg_batch_cost: 0.25178 s, avg_samples: 5.56, ips: 22.08284 samples/s, eta: 6 days, 20:00:31, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:04:23] ppocr INFO: epoch: [1/2], global_step: 4300, lr: 0.000001, CTCLoss: 396.124695, NRTRLoss: 9.411137, loss: 405.608521, avg_reader_cost: 0.00196 s, avg_batch_cost: 0.26127 s, avg_samples: 5.59, ips: 21.39521 samples/s, eta: 6 days, 19:57:59, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:04:57] ppocr INFO: epoch: [1/2], global_step: 4400, lr: 0.000001, CTCLoss: 396.036011, NRTRLoss: 9.348650, loss: 405.292023, avg_reader_cost: 0.00307 s, avg_batch_cost: 0.25872 s, avg_samples: 5.49, ips: 21.21994 samples/s, eta: 6 days, 19:53:24, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:05:31] ppocr INFO: epoch: [1/2], global_step: 4500, lr: 0.000001, CTCLoss: 394.853760, NRTRLoss: 9.629125, loss: 404.182831, avg_reader_cost: 0.00192 s, avg_batch_cost: 0.25663 s, avg_samples: 5.79, ips: 22.56167 samples/s, eta: 6 days, 19:47:15, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:06:04] ppocr INFO: epoch: [1/2], global_step: 4600, lr: 0.000001, CTCLoss: 394.797150, NRTRLoss: 9.572542, loss: 404.240997, avg_reader_cost: 0.00218 s, avg_batch_cost: 0.25346 s, avg_samples: 5.76, ips: 22.72562 samples/s, eta: 6 days, 19:38:47, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:06:39] ppocr INFO: epoch: [1/2], global_step: 4700, lr: 0.000001, CTCLoss: 393.903870, NRTRLoss: 9.635122, loss: 403.733826, avg_reader_cost: 0.00451 s, avg_batch_cost: 0.26626 s, avg_samples: 5.66, ips: 21.25740 samples/s, eta: 6 days, 19:40:50, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:07:13] ppocr INFO: epoch: [1/2], global_step: 4800, lr: 0.000001, CTCLoss: 393.230835, NRTRLoss: 9.445354, loss: 402.580292, avg_reader_cost: 0.00382 s, avg_batch_cost: 0.25634 s, avg_samples: 5.66, ips: 22.08021 samples/s, eta: 6 days, 19:35:03, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:07:46] ppocr INFO: epoch: [1/2], global_step: 4900, lr: 0.000001, CTCLoss: 391.972961, NRTRLoss: 9.429935, loss: 401.075256, avg_reader_cost: 0.00201 s, avg_batch_cost: 0.25507 s, avg_samples: 5.61, ips: 21.99439 samples/s, eta: 6 days, 19:28:32, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:08:21] ppocr INFO: epoch: [1/2], global_step: 5000, lr: 0.000001, CTCLoss: 392.572388, NRTRLoss: 9.271074, loss: 402.017487, avg_reader_cost: 0.00198 s, avg_batch_cost: 0.25798 s, avg_samples: 5.86, ips: 22.71459 samples/s, eta: 6 days, 19:24:25, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:08:56] ppocr INFO: epoch: [1/2], global_step: 5100, lr: 0.000001, CTCLoss: 390.078430, NRTRLoss: 9.474798, loss: 399.878479, avg_reader_cost: 0.00612 s, avg_batch_cost: 0.26929 s, avg_samples: 5.93, ips: 22.02116 samples/s, eta: 6 days, 19:28:44, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:09:30] ppocr INFO: epoch: [1/2], global_step: 5200, lr: 0.000001, CTCLoss: 391.718140, NRTRLoss: 9.447325, loss: 400.623383, avg_reader_cost: 0.00192 s, avg_batch_cost: 0.25832 s, avg_samples: 5.76, ips: 22.29763 samples/s, eta: 6 days, 19:24:59, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:10:05] ppocr INFO: epoch: [1/2], global_step: 5300, lr: 0.000001, CTCLoss: 391.513367, NRTRLoss: 9.411615, loss: 401.181946, avg_reader_cost: 0.00474 s, avg_batch_cost: 0.27940 s, avg_samples: 5.68, ips: 20.32908 samples/s, eta: 6 days, 19:36:12, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:10:44] ppocr INFO: epoch: [1/2], global_step: 5400, lr: 0.000001, CTCLoss: 389.666229, NRTRLoss: 9.352569, loss: 399.039429, avg_reader_cost: 0.00258 s, avg_batch_cost: 0.29844 s, avg_samples: 5.89, ips: 19.73623 samples/s, eta: 6 days, 20:00:09, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:11:18] ppocr INFO: epoch: [1/2], global_step: 5500, lr: 0.000001, CTCLoss: 390.512390, NRTRLoss: 9.492992, loss: 400.076111, avg_reader_cost: 0.00206 s, avg_batch_cost: 0.26427 s, avg_samples: 5.67, ips: 21.45525 samples/s, eta: 6 days, 20:00:01, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:11:53] ppocr INFO: epoch: [1/2], global_step: 5600, lr: 0.000001, CTCLoss: 388.234558, NRTRLoss: 9.447018, loss: 397.642517, avg_reader_cost: 0.00557 s, avg_batch_cost: 0.26956 s, avg_samples: 5.85, ips: 21.70234 samples/s, eta: 6 days, 20:03:24, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:12:28] ppocr INFO: epoch: [1/2], global_step: 5700, lr: 0.000001, CTCLoss: 388.416748, NRTRLoss: 9.419914, loss: 397.355530, avg_reader_cost: 0.00204 s, avg_batch_cost: 0.27204 s, avg_samples: 5.56, ips: 20.43798 samples/s, eta: 6 days, 20:08:17, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:13:03] ppocr INFO: epoch: [1/2], global_step: 5800, lr: 0.000001, CTCLoss: 389.557190, NRTRLoss: 9.427664, loss: 398.989441, avg_reader_cost: 0.00516 s, avg_batch_cost: 0.27135 s, avg_samples: 5.76, ips: 21.22755 samples/s, eta: 6 days, 20:12:32, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:13:38] ppocr INFO: epoch: [1/2], global_step: 5900, lr: 0.000001, CTCLoss: 385.087982, NRTRLoss: 9.198366, loss: 394.331665, avg_reader_cost: 0.00213 s, avg_batch_cost: 0.26089 s, avg_samples: 5.9, ips: 22.61533 samples/s, eta: 6 days, 20:10:00, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:14:13] ppocr INFO: epoch: [1/2], global_step: 6000, lr: 0.000001, CTCLoss: 384.622284, NRTRLoss: 9.201010, loss: 393.830872, avg_reader_cost: 0.00229 s, avg_batch_cost: 0.27262 s, avg_samples: 5.67, ips: 20.79813 samples/s, eta: 6 days, 20:14:50, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:16:11] ppocr INFO: epoch: [1/2], global_step: 6100, lr: 0.000001, CTCLoss: 385.888947, NRTRLoss: 9.260035, loss: 395.053009, avg_reader_cost: 0.00741 s, avg_batch_cost: 0.26461 s, avg_samples: 5.7, ips: 21.54075 samples/s, eta: 6 days, 20:14:37, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:16:46] ppocr INFO: epoch: [1/2], global_step: 6200, lr: 0.000001, CTCLoss: 386.948914, NRTRLoss: 9.118103, loss: 396.114807, avg_reader_cost: 0.00277 s, avg_batch_cost: 0.27552 s, avg_samples: 5.64, ips: 20.47059 samples/s, eta: 6 days, 20:20:56, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:17:21] ppocr INFO: epoch: [1/2], global_step: 6300, lr: 0.000001, CTCLoss: 384.159241, NRTRLoss: 9.358393, loss: 393.390594, avg_reader_cost: 0.00203 s, avg_batch_cost: 0.26690 s, avg_samples: 5.49, ips: 20.56917 samples/s, eta: 6 days, 20:21:56, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:17:55] ppocr INFO: epoch: [1/2], global_step: 6400, lr: 0.000001, CTCLoss: 381.691772, NRTRLoss: 9.211382, loss: 390.990479, avg_reader_cost: 0.00204 s, avg_batch_cost: 0.26412 s, avg_samples: 5.74, ips: 21.73258 samples/s, eta: 6 days, 20:21:17, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:18:29] ppocr INFO: epoch: [1/2], global_step: 6500, lr: 0.000001, CTCLoss: 383.067505, NRTRLoss: 9.240335, loss: 392.569336, avg_reader_cost: 0.00208 s, avg_batch_cost: 0.25868 s, avg_samples: 5.54, ips: 21.41669 samples/s, eta: 6 days, 20:17:30, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:19:03] ppocr INFO: epoch: [1/2], global_step: 6600, lr: 0.000001, CTCLoss: 382.535767, NRTRLoss: 9.244854, loss: 391.394440, avg_reader_cost: 0.00200 s, avg_batch_cost: 0.25505 s, avg_samples: 5.67, ips: 22.23095 samples/s, eta: 6 days, 20:11:47, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:19:38] ppocr INFO: epoch: [1/2], global_step: 6700, lr: 0.000001, CTCLoss: 381.917664, NRTRLoss: 9.062761, loss: 390.802917, avg_reader_cost: 0.00196 s, avg_batch_cost: 0.26354 s, avg_samples: 5.48, ips: 20.79402 samples/s, eta: 6 days, 20:10:56, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:20:13] ppocr INFO: epoch: [1/2], global_step: 6800, lr: 0.000001, CTCLoss: 380.590637, NRTRLoss: 9.080160, loss: 390.018097, avg_reader_cost: 0.00507 s, avg_batch_cost: 0.27640 s, avg_samples: 6.11, ips: 22.10564 samples/s, eta: 6 days, 20:17:10, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:20:48] ppocr INFO: epoch: [1/2], global_step: 6900, lr: 0.000001, CTCLoss: 378.476593, NRTRLoss: 8.985798, loss: 387.645203, avg_reader_cost: 0.00212 s, avg_batch_cost: 0.26491 s, avg_samples: 5.45, ips: 20.57319 samples/s, eta: 6 days, 20:16:59, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:21:22] ppocr INFO: epoch: [1/2], global_step: 7000, lr: 0.000001, CTCLoss: 378.699402, NRTRLoss: 9.056095, loss: 387.686737, avg_reader_cost: 0.00285 s, avg_batch_cost: 0.25652 s, avg_samples: 5.33, ips: 20.77831 samples/s, eta: 6 days, 20:12:20, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:21:55] ppocr INFO: epoch: [1/2], global_step: 7100, lr: 0.000001, CTCLoss: 377.567657, NRTRLoss: 8.949715, loss: 386.298645, avg_reader_cost: 0.00200 s, avg_batch_cost: 0.25982 s, avg_samples: 5.57, ips: 21.43759 samples/s, eta: 6 days, 20:09:31, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:22:30] ppocr INFO: epoch: [1/2], global_step: 7200, lr: 0.000001, CTCLoss: 376.851562, NRTRLoss: 8.789501, loss: 385.556824, avg_reader_cost: 0.00249 s, avg_batch_cost: 0.25844 s, avg_samples: 5.98, ips: 23.13894 samples/s, eta: 6 days, 20:06:04, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:23:06] ppocr INFO: epoch: [1/2], global_step: 7300, lr: 0.000001, CTCLoss: 375.163330, NRTRLoss: 9.008641, loss: 384.158752, avg_reader_cost: 0.00244 s, avg_batch_cost: 0.27335 s, avg_samples: 5.82, ips: 21.29164 samples/s, eta: 6 days, 20:10:19, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:23:40] ppocr INFO: epoch: [1/2], global_step: 7400, lr: 0.000001, CTCLoss: 373.567627, NRTRLoss: 8.986026, loss: 382.471893, avg_reader_cost: 0.00214 s, avg_batch_cost: 0.26562 s, avg_samples: 5.74, ips: 21.60989 samples/s, eta: 6 days, 20:10:32, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:24:14] ppocr INFO: epoch: [1/2], global_step: 7500, lr: 0.000001, CTCLoss: 373.707336, NRTRLoss: 8.994013, loss: 382.640991, avg_reader_cost: 0.00213 s, avg_batch_cost: 0.25879 s, avg_samples: 5.52, ips: 21.32996 samples/s, eta: 6 days, 20:07:21, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:24:51] ppocr INFO: epoch: [1/2], global_step: 7600, lr: 0.000001, CTCLoss: 374.491699, NRTRLoss: 9.034416, loss: 383.678406, avg_reader_cost: 0.00200 s, avg_batch_cost: 0.27741 s, avg_samples: 5.96, ips: 21.48468 samples/s, eta: 6 days, 20:13:22, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:25:25] ppocr INFO: epoch: [1/2], global_step: 7700, lr: 0.000001, CTCLoss: 373.003479, NRTRLoss: 8.940775, loss: 382.177246, avg_reader_cost: 0.00198 s, avg_batch_cost: 0.26221 s, avg_samples: 5.84, ips: 22.27197 samples/s, eta: 6 days, 20:11:51, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:26:01] ppocr INFO: epoch: [1/2], global_step: 7800, lr: 0.000001, CTCLoss: 372.563416, NRTRLoss: 8.999224, loss: 381.335297, avg_reader_cost: 0.00198 s, avg_batch_cost: 0.27072 s, avg_samples: 5.9, ips: 21.79360 samples/s, eta: 6 days, 20:14:26, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:26:36] ppocr INFO: epoch: [1/2], global_step: 7900, lr: 0.000001, CTCLoss: 372.824219, NRTRLoss: 9.039875, loss: 382.021118, avg_reader_cost: 0.00238 s, avg_batch_cost: 0.27731 s, avg_samples: 5.74, ips: 20.69899 samples/s, eta: 6 days, 20:20:03, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:27:11] ppocr INFO: epoch: [1/2], global_step: 8000, lr: 0.000001, CTCLoss: 370.048279, NRTRLoss: 8.854296, loss: 378.577545, avg_reader_cost: 0.00190 s, avg_batch_cost: 0.26518 s, avg_samples: 5.69, ips: 21.45672 samples/s, eta: 6 days, 20:19:52, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:29:10] ppocr INFO: epoch: [1/2], global_step: 8100, lr: 0.000001, CTCLoss: 369.931732, NRTRLoss: 9.039619, loss: 379.149536, avg_reader_cost: 0.00251 s, avg_batch_cost: 0.26537 s, avg_samples: 6.06, ips: 22.83607 samples/s, eta: 6 days, 20:19:46, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:29:45] ppocr INFO: epoch: [1/2], global_step: 8200, lr: 0.000001, CTCLoss: 369.772705, NRTRLoss: 8.865396, loss: 378.834198, avg_reader_cost: 0.00199 s, avg_batch_cost: 0.26690 s, avg_samples: 5.59, ips: 20.94417 samples/s, eta: 6 days, 20:20:21, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:30:19] ppocr INFO: epoch: [1/2], global_step: 8300, lr: 0.000001, CTCLoss: 365.449036, NRTRLoss: 8.667650, loss: 373.802765, avg_reader_cost: 0.00216 s, avg_batch_cost: 0.26500 s, avg_samples: 5.51, ips: 20.79255 samples/s, eta: 6 days, 20:20:03, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:30:53] ppocr INFO: epoch: [1/2], global_step: 8400, lr: 0.000001, CTCLoss: 365.448059, NRTRLoss: 8.798958, loss: 374.566315, avg_reader_cost: 0.00190 s, avg_batch_cost: 0.26062 s, avg_samples: 5.83, ips: 22.36964 samples/s, eta: 6 days, 20:17:48, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:31:28] ppocr INFO: epoch: [1/2], global_step: 8500, lr: 0.000002, CTCLoss: 364.957245, NRTRLoss: 8.892164, loss: 373.899353, avg_reader_cost: 0.00199 s, avg_batch_cost: 0.26666 s, avg_samples: 5.63, ips: 21.11293 samples/s, eta: 6 days, 20:18:15, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:32:01] ppocr INFO: epoch: [1/2], global_step: 8600, lr: 0.000002, CTCLoss: 363.196930, NRTRLoss: 8.831235, loss: 372.035492, avg_reader_cost: 0.00218 s, avg_batch_cost: 0.25541 s, avg_samples: 5.28, ips: 20.67284 samples/s, eta: 6 days, 20:13:48, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:32:35] ppocr INFO: epoch: [1/2], global_step: 8700, lr: 0.000002, CTCLoss: 363.983246, NRTRLoss: 8.799014, loss: 372.707977, avg_reader_cost: 0.00203 s, avg_batch_cost: 0.26001 s, avg_samples: 5.62, ips: 21.61433 samples/s, eta: 6 days, 20:11:25, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:33:10] ppocr INFO: epoch: [1/2], global_step: 8800, lr: 0.000002, CTCLoss: 362.398926, NRTRLoss: 8.753778, loss: 371.114227, avg_reader_cost: 0.00209 s, avg_batch_cost: 0.26262 s, avg_samples: 5.69, ips: 21.66631 samples/s, eta: 6 days, 20:10:11, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:33:45] ppocr INFO: epoch: [1/2], global_step: 8900, lr: 0.000002, CTCLoss: 361.002136, NRTRLoss: 8.770383, loss: 369.739197, avg_reader_cost: 0.00208 s, avg_batch_cost: 0.27102 s, avg_samples: 5.55, ips: 20.47825 samples/s, eta: 6 days, 20:12:29, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:34:20] ppocr INFO: epoch: [1/2], global_step: 9000, lr: 0.000002, CTCLoss: 359.807007, NRTRLoss: 8.891740, loss: 368.952026, avg_reader_cost: 0.00206 s, avg_batch_cost: 0.26330 s, avg_samples: 5.82, ips: 22.10381 samples/s, eta: 6 days, 20:11:32, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:34:56] ppocr INFO: epoch: [1/2], global_step: 9100, lr: 0.000002, CTCLoss: 357.975098, NRTRLoss: 8.590385, loss: 366.844421, avg_reader_cost: 0.00194 s, avg_batch_cost: 0.26765 s, avg_samples: 5.85, ips: 21.85728 samples/s, eta: 6 days, 20:12:22, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:35:29] ppocr INFO: epoch: [1/2], global_step: 9200, lr: 0.000002, CTCLoss: 357.741455, NRTRLoss: 8.674175, loss: 366.438812, avg_reader_cost: 0.00202 s, avg_batch_cost: 0.25700 s, avg_samples: 5.41, ips: 21.05063 samples/s, eta: 6 days, 20:08:51, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:36:04] ppocr INFO: epoch: [1/2], global_step: 9300, lr: 0.000002, CTCLoss: 356.887573, NRTRLoss: 8.622353, loss: 365.602539, avg_reader_cost: 0.00204 s, avg_batch_cost: 0.26423 s, avg_samples: 5.59, ips: 21.15566 samples/s, eta: 6 days, 20:08:19, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:36:38] ppocr INFO: epoch: [1/2], global_step: 9400, lr: 0.000002, CTCLoss: 357.669067, NRTRLoss: 8.845102, loss: 366.515747, avg_reader_cost: 0.00203 s, avg_batch_cost: 0.26103 s, avg_samples: 5.43, ips: 20.80227 samples/s, eta: 6 days, 20:06:30, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:37:12] ppocr INFO: epoch: [1/2], global_step: 9500, lr: 0.000002, CTCLoss: 355.997437, NRTRLoss: 8.847376, loss: 364.532776, avg_reader_cost: 0.00206 s, avg_batch_cost: 0.26688 s, avg_samples: 5.65, ips: 21.17032 samples/s, eta: 6 days, 20:07:01, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:37:46] ppocr INFO: epoch: [1/2], global_step: 9600, lr: 0.000002, CTCLoss: 353.975555, NRTRLoss: 8.639709, loss: 362.645111, avg_reader_cost: 0.00207 s, avg_batch_cost: 0.26248 s, avg_samples: 5.55, ips: 21.14478 samples/s, eta: 6 days, 20:05:48, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:38:21] ppocr INFO: epoch: [1/2], global_step: 9700, lr: 0.000002, CTCLoss: 353.316681, NRTRLoss: 8.692984, loss: 362.085175, avg_reader_cost: 0.00203 s, avg_batch_cost: 0.26891 s, avg_samples: 5.82, ips: 21.64296 samples/s, eta: 6 days, 20:07:04, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:38:56] ppocr INFO: epoch: [1/2], global_step: 9800, lr: 0.000002, CTCLoss: 353.826080, NRTRLoss: 8.561289, loss: 362.419647, avg_reader_cost: 0.00298 s, avg_batch_cost: 0.26914 s, avg_samples: 5.59, ips: 20.76973 samples/s, eta: 6 days, 20:08:24, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:39:31] ppocr INFO: epoch: [1/2], global_step: 9900, lr: 0.000002, CTCLoss: 349.449860, NRTRLoss: 8.583508, loss: 357.899445, avg_reader_cost: 0.00193 s, avg_batch_cost: 0.26822 s, avg_samples: 5.68, ips: 21.17648 samples/s, eta: 6 days, 20:09:20, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:40:05] ppocr INFO: epoch: [1/2], global_step: 10000, lr: 0.000002, CTCLoss: 345.930420, NRTRLoss: 8.705133, loss: 354.210388, avg_reader_cost: 0.00203 s, avg_batch_cost: 0.25990 s, avg_samples: 5.39, ips: 20.73900 samples/s, eta: 6 days, 20:07:09, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:42:03] ppocr INFO: epoch: [1/2], global_step: 10100, lr: 0.000002, CTCLoss: 347.878510, NRTRLoss: 8.789956, loss: 356.577545, avg_reader_cost: 0.00208 s, avg_batch_cost: 0.27232 s, avg_samples: 5.71, ips: 20.96785 samples/s, eta: 6 days, 20:09:35, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:42:38] ppocr INFO: epoch: [1/2], global_step: 10200, lr: 0.000002, CTCLoss: 347.717224, NRTRLoss: 8.817150, loss: 356.264282, avg_reader_cost: 0.00201 s, avg_batch_cost: 0.26562 s, avg_samples: 5.74, ips: 21.60954 samples/s, eta: 6 days, 20:09:30, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:43:12] ppocr INFO: epoch: [1/2], global_step: 10300, lr: 0.000002, CTCLoss: 348.153748, NRTRLoss: 8.501930, loss: 356.501251, avg_reader_cost: 0.00608 s, avg_batch_cost: 0.26630 s, avg_samples: 5.63, ips: 21.14187 samples/s, eta: 6 days, 20:09:40, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:43:46] ppocr INFO: epoch: [1/2], global_step: 10400, lr: 0.000002, CTCLoss: 342.761414, NRTRLoss: 8.543266, loss: 351.291290, avg_reader_cost: 0.00207 s, avg_batch_cost: 0.26078 s, avg_samples: 5.17, ips: 19.82492 samples/s, eta: 6 days, 20:07:51, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:44:20] ppocr INFO: epoch: [1/2], global_step: 10500, lr: 0.000002, CTCLoss: 341.775818, NRTRLoss: 8.573007, loss: 350.151215, avg_reader_cost: 0.00198 s, avg_batch_cost: 0.25666 s, avg_samples: 5.64, ips: 21.97502 samples/s, eta: 6 days, 20:04:35, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:44:55] ppocr INFO: epoch: [1/2], global_step: 10600, lr: 0.000002, CTCLoss: 346.363220, NRTRLoss: 8.634343, loss: 355.000610, avg_reader_cost: 0.00201 s, avg_batch_cost: 0.26708 s, avg_samples: 5.63, ips: 21.08013 samples/s, eta: 6 days, 20:05:03, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:45:31] ppocr INFO: epoch: [1/2], global_step: 10700, lr: 0.000002, CTCLoss: 341.860535, NRTRLoss: 8.483498, loss: 350.368286, avg_reader_cost: 0.00794 s, avg_batch_cost: 0.27509 s, avg_samples: 6.05, ips: 21.99289 samples/s, eta: 6 days, 20:08:16, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:46:07] ppocr INFO: epoch: [1/2], global_step: 10800, lr: 0.000002, CTCLoss: 342.920593, NRTRLoss: 8.606590, loss: 351.063599, avg_reader_cost: 0.00202 s, avg_batch_cost: 0.27397 s, avg_samples: 5.75, ips: 20.98782 samples/s, eta: 6 days, 20:11:02, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:46:42] ppocr INFO: epoch: [1/2], global_step: 10900, lr: 0.000002, CTCLoss: 339.964661, NRTRLoss: 8.407807, loss: 348.575195, avg_reader_cost: 0.00200 s, avg_batch_cost: 0.26814 s, avg_samples: 5.43, ips: 20.25028 samples/s, eta: 6 days, 20:11:46, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:47:17] ppocr INFO: epoch: [1/2], global_step: 11000, lr: 0.000002, CTCLoss: 339.457214, NRTRLoss: 8.648579, loss: 347.969482, avg_reader_cost: 0.00202 s, avg_batch_cost: 0.26858 s, avg_samples: 5.68, ips: 21.14841 samples/s, eta: 6 days, 20:12:36, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:47:51] ppocr INFO: epoch: [1/2], global_step: 11100, lr: 0.000002, CTCLoss: 336.576538, NRTRLoss: 8.635804, loss: 345.281952, avg_reader_cost: 0.00197 s, avg_batch_cost: 0.26174 s, avg_samples: 5.7, ips: 21.77763 samples/s, eta: 6 days, 20:11:08, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:48:25] ppocr INFO: epoch: [1/2], global_step: 11200, lr: 0.000002, CTCLoss: 334.776001, NRTRLoss: 8.335504, loss: 342.895081, avg_reader_cost: 0.00215 s, avg_batch_cost: 0.25999 s, avg_samples: 5.59, ips: 21.50080 samples/s, eta: 6 days, 20:09:07, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:48:59] ppocr INFO: epoch: [1/2], global_step: 11300, lr: 0.000002, CTCLoss: 334.675964, NRTRLoss: 8.518665, loss: 343.281525, avg_reader_cost: 0.00199 s, avg_batch_cost: 0.25684 s, avg_samples: 5.52, ips: 21.49219 samples/s, eta: 6 days, 20:06:04, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:49:35] ppocr INFO: epoch: [1/2], global_step: 11400, lr: 0.000002, CTCLoss: 333.580719, NRTRLoss: 8.529530, loss: 341.678223, avg_reader_cost: 0.00204 s, avg_batch_cost: 0.27269 s, avg_samples: 5.92, ips: 21.70972 samples/s, eta: 6 days, 20:08:15, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:50:09] ppocr INFO: epoch: [1/2], global_step: 11500, lr: 0.000002, CTCLoss: 334.276001, NRTRLoss: 8.339378, loss: 342.891357, avg_reader_cost: 0.00329 s, avg_batch_cost: 0.27006 s, avg_samples: 5.68, ips: 21.03204 samples/s, eta: 6 days, 20:09:32, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:50:45] ppocr INFO: epoch: [1/2], global_step: 11600, lr: 0.000002, CTCLoss: 334.626038, NRTRLoss: 8.421999, loss: 343.542908, avg_reader_cost: 0.00210 s, avg_batch_cost: 0.27736 s, avg_samples: 5.71, ips: 20.58714 samples/s, eta: 6 days, 20:13:07, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:51:19] ppocr INFO: epoch: [1/2], global_step: 11700, lr: 0.000002, CTCLoss: 334.525391, NRTRLoss: 8.432699, loss: 342.459717, avg_reader_cost: 0.00198 s, avg_batch_cost: 0.26085 s, avg_samples: 5.41, ips: 20.73977 samples/s, eta: 6 days, 20:11:24, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:51:55] ppocr INFO: epoch: [1/2], global_step: 11800, lr: 0.000002, CTCLoss: 331.242432, NRTRLoss: 8.443002, loss: 339.671692, avg_reader_cost: 0.00413 s, avg_batch_cost: 0.27923 s, avg_samples: 5.85, ips: 20.95075 samples/s, eta: 6 days, 20:15:29, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:52:30] ppocr INFO: epoch: [1/2], global_step: 11900, lr: 0.000002, CTCLoss: 324.933990, NRTRLoss: 8.380083, loss: 333.283569, avg_reader_cost: 0.00358 s, avg_batch_cost: 0.26628 s, avg_samples: 5.63, ips: 21.14291 samples/s, eta: 6 days, 20:15:27, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:53:04] ppocr INFO: epoch: [1/2], global_step: 12000, lr: 0.000002, CTCLoss: 323.600708, NRTRLoss: 8.229788, loss: 331.966675, avg_reader_cost: 0.00205 s, avg_batch_cost: 0.26465 s, avg_samples: 5.51, ips: 20.81969 samples/s, eta: 6 days, 20:14:55, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:55:01] ppocr INFO: epoch: [1/2], global_step: 12100, lr: 0.000002, CTCLoss: 329.255554, NRTRLoss: 8.512887, loss: 337.524536, avg_reader_cost: 0.00835 s, avg_batch_cost: 0.27368 s, avg_samples: 5.84, ips: 21.33878 samples/s, eta: 6 days, 20:17:09, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:55:36] ppocr INFO: epoch: [1/2], global_step: 12200, lr: 0.000002, CTCLoss: 322.391357, NRTRLoss: 8.227340, loss: 330.386719, avg_reader_cost: 0.00271 s, avg_batch_cost: 0.27283 s, avg_samples: 5.35, ips: 19.60950 samples/s, eta: 6 days, 20:19:04, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:56:11] ppocr INFO: epoch: [1/2], global_step: 12300, lr: 0.000002, CTCLoss: 323.497711, NRTRLoss: 8.279412, loss: 331.616821, avg_reader_cost: 0.00257 s, avg_batch_cost: 0.27067 s, avg_samples: 5.71, ips: 21.09552 samples/s, eta: 6 days, 20:20:19, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:56:47] ppocr INFO: epoch: [1/2], global_step: 12400, lr: 0.000002, CTCLoss: 323.200806, NRTRLoss: 8.423332, loss: 331.565125, avg_reader_cost: 0.00221 s, avg_batch_cost: 0.27372 s, avg_samples: 6.11, ips: 22.32225 samples/s, eta: 6 days, 20:22:26, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:57:21] ppocr INFO: epoch: [1/2], global_step: 12500, lr: 0.000002, CTCLoss: 321.837219, NRTRLoss: 8.518212, loss: 330.278656, avg_reader_cost: 0.00202 s, avg_batch_cost: 0.25818 s, avg_samples: 5.59, ips: 21.65137 samples/s, eta: 6 days, 20:19:54, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:57:55] ppocr INFO: epoch: [1/2], global_step: 12600, lr: 0.000002, CTCLoss: 319.185486, NRTRLoss: 8.256691, loss: 327.026550, avg_reader_cost: 0.00213 s, avg_batch_cost: 0.25906 s, avg_samples: 5.63, ips: 21.73228 samples/s, eta: 6 days, 20:17:39, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:58:30] ppocr INFO: epoch: [1/2], global_step: 12700, lr: 0.000002, CTCLoss: 319.555206, NRTRLoss: 8.278657, loss: 327.524963, avg_reader_cost: 0.00554 s, avg_batch_cost: 0.27675 s, avg_samples: 5.89, ips: 21.28240 samples/s, eta: 6 days, 20:20:37, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:59:05] ppocr INFO: epoch: [1/2], global_step: 12800, lr: 0.000002, CTCLoss: 318.133118, NRTRLoss: 8.201355, loss: 326.310852, avg_reader_cost: 0.00200 s, avg_batch_cost: 0.26965 s, avg_samples: 5.5, ips: 20.39705 samples/s, eta: 6 days, 20:21:27, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 11:59:38] ppocr INFO: epoch: [1/2], global_step: 12900, lr: 0.000002, CTCLoss: 316.876099, NRTRLoss: 8.146032, loss: 325.264343, avg_reader_cost: 0.00271 s, avg_batch_cost: 0.25679 s, avg_samples: 5.34, ips: 20.79481 samples/s, eta: 6 days, 20:18:34, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 12:00:13] ppocr INFO: epoch: [1/2], global_step: 13000, lr: 0.000002, CTCLoss: 314.106812, NRTRLoss: 8.332592, loss: 322.563416, avg_reader_cost: 0.00215 s, avg_batch_cost: 0.26525 s, avg_samples: 5.52, ips: 20.81063 samples/s, eta: 6 days, 20:18:09, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 12:00:47] ppocr INFO: epoch: [1/2], global_step: 13100, lr: 0.000002, CTCLoss: 317.016754, NRTRLoss: 8.376856, loss: 324.358826, avg_reader_cost: 0.00199 s, avg_batch_cost: 0.26382 s, avg_samples: 5.86, ips: 22.21210 samples/s, eta: 6 days, 20:17:19, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB
[2024/10/30 12:01:23] ppocr INFO: epoch: [1/2], global_step: 13200, lr: 0.000002, CTCLoss: 313.327271, NRTRLoss: 8.192154, loss: 321.335999, avg_reader_cost: 0.00220 s, avg_batch_cost: 0.27408 s, avg_samples: 5.78, ips: 21.08863 samples/s, eta: 6 days, 20:19:23, max_mem_reserved: 3462 MB, max_mem_allocated: 2774 MB

@nissansz
Copy link
Author

nissansz commented Oct 30, 2024

yml 改了 cal_metric_during_train: True,可以显示了 acc: , norm_edit_dis: 0.864818 了

[2024/10/30 17:40:19] ppocr INFO: epoch: [1/2], global_step: 100, lr: 0.000000, acc: 0.000000, norm_edit_dis: 0.000001, CTCLoss: 65.103783, NRTRLoss: 7.208924, loss: 72.417091, avg_reader_cost: 0.20637 s, avg_batch_cost: 0.89114 s, avg_samples: 10.9, ips: 12.23155 samples/s, eta: 11 days, 13:38:42, max_mem_reserved: 4945 MB, max_mem_allocated: 4128 MB
[2024/10/30 17:41:39] ppocr INFO: epoch: [1/2], global_step: 200, lr: 0.000000, acc: 0.000000, norm_edit_dis: 0.000001, CTCLoss: 75.929169, NRTRLoss: 7.318246, loss: 83.308685, avg_reader_cost: 0.17139 s, avg_batch_cost: 0.80754 s, avg_samples: 11.28, ips: 13.96836 samples/s, eta: 11 days, 0:35:53, max_mem_reserved: 4945 MB, max_mem_allocated: 4128 MB
[2024/10/30 17:43:02] ppocr INFO: epoch: [1/2], global_step: 300, lr: 0.000000, acc: 0.000000, norm_edit_dis: 0.000001, CTCLoss: 66.579941, NRTRLoss: 7.028238, loss: 73.812561, avg_reader_cost: 0.13904 s, avg_batch_cost: 0.82156 s, avg_samples: 11.74, ips: 14.28995 samples/s, eta: 10 days, 21:41:23, max_mem_reserved: 4945 MB, max_mem_allocated: 4128 MB
[2024/10/30 17:44:20] ppocr INFO: epoch: [1/2], global_step: 400, lr: 0.000000, acc: 0.000000, norm_edit_dis: 0.000001, CTCLoss: 65.599457, NRTRLoss: 6.875330, loss: 72.492935, avg_reader_cost: 0.12426 s, avg_batch_cost: 0.78142 s, avg_samples: 11.06, ips: 14.15381 samples/s, eta: 10 days, 17:05:54, max_mem_reserved: 4945 MB, max_mem_allocated: 4167 MB

@Topdu
Copy link
Owner

Topdu commented Oct 30, 2024

This requires manual configuration of the pre-trained model for SVTRv2.

@nissansz
Copy link
Author

This requires manual configuration of the pre-trained model for SVTRv2.

I train new model for Chinese, Korean, Japanese, English model,
pre-trained model is not usable, right?

@nissansz
Copy link
Author

CTCLoss: 到了60左右,好像很难改善了。一般8G显卡要训练多久才能改善到10以内?

[2024/10/31 07:53:32] ppocr INFO: epoch: [1/2], global_step: 45800, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.000002, CTCLoss: 60.874081, NRTRLoss: 6.455424, loss: 67.166962, avg_reader_cost: 0.00234 s, avg_batch_cost: 0.39864 s, avg_samples: 5.94, ips: 14.90062 samples/s, eta: 9 days, 13:23:11, max_mem_reserved: 3465 MB, max_mem_allocated: 2774 MB
[2024/10/31 07:54:11] ppocr INFO: epoch: [1/2], global_step: 45900, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.000002, CTCLoss: 63.886902, NRTRLoss: 6.652631, loss: 70.947906, avg_reader_cost: 0.00219 s, avg_batch_cost: 0.38966 s, avg_samples: 5.76, ips: 14.78221 samples/s, eta: 9 days, 13:23:40, max_mem_reserved: 3465 MB, max_mem_allocated: 2774 MB
[2024/10/31 07:54:49] ppocr INFO: epoch: [1/2], global_step: 46000, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.000002, CTCLoss: 60.768234, NRTRLoss: 6.741760, loss: 67.918365, avg_reader_cost: 0.00250 s, avg_batch_cost: 0.37633 s, avg_samples: 5.49, ips: 14.58837 samples/s, eta: 9 days, 13:23:05, max_mem_reserved: 3465 MB, max_mem_allocated: 2774 MB
[2024/10/31 07:55:27] ppocr INFO: epoch: [1/2], global_step: 46100, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.000002, CTCLoss: 60.215626, NRTRLoss: 6.481076, loss: 65.829453, avg_reader_cost: 0.00240 s, avg_batch_cost: 0.38033 s, avg_samples: 5.61, ips: 14.75027 samples/s, eta: 9 days, 13:22:49, max_mem_reserved: 3465 MB, max_mem_allocated: 2774 MB
[2024/10/31 07:56:05] ppocr INFO: epoch: [1/2], global_step: 46200, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.000002, CTCLoss: 63.288437, NRTRLoss: 6.536931, loss: 70.091736, avg_reader_cost: 0.00212 s, avg_batch_cost: 0.38071 s, avg_samples: 5.39, ips: 14.15758 samples/s, eta: 9 days, 13:22:35, max_mem_reserved: 3465 MB, max_mem_allocated: 2774 MB
[2024/10/31 07:56:43] ppocr INFO: epoch: [1/2], global_step: 46300, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.000002, CTCLoss: 61.439026, NRTRLoss: 6.389464, loss: 67.899994, avg_reader_cost: 0.00211 s, avg_batch_cost: 0.38261 s, avg_samples: 5.68, ips: 14.84540 samples/s, eta: 9 days, 13:22:30, max_mem_reserved: 3465 MB, max_mem_allocated: 2774 MB
[2024/10/31 07:57:22] ppocr INFO: epoch: [1/2], global_step: 46400, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.000002, CTCLoss: 68.265915, NRTRLoss: 6.538834, loss: 74.272949, avg_reader_cost: 0.00237 s, avg_batch_cost: 0.38132 s, avg_samples: 5.66, ips: 14.84300 samples/s, eta: 9 days, 13:22:18, max_mem_reserved: 3465 MB, max_mem_allocated: 2774 MB
[2024/10/31 07:57:59] ppocr INFO: epoch: [1/2], global_step: 46500, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.000002, CTCLoss: 55.310104, NRTRLoss: 6.450691, loss: 62.328751, avg_reader_cost: 0.00225 s, avg_batch_cost: 0.37691 s, avg_samples: 5.6, ips: 14.85752 samples/s, eta: 9 days, 13:21:46, max_mem_reserved: 3465 MB, max_mem_allocated: 2774 MB

@nissansz
Copy link
Author

For hardcase characters, do I need to add more images for them as ResNet34 after finished training with normal text or add them from the beginning?

@Topdu
Copy link
Owner

Topdu commented Nov 1, 2024

It is recommended to add them from the beginning.

@nissansz
Copy link
Author

nissansz commented Nov 1, 2024

When I add 20% hardcases from the beginning, the acc. is very slow to improve.
After removing them, speed seems a little faster.

Another question is: under ResNet34, English letters in some images may be easily mis-recognized.
Is it due to font different? Will this new yml improve English recognition?

Below is current progress.

[2024/11/01 13:15:22] ppocr INFO: cur metric, acc: 0.0780839889889502, norm_edit_dis: 0.39439033382906086, fps: 43.12335596370007
[2024/11/01 13:15:24] ppocr INFO: save model in C:/Users/Administrator/Desktop/svtr2\latest
[2024/11/01 13:15:28] ppocr INFO: save best model is to C:/Users/Administrator/Desktop/svtr2\best_accuracy
[2024/11/01 13:15:28] ppocr INFO: best metric, acc: 0.0780839889889502, is_float16: False, norm_edit_dis: 0.39439033382906086, fps: 43.12335596370007, best_epoch: 1

@Topdu
Copy link
Owner

Topdu commented Nov 1, 2024

That could be because it's the amount of data that's too small, causing the training not to converge. In general, training a model with a respectable accuracy requires 50w+ of data and training with 200 epoch using data augmentation.

@nissansz
Copy link
Author

nissansz commented Nov 1, 2024

Ok.
I use several font, random text line, 500w lines to train.
One epoch takes about 4 days

@nissansz
Copy link
Author

nissansz commented Nov 1, 2024

if 50w+ of data -> training with 200 epoch

So, if 500W+data->20 epoch?

@nissansz
Copy link
Author

nissansz commented Nov 1, 2024

Is it useful to increase accuracy if I rotate each character by a random angle on each image?

image

@Topdu
Copy link
Owner

Topdu commented Nov 1, 2024

It is not clear if this is useful and ablation experiments need to be performed to validate it.

@nissansz
Copy link
Author

nissansz commented Nov 1, 2024

How to perform ablation experiments ?

if 50w+ of data -> training with 200 epoch

So, if 500W+data->20 epoch?

@nissansz
Copy link
Author

nissansz commented Nov 2, 2024

acc seems half of norm_edit_dis. Is this normal?

[2024/11/03 06:16:08] ppocr INFO: best metric, acc: 0.4002624645652069, is_float16: False, norm_edit_dis: 0.7241033262538159, fps: 44.641736053984985, best_epoch: 1
[2024/11/03 06:16:43] ppocr INFO: epoch: [1/2], global_step: 236100, lr: 0.000050, CTCLoss: 29.011494, NRTRLoss: 4.175393, loss: 33.616753, avg_reader_cost: 0.00192 s, avg_batch_cost: 0.26206 s, avg_samples: 5.82, ips: 22.20878 samples/s, eta: 5 days, 1:51:47, max_mem_reserved: 3474 MB, max_mem_allocated: 2774 MB
[2024/11/03 06:17:17] ppocr INFO: epoch: [1/2], global_step: 236200, lr: 0.000050, CTCLoss: 23.255911, NRTRLoss: 3.637556, loss: 27.158762, avg_reader_cost: 0.00230 s, avg_batch_cost: 0.25642 s, avg_samples: 5.62, ips: 21.91724 samples/s, eta: 5 days, 1:51:15, max_mem_reserved: 3474 MB, max_mem_allocated: 2774 MB
[2024/11/03 06:17:51] ppocr INFO: epoch: [1/2], global_step: 236300, lr: 0.000050, CTCLoss: 24.377645, NRTRLoss: 3.557321, loss: 27.860359, avg_reader_cost: 0.00192 s, avg_batch_cost: 0.25757 s, avg_samples: 5.63, ips: 21.85830 samples/s, eta: 5 days, 1:50:43, max_mem_reserved: 3474 MB, max_mem_allocated: 2774 MB

@nissansz
Copy link
Author

nissansz commented Nov 3, 2024

There are thousands of fonts, is it necessary to include all fonts in model?

@nissansz
Copy link
Author

nissansz commented Nov 3, 2024

Training speed seems similar to ResNet34.

[2024/11/03 17:51:18] ppocr INFO: best metric, acc: 0.4960629888709778, is_float16: False, norm_edit_dis: 0.783771419196146, fps: 43.44646117132341, best_epoch: 1
[2024/11/03 17:51:53] ppocr INFO: epoch: [1/2], global_step: 64100, lr: 0.000014, CTCLoss: 22.398418, NRTRLoss: 3.633534, loss: 26.019617, avg_reader_cost: 0.00207 s, avg_batch_cost: 0.27079 s, avg_samples: 5.89, ips: 21.75136 samples/s, eta: 5 days, 15:46:07, max_mem_reserved: 3476 MB, max_mem_allocated: 2774 MB
[2024/11/03 17:52:28] ppocr INFO: epoch: [1/2], global_step: 64200, lr: 0.000014, CTCLoss: 17.576801, NRTRLoss: 3.051838, loss: 20.692932, avg_reader_cost: 0.00200 s, avg_batch_cost: 0.26254 s, avg_samples: 5.66, ips: 21.55867 samples/s, eta: 5 days, 15:45:26, max_mem_reserved: 3476 MB, max_mem_allocated: 2774 MB
[2024/11/03 17:53:03] ppocr INFO: epoch: [1/2], global_step: 64300, lr: 0.000014, CTCLoss: 23.985691, NRTRLoss: 3.552601, loss: 27.266720, avg_reader_cost: 0.00209 s, avg_batch_cost: 0.27263 s, avg_samples: 5.93, ips: 21.75114 samples/s, eta: 5 days, 15:45:13, max_mem_reserved: 3476 MB, max_mem_allocated: 2774 MB
[2024/11/03 17:53:39] ppocr INFO: epoch: [1/2], global_step: 64400, lr: 0.000014, CTCLoss: 19.979275, NRTRLoss: 3.278076, loss: 23.216358, avg_reader_cost: 0.00218 s, avg_batch_cost: 0.26658 s, avg_samples: 6.14, ips: 23.03255 samples/s, eta: 5 days, 15:44:44, max_mem_reserved: 3476 MB, max_mem_allocated: 2774 MB
[2024/11/03 17:54:14] ppocr INFO: epoch: [1/2], global_step: 64500, lr: 0.000014, CTCLoss: 22.323454, NRTRLoss: 3.573228, loss: 25.727798, avg_reader_cost: 0.00210 s, avg_batch_cost: 0.27141 s, avg_samples: 5.45, ips: 20.08047 samples/s, eta: 5 days, 15:44:28, max_mem_reserved: 3476 MB, max_mem_allocated: 2774 MB

@nissansz
Copy link
Author

nissansz commented Nov 4, 2024

When loading pretrained model, is it necessary to specify a learning rate?
What is the learning rate to specify?

@nissansz
Copy link
Author

The computer automatically restarts during eval after training, what is the possible reason? any solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants