Code to Speed Up Image Generation Using Parallelization

[amaank123456]

Dear authors,

I recently wrote a script to parallelize the image generation process using Python's multiprocessing package, speeding it up significantly. To illustrate, I tested the speed of the new script (dubbed gen_ecg_images_from_data_batch_parallel.py) compared to the old one (dubbed gen_ecg_images_from_data_batch.py) to generate 293 images with augmentations.
The code for gen_ecg_images_from_data_batch_parallel.py is similar to its predecessor with the exception of the parallel processing code, and I included an argument called cpu_count to allow the user to delegate how many cores they want to use to generate the data (the default is the maximum number of cores available).

This was the hardware setup for the experiments:
• Lenovo ThinkPad P16 Gen 2
• Intel Core i9-13950HX processor
• 32 CPUs
• 32 GB of RAM

This was the software setup for the experiments:
• OS: Windows 11 Pro
• Environment: Windows/WSL2 with Ubuntu 24.04 (tested in Windows and Linux environments)
• Python version: 3.9

I tested these scripts within a Windows environment on 3 separate laptops of the same kind and within a Linux environment on 1 laptop using WSL2 (OS: Ubuntu 24.04 LTS). One last thing is that I only used 16 out of the 32 CPUs available when performing the parallelized data generation.

Here are the execution times:
• gen_ecg_images_from_data_batch.py: Approximately 19 minutes
• gen_ecg_images_from_data_batch_parallel.py: Approximately 2 minutes

As you can see, the parallel data generation process was approximately 89% faster than the original process!

One limitation of this is that images with handwritten text cannot be generated due to memory issues loading the spacy model. Thus, I modified the logic in some of the existing scripts (gen_ecg_images_from_data_batch.py and gen_ecg_images_from_data.py) to account for this so that if someone wants to generate images with handwritten text, they can run the original scripts without any issues.

This experiment can be reproduced by setting up a conda environment using the environment_droplet.yml file which has been updated and following the instructions within image_generation_experiment_instructions.md.

Please let me know if you have any comments or questions, and I hope this can help speed up any processing!