Producer Platform - Don't require the user to remain online during all the duration of the upload of images

[teolemon]

Ask: Don't require the user to remain online during all the duration of the upload of images on the manufacturer platform.
Rationale: The design choice of using Javascript is questionable. It stalls on large uploads, maxxing out CPU and Memory, even on brand new upscale computers (the one corporates don't have), and requires the user to stay connected for 10 hours +, with no resume solution.

• A proposed solution would be to upload all the files in parallel or to upload a complete ZIP archive.
• Having the possibility to pre-upload via FTP for power users would be nice

Part of

• 🏭 Producers platform (Tracker issue) #5521

[teolemon]

Case in point, after checking, the whole image folder is 1,1GB unzipped, something that uploads quickly on a modern connexion and could be handled afterwards by the server, and then the user could be mailed once it's ready for review.
I've been at it since 2PM yesterday. File by file upload is flawed.

[hangy]

The design choice of using Javascript is questionable. It stalls on large uploads, maxxing out CPU and Memory, even on brand new upscale computers (the one corporates don't have)

What do we do during the upload? Maxing out memory and CPU for a JPEG upload sounds weird. Any idea what the bottleneck is? I haven't used/seen the 'producer' platform, yet. Do we have any browser benchmarks/traces?

and requires the user to stay connected for 10 hours +, with no resume solution.

How many images (at which average size) are we talking about? At 10 Mbit/s upload speed, you could theoretically upload up to 45 GB.

A proposed solution would be to upload all the files in parallel

If the PC is at 100% utilisation, parallel upload won't help.

or to upload a complete ZIP archive.

That could work, unless we do any preprocessing in the browser. Again the file size would be of interest. HTML uploads can be made resumable, with the help of some server adjustments.

Having the possibility to pre-upload via FTP for power users would be nice

That would add another infrastructure component that needs to be maintained. Would we want individual accounts for every off user?

[stephanegigandet]

@hangy You can try the pro platform on https://world.pro.openfoodfacts.dev

This was to upload 10 000 images for a total size of 1.1 Gb.

This image upload form is clearly not targeted for that kind of use (but it's great that it does work anyway, even if it is slow and not convenient), it's expected that most users will send less than 1000 photos at once. We show them thumbnails so that they can check they are uploading the right images etc.

The form creates a table, with thumbnails of the images. Browsers are not good at handling tables with 10 000 rows, with 10 000 images...

We could create another feature to upload a zip file with tons of images, and process it afterwards.

[hangy]

The form creates a table, with thumbnails of the images. Browsers are not good at handling tables with 10 000 rows, with 10 000 images...

Not really, no. 😄