Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python Flask back-end: final file corrupted due to insufficient chunks testing to define upload completed #619

Open
eanon opened this issue Aug 18, 2024 · 0 comments

Comments

@eanon
Copy link

eanon commented Aug 18, 2024

Relying on the provided python back-end example as starting point for my own one, I discovered that sometimes the final file on the server side was corrupted (e.g., a chunk could have been merged twice). So, debugging through well-placed print's, I finally understood that talking about chunks which could be uploaded in parallel, checking the existence of the chunks file was not enough to indicate that the entire upload was achieved; some chunks could be there but still in progress.

So, I changed this code block:

    # check if the upload is complete
    chunk_paths = [os.path.join(temp_dir, get_chunk_name(resumableFilename, x)) for x in range(1, resumableTotalChunks+1)]
    upload_complete = all([os.path.exists(p) for p in chunk_paths])

    # combine all the chunks to create the final file
    if upload_complete:

By this one:

        # check if all chunks fully downloaded
        chunk_paths = [os.path.join(temp_dir, get_chunk_name(resumableFilename, x)) for x in range(1, resumableTotalChunks+1)]
        chunks_total = 0
        chunks_size = 0
        for chunk_path in chunk_paths:
            if not os.path.exists(chunk_path):
                break
            chunks_total += 1
            chunks_size += os.stat(chunk_path).st_size

        # combine all the chunks to create the final file
        if (chunks_total == resumableTotalChunks) and (chunks_size == resumableTotalSize):

This way, we ensure that reassembling the chunks is only operated when all the chunks are here and terminated for sure. Well... Sorry, I've no time to provide a pull request, but I wanted to let you know through this quick report (feel free to use it in your example if you want).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant