Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scanner support for zip archives containing multiple files #165

Open
maxtheaxe opened this issue Jan 20, 2025 · 0 comments
Open

Scanner support for zip archives containing multiple files #165

maxtheaxe opened this issue Jan 20, 2025 · 0 comments
Labels
enhancement New feature or request needs investigation It looks as though have all the information needed but investigation is required scanners Any tasks or issues relating specifically to scanners

Comments

@maxtheaxe
Copy link

Had the following exchange in the Redpanda Community Slack, where we (@mihaitodor, realistically) realized that the existing scanners don't support zip archives containing multiple files.

In case it helps, just to break down a use case: I'm grabbing a data dump a few times per day via SFTP that has a bunch of CSV files. It's pretty sizeable, so opening everything in memory (as @mihaitodor pointed out I can already do with the unarchive processor) is pretty expensive (memory-wise, I mean).

This is my first time using benthos/redpanda connect, but I'm assuming that this hypothetical scanner will need to either add some metadata to each line/message to indicate which file it's coming from or otherwise allow me to batch them in a way that I can tell which is which for subsequent processing.

If there's any way I can be of assistance, please let me know.

@mihaitodor mihaitodor added enhancement New feature or request needs investigation It looks as though have all the information needed but investigation is required scanners Any tasks or issues relating specifically to scanners labels Jan 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request needs investigation It looks as though have all the information needed but investigation is required scanners Any tasks or issues relating specifically to scanners
Projects
None yet
Development

No branches or pull requests

2 participants