file input deletes empty files but doesn't actually process them

[ligfx]

For instance, try echo -n > inputs/emptyfile vs echo > inputs/newlinefile. The empty file will send the message Consuming from file 'inputs/emptyfile' to the debug logger and will delete the file, but won't actually run any processors or outputs.

Example connect config:

logger:
  level: DEBUG

input:
  file:
    delete_on_finish: true
    paths:
    - inputs/*

pipeline:
  processors:
  - log:
      message: "This won't appear for empty files!"

[mihaitodor]

Hey @ligfx 👋 The log line you're seeing is a debug level message. While we could improve that, I think the current behaviour (deleting empty files and not producing any messages) is problematic. I'm leaning towards treating this as a special case and actually emitting a single message with no contents, but with the standard file metadata attached, so users can decide what action to take when such files are picked up.

PS: Moving this to the benthos repo, since that's where the file input is implemented.