Benthos Fails to Fully Process Environment Values Containing #

[pv123444]

Slack Reference:

Channel: Redpanda Community
Message Link: Slack Message

Issue:

We found that Benthos fails to fully process environment variable values containing the # character. When fetching such values, it stops reading at the # and ignores the remaining part of the value.

Example:
Given an environment variable set as:
MY_ENV_VAR=abc#def#ghi

Benthos only reads abc, and everything after the first # (def#ghi) is ignored.

Expected Behavior:
Benthos should read and process the entire value, abc#def#ghi.

Actual Behavior:
Benthos only reads abc, causing issues for workflows that need the full value.

[mihaitodor]

Hey @pv123444, can you please confirm that MY_ENV_VAR="abc#def#ghi" works for you? As mentioned on Slack, we can potentially look into adjusting the env file parser, but we need to know which standard to follow.

[pv123444]

It works for values with #, but when using strings with special characters like \n or \t, they are treated as escape characters and don't work as expected.

…

On Tue, Dec 24, 2024 at 6:31 PM Mihai Todor ***@***.***> wrote: Hey @pv123444 <https://github.com/pv123444>, can you please confirm that MY_ENV_VAR="abc#def#ghi" works for you? As mentioned on Slack, we can potentially look into adjusting the env file parser, but we need to know which standard to follow. — Reply to this email directly, view it on GitHub <#147 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BN6AYE6TWSLYM62W74I72CL2HFLMPAVCNFSM6AAAAABUEGLYVOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRRGEYDMNRVGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *Disclaimer:* The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. If you are not the intended recipient, please notify the sender at Crest Data Systems immediately and destroy all copies of this message and any attachments.

[mihaitodor]

I'm not sure I follow... Can you please provide an example? Also, as mentioned, if you believe we need to change something, please provide a reference to a detailed spec that we should follow.

I added the following in test.env:

FOOBAR="abc\tdef\nghi"

And then running this YAML:

input:
  generate:
    count: 1
    mapping: |
      root = "${FOOBAR}"

I get:

$ benthos run -e foo.env foo.yaml
INFO Running main config from specified file       @service=redpanda-connect benthos_version="" path=foo.yaml
INFO Listening for HTTP requests at: http://0.0.0.0:4195  @service=redpanda-connect
INFO Launching a Redpanda Connect instance, use CTRL+C to close  @service=redpanda-connect
INFO Input type generate is now active             @service=redpanda-connect label="" path=root.input
INFO Output type stdout is now active              @service=redpanda-connect label="" path=root.output
abc	def
ghi
INFO Pipeline has terminated. Shutting down the service  @service=redpanda-connect

Also, note that we do support raw strings as well via """...""". See an example here.

[pv123444]

If we support raw strings, that’s great! However, I’d like to know if there’s a way to store an environment variable without using string or raw string format, but still fetch the exact value as is.

For example, when I set the environment variable FOOBAR=abc\tdef\nghi, I’d like to retrieve( env("FOOBAR") ) it in its literal form with the escape sequences intact, as abc\tdef\nghi, without interpreting them as tab or newline characters.

Is there a specific method we can use to achieve this? If there’s a detailed specification or guide that outlines how to handle such cases, I’d greatly appreciate a reference to it.

[mihaitodor]

Setting FOOBAR=abc\tdef\nghi inside an env file works just fine. You can also do FOOBAR='abc\tdef\nghi' benthos run test.yaml and that should also work (note the ' instead of " to prevent the terminal from interpreting escape characters). If you really need to use env("FOOBAR") instead of ${FOOBAR} in your bloblang code, you might have to do env("FOOBAR").replace_all_many(["\\t", "\t", "\\n", "\n"]) (see the replace_all_many() method docs).

For #, the issue is defining what should the parser interpret as a comment. If the current implementation isn't suitable, then we'll need a reference spec to consult and determine if there's a way to adhere to it without breaking existing code.

[pv123444]

Hello @mihaitodor
We provide credentials through the UI and store them in AWS Secrets Manager. These credentials are then exported as raw strings using a command like: export variable_name="\"\"\" aws_secret_manager_variable_path \"\"\"" and saved in the environment variables. Benthos is expected to read the value from these environment variables, but it does not interpret the value correctly. For example, when we set a value like ##%'#Benthos##", we checked its behavior using logged output with env("variable_name"), which shows additional raw string characters, such as \"\\\"\\\"\\\"##%'#Benthos##\\\"'\\\"\\\"\\\"\". This indicates that Benthos is not properly parsing the value.

[mihaitodor]

@pv123444 the logging can be misleading since it needs to do all sorts of string escaping. This looks fine to me:

input:
  generate:
    count: 1
    mapping: root = env("FOO")

output:
  stdout: {}

$ export FOO="##%'#Databee##\""
$ rpk connect run test.yaml
...
##%'#Databee##"
INFO Pipeline has terminated. Shutting down the service  @service=redpanda-connect

With an env file, you'll have to use a raw string:

FOO="""##%'#Databee##" """

$ rpk connect run -e test.env test.yaml
....
##%'#Databee##" 
INFO Pipeline has terminated. Shutting down the service  @service=redpanda-connect

Not ideal if you have a " right at the end since you'll get an extra space after the password, but that feels like a niche edge case... We could look into handling this particular situation better if need be.

Have you considered base64-encoding these passwords first? It might save you a lot of headaches...