Bloblang interpolation with custom binary plugin

[Arenash13]

Hello,

I have a case where I need to assign messages to a specific Kafka topic in the same style as https://www.benthos.dev/docs/configuration/interpolation#bloblang-queries

output:
  kafka:
    addresses: [ "TODO:6379" ]
    topic: 'dope-${! json("topic") }'

The only difference is that these are binary payloads where I search for specific bytes to determine the route.
I started to write a bloblang plugin but I'm not sure if it works out of the box and if I have to do anything to keep the original payload "untouched".

Is there a specific example I can rely on with similar constraints ?

Thanks in advance

[mihaitodor]

Hey @Arenash13

I started to write a bloblang plugin but I'm not sure if it works out of the box and if I have to do anything to keep the original payload "untouched".

You can simply return the original message similar to the noop processor. See here.

However, depending on the binary format, bloblang should have the required utilities you'd need to do basic binary parsing and comparisons. Have a look at the content() function and then you can use something like the encode() method to get a string representation or slice() to get byte chunks or individual bytes.

PS: Converted this to a discussion as per #2026.

[Arenash13]

Hi @mihaitodor,

I looked at the functions, unfortunately my use case contains more advanced binary logic (with ASN.1 constraints checks, individual values retrieval, etc.).
Moreover, I would like to keep this business logic in a dedicated module (also used by other components in my data pipeline, with dedicated unit tests,etc.) if possible.

Thus I followed the example of the noop processor. My question is : is there a way to save a value in the processor and call it in the yaml file (via variable, string interpolation,...) in an idiomatic way ? Or am I supposed to implement it differently?

Thanks again

[mihaitodor]

is there a way to save a value in the processor and call it in the yaml file

Sure! You can set it in the metadata (if you don't want to modify or overwrite the message). See here: https://pkg.go.dev/github.com/benthosdev/benthos/[email protected]/public/service#Message.MetaSetMut Then you can access it in the config using the metadata() function or the @ operator (see here). Note that the meta() function is still available, but it always returns strings instead of whatever type the original value had.

[Arenash13]

Thanks for your help @mihaitodor, I'll will implement it that way then