Improve stackdriver logs sink

[fbs]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

The stackdriver sink support only a small set of the fields Google supports. Useful fields like labels and the trace correlation fields are not supported

Attempted Solutions

Proposal

The sink currently seems to have two ways of setting fields:

• resource are fields in the sink config, the name is fixed but the value can be templated. To avoid having them also in the actual jsonPayload encoding.except_fields needs to be used. E.g.

# example data
 { "message": "some log message", "resources": { "namespace": aaa}}

# sink
gcp:
  type: gcp_stackdriver_logs
  inputs:
  - remap_gcp
  encoding:
    except_fields:
    - resource
  resource:
    type: k8s_container
    namespace: "{{ .resource.namespace }}"

• severity, which fetches and removes the value from the log message,

# example data
 { "message": "some log message", "severity": 1}

# sink
gcp:
  type: gcp_stackdriver_logs
  inputs:
  - remap_gcp
  encoding:
    except_fields:
    - resource
  severity_key: severity

Both approaches seem like they can work, but neither is ideal. I'd prefer it if the sink would be opinionated and performs fixed format/mapping from the input to LogEntry.
However doing that has 2 issues:

1. it introduces a 3rd way of working, but fixing the others would be a breaking change
2. its a breaking change, as it potentially removes fields that used to end up in the jsonPayload.

What way does best fit the Vector?

References

• Support user-defined labels in gcp_stackdriver_logs sink #8516

Version

v0.44.0

[fbs]

I'm interested in implementing this, but need some guidance on how the config should look (VRL&sink).