loki_out: add stuctured_metadata_map_keys

[0x006EA1E5]

Resolves #9463

• Adds stuctured_metadata_map_keys config to dynamically populate stuctured_metadata from a map

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

fluent/fluent-bit-docs#1527

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[edsiper]

flb_ra_* API do not use msgpack-c return codes like MSGPACK_UNPACK_CONTINUE, need to check the return values and adjust it

[0x006EA1E5]

I'm a bit confused as to the meaning of the return code from flb_ra_get_kv_pair. I see from the source it is commented that this should

Returns FLB_TRUE if the pattern matched a kv pair, otherwise it returns FLB_FALSE

Where FLB_TRUE is 1 and FLB_FALSE is 0

However, actually reading the code, if this doesn't hit the FLB_FALSE code path, it actually just returns whatever the following call to flb_ra_key_value_get returns, and function flb_ra_key_value_get actually seems to return 0 for okay, and -1 for not okay..

It looks like we are getting return codes as follows:

• Valid RA, normal happy path - this will delegate to flb_ra_key_value_get and return 0
• Syntactically invalid RA key, such as $$ - this will return FLB_FALSE (also 0) via get_ra_parser.
  • This PR then proceeds assuming things are okay, then to check out_val and finally log No valid map data found for key $ (test case flb_test_structured_metadata_map_invalid_ra_key)
• Syntactically valid RA key, such as $missing_map which doesn't reference anything - this will delegate to flb_ra_key_value_get and return -1 (test case structured_metadata_map_single_missing_map)

It seems like the main option here to to test for not equal to -1, and then check out_val.

Otherwise, we have to change how flb_ra_get_kv_pair works to agree with its comment, which I assume would break some important things elsewhere?

[0x006EA1E5]

Do the comment start positions need to be aligned, or is it okay for this one to stick out further?

[patrick-stephens]

I'd say line them up if you can

[0x006EA1E5]

This pushed a couple of the lines over 90 characters long.
I could align the righthand side instead, like this?

    int ra_used;                           /* number of record accessor label keys */
    struct flb_record_accessor *ra_k8s;              /* kubernetes record accessor */
    struct mk_list labels_list;                       /* list of flb_loki_kv nodes */
    struct mk_list structured_metadata_list;          /* list of flb_loki_kv nodes */
    struct mk_list structured_metadata_map_keys_list; /* list of flb_loki_kv nodes */
    struct mk_list remove_keys_derived;              /* remove_keys with label RAs */
    struct flb_mp_accessor *remove_mpa;      /* remove_keys multi-pattern accessor */
    struct flb_record_accessor *ra_tenant_id_key;         /* dynamic tenant id key */

[patrick-stephens]

seems reasonable to me