Components with multiple roles (component that is both a sink and a source)

[esensar]

Recently, my PR has been merged (#21348), which added a new type of enrichment table, which can also act as a sink. That has caused some changes in the topology, because enrichment tables were not meant to act as sinks and were supposed to be accessible only from VRL remap transform.

We have had a brief discussion on how to handle this in other ways in the PR (#21348 (comment)) - more granular approach to building the topology, giving each component parts that might make it a sink, source, transform, or other things that support each of these roles, turning all of them into generic nodes, which can then be combined into a topology in more ways, depending on what each of the components support.

Right now, I am maintaining a branch which also enables this memory enrichment table to act as a source. You can see the changes here: master...esensar:vector:feat/vrl-cache-as-source (it is not very clean, since I assumed that something like this would not be accepted as a contribution, at least not in the near future, but I might be wrong). This enables some interesting new ways to use this table and I assume it might make sense for many more components.

My question is, does this sound like a direction it makes sense for Vector to move in, or does it introduce too much complexity? Does the current state of that memory enrichment table as a source branch make any sense? Would it be acceptable to merge with some changes? If not, do you have a suggestion on how to provide similar functionality in an acceptable way (dumping data from the cache)?

[pront]

Hi @esensar, thank you for sharing this. I will definitely take a look at your branch and comment. I agree that having generic nodes in the topology opens up new capabilities and use cases. For example, we can inspect sink output. Now, how to do this refactoring elegantly is another big challenge :)

In the meantime, sharing this prior art in case it helps:

• Support dead letter queue on sinks #1772 (dead letter queue, chaining sinks)
• chore: RFC for handling discarded events #14708
• Various linked issues contain more context