Feature: Replace Nginx on LBs with native L4 LB solution

[bernardhalas]

Motivation

There are multiple issues with the way we're using nginx currently on the LBs, e.g. throughput problems, source IP replacement, etc. (you can find the history on the #claudie channel on Kubernetes slack). All of these are of course configurable, but for each change, we'd need to make a new release, which is taking too long. I'd like to see us using raw L4 load-balancing, as we don't need anything more or anything less.

Description

Replace nginx on the LBs with a native L4 LB solution like HAproxy. I'd also consider running the LB app containerized so that the deployment unit is self-contained.

Exit criteria

• Choose LB solution
• Consider proposing a way how to override the default configuration (links to Feature: Inject custom manifest configuration #1187)
• Build configuration for exposing metrics
• Implement the solution
• Test/verify upgrade path