Support NLB type of load balancer (AWS)

[pbochynski]

Description
Support NLB type of load balancer (AWS). At the beginning, it could be a configuration (or feature toggle). In the future maybe even a default option.

Reasons
The default LoadBalancer for the AWS cluster is ELB. There is a newer and more performant solution: NLB. The connectivity proxy team requested it as the latency for NLB is much lower than for the classic load balancer.

Ideas
I tried to change the LoadBalancer type, using the annotation in the ingress gateway service (https://cloud-provider-aws.sigs.k8s.io/service_controller/):

   service.beta.kubernetes.io/aws-load-balancer-type: nlb

The new LB was provisioned and used as external-ip in the ingress gateway service.
Of course it has to be added on the istio level:
https://istio.io/v1.14/blog/2018/aws-nlb/#generate-the-istio-manifest

[strekm]

we might want to consider setting NLB type default. Before making decision we need to check how this change will take effect on existing installations. obviously downtime and IP change needs to be avoided