Investigation/design for network traffic restrictions in ambient mode

[mjnagel]

Related to #681

During initial investigations of istio ambient we discovered that traffic appears to primarily/exclusively flow through the ztunnel port. This seems to make some of our network policies not have an effect. We should ensure that we are able to provide the same restrictions around ingress/egress and make a design for how to do this with ambient.

Definition of done should be a design doc/proposal that ensures traffic lock down, considering these items in particular:

• Support for hybrid mesh (sidecar and ambient)
• Usage of Istio AuthorizationPolicies?
• Easy Migration/Upgrade Process

[noahpb]

I found this doc related to the topic:
https://istio.io/latest/docs/ambient/usage/networkpolicy/

Additionally, L4 AuthorizationPolicies do not currently apply to Waypoints.

[sgettys]

Looks like ambient should work with existing netpols with 2 caveats:

Once you have added applications to the ambient mesh, ambient’s secure L4 overlay will tunnel traffic between your pods over port 15008. Once secured traffic enters the target pod with a destination port of 15008, the traffic will be proxied back to the original destination port.

However, NetworkPolicy is enforced on the host, outside the pod. This means that if you have preexisting NetworkPolicy in place that, for example, will deny list inbound traffic to an ambient pod on every port but 443, you will have to add an exception to that NetworkPolicy for port 15008.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
spec:
  ingress:
  - ports:
    - port: 8080
      protocol: TCP
    - port: 15008
      protocol: TCP
  podSelector:
    matchLabels:
      app.kubernetes.io/name: my-app

And

In Istio ambient, this problem is solved by using a combination of iptables rules and source network address translation (SNAT) to rewrite only packets that provably originate from the local node with a fixed link-local IP, so that they can be explicitly ignored by Istio policy enforcement as unsecured health probe traffic. A link-local IP was chosen as the default since they are typically ignored for ingress-egress controls, and by IETF standard are not routable outside of the local subnetwork.

This behavior is transparently enabled when you add pods to the ambient mesh, and by default ambient uses the link-local address 169.254.7.127 to identify and correctly allow kubelet health probe packets.

However if your workload, namespace or cluster has a preexisting ingress or egress NetworkPolicy, depending on the CNI you are using, packets with this link-local address may be blocked by the explicit NetworkPolicy, which will cause your application pod health probes to begin failing when you add your pods to the ambient mesh.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-ingress-allow-kubelet-healthprobes
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: my-app
  ingress:
    - from:
      - ipBlock:
          cidr: 169.254.7.127/32