No connectivity with AKS and Calico CNI

[ed-boykin]

Hey all, I'm trying to implement a POC with EgressGateway on Azure Kubernetes. I have built a cluster as a 'bring your own CNI' and installed Calico. I applied the patch to Calico to set chainInsertMode to Append.
I've verified connectivity inside and outside of the cluster.
I installed EgressGateway for Calico using the docs. I have setup an app to test the egress.
Before adding an egress policy, I verified I can curl to a test location outside the cluster, and it gives me the correct source IP which is the expected Node IP.
After I create my egress policy, my test pod can no longer curl to the test location. Removing the policy and I can connect again. I've provided some information that might help but I'm not sure where to go from here to diagnose my issue.

kubectl get nodes -o wide
NAME                                STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-agentpool-31736691-vmss000000   Ready    <none>   21h   v1.29.4   10.47.10.12   <none>        Ubuntu 22.04.5 LTS   5.15.0-1073-azure   containerd://1.7.20-1
aks-agentpool-31736691-vmss000001   Ready    <none>   21h   v1.29.4   10.47.10.13   <none>        Ubuntu 22.04.5 LTS   5.15.0-1073-azure   containerd://1.7.20-1
aks-agentpool-31736691-vmss000002   Ready    <none>   21h   v1.29.4   10.47.10.5    <none>        Ubuntu 22.04.5 LTS   5.15.0-1073-azure   containerd://1.7.20-1
aks-agentpool-31736691-vmss000003   Ready    <none>   21h   v1.29.4   10.47.10.4    <none>        Ubuntu 22.04.5 LTS   5.15.0-1073-azure   containerd://1.7.20-1
aks-agentpool-31736691-vmss000004   Ready    <none>   21h   v1.29.4   10.47.10.14   <none>        Ubuntu 22.04.5 LTS   5.15.0-1073-azure   containerd://1.7.20-1
aks-agentpool-31736691-vmss000005   Ready    <none>   21h   v1.29.4   10.47.10.11   <none>        Ubuntu 22.04.5 LTS   5.15.0-1073-azure   containerd://1.7.20-1

kubectl get egt -o wide
NAME TUNNELMAC TUNNELIPV4 TUNNELIPV6 MARK PHASE
aks-agentpool-31736691-vmss000000 66:fa:f6:23:10:c5 10.47.10.222 0x2648f281 Ready
aks-agentpool-31736691-vmss000001 66:c0:42:77:63:a0 10.47.10.212 0x26ab2146 Ready
aks-agentpool-31736691-vmss000002 66:35:9e:da:e8:02 10.47.10.221 0x265a159b Ready
aks-agentpool-31736691-vmss000003 66:b7:4d:32:f4:1c 10.47.10.217 0x26f4dd64 Ready
aks-agentpool-31736691-vmss000004 66:00:88:47:ed:4a 10.47.10.216 0x26a171c1 Ready
aks-agentpool-31736691-vmss000005 66:51:db:f4:62:3a 10.47.10.215 0x260c7edb Ready

kubectl get pods -o wide -A | grep ubuntu
NAMESPACE    NAME              READY   STATUS    RESTARTS      AGE     IP                NODE                                NOMINATED NODE   READINESS GATES
default      ubuntu            1/1     Running   0             8h      192.168.239.196   aks-agentpool-31736691-vmss000004   <none>           <none>

kubectl get egresspolicy -o yaml
apiVersion: v1
items:

• apiVersion: egressgateway.spidernet.io/v1beta1
  kind: EgressPolicy
  metadata:
  annotations:
  kubectl.kubernetes.io/last-applied-configuration: |
  {"apiVersion":"egressgateway.spidernet.io/v1beta1","kind":"EgressPolicy","metadata":{"annotations":{},"name":"test","namespace":"default"},"spec":{"appliedTo":{"podSelector":{"matchLabels":{"run":"ubuntu"}}},"egressGatewayName":"default"}}
  creationTimestamp: "2024-10-03T18:42:15Z"
  generation: 1
  name: test
  namespace: default
  resourceVersion: "363892"
  uid: 37d869e5-11ee-4494-9f37-94f20e607e30
  spec:
  appliedTo:
  podSelector:
  matchLabels:
  run: ubuntu
  egressGatewayName: default
  egressIP:
  allocatorPolicy: default
  useNodeIP: false
  status:
  eip:
  ipv4: 10.47.10.202
  node: aks-agentpool-31736691-vmss000002
  kind: List
  metadata:
  resourceVersion: ""

Please let me know what other information I can provide. I'm kind of stuck at this point. Thanks ahead of time