Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase BPF Dataplane Endpoints #9304

Closed
legolego621 opened this issue Oct 3, 2024 · 15 comments
Closed

Increase BPF Dataplane Endpoints #9304

legolego621 opened this issue Oct 3, 2024 · 15 comments
Assignees
@sridhartigera
Labels
area/bpf eBPF Dataplane issues kind/bug

Comments

@legolego621
Copy link

Context

After updating calico v3.28.2, we observe an increase in the felix_bpf_dataplane_endpoints metric from the exporter (Felix BPF Dataplane Endpoints).
Is this normal behavior? It only happened after the update.
The number of real endpoints resources in kubernetes has not been increased.
2024-10-03_13-15
The motivation for the update is to fix the bug #8897

These are the logs that are observed on calico-node. Constant updating of endpoints is done intentionally.

2024-10-03 10:23:16.196 [INFO][74] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"}
2024-10-03 10:23:16.196 [INFO][74] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"}
2024-10-03 10:23:16.196 [INFO][74] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"}
2024-10-03 10:23:16.201 [INFO][74] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-03 10:23:16.201 [INFO][74] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"} status="up"
2024-10-03 10:23:17.388 [INFO][74] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/newfmm-db-newfm-0, name=eth0)
2024-10-03 10:23:17.389 [INFO][74] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/newfmm-db-newfm-0" endpoint_id:"eth0" > endpoint:<state:"active" name:"cali5ab97634bcf" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.230/32" > 
2024-10-03 10:23:17.395 [INFO][74] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"}
2024-10-03 10:23:17.395 [INFO][74] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"}
2024-10-03 10:23:17.395 [INFO][74] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"}
2024-10-03 10:23:17.396 [INFO][74] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"}
2024-10-03 10:23:17.402 [INFO][74] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-03 10:23:17.402 [INFO][74] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"} status="up"
2024-10-03 10:23:18.318 [INFO][74] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/newfmm-db-newfm-0, name=eth0)
2024-10-03 10:23:18.318 [INFO][74] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/newfmm-db-newfm-0" endpoint_id:"eth0" > endpoint:<state:"active" name:"cali5ab97634bcf" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.230/32" > 
2024-10-03 10:23:18.325 [INFO][74] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"}
2024-10-03 10:23:18.325 [INFO][74] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"}
2024-10-03 10:23:18.325 [INFO][74] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"}
2024-10-03 10:23:18.326 [INFO][74] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"}
2024-10-03 10:23:18.331 [INFO][74] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-03 10:23:18.331 [INFO][74] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/newfmm-db-newfm-0", EndpointId:"eth0"} status="up"

Expected Behavior

Real display of the number of bpf dataplane endpoints.

Current Behavior

Increase bpf dataplane endpoints.

Possible Solution

Steps to Reproduce (for bugs)

Your Environment

  • Calico version: 3.28.2 on ebpf dataplane.
    Felix configuration
apiVersion: crd.projectcalico.org/v1
kind: FelixConfiguration
metadata:
  annotations:
  name: default
  resourceVersion: "283159629"
  uid: 222bf9d5-6012-4cb4-9d49-2b3575416a14
spec:
  bpfEnabled: true
  bpfExternalServiceMode: Tunnel
  bpfLogLevel: ""
  logSeverityScreen: Debug
  prometheusMetricsEnabled: true
  reportingInterval: 0s
  vxlanEnabled: true
  vxlanPort: 4799
  vxlanVNI: 4096
  • Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes v1.22.15
  • Operating System and version: Ubuntu 20.04.4 LTS, kernel: 5.15.0-67-generic, cri: containerd://1.5.5
@tomastigera tomastigera added kind/support area/bpf eBPF Dataplane issues labels Oct 3, 2024
@tomastigera
Copy link
Contributor

Does the count match the number of end points in your cluster? Is it ever growing?

@legolego621
Copy link
Author

@tomastigera Hello, the metric does not reflect the actual number of endpoints in the cluster. The metric is constantly growing, but the actual number of endpoints remains unchanged.

@tomastigera
Copy link
Contributor

tomastigera commented Oct 3, 2024 •
edited
Loading

Are endpoints being deleted and created? Or you have just the same endpoints all the time and those endpoints only receive updates?

We have reproduced a leak created by recent change, but some pods churn is necessary for that.

@legolego621
Copy link
Author

the list of endpoints does not change, only updates occur

@tomastigera
Copy link
Contributor

Would you be able to try this calico-node image thruby/node:3-28-leak-fix-1 ? It is 3.28 with this fix #9308 I still cannot see how the value can grow as it is the size of the list of all interfaces. So unless the interfaces change and we forget to remove some (that is what the fix addresses), this should be constant.

@legolego621
Copy link
Author

Would you be able to try this calico-node image thruby/node:3-28-leak-fix-1 ? It is 3.28 with this fix #9308 I still cannot see how the value can grow as it is the size of the list of all interfaces. So unless the interfaces change and we forget to remove some (that is what the fix addresses), this should be constant.

Hello, thanks, I am going to try it on Monday and get back with response

@legolego621
Copy link
Author

@tomastigera Hello!

Thank you for your time, I've been testing the new image for 5 hours now and it seems to solve the problem.
I want to wait 1 day for full confirmation.

I'll be back with the result tomorrow :)

calico

@tomastigera
Copy link
Contributor

@legolego621 tanks for testing 🤞 @sridhartigera thanks for the patch!

@legolego621
Copy link
Author

@tomastigera

Good day.
Unfortunately, the problem has not been solved. But I was able to compare the number of pods on the node and the felix_bpf_dataplane_endpoints metric.

When the number of pods increases, the felix_bpf_dataplane_endpoints metric increases, but when the number of pods decreases, the metric does not change its value, it seems that it works as a counter.

If you restart calico-node, the value becomes correct until the first increase in pods.

calico-debug

@sridhartigera
Copy link
Member

@legolego621 can you provide the felix debug logs when pods are added and removed.

@legolego621
Copy link
Author

@sridhartigera

I am attaching a log at the moment of creation and deletion of pods logical-backup-newfmm-db-newfm-manual*.

$ kubectl -n kube-system logs --tail 1000 calico-node-ntn5l --all-containers=true -f | grep logical-backup-newfmm-db-newfm-manual

2024-10-08 06:42:31.558 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4, name=eth0)
2024-10-08 06:42:31.558 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4" endpoint_id:"eth0" > endpoint:<state:"active" name:"caliab9cc0f9188" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.238/32" > 
2024-10-08 06:42:31.559 [INFO][60] felix/bpf_ep_mgr.go 2059: Adding workload interface to iptables allow list. id=orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4" endpoint_id:"eth0"  iface="caliab9cc0f9188"
2024-10-08 06:42:31.559 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:31.559 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:31.559 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:31.559 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:31.570 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:42:31.570 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} status="down"
2024-10-08 06:42:31.642 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:31.642 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:31.648 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:42:31.648 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} status="down"
2024-10-08 06:42:31.730 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:31.730 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:31.736 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:42:31.736 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} status="up"
2024-10-08 06:42:32.809 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4, name=eth0)
2024-10-08 06:42:32.809 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4" endpoint_id:"eth0" > endpoint:<state:"active" name:"caliab9cc0f9188" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.238/32" > 
2024-10-08 06:42:32.819 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:32.819 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:32.820 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:32.820 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:32.827 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:42:32.827 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} status="up"
2024-10-08 06:42:33.833 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4, name=eth0)
2024-10-08 06:42:33.833 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4" endpoint_id:"eth0" > endpoint:<state:"active" name:"caliab9cc0f9188" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.238/32" > 
2024-10-08 06:42:33.839 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:33.839 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:33.840 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:33.841 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:33.846 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:42:33.846 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} status="up"
2024-10-08 06:42:34.908 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4, name=eth0)
2024-10-08 06:42:34.908 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4" endpoint_id:"eth0" > endpoint:<state:"active" name:"caliab9cc0f9188" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.238/32" > 
2024-10-08 06:42:34.964 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:34.964 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:34.964 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=true known=true operUp=true status="error" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:34.965 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="error" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:34.970 [INFO][60] felix/status_combiner.go 75: Endpoint is in error, will report error id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} ipVersion=0x4 status="error"
2024-10-08 06:42:34.970 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} status="error"
2024-10-08 06:42:35.011 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:35.011 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:35.017 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:42:35.017 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} status="down"
2024-10-08 06:42:35.060 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:35.060 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:35.069 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:42:35.069 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"} status="down"
2024-10-08 06:42:35.822 [INFO][60] felix/calc_graph.go 505: Local endpoint deleted id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4, name=eth0)
2024-10-08 06:42:35.822 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointRemove update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4" endpoint_id:"eth0" > 
2024-10-08 06:42:35.824 [INFO][60] felix/endpoint_mgr.go 792: Workload removed, deleting its chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:35.824 [INFO][60] felix/endpoint_mgr.go 654: Workload removed, deleting old state. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:35.824 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=false failed=false known=false operUp=false status="" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:35.824 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:35.839 [INFO][60] felix/status_combiner.go 86: Reporting endpoint removed. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-hpvg4", EndpointId:"eth0"}
2024-10-08 06:42:36.389 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5, name=eth0)
2024-10-08 06:42:36.389 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5" endpoint_id:"eth0" > endpoint:<state:"active" name:"calicbf47a25eb7" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.224/32" > 
2024-10-08 06:42:36.390 [INFO][60] felix/bpf_ep_mgr.go 2059: Adding workload interface to iptables allow list. id=orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5" endpoint_id:"eth0"  iface="calicbf47a25eb7"
2024-10-08 06:42:36.390 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.390 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.390 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.390 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.402 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:42:36.402 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} status="down"
2024-10-08 06:42:36.463 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.463 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.469 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:42:36.469 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} status="down"
2024-10-08 06:42:36.534 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.534 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.541 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:42:36.541 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} status="up"
2024-10-08 06:42:36.824 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5, name=eth0)
2024-10-08 06:42:36.824 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5" endpoint_id:"eth0" > endpoint:<state:"active" name:"calicbf47a25eb7" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.224/32" > 
2024-10-08 06:42:36.830 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.830 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.830 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.830 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:36.840 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:42:36.840 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} status="up"
2024-10-08 06:42:38.831 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5, name=eth0)
2024-10-08 06:42:38.831 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5" endpoint_id:"eth0" > endpoint:<state:"active" name:"calicbf47a25eb7" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.224/32" > 
2024-10-08 06:42:38.839 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:38.839 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:38.839 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:38.839 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:38.847 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:42:38.847 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} status="up"
2024-10-08 06:42:39.939 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5, name=eth0)
2024-10-08 06:42:39.939 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5" endpoint_id:"eth0" > endpoint:<state:"active" name:"calicbf47a25eb7" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.224/32" > 
2024-10-08 06:42:40.027 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.027 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.027 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=true known=true operUp=true status="error" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.028 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="error" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.035 [INFO][60] felix/status_combiner.go 75: Endpoint is in error, will report error id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} ipVersion=0x4 status="error"
2024-10-08 06:42:40.035 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} status="error"
2024-10-08 06:42:40.045 [INFO][60] felix/bpf_ep_mgr.go 2059: Adding workload interface to iptables allow list. id=orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5" endpoint_id:"eth0"  iface="calicbf47a25eb7"
2024-10-08 06:42:40.045 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.045 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.053 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:42:40.053 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} status="down"
2024-10-08 06:42:40.055 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.055 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.062 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:42:40.062 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"} status="down"
2024-10-08 06:42:40.843 [INFO][60] felix/calc_graph.go 505: Local endpoint deleted id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5, name=eth0)
2024-10-08 06:42:40.843 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointRemove update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5" endpoint_id:"eth0" > 
2024-10-08 06:42:40.845 [INFO][60] felix/endpoint_mgr.go 792: Workload removed, deleting its chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.845 [INFO][60] felix/endpoint_mgr.go 654: Workload removed, deleting old state. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.845 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=false failed=false known=false operUp=false status="" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.845 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:40.855 [INFO][60] felix/status_combiner.go 86: Reporting endpoint removed. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-444s5", EndpointId:"eth0"}
2024-10-08 06:42:46.414 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd, name=eth0)
2024-10-08 06:42:46.414 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd" endpoint_id:"eth0" > endpoint:<state:"active" name:"cali63f1de1eb26" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.195/32" > 
2024-10-08 06:42:46.415 [INFO][60] felix/bpf_ep_mgr.go 2059: Adding workload interface to iptables allow list. id=orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd" endpoint_id:"eth0"  iface="cali63f1de1eb26"
2024-10-08 06:42:46.415 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:46.415 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:46.415 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:46.415 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:46.424 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:42:46.424 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} status="down"
2024-10-08 06:42:46.566 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:46.566 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:46.572 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:42:46.572 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} status="up"
2024-10-08 06:42:46.855 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd, name=eth0)
2024-10-08 06:42:46.856 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd" endpoint_id:"eth0" > endpoint:<state:"active" name:"cali63f1de1eb26" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.195/32" > 
2024-10-08 06:42:46.864 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:46.864 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:46.865 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:46.865 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:46.873 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:42:46.873 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} status="up"
2024-10-08 06:42:48.870 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd, name=eth0)
2024-10-08 06:42:48.870 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd" endpoint_id:"eth0" > endpoint:<state:"active" name:"cali63f1de1eb26" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.195/32" > 
2024-10-08 06:42:48.876 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:48.876 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:48.876 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:48.876 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:48.882 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:42:48.882 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} status="up"
2024-10-08 06:42:49.956 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd, name=eth0)
2024-10-08 06:42:49.957 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd" endpoint_id:"eth0" > endpoint:<state:"active" name:"cali63f1de1eb26" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.195/32" > 
2024-10-08 06:42:50.029 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.029 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.029 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=true known=true operUp=true status="error" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.029 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="error" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.038 [INFO][60] felix/status_combiner.go 75: Endpoint is in error, will report error id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} ipVersion=0x4 status="error"
2024-10-08 06:42:50.038 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} status="error"
2024-10-08 06:42:50.061 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.061 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.067 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:42:50.067 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} status="down"
2024-10-08 06:42:50.124 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.124 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.131 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:42:50.131 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"} status="down"
2024-10-08 06:42:50.874 [INFO][60] felix/calc_graph.go 505: Local endpoint deleted id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd, name=eth0)
2024-10-08 06:42:50.875 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointRemove update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd" endpoint_id:"eth0" > 
2024-10-08 06:42:50.876 [INFO][60] felix/endpoint_mgr.go 792: Workload removed, deleting its chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.877 [INFO][60] felix/endpoint_mgr.go 654: Workload removed, deleting old state. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.877 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=false failed=false known=false operUp=false status="" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.877 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:42:50.886 [INFO][60] felix/status_combiner.go 86: Reporting endpoint removed. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-t45xd", EndpointId:"eth0"}
2024-10-08 06:43:06.413 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p, name=eth0)
2024-10-08 06:43:06.413 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p" endpoint_id:"eth0" > endpoint:<state:"active" name:"calif94a797649c" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.221/32" > 
2024-10-08 06:43:06.414 [INFO][60] felix/bpf_ep_mgr.go 2059: Adding workload interface to iptables allow list. id=orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p" endpoint_id:"eth0"  iface="calif94a797649c"
2024-10-08 06:43:06.414 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:06.414 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:06.414 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:06.414 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:06.422 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:43:06.423 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} status="down"
2024-10-08 06:43:06.562 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:06.562 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:06.569 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:43:06.569 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} status="up"
2024-10-08 06:43:06.913 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p, name=eth0)
2024-10-08 06:43:06.913 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p" endpoint_id:"eth0" > endpoint:<state:"active" name:"calif94a797649c" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.221/32" > 
2024-10-08 06:43:06.920 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:06.920 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:06.920 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:06.920 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:06.928 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:43:06.928 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} status="up"
2024-10-08 06:43:08.931 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p, name=eth0)
2024-10-08 06:43:08.931 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p" endpoint_id:"eth0" > endpoint:<state:"active" name:"calif94a797649c" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.221/32" > 
2024-10-08 06:43:08.938 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:08.938 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:08.939 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:08.939 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:08.946 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:43:08.946 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} status="up"
2024-10-08 06:43:10.027 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p, name=eth0)
2024-10-08 06:43:10.027 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p" endpoint_id:"eth0" > endpoint:<state:"active" name:"calif94a797649c" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.221/32" > 
2024-10-08 06:43:10.080 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.080 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.080 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=true known=true operUp=true status="error" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.080 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="error" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.086 [INFO][60] felix/status_combiner.go 75: Endpoint is in error, will report error id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} ipVersion=0x4 status="error"
2024-10-08 06:43:10.086 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} status="error"
2024-10-08 06:43:10.130 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.130 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.137 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:43:10.137 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} status="down"
2024-10-08 06:43:10.175 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.175 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.181 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:43:10.181 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"} status="down"
2024-10-08 06:43:10.936 [INFO][60] felix/calc_graph.go 505: Local endpoint deleted id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p, name=eth0)
2024-10-08 06:43:10.937 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointRemove update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p" endpoint_id:"eth0" > 
2024-10-08 06:43:10.940 [INFO][60] felix/endpoint_mgr.go 792: Workload removed, deleting its chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.940 [INFO][60] felix/endpoint_mgr.go 654: Workload removed, deleting old state. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.940 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=false failed=false known=false operUp=false status="" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.940 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:10.949 [INFO][60] felix/status_combiner.go 86: Reporting endpoint removed. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-6nx9p", EndpointId:"eth0"}
2024-10-08 06:43:46.481 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td, name=eth0)
2024-10-08 06:43:46.481 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td" endpoint_id:"eth0" > endpoint:<state:"active" name:"calie7e7aa36267" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.209/32" > 
2024-10-08 06:43:46.482 [INFO][60] felix/bpf_ep_mgr.go 2059: Adding workload interface to iptables allow list. id=orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td" endpoint_id:"eth0"  iface="calie7e7aa36267"
2024-10-08 06:43:46.482 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:46.482 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:46.482 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:46.482 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:46.491 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:43:46.491 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} status="down"
2024-10-08 06:43:46.564 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:46.564 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:46.574 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:43:46.574 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} status="down"
2024-10-08 06:43:46.634 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:46.634 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:46.640 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:43:46.640 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} status="up"
2024-10-08 06:43:47.030 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td, name=eth0)
2024-10-08 06:43:47.031 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td" endpoint_id:"eth0" > endpoint:<state:"active" name:"calie7e7aa36267" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.209/32" > 
2024-10-08 06:43:47.037 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:47.037 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:47.037 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:47.037 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:47.045 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:43:47.045 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} status="up"
2024-10-08 06:43:49.046 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td, name=eth0)
2024-10-08 06:43:49.055 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td" endpoint_id:"eth0" > endpoint:<state:"active" name:"calie7e7aa36267" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.209/32" > 
2024-10-08 06:43:49.063 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:49.064 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:49.064 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:49.064 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:49.072 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:43:49.072 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} status="up"
2024-10-08 06:43:50.136 [INFO][60] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td, name=eth0)
2024-10-08 06:43:50.136 [INFO][60] felix/int_dataplane.go 1916: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td" endpoint_id:"eth0" > endpoint:<state:"active" name:"calie7e7aa36267" profile_ids:"kns.test-postgres-instance" profile_ids:"ksa.test-postgres-instance.postgres-pod" ipv4_nets:"10.222.56.209/32" > 
2024-10-08 06:43:50.141 [INFO][60] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:50.141 [INFO][60] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:50.142 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:50.142 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:50.207 [INFO][60] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} ipVersion=0x4 status="up"
2024-10-08 06:43:50.207 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} status="up"
2024-10-08 06:43:50.244 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:50.244 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:50.250 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:43:50.250 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} status="down"
2024-10-08 06:43:50.308 [INFO][60] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:50.308 [INFO][60] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"}
2024-10-08 06:43:50.314 [INFO][60] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} ipVersion=0x4 status="down"
2024-10-08 06:43:50.314 [INFO][60] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td", EndpointId:"eth0"} status="down"
2024-10-08 06:43:51.047 [INFO][60] felix/calc_graph.go 505: Local endpoint deleted id=WorkloadEndpoint(node=group1-65f98cfcb6xx8nzd-zgw7b, orchestrator=k8s, workload=test-postgres-instance/logical-backup-newfmm-db-newfm-manual-9c5td, name

calico-debug

@sridhartigera
Copy link
Member

@legolego621 we need the logs with debug turned on. logSeverityScreen in felix configuration must be set to debug.

@sridhartigera
Copy link
Member

sridhartigera commented Oct 8, 2024 •
edited by tomastigera
Loading

@legolego621 Would you be able to test with this image? sridhartigera/node:fix-bpf-eps. This is based out of 3.28. nIt carries an additional fix #9316

@legolego621
Copy link
Author

@sridhartigera

Hi, I tried the image sridhartigera/node:fix-bpf-eps and it seems to solve the issue.

calico-debug

I'm attaching debug logs during pod creation and pod deletion just in case.
nodename: group1-65f98cfcb6xx8nzd-zgw7b
pods_prefix: logical-backup-newfmm-db-newfm-manual

time creating pods ~ 04.41
time deleting pods ~ 04.45

logs.zip

@sridhartigera
Copy link
Member

Thanks for testing. The fix is merged into 3.28 branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sridhartigera sridhartigera

Labels
area/bpf eBPF Dataplane issues kind/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@legolego621 @tomastigera @sridhartigera