Form hazelcast cluster between pods across mulitple kubernetes namespace.

[p4pramod]

I have a requirement of forming a cluster across name kubernetes namespaces. I have to use the DNS based discovery using headless service. I have three app in three namespaces that needs to form the cluster.

I created a headless service ( HeadLess-Srv-1 ) in App-1 in Ns-1, and using the following config in application App-1 in Ns-1 and application App-2 in Ns-2 I can see that the cluster is getting formed;

Config App-1 in Ns-1 :

hazelcast:
network:
join:
multicast:
enabled: false
kubernetes:
enabled: true
namespace: NS-1
service-name: HeadLess-Srv-1

Config in App-2 in Ns-2 :

hazelcast:
network:
join:
multicast:
enabled: false
kubernetes:
enabled: true
namespace: NS-1
service-name: HeadLess-Srv-1

I can see th cluster getting formed. The problem arises when App-3 in Ns-3 tries to use the same config and I cand see that the cluster showing three apps. But after while App-2 crashes. Seems it is not able to reach out to App-2 in Ns-2.

But theoratically if A is connected to B and A is connected to C. Then B and C can also be discovered by each other. Seems it is not able to do that.

Is there any thing you can suggest. How to solve this.?

Thanks,
Pramod