Inquiring About Supported Network Topologies in Submariner

[adibrastegarnia]

I am currently evaluating Submariner for networking across multiple Kubernetes clusters and am seeking clarification on the types of network topologies that Submariner supports.

From the documentation, it is clear that Submariner facilitates secure inter-cluster communication, but I would like to understand more about the specific topologies that can be implemented. My questions are as follows:

• What network topologies does Submariner support out of the box? Is it limited to a full mesh, or are there other options available?
• If alternative topologies are supported, such as a brokered or hub-and-spoke model where a central cluster is not in the data path, could you point me to the relevant documentation or examples for setting up such configurations?
• For scenarios involving a large number of clusters, what topology would you recommend to ensure scalability and manageability? Are there any best practices or patterns that Submariner recommends?
• In the default configuration, are IPsec tunnels between clusters pre-created, or are they created lazily on demand? Could you provide some insight into how tunnel creation is managed and whether this behavior can be configured?

[yboaron]

Short background on Submariner design,

Submariner implements two main functions:

1. L3 connectivity between clusters
2. Service discovery (project Lighthouse) for Kubernetes clusters connected via Submariner in multi-cluster environments. Lighthouse implements the Kubernetes Multi-Cluster Service APIs

Control plane:
Submariner Broker is used to facilitate the exchange of metadata information between the Gateway Engines deployed in the participant clusters.
The Gateway Engine components deployed in each participating cluster are configured with the information to securely connect to the broker cluster API.

The availability of the broker cluster does not affect the operation of the data plane in the participating clusters, meaning that the data plane will continue to route traffic using the last known information while the broker is unavailable.

However, during this time, the control plane components will not be able to publish new or updated information to other clusters and learn about new or updated information from other clusters. When the connection is reestablished to the broker, each component will automatically resynchronize its local information with the broker and update the data plane if necessary.

Data plane:
Submariner Gateway creates a tunnel for each remote cluster (full mesh) in the cluster set, the tunnel is created when the remote cluster joins the cluster set.

The tunnels are created using a pluggable cable driver, currently, IPSec(default), VxLAN and WireGuard are supported.

If you are interested in other cable driver, we always welcome contributions.
Cable driver addition should be smooth and simple, with Submariner supports pluggable cable drivers.

Active/Passive HA for Gateway engine is supported.

Additionally, Submariner periodically monitors the tunnels health and reflects this information into Lighthouse (for example: Lighthouse will not resolve remote service hostname to a remote cluster to which the tunnel is unhealthy).

Can you elaborate on your environment?
How many clusters are you going to have in clusterset?
Are you going to use multi-cluster service discovery (Lighthouse) ?
How many services do you plan to export?

[adibrastegarnia]

Thank you for your reply. Assuming we have a set of 1000 clusters, this would result in 1,000,000 (1000 * 1000) potential tunnels. We plan to use Lighthouse and GlobalNet for this setup. Could you provide any scalability metrics or figures that indicate the level of scalability we can anticipate from Submariner in such a scenario?

[tpantelis]

On Thu, Sep 5, 2024 at 11:24 AM Adib Rastegarnia ***@***.***> wrote: Thank you for your reply. Assuming we have a set of 1000 clusters, this would result in 1,000,000 (1000 * 1000) potential tunnels. We plan to use Lighthouse and GlobalNet for this setup. Could you provide any scalability metrics or figures that indicate the level of scalability we can anticipate from Submariner in such a scenario?

We currently don't have scalability metrics for that many clusters. If possible, you may want to consider creating smaller clustersets to reduce the number of connections between the clusters. However, keep in mind that each cluster can only be part of one clusterset. A clusterset basically equates to a broker namespace.

…

— Reply to this email directly, view it on GitHub <#3154 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADDICZB3TUDFHWBJTAWPHYDZVBZT3AVCNFSM6AAAAABNVHOJBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZSGAYDKOJSGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>