Support for CrossSubnet in Calico configuration

[zanetworker]

How to categorize this issue?

/kind enhancement
/priority normal
/area networking

What would you like to be added:
Currently, we are using IPIP all the time for GCP and AWS providers. Maybe we should consider using IPIP only for cross subnet network traffic.

Test Cases (Validate that this possible in the first place) enabling / disabling cross subnet:

• Old shoot -> new shoot with Cross Subnet (are the old route there? are se able to reach all new nodes and new pods from old nodes)
• Completely new shoots.
• When switching from CS to normal will calico add the routes back and will everything work as expected or do we need to do something (e.g., roll the nodes)?
• Check whether the IPPool IPIP configuration is automatically updated or do we have to manually pach it (if so we might aim for a short-term solution with init-containers and raise in issue on calico/node).
  • Related issue: https://github.com/projectcalico/node/issues/15

Expose the src/destination check configuration:

• Expose it in the MCM (raise a PR or create an issue).
• Set it when cross-subnet is enabled in the provider-aws extension (probably in the worker configuration, we need to access the settings from the cluster resource).
• When it is disabled...are the checks now working.

Some concerns:

• What happens when CS is disabled?
• Routes will only exist on new nodes probably.
• How do we enforce node rolling? this usually happens during the maintenance time-window.
• Maybe we don't need to roll the nodes, but adding a Daemonset that would delete the old routes would be enough.

Why is this needed:
Improve network performance.

[gardener-robot]

@zanetworker Label area/todo does not exist.

[gardener-robot]

@zanetworker Label area/todo does not exist.

[gardener-robot]

@zanetworker Label area/todo does not exist.