IPv4: Need to standardize how public IPv4 networking behaves by default #167

garloff · 2022-09-26T08:43:51Z

As a IaaS user (DevOps person), I need to make sure the way to get a public IPv4 address to a VM or Loadbalancer is standardized across SCS clouds.

Definition of Ready:

User Story is small enough to be finished within one sprint
User Story is clear and understood by the whole team
Acceptance criteria are defined
Acceptance criteria are clear and understood by the whole team

Definition of Done:

All acceptance criteria are met
Changes have been reviewed
CI tests have run successfully
Documentation has been updated
Release Notes have been updated

fkr · 2022-09-26T08:46:43Z

This is important, since we don't want to fall into this category:

https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/cloud/meta.py#L160

garloff · 2022-09-27T14:03:29Z

90+% of existing OpenStack clouds have an external virtual network (called external or ext01 or ext-net or internet or public or so) from which floating IP addresses can be allocated and which can be attached to fixed IP addresses from internal virtual networks.

garloff · 2022-09-27T14:09:22Z

So I favor the

* Cloud only has private tenant network provided by neutron and requires
   floating-ip for external routing (dreamhost, hp)

approach. (Maybe this is not used by 90+% of all OpenStack clouds?)

garloff · 2022-12-07T08:08:51Z

Plan to standardize IPv4 networking:

Created networks, subnets are private
Networks can be connected to other networks with a router
Router can have an external network (gateway) configured (opt-in!), then
- SNAT is enabled, so outgoing IPv4 works as soon as router has external gateway set (disabling connectivity in such setups via security groups, disabling the router's SNAT is unfortunately a privileged operation on most clouds)
- Floating IPs can be attached to existing private ports in subnets connected to a such connected router
  - Outgoing connections are now using this floating IP (instead of a shared SNAT IP)
  - Incoming connections to this floating IP reach the VM (subject to security groups of course) or the Loadbalancer
OPTIONAL: Additional networks (provider networks), which may provide access to provider-specific networks and possibly as well to the internet

garloff · 2022-12-07T08:21:39Z

Open question: Should the name of the external network be standardized? external (wave)? ext01 (pco)? internet (sxperts)? ext-net (cleura)? admin_external_net (otc) Ext-Net (ovh), extern (teuto)?
Or maybe at least a recommendation (SHOULD)?
And make it transparent in the self-description?
(Note: It is trivially discoverable for those with access: openstack network list --external. However, still adds complexity for terraform, ansible, heat automation.)

garloff · 2022-12-07T11:42:28Z

Sidenote: IPv6 connectivity is dealt with in a separate issue (#166).

fkr · 2023-06-14T13:37:24Z

@garloff @berendt is this something that should be done as part of VP04 (and the work of VP04 taken into consideration for that)?

garloff · 2023-10-17T10:55:27Z

I don't think that any work in VP04 will change the user-visible approach to networking in OpenStack.

berendt · 2023-10-17T10:56:51Z

I don't think that any work in VP04 will change the user-visible approach to networking in OpenStack.

I would agree with that

fkr · 2023-11-14T09:52:50Z

Open question: Should the name of the external network be standardized? external (wave)? ext01 (pco)? internet (sxperts)? ext-net (cleura)? admin_external_net (otc) Ext-Net (ovh), extern (teuto)? Or maybe at least a recommendation (SHOULD)? And make it transparent in the self-description? (Note: It is trivially discoverable for those with access: openstack network list --external. However, still adds complexity for terraform, ansible, heat automation.)

Not really sure wether agreeing on a standard name is helpful (especially since there are also clouds with more than one (1) external network. Furthermore this is - yet another topic - that will cause grief with existing clouds.
Much rather I'd like to see that we standardize the functional default behaviour - so that things behave the same way on scs compliant clouds.

fkr · 2023-11-14T09:53:13Z

Router can have an external network (gateway) configured (opt-in!), then

SNAT is enabled, so outgoing IPv4 works as soon as router has external gateway set (disabling connectivity in such setups via security groups, disabling the router's SNAT is unfortunately a privileged operation on most clouds)

Floating IPs can be attached to existing private ports in subnets connected to a such connected router

Outgoing connections are now using this floating IP (instead of a shared SNAT IP)

Incoming connections to this floating IP reach the VM (subject to security groups of course) or the Loadbalancer

OPTIONAL: Additional networks (provider networks), which may provide access to provider-specific networks and possibly as well to the internet

This seems to be a good starting point and could be drafted.

josephineSei · 2024-03-12T14:19:10Z

After looking through this issue and discussing with @markus-hentsch , I have identified more than one thing to standardize:

standardized network architecture / topology:
- this means, we want to have a picture about how network topology should look like
- e.g. always use private networks to attach VMs to, and always use a neutron router to gain access from the private network to an external network.
- open questions: is the "external" flag enough to mark provider networks?
- do we allow shared networks (between projects)? or do we just not care?
- how many external networks are okay? (e.g. one for IPv4, one for IPv6, and maybe one for a special VPN case?... )
- how to choose the fitting network.
if we have a coarse picture of the topology: standardized workflow / guideline to work with Floating IPs:
- necessary because of some strange behavior of Floating IPs: it is possible to steal a floating ip from a VM in the same project and that should be avoided.

I would add a separate issue for the second part, with a detailed description, of the behavior, if you want @fkr

josephineSei · 2024-03-13T08:59:28Z

I described the behavior in here:
https://input.scs.community/gRSKVYboQRebi62Lx_eSMA?both

josephineSei · 2024-03-19T08:50:36Z

I described the behavior in here: https://input.scs.community/gRSKVYboQRebi62Lx_eSMA?both

I edited some parts of the etherpad in preperation of the IaaS call on Wednesday 20th of March

josephineSei · 2024-03-27T13:12:05Z

We (@cah-patrickthiem , @anjastrunk , @josephineSei , @markus-hentsch , @kgube , @martinmo , @tonifinger , @mbuechse ) had a lengthy discussion about this issue and possible splitting some of the open questions mentioned in the IaaS-call today into different issues. That would mainly concern the loadbalancer. Some other issues for security groups, floating IPs, DNS, etc... already exist.

josephineSei · 2024-03-27T14:02:00Z

I wanted to look into the naming, when I stumbled upon this:

stack@devstack:~/devstack$ openstack network show public
+---------------------------+----------------------------------------------------------------------------+
| Field                     | Value                                                                      |
+---------------------------+----------------------------------------------------------------------------+
| admin_state_up            | UP                                                                         |
| availability_zone_hints   |                                                                            |
| availability_zones        |                                                                            |
| created_at                | 2024-01-24T16:12:31Z                                                       |
| description               |                                                                            |
| dns_domain                | None                                                                       |
| id                        | 73edb86b-d7ab-4db3-82b7-25fa8b012e40                                       |
| ipv4_address_scope        | None                                                                       |
| ipv6_address_scope        | None                                                                       |
| is_default                | True                                                                       |
| is_vlan_transparent       | None                                                                       |
| mtu                       | 1500                                                                       |
| name                      | public                                                                     |
| port_security_enabled     | True                                                                       |
| project_id                | 15f2ab0eaa5b4372b759bde609e86224                                           |
| provider:network_type     | flat                                                                       |
| provider:physical_network | public                                                                     |
| provider:segmentation_id  | None                                                                       |
| qos_policy_id             | None                                                                       |
| revision_number           | 3                                                                          |
| router:external           | External                                                                   |
| segments                  | None                                                                       |
| shared                    | False                                                                      |
| status                    | ACTIVE                                                                     |
| subnets                   | 3e0206bc-53c8-44ca-a0f1-2c2548bba766, 84dffd43-6d7f-4c2f-9180-8f0f0b83c9d4 |
| tags                      |                                                                            |
| tenant_id                 | 15f2ab0eaa5b4372b759bde609e86224                                           |
| updated_at                | 2024-01-24T16:12:52Z                                                       |
+---------------------------+----------------------------------------------------------------------------+
stack@devstack:~/devstack$ openstack subnet show 3e0206bc-53c8-44ca-a0f1-2c2548bba766
+----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
| allocation_pools     | 172.24.4.2-172.24.4.254              |
| cidr                 | 172.24.4.0/24                        |
| created_at           | 2024-01-24T16:12:38Z                 |
| description          |                                      |
| dns_nameservers      |                                      |
| dns_publish_fixed_ip | None                                 |
| enable_dhcp          | False                                |
| gateway_ip           | 172.24.4.1                           |
| host_routes          |                                      |
| id                   | 3e0206bc-53c8-44ca-a0f1-2c2548bba766 |
| ip_version           | 4                                    |
| ipv6_address_mode    | None                                 |
| ipv6_ra_mode         | None                                 |
| name                 | public-subnet                        |
| network_id           | 73edb86b-d7ab-4db3-82b7-25fa8b012e40 |
| project_id           | 15f2ab0eaa5b4372b759bde609e86224     |
| revision_number      | 0                                    |
| segment_id           | None                                 |
| service_types        |                                      |
| subnetpool_id        | None                                 |
| tags                 |                                      |
| updated_at           | 2024-01-24T16:12:38Z                 |
+----------------------+--------------------------------------+
stack@devstack:~/devstack$ openstack subnet show 84dffd43-6d7f-4c2f-9180-8f0f0b83c9d4
+----------------------+-------------------------------------------------------------------+
| Field                | Value                                                             |
+----------------------+-------------------------------------------------------------------+
| allocation_pools     | 2001:db8::1-2001:db8::1,2001:db8::3-2001:db8::ffff:ffff:ffff:ffff |
| cidr                 | 2001:db8::/64                                                     |
| created_at           | 2024-01-24T16:12:52Z                                              |
| description          |                                                                   |
| dns_nameservers      |                                                                   |
| dns_publish_fixed_ip | None                                                              |
| enable_dhcp          | False                                                             |
| gateway_ip           | 2001:db8::2                                                       |
| host_routes          |                                                                   |
| id                   | 84dffd43-6d7f-4c2f-9180-8f0f0b83c9d4                              |
| ip_version           | 6                                                                 |
| ipv6_address_mode    | None                                                              |
| ipv6_ra_mode         | None                                                              |
| name                 | ipv6-public-subnet                                                |
| network_id           | 73edb86b-d7ab-4db3-82b7-25fa8b012e40                              |
| project_id           | 15f2ab0eaa5b4372b759bde609e86224                                  |
| revision_number      | 0                                                                 |
| segment_id           | None                                                              |
| service_types        |                                                                   |
| subnetpool_id        | None                                                              |
| tags                 |                                                                   |
| updated_at           | 2024-01-24T16:12:52Z                                              |
+----------------------+-------------------------------------------------------------------+
stack@devstack:~/devstack$ openstack floating ip create --subnet 84dffd43-6d7f-4c2f-9180-8f0f0b83c9d4 public
Error while executing command: BadRequestException: 400, Unable to find any IP address on external network 73edb86b-d7ab-4db3-82b7-25fa8b012e40.
stack@devstack:~/devstack$ openstack floating ip create public
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| created_at          | 2024-03-27T13:35:59Z                 |
| description         |                                      |
| dns_domain          |                                      |
| dns_name            |                                      |
| fixed_ip_address    | None                                 |
| floating_ip_address | 172.24.4.107                         |
| floating_network_id | 73edb86b-d7ab-4db3-82b7-25fa8b012e40 |
| id                  | 9e193d33-17f9-400b-b639-b51750d41bc0 |
| name                | 172.24.4.107                         |
| port_details        | None                                 |
| port_id             | None                                 |
| project_id          | 15f2ab0eaa5b4372b759bde609e86224     |
| qos_policy_id       | None                                 |
| revision_number     | 0                                    |
| router_id           | None                                 |
| status              | DOWN                                 |
| subnet_id           | None                                 |
| tags                | []                                   |
| updated_at          | 2024-03-27T13:35:59Z                 |
+---------------------+--------------------------------------+

The error with the floating ip for ipv6 can be a local issue.

A single (external) network can have multiple subnets. This is the same as physical behavior: on the same Layer 2 network there can be multiple layer three subnets.
But within the scope of this ticket this leads to the following question:

Do we allow multiple / different subnets for external networks (e.g. combining IPv4 and IPv6)?
- this would lead to users having to specify the correct subnet when asking for a floating IP
How do we let user know what to expect from a network (e.g. will it grant IPv4 or IPv6 floating IP at default? Which subnet is used defaultly?)

markus-hentsch · 2024-04-09T11:53:00Z

Port Security and Shared Networks

@cah-patrickthiem, @josephineSei in the context of our discussion today about port security and security groups I had a closer look at this on a DevStack and found some interesting things.

Context recap: we consider disabled port security on ports within a network shared by multiple tenants a threat because then malicious tenants can spoof their addresses to be that of other tenants in the network. With port security enabled, OpenStack makes sure that only traffic from/to known assigned addresses is allowed. Also port security controls whether security group rules are applied or not (enabled port security = security groups enabled).

Changing port security on admin side:

an admin can create a shared or public/external¹ network with --disable-port-security which disables port security for all ports created in this network per default
- if tenants create VMs within such network while not specifying security groups, no default security group will be assigned to the VM
- if tenants create VMs within such network while specifying at least one security group, the server will reach ERROR state but the message about it not being able to assign the security group because of the network enforcing disabled port security seems to be only visible in Nova's logs (see the bottom of this post), not in the server status

We should add "Port security MUST NOT be disabled for networks created as shared (--share on CLI) or with router:external (--external on CLI)" or something similar to the standard.
Note: how we phrase this depends on how the glossary turns out, i.e. whether we state the API attributes here (shared, router:external) or if we reference well-defined network classification terminology established in the glossary.

Changing port security on user side:

if a tenant uses openstack port set --disable-port-security ... on a port in a shared or public/external¹ network, the default Neutron API policies will deny this request:
```
ForbiddenException: 403: Client Error for url: $NEUTRON:9696/networking/v2.0/ports/b4a5ea84-6084-43dd-a5bc-8b8bf40cba0f, (rule:update_port and rule:update_port:port_security_enabled) is disallowed by policy
```
- the default policy² restricts this to admins or users of the same project the network was created in

Since in the default policy creation of shared or external networks is an admin-only action, we should be safe here in most cases. However, since admins can create networks in specific projects (e.g. openstack network create --project ...) in conjunction with the external or shared flag, this might introduce loopholes.

Therefore, I think we should also add "Networks marked as shared or external (router:external attribute) MUST NOT be created within a tenant's project" or something. Again, phrasing should be in line with the terminology established.

Here is the nova-compute log entry when a tenant tries to create a VM with a security group assigned in a network where the admin has disabled port security:

nova.exception.SecurityGroupCannotBeApplied: Network requires port_security_enabled and subnet associated in order to apply security groups

... and this is what the user sees in openstack server show:

| fault                               | {'code': 500, 'created': '2024-04-09T11:45:40Z', 'message':        |
|                                     | 'Exceeded maximum number of retries. Exhausted all hosts available |
|                                     | for retrying build failures for instance                           |
|                                     | 4dd38355-7bdf-4c84-b07f-1516c3ae0925.'}                            |
| status                              | ERROR                                                              |

public/external as in --external on CLI or router:external in the API. ↩ ↩²
https://docs.openstack.org/neutron/latest/configuration/policy.html ↩

fkr added this to Sovereign Cloud Stack Sep 26, 2022

fkr moved this to Backlog in Sovereign Cloud Stack Sep 26, 2022

fkr added the IaaS Issues or pull requests relevant for Team1: IaaS label Sep 26, 2022

fkr added this to the R4 (v5.0.0) milestone Sep 26, 2022

fkr added SCS is standardized SCS is standardized and removed SCS is standardized SCS is standardized labels Oct 4, 2022

fkr added epic Issues that are spread across multiple sprints SCS is standardized SCS is standardized labels Oct 18, 2022

garloff mentioned this issue Dec 7, 2022

Write ADR to standardize IaaS IPv4 connectivity #228

Closed

9 tasks

garloff mentioned this issue Dec 7, 2022

IaaS DR/Standard for tenant VM DNS #229

Open

9 tasks

garloff self-assigned this Dec 7, 2022

tibeer mentioned this issue Mar 29, 2023

Stale issues #11

Closed

fkr removed this from the R4 (v5.0.0) milestone Apr 27, 2023

mbuechse mentioned this issue May 10, 2023

[EPIC] IaaS standards SovereignCloudStack/standards#285

Open

59 tasks

fkr changed the title ~~IPv4: Need to standardize how public IPv4 addresses can be assigned to servers/VMs (IaaS)~~ IPv4: Need to standardize how public IPv4 networking behaves by default Nov 14, 2023

cah-patrickthiem self-assigned this Feb 22, 2024

anjastrunk added the SCS-VP10 Related to tender lot SCS-VP10 label Mar 25, 2024

anjastrunk moved this from Backlog to Doing in Sovereign Cloud Stack Mar 25, 2024

anjastrunk assigned kgube Mar 27, 2024

josephineSei mentioned this issue Mar 28, 2024

Neutron Policies Standard SovereignCloudStack/standards#543

Closed

8 tasks

kgube mentioned this issue Apr 2, 2024

IPv6 support for IaaS #166

Open

10 tasks

markus-hentsch linked a pull request Apr 9, 2024 that will close this issue

standardizing IPv4 networking in SCS SovereignCloudStack/standards#522

Draft

kgube mentioned this issue Apr 19, 2024

Add standard for provider networks SovereignCloudStack/standards#572

Open

anjastrunk unassigned cah-patrickthiem Aug 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IPv4: Need to standardize how public IPv4 networking behaves by default #167

IPv4: Need to standardize how public IPv4 networking behaves by default #167

garloff commented Sep 26, 2022 •

edited by mbuechse

Loading

fkr commented Sep 26, 2022

garloff commented Sep 27, 2022

garloff commented Sep 27, 2022

garloff commented Dec 7, 2022 •

edited

Loading

garloff commented Dec 7, 2022 •

edited

Loading

garloff commented Dec 7, 2022

fkr commented Jun 14, 2023

garloff commented Oct 17, 2023

berendt commented Oct 17, 2023

fkr commented Nov 14, 2023

fkr commented Nov 14, 2023

josephineSei commented Mar 12, 2024

josephineSei commented Mar 13, 2024 •

edited

Loading

josephineSei commented Mar 19, 2024

josephineSei commented Mar 27, 2024

josephineSei commented Mar 27, 2024

markus-hentsch commented Apr 9, 2024

IPv4: Need to standardize how public IPv4 networking behaves by default #167

IPv4: Need to standardize how public IPv4 networking behaves by default #167

Comments

garloff commented Sep 26, 2022 • edited by mbuechse Loading

Definition of Ready:

Definition of Done:

fkr commented Sep 26, 2022

garloff commented Sep 27, 2022

garloff commented Sep 27, 2022

garloff commented Dec 7, 2022 • edited Loading

garloff commented Dec 7, 2022 • edited Loading

garloff commented Dec 7, 2022

fkr commented Jun 14, 2023

garloff commented Oct 17, 2023

berendt commented Oct 17, 2023

fkr commented Nov 14, 2023

fkr commented Nov 14, 2023

josephineSei commented Mar 12, 2024

josephineSei commented Mar 13, 2024 • edited Loading

josephineSei commented Mar 19, 2024

josephineSei commented Mar 27, 2024

josephineSei commented Mar 27, 2024

markus-hentsch commented Apr 9, 2024

Port Security and Shared Networks

Changing port security on admin side:

Changing port security on user side:

Footnotes

garloff commented Sep 26, 2022 •

edited by mbuechse

Loading

garloff commented Dec 7, 2022 •

edited

Loading

garloff commented Dec 7, 2022 •

edited

Loading

josephineSei commented Mar 13, 2024 •

edited

Loading