Network Boot IPv6 does not work with remote tftp server and DHCPv6 Relay

[owendelong]

---

name: Bug: IPv6 Netboot using DHCPv6 Relay and TFTP server on far side of router(s)
about: I'm trying to get a Pi 5 to boot via IPv6.

---

Describe the bug
Attempting to network boot Pi 5 with USE_IPV6=1 and boot order 0xf417 on an IPv6-only network.

DHCPv6 completes, but the PI does not send a request to the tftp server provided in option 61 (Bootfile-URL).

Instead, it sends out a series of neighbor solicitation packets to ff02::1:ff00:0 asking "who has ::".

To reproduce
Environment is as follows:
IPv6 Prefix redacted to xxxx:xxxx:xxxx. Everything is within the same /48 prefix. All other IPv6 quartets are as spelled out below.

Raspberry Pi 5 8GB ethernet (wired) connection to v6-only VLAN on Juniper ex4200 (xxxx:xxxx:xxxx:2000::/64). v6only VLAN on ex4200 is on three interfaces: ge-0/0/15 (direct to pi, access port) and ae0/ae1 which are 2x10G LAG (LACP/802.3ad) connections to a pair of Juniper MX-240 routers.
Routers have multiple virtual circuits connecting to another MX-240 located at my other site. TFTP Server is located at this site and connected (L2) through another ex4200 to the MX-240 at that site. TFTP Server is xxxx:xxxx:xxxx::200:2/64. There is an analyzer port configured on the ex4200 allowing me to sniff traffic to/from the raspberry pi.

DHCP is KEA running in an HA config on a pair of Nano-pi R6S units which are connected on a different VLAN to the same ex4200 switch and routed via the MX-240 routers.

Relevant KEA configuration fragments are as follows:

    "option-def": [
      {
        "name": "PXEDiscoveryControl",
        "code": 6,
        "space": "vendor-40712",
        "type": "uint8",
        "array": false
      },
      {
        "name": "PXEMenuPrompt",
        "code": 10,
        "space": "vendor-40712",
        "type": "record",
        "array": false,
        "record-types": "uint8,string"
      },
      {
        "name": "PXEBootMenu",
        "code": 9,
        "space": "vendor-40712",
        "type": "record",
        "array": false,
        "record-types": "uint16,uint8,string"
      }
    ],
    "option-data": [
      # Enable RFC 5007 support (same than for DHCPv4)
      #allow leasequery;
      # Global definitions for name server address(es) and domain search list
      {
        "space": "dhcp6",
        "name": "dns-servers",
        "code": 23,
        "data": "xxxx:xxxx:xxxx:1::200:2, 2001:4860:4860::8888"
      },
      {
        "space": "dhcp6",
        "name": "domain-search",
        "code": 24,
        "data": "delong.com"
      },
      # Set preference to 255 (maximum) in order to avoid waiting for
      # additional servers when there is only one
      {
        "space": "dhcp6",
        "name": "preference",
        "code": 7,
        "data": "255"
      },
      # Server side command to enable rapid-commit (2 packet exchange)
      {
        "space": "dhcp6",
        "name": "rapid-commit",
        "code": 14,
        "data": ""
      },
      # The delay before information-request refresh
      #  (minimum is 10 minutes, maximum one day, default is to not refresh)
      #  (set to 6 hours)
      {
        "space": "dhcp6",
        "name": "information-refresh-time",
        "code": 32,
        "data": "21600"
      }
    ],
    "client-classes": [
      {
        # Support for RasPi IPv6 Boot
        "name": "rpi-pxe",
        "test": "option[client-arch-type].hex == 0x0029",
        "option-data": [
          {
            "name": "bootfile-url",
            "data": "tftp://[xxxx:xxxx:xxxx::200:2]/rpi/start.elf"
          },
          {
            "name": "vendor-opts",
            "always-send": true,
            "data": "40712"
          },
          {
            "name": "PXEBootMenu",
            "always-send": true,
            "csv-format": true,
            "data": "0,17,Raspberry Pi Boot",
            "space": "vendor-40712"
          },
          {
            "name": "PXEDiscoveryControl",
            "always-send": true,
            "data": "3",
            "space": "vendor-40712"
          },
          {
            "name": "PXEMenuPrompt",
            "always-send": true,
            "csv-format": true,
            "data": "0,pxe",
            "space": "vendor-40712"
          }
        ]
      }
    ],
    ....
    "subnet6": [
      ....
      {
        "id": 2,
        "user-context": {
          "vlan": "v6only"
        },
        "subnet": "xxxx:xxxx:xxxx:2000::/64",
        "rapid-commit": true,
        "pools": [
          {
            # Use a broad range for addres assignment, but avoid conflicts with SLAAC
            "pool": "xxxx:xxxx:xxxx:2000::d8c:0:0 - xxxx:xxxx:xxxx:2000::d8c:ffff:ffff"
          }
        ],
        "interface": "eth0",
        "reservations": [
          {
            "hostname": "test-pi1",
            "duid": "00:03:00:01:d8:3a:dd:e8:2b:d8",
            #"hw-address": "d8:3a:dd:e8:2b:d8",
            "ip-addresses": [
                "xxxx:xxxx:xxxx:2000::200"
            ]
          },
          {
            "hostname": "test-pi2",
            "duid": "00:04:f3:8f:6b:10:ea:6c:47:17:43:43:ba:1f:27:48:20:7f",
            #"hw-address": "d8:3a:dd:e8:2b:d8",
            "ip-addresses": [
                "xxxx:xxxx:xxxx:2000::201"
            ]
          }
        ]
      }
    ]
  }

I can see the Pi sending out a DHCP request and I get a proper response (via the MX-240 relays):
(tcpdump output from analyzer port)

20:05:32.615220 d8:3a:dd:e8:2b:d8 > 33:33:00:01:00:02, ethertype IPv6 (0x86dd), length 132: (hlim 255, next-header UDP (17) payload length: 78) fe80::da3a:ddff:fee8:2bd8.dhcpv6-client > ff02::1:2.dhcpv6-server: [udp sum ok] dhcp6 request (xid=d167f2 (client-ID hwaddr type 1 d83adde82bd8) (server-ID hwaddr/time type 1 time 754210615 b2711ca3cd44) (IA_NA IAID:0 T1:0 T2:0) (option-request Bootfile-URL) (opt_61) (elapsed-time 2))
	0x0000:  6000 0000 004e 11ff fe80 0000 0000 0000
	0x0010:  da3a ddff fee8 2bd8 ff02 0000 0000 0000
	0x0020:  0000 0000 0001 0002 0222 0223 004e ab5f
	0x0030:  03d1 67f2 0001 000a 0003 0001 d83a dde8
	0x0040:  2bd8 0002 000e 0001 0001 2cf4 5737 b271
	0x0050:  1ca3 cd44 0003 000c 0000 0000 0000 0000
	0x0060:  0000 0000 0006 0002 003b 003d 0002 0029
	0x0070:  0008 0002 0002
20:05:32.662523 80:71:1f:4f:95:f0 > d8:3a:dd:e8:2b:d8, ethertype IPv6 (0x86dd), length 231: (hlim 64, next-header UDP (17) payload length: 177) fe80::8271:1f00:64f:95f0.dhcpv6-server > fe80::da3a:ddff:fee8:2bd8.dhcpv6-client: [udp sum ok] dhcp6 reply (xid=d167f2 (client-ID hwaddr type 1 d83adde82bd8) (server-ID hwaddr/time type 1 time 754210615 b2711ca3cd44) (IA_NA IAID:0 T1:43200 T2:69120 (IA_ADDR xxxx:xxxx:xxxx:2000::200 pltime:86400 vltime:604800)) (vendor-specific-info) (Bootfile-URL tftp://[xxxx:xxxx:xxxx::200:2]/rpi/start.elf))
	0x0000:  6000 0000 00b1 1140 fe80 0000 0000 0000
	0x0010:  8271 1f00 064f 95f0 fe80 0000 0000 0000
	0x0020:  da3a ddff fee8 2bd8 0223 0222 00b1 1497
	0x0030:  07d1 67f2 0001 000a 0003 0001 d83a dde8
	0x0040:  2bd8 0002 000e 0001 0001 2cf4 5737 b271
	0x0050:  1ca3 cd44 0003 0028 0000 0000 0000 a8c0
	0x0060:  0001 0e00 0005 0018 2620 0000 0930 2000
	0x0070:  0000 0000 0000 0200 0001 5180 0009 3a80
	0x0080:  0011 0029 0000 9f08 0006 0001 0300 0900
	0x0090:  1400 0011 5261 7370 6265 7272 7920 5069
	0x00a0:  2042 6f6f 7400 0a00 0400 7078 6500 3b00
	0x00b0:  2874 6674 703a 2f2f 5b32 3632 303a 303a
	0x00c0:  3933 303a 3a32 3030 3a32 5d2f 7270 692f
	0x00d0:  7374 6172 742e 656c 66

However, after this, instead of sending a tftp RRQ to the intended server, it starts sending out IPv6 NS packets looking for neighbor :: as follows:

20:05:33.615082 d8:3a:dd:e8:2b:d8 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::da3a:ddff:fee8:2bd8 > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has ::
	  source link-address option (1), length 8 (1): d8:3a:dd:e8:2b:d8
	    0x0000:  d83a dde8 2bd8
	0x0000:  6000 0000 0020 3aff fe80 0000 0000 0000
	0x0010:  da3a ddff fee8 2bd8 ff02 0000 0000 0000
	0x0020:  0000 0001 ff00 0000 8700 b626 0000 0000
	0x0030:  0000 0000 0000 0000 0000 0000 0000 0000
	0x0040:  0101 d83a dde8 2bd8
20:05:34.615140 d8:3a:dd:e8:2b:d8 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::da3a:ddff:fee8:2bd8 > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has ::
	  source link-address option (1), length 8 (1): d8:3a:dd:e8:2b:d8
	    0x0000:  d83a dde8 2bd8
	0x0000:  6000 0000 0020 3aff fe80 0000 0000 0000
	0x0010:  da3a ddff fee8 2bd8 ff02 0000 0000 0000
	0x0020:  0000 0001 ff00 0000 8700 b626 0000 0000
	0x0030:  0000 0000 0000 0000 0000 0000 0000 0000
	0x0040:  0101 d83a dde8 2bd8
20:05:35.615281 d8:3a:dd:e8:2b:d8 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::da3a:ddff:fee8:2bd8 > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has ::
	  source link-address option (1), length 8 (1): d8:3a:dd:e8:2b:d8
	    0x0000:  d83a dde8 2bd8
	0x0000:  6000 0000 0020 3aff fe80 0000 0000 0000
	0x0010:  da3a ddff fee8 2bd8 ff02 0000 0000 0000
	0x0020:  0000 0001 ff00 0000 8700 b626 0000 0000
	0x0030:  0000 0000 0000 0000 0000 0000 0000 0000
	0x0040:  0101 d83a dde8 2bd8

These continue to repeat until the connection times out and it falls through to HTTP which fails because there is no DHCP4 or IPv4 configuration at all on this subnet and then it falls through and boots from the SD card.

Expected behaviour
Everything is as expected up to the DHCP6 reply.
After the DHCP6 reply is received, I expect the host to send NS packets for the VRRP address provided in the RA advertisements from both routers and then send tftp RRQ packets to the tftp server defined in the DHCP reply, then boot from the downloaded file.

Actual behaviour
As stated above, the R.Pi starts sending ICMP6 NS packets requesting the MAC address for :: instead of trying to reach out to the given TFTP server.

System

• Which model of Raspberry Pi? Pi5 8GB

• Which OS and version (cat /etc/rpi-issue)?
  Raspberry Pi reference 2024-11-19
  Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 891df1e21ed2b6099a2e6a13e26c91dea44b34d4, stage4

• Which firmware version (vcgencmd version)? I will have to add this later. It's the latest firmware I was able to obtain (Dec. 2024 IIRC)

• Which kernel version (uname -a)? 6.6.51

Logs
Relevant logs are included above.

Additional context
Nothing to add at this time, but happy to answer any questions or cooperate in any way possible in getting this solved. Willing to run alpha or beta firmware to test solutions.

[timg236]

Thanks for the bug report. Unfortunately, IPV6 support is not being actively developed in the bootloader right now so this is unlikely to change in the near future.

[owendelong]

It feels like this is probably a small bug in the code that parses the dhxpv6 advertise message. It’s tragic that the bootloader code isn’t open source. I’m sure there are people who would be willing to work on this if they could.

I am going to try some further diagnostics today to get more information and will post what I find here. I may even go as far as implementing a NAT66 against a LL address to see if that works.

[owendelong]

So by changing the tftp:// URL on the DHCP server to "tftp://[xxxx:xxxx:xxxx:2000::200:2]/start.elf" I do see the PI requesting NS for xxxx:xxxx:xxxx:2000::200:2 instead of :: which I believe confirms that there is a bug in handling an off-net tftp server.

[owendelong]

The plot thickens... It appears that after the DHCP process occurs, there's an NS packet sent by the DHCP Relay:

06:43:39.353956 IP6 fe80::6:feed:0:1 > ff02::1: ICMP6, router advertisement, length 144
06:43:46.998513 IP6 fe80::da3a:ddff:fee8:2bd8.546 > ff02::1:2.547: dhcp6 solicit
06:43:47.011537 IP6 fe80::da3a:ddff:fee8:2bd8.546 > ff02::1:2.547: dhcp6 request
06:43:48.011532 IP6 fe80::da3a:ddff:fee8:2bd8 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has 2620:0:930:2000::200:2, length 32
06:43:48.011577 IP6 2620:0:930:2000::200:2 > fe80::da3a:ddff:fee8:2bd8: ICMP6, neighbor advertisement, tgt is 2620:0:930:2000::200:2, length 32

Which the Raspberry pi answers (above). However, the pi then refuses to respond to any further NS requests from any other host, thus making it essentially mandatory for the DHCP server and the TFTP server to be one and the same and on-link in order to work. This seems like a completely untenable situation in all but the most basic of test or laboratory networks.

06:43:48.035230 IP6 2620:0:930:2000::200.46819 > 2620:0:930:2000::200:2.69: TFTP, length 49, RRQ "530cf66c/config.txt" octet tsize 0 blksize 1468
06:43:48.039817 IP6 2620:0:930:2000::200:2 > ff02::1:ff00:200: ICMP6, neighbor solicitation, who has 2620:0:930:2000::200, length 32
06:43:49.046676 IP6 2620:0:930:2000::200:2 > ff02::1:ff00:200: ICMP6, neighbor solicitation, who has 2620:0:930:2000::200, length 32

This trio of packets repeats over and over until the Pi exceeds its retry count for 530cf66c/config.txt (and subsequently config.txt is repeated with the same pattern and behavior).

So in short, if the IPv6 netboot capability of a pi functions at all, the only scenario in which it is apparently possible is if one puts the entire boot functionality (tftp server, dhcp server at least) on the same host and puts them on-link with the pi. This falls well short of the RFC specificaiton for IPv6 netboot and also well short of most real-world requirements for same.