Proxy doesn't remember which IP address it received on when sending packets back to the client

[AndrewGambell]

What happened?:

Quilkin socket opens and listens across all address families and interfaces/ip addresses configured by default, however, it does not appear to remember which IP address the client originally sent to, proxying works as expected if packets are sent to the address of return interface, but in the case where these are different the source of the packet returned to the client won't match, breaking the flow.

What you expected to happen:

Quilkin would hold the dst address of the packet it received from the client in its session state, and use it explicitly as the src when sending packets back to the client

How to reproduce it (as minimally and precisely as possible):

In our scenario we have a VIP (10.19.16.209/32) configured as a loopback interface on a host where its physical network interface has the address 10.9.16.1/31. We have a simple echo server listening on localhost 26000 (see quilkin config below). We send a test packet from another host on the network 10.9.16.3 to the VIP address 10.19.16.209. Quilkin happily receives and proxies this packet however, the returned packet towards the client is sourced from the outgoing interface address 10.9.16.1 (Shown in the packet capture below)

# tcpdump -n -i eth0 udp port 7777
dropped privs to pcap
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on rack, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:54:13.098466 IP 10.9.16.3.42448 > 10.9.16.209.7777: UDP, length 6
13:54:13.099235 IP 10.9.16.1.7777 > 10.9.16.3.42448: UDP, length 6

version: v1alpha1
clusters:
  - endpoints:
    - address: 127.0.0.1:26000

Anything else we need to know?:

Environment:

• Quilkin version: 0.8.0
• Execution environment (binary, container, etc): docker container in host network mode
• Operating system: linux kernel 6.1.53
• Custom filters? (Yes/No - if so, what do they do?): No
• Log(s):
• Others:

[XAMPPRocky]

Thank you for your issue! I must admit I'm a bit lost, could you clarify this further for me?

Are you saying that instead of sending traffic back through ip, port that's created for a session, you'd like to be able to respond to traffic by sending to the same listen address as the clients?

Can I ask why would this behaviour be desirable? My initial concern is that UDP packets can have their src and dest addresses forged, so it would allow someone to gain information or reflect data by IP knocking.

[AndrewGambell]

A picture probably helps.

The problem lays with the source IP address that is selected for the return packet back to the client, default kernel behaviour will use the outgoing interface address for the SRC IP of the packet if its not told to do otherwise, this can lead to some breaking behaviour.

Note: Quilkin is the "server" in the diagram

[XAMPPRocky]

Ah, that is much clearer. Thank you for the diagram. Do you know what you need to configure on the socket level to tell it to use Lo as the source address in this case? If know what we need to tell the kernel, I can figure out how to call that in Rust.

[AndrewGambell]

So there is a nice cloudflare blog post on exactly this issue.

https://blog.cloudflare.com/everything-you-ever-wanted-to-know-about-udp-sockets-but-were-afraid-to-ask-part-1

I think the easiest path forward here would be to use the sendmsg syscall and pass in the control message struct with the source address it was sent to (as far as I can tell we keep that in the session data)

The only problem I see is that sendmsg doesn't seem to be supported in the socket2 lib.

rust-lang/socket2#493

The two other options established-over-unconnected technique described in the doc, which might be a bit dangerous in the event of a DDoS type attack, or have the options to just bind to an address rather than wildcard, however with this option it might still be nice to be able to bind to several different addresses.

[markmandel]

Oh that's a joy and a delight - and hunting around, socket2 doesn't support IP_RECVPKTINO or IPV6_RECVPKTINFO (although that might be easy enough to add either).

or have the options to just bind to an address rather than wildcard, however with this option it might still be nice to be able to bind to several different addresses.

That does seem like the easiest option (not the best, but definitely the easiest) - but to check, would it just be the listening port we would bind, or would the sockets for each upstream session also need to be bound to that IP? (or maybe you can specify both listening and session socket bind addresses?)

[AndrewGambell]

I was more concerned with the listening side. However...

Being able to specify where you're traffic is being proxied from might be nice. But as I was thinking through this, quilkin has a 1:1 relationship with a session and the socket used to proxy upstream communication, this probably means you have a hard limit of connections based on the number of ephemeral ports you have access to. Perhaps its nice to be able to bind the upstream to a particular address or even list of addresses? As it lets you either contain the ephemeral port use or expand past what a single IP address can support.

[markmandel]

@XAMPPRocky solved the ephemeral port limit a while ago with #815 😄 a very clever solution!

So we shouldn't need a list of addresses to solve that issue.

[XAMPPRocky]

quilkin has a 1:1 relationship with a session and the socket used to proxy upstream communication, this probably means you have a hard limit of connections based on the number of ephemeral ports you have access to

Quilkin has a 1:1 relationship per gameserver. So this means for example if we had 1000 upstream gameservers on a proxy that had one player per server, we only use one port/socket for all servers.

We do this by muxing and demuxing packets by the server ip address.

So technically there is a limit of ~28,231 players per gameserver, but if you're hitting that limit you could increase the port range, and you probably have a lot of bigger problems to solve :)

[XAMPPRocky]

I believe sendmsg and recvmsg have now been added to tokio-uring, so we could probably fix this when upgrade to the newest version. https://github.com/tokio-rs/tokio-uring/releases/tag/v0.5.0