Reverse dialing registration of backend hosts

[bradfitz]

At home and in Go, I have a number of backends that are a pain to route to because they're buried behind NAT or other firewalls.

In the Go build system, we solve this using an old package I wrote (http://godoc.org/golang.org/x/build/revdial) that lets a backend connect to the server, and then turn the single TCP connection around (after authenticating) and let the server open up and multiplex many TCP connections as needed over that single TCP connection.

That code (first added Sep 2015 in golang/build@1f0d8f2) replaced an earlier "reverse roundtripper" that did the same thing, but only allowed a single HTTP request at a time per connection.

In any case, this is a model I keep returning to and finding super convenient.

To minimize this pain managing backends, I'd like some "revdial" or "backpain" mode in tcpproxy that lets backends register themselves.

The revdial package has been in production for a long time and might be a good start, but it doesn't do back pressure, so it wouldn't be good as a general solution. It only works for us because I know we won't have streams starving each other.

@danderson was suggesting re-using Go's existing HTTP/2 server is probably a better idea, even if it's a bit more work.

The http2 code already has unit tests to verify that full duplex CONNECT requests work over it.

We can just do an HTTP/1.x protocol upgrade over HTTPS with auth to the server proxy which can then Hijack the conn, turn it around, and be an HTTP/2 client to the HTTP/2 server running on the backends. The backend would then handle the incoming CONNECT requests from the server, do the proxying where needed, and let the io.Copy to the http.ResponseWriter handle all the flow control automatically via the http2 package.

The code on the backend (which could be an embeddable Go package + a binary for non-Go users) would look at lot this code from Go's build system:

https://github.com/golang/build/blob/d925a7bd2b3ee25956efb3dd98a87c8d3fee7ea6/cmd/buildlet/reverse.go#L72

(but using http2+CONNECT instead of revdial)

Note that it just writes an HTTP/1.x request over HTTPS with a token in it, expects a "101 Switching Protocols", and then switches into becoming a server itself.

The token it sends as auth can also register it as a new Target (https://godoc.org/github.com/google/tcpproxy#Target) implementation on the server side, so we can say "anything matching SNI foo.com should go to backend with token XYZFOOBAR".

@danderson, thoughts?

[danderson]

General idea LGTM. Some concerns on the authn/authz side of things, which we didn't touch on previously.

But first, here's some terminology I made up, because I kept confusing myself by calling different bits the client: backend is the thing initiating the revdial, and which ultimately lives "behind" the proxy. proxy is the proxy. client is the thing that wishes to talk to backend, and does so through proxy.

Passing auth tokens around implies either doing TLS for the backend to proxy connection, or using some form of replay-proof challenge-response thing to prevent backend hijacking.

TLS makes the auth token design simple, but means backend will be doing double-TLS decryption, once for client→backend and once for proxy→backend.

Not TLS-ing the backend to proxy connection avoids double-crypto, but makes the auth much more challenging.

On balance, probably better to just eat the double crypto costs. AES-NI makes the crypto super fast on x86, and ChaCha20 makes it super fast on everything else.

[bradfitz]

Yeah, I didn't consider the double encryption. In Go's build system, we do TLS from backend to proxy, but then the "proxy" (which is the build system itself) speaks regular HTTP/1.x over it, which would be plaintext if it weren't already encrypted by revdial.

I agree double encryption works for now. We can do a fancier thing later as an option. Or we could even add a mode where the proxy does the LetsEncrypt+TLS part and does "plaintext" over the already-TLS connection back to the backend. That might be feature creep upon both of our original plans, though. I'm only proposing that if the design we've end up with at the time makes it trivial. But it's not a goal.