Cannot intercept protocol in which SSL connection is initiated by the server

[symexec]

I am trying to intercept a custom, SSL-encrypted protocol where the client (192.168.150.201) establishes an initial TCP connection from port 9999 to the server (192.168.150.150) at port 541, and then the server send a "Client Hello" in return. The issue is, as the title says, the SSL traffic is not getting intercepted by the server. The protocol looks "switched" to me in the sense that the server sends the "Client Hello" here. Can that be the reason sslsplit is failing to work?

sslsplit is running on a Ubuntu 22.04 Server (5.15.0-101-generic x86-64) with a host IP of 192.168.150.1. The server and client are running on two VMs with the IPs listed above. The physical server, the server, and the client are a part of KVM NAT network (192.168.150.0/24). The physical server, which sslsplit is running on, acts like a router.

The traffic redirection is done via iptables:
iptables -t nat -A PREROUTING -p tcp --dport 541 -j REDIRECT --to-ports 10541

sslsplit is running as follows:
sslsplit -D -l connections.log -j . -M ssl_key_logfile -S log -k server.key -c server.crt -b client.key -a client.crt ssl 192.168.150.1 10541

Communication between the client and the server:

To rule out any error on my part, I have tested my sslsplit setup on a simple SSL server-client app. It works as expected, i.e., I am able to intercept and view SSL-encrypted traffic.

[sonertari]

It may be because sslsplit does not support TLS 1.3 yet. Can you check the output of the -V option?
If that's the issue, I'd recommend trying SSLproxy in split mode, as it supports TLS 1.3.

[symexec]

$ sslsplit -V

SSLsplit 0.5.5 (built 2021-12-26)
Copyright (c) 2009-2019, Daniel Roethlisberger <[email protected]>
https://www.roe.ch/SSLsplit
Build info: V:FILE HDIFF:1 N:83c4edf
Features: -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT IP6T_SO_ORIGINAL_DST
Local process info support: no
compiled against OpenSSL 3.0.1 14 Dec 2021 (30000010)
rtlinked against OpenSSL 3.0.2 15 Mar 2022 (30000020)
OpenSSL has support for TLS extensions
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
OpenSSL has engine support
Using SSL_MODE_RELEASE_BUFFERS
SSL/TLS protocol availability: tls10 tls11 tls12
SSL/TLS algorithm availability: !SHA0 RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.1.12-stable
rtlinked against libevent 2.1.12-stable
compiled against libnet 1.1.6
rtlinked against libnet 1.1.6
compiled against libpcap n/a
rtlinked against libpcap 1.10.1 (with TPACKET_V3)
24 CPU cores detected

It indeed looks like TLS v1.3 support is not there. Though I was running sslsplit with -D switch, it did not complain about the lack of protocol support in the debug log. For the future, is there any other switch that I can use to enable more verbose debug logging, something like that reports each incoming packets and action taken?

I also notice that there is a tls13 branch in sslsplit? Is it incomplete/experimental?

[sonertari]

Your version of sslsplit is too old. So I'd recommend you build and use the develop branch instead.
Also, make sure you compile and link the same OpenSSL version (not that this may be the issue in your case).

When you're building you can enable the DEBUG_PROXY and other switches in GNUmakefile, and then pass the -D option to sslsplit for debug logs.

I probably should port the TLS 1.3 changes from sslproxy to the develop branch of sslsplit. But I don't think I can do that soon.

[symexec]

I have built commit b2128fc9a7cbd715cbf15595ebd564a74100b602 which yields the following:

$ sslsplit -V

SSLsplit 0.5.5-38-gb2128fc (built 2024-03-20)
Copyright (c) 2009-2019, Daniel Roethlisberger <[email protected]>
https://www.roe.ch/SSLsplit
Build info: V:GIT
Features: -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT IP6T_SO_ORIGINAL_DST
Local process info support: no
compiled against OpenSSL 3.0.2 15 Mar 2022 (30000020)
rtlinked against OpenSSL 3.0.2 15 Mar 2022 (30000020)
OpenSSL has support for TLS extensions
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
OpenSSL has engine support
Using SSL_MODE_RELEASE_BUFFERS
SSL/TLS protocol availability: tls10 tls11 tls12
SSL/TLS algorithm availability: !SHA0 RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.1.12-stable
rtlinked against libevent 2.1.12-stable
compiled against libnet 1.1.6
rtlinked against libnet 1.1.6
compiled against libpcap n/a
rtlinked against libpcap 1.10.1 (with TPACKET_V3)
24 CPU cores detected

However, it does not make any difference to the issue at hand. The one I was using before (Ubuntu repo) and the one I build look almost similar to me. Especially, if the lack of TLS v1.3 is the main issue, then that is not going to get solved even with the updated version, right?

I have built with DEBUG_PROXY set. So now I do see the following on the debug log which at least confirms that sslsplit receives the setup/teardown packets:

0x55f3c0ef36b0             pxy_conn_ctx_new
0x55f3c0ef36b0             pxy_conn_ctx_free

Is there any option that I can turn on for even verbose logging, for example, what packets are received and what actions are being taken? In general, what does it do when a TLS v1.3 ClientHello is received? Just drops it as an unrecognized packet?

[symexec]

Per your previous suggestion, I built sslproxy (commit dfb783d7ba187187453b44f316a849b789c4f7e6) with DEBUG_PROXY on.

$ sslproxy -V

SSLproxy v0.9.5-dirty (built 2024-03-20)
Copyright (c) 2017-2024, Soner Tari <[email protected]>
https://github.com/sonertari/SSLproxy
Copyright (c) 2009-2019, Daniel Roethlisberger <[email protected]>
https://www.roe.ch/SSLsplit
Build info: V:GIT
Features: -DDEBUG_PROXY -DHAVE_NETFILTER -DWITHOUT_USERAUTH
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT IP6T_SO_ORIGINAL_DST
Local process info support: no
compiled against OpenSSL 3.0.2 15 Mar 2022 (30000020)
rtlinked against OpenSSL 3.0.2 15 Mar 2022 (30000020)
OpenSSL has support for TLS extensions
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
OpenSSL has engine support
Using SSL_MODE_RELEASE_BUFFERS
SSL/TLS protocol availability: tls10 tls11 tls12 tls13
SSL/TLS algorithm availability: !SHA0 RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.1.12-stable
rtlinked against libevent 2.1.12-stable
compiled against libnet 1.1.6
rtlinked against libnet 1.1.6
compiled against libpcap n/a
rtlinked against libpcap 1.10.1 (with TPACKET_V3)
24 CPU cores detected

It has got TLS v1.3 support, as expected. Since sslproxy is argument-compatible with sslsplit, I ran it with:
sslproxy -D -l connections.log -j . -M ssl_key_logfile -S log -k server.key -c server.crt -b client.key -a client.crt ssl 192.168.150.1 10541

My iptables redirection rules was same as before:
6 360 REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:541 redir ports 10541

However, the client still hangs as before, though I see iptables packet counter increase by one every time I try to connect to the server. On sslproxy console, it shows the following when the connection tears up (I abort the client):
Socket got closed while waiting. I see the following, too, which confirms that the proxy is on: - listen=[192.168.150.1]:10541 ssl netfilter.

Is there anything else to do to put sslproxy to split mode?

[symexec]

I further tested sslproxy with my Python-based demo server-client which were at least working with sslsplit before (refer to above). However, my server script now complains: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1002). On sslproxy log, the following appears:

Client-side BEV_EVENT_ERROR
Error from bufferevent: 0:- 2147483650:2:-:2:system library:0:-
Additional SSL error: 369098857:105:unregistered scheme:44:STORE routines:0:-
Additional SSL error: 1:1:-:0:-:0:-
SSL_free() in state 00000004 = 0004 = SSLERR (error) [connect socket]

[sonertari]

The DEBUG_CERTIFICATE and DEBUG_CLIENTHELLO_PARSER switches may help too.

You may see more details if you start sslproxy with -D4, very verbose logging. I've never seen the error reason that sslproxy reports, unregistered scheme, before.

You can pass -n to sslproxy to force split mode, but I guess that's not the reason.

Your python client complains about the CA cert you use with sslsplit/sslproxy. I think this may be the reason. You need to add that CA cert to the trust store of that python client, or on that system. Or you should somehow disable cert verification in that client.

Cert verification in sslproxy is on by default, you should disable it too, by something like -o VerifyPeer=no.

[symexec]

Passing -o VerifyPeer=no solves the connection issue between my demo Python-based server-client application, but the issue with the actual server-client still persists. Passing -D4 with the actual server-client produces this log. Does it ring any bell?

[FINEST] [0.0 fd=208 cfd=0] pxy_conn_init: srcaddr= [192.168.150.201]:40682
[FINEST] pxy_thr_timer_cb: thr=12, load=0, to=0
[FINEST] pxy_thr_timer_cb: thr=20, load=0, to=0
.
.
.
[FINEST] [0.0 fd=208 cfd=0] protossl_fd_readcb: ENTER
Socket got closed while waiting
[FINE] [0.0 fd=208 cfd=0] protossl_fd_readcb: Socket got closed while waiting
[FINEST] [0.0 fd=208 cfd=0] pxy_conn_ctx_free: ENTER
[FINEST] [0.0 fd=208 cfd=0] pxy_thr_detach: Removing conn
[FINEST] [0.0 fd=208 cfd=0] pxy_thr_detach: Cannot find conn in thr conns, empty
.
.
.

[sonertari]

I still think that the issue may be the client not liking the CA cert you pass to sslproxy (isn't it self-signed?). Can you add the CA cert to the trust store of that client? Or disable that client's cert verification?

You can even follow the function calls for each connection if you inspect verbose logs with -D4. But the logs you posted do not help much.

[symexec]

Yes, the certificate is self-signed.

Is there any way to turn on DEBUG_CERTIFICATE and DEBUG_CLIENTHELLO_PARSER from the command-line during make?

I missed the call stack log from -D4 because it's very verbose. Here it is.

[FINEST] proxy_listener_acceptcb: ENTER, fd=208
[FINEST] proxy_conn_ctx_new: ENTER, fd=208
[FINEST] [0.2 fd=208 cfd=0] proxy_conn_ctx_new: Created new conn
[FINEST] [0.2 fd=208 cfd=0] pxy_thrmgr_assign_thr: ENTER
[FINEST] [0.2 fd=208 cfd=0] protossl_init_conn: ENTER
[FINEST] [0.2 fd=208 cfd=0] pxy_conn_init: ENTER
[FINEST] [0.2 fd=208 cfd=0] pxy_thr_attach: Adding conn
[FINER] [0.2 fd=208 cfd=0] check_fd_usage: descriptor_table_size=1024, dtablecount=0, reserve=10
[FINEST] [0.2 fd=208 cfd=0] pxy_conn_init: srcaddr= [192.168.150.201]:40294.
.
.
.

[symexec]

For now, I have enabled the FEATURES+= -DDEBUG_CLIENTHELLO_PARSER option by uncommenting the relevant line in Mk/main.mk. Unfortunately, the debug log (-D4) stays the same as above. I don't see any message prefixed with ClientHello parser: in the log.

As I pointed out in my original post, it's the "supposed" server that sends the ClientHello here. Can that make any difference? In protocssl.c:1424, I found the following comment: /* for SSL, defer dst connection setup to initial_readcb */. In my case, the ClientHello will never be sent by the server until the initial connection is made, hence a deadlock. Is my interpretation incorrect?

[sonertari]

No, I think you're right. I've been trying to rule out all usual suspects, and I had assumed you were confused about the "switched" roles of client and server.

But if that's true, then it may explain the issue. So, as you say, if the client establishes the TCP connection only, and then goes silent, the server side does not get connected. In fact, as you have seen, sslsplit/sslproxy assumes the client has closed the connection. So, sslsplit or sslproxy may not help in this case, as they are.

But before givin up, you can comment out the whole "if (n == 0)" condition block where "Socket got closed while waiting" log is written in protossl_fd_readcb(), then build and try again, i.e.:

//	if (n == 0) {
//		/* socket got closed while we were waiting */
//		log_err_printf("Socket got closed while waiting\n");
//		log_fine("Socket got closed while waiting");
//		goto out;
//	}

This may establish the server side connection, hopefully.

[symexec]

Commenting the block produces this:

[FINEST] proxy_listener_acceptcb: ENTER, fd=208
[FINEST] proxy_conn_ctx_new: ENTER, fd=208
[FINEST] [0.1 fd=208 cfd=0] proxy_conn_ctx_new: Created new conn
[FINEST] [0.1 fd=208 cfd=0] pxy_thrmgr_assign_thr: ENTER
[FINEST] [0.1 fd=208 cfd=0] protossl_init_conn: ENTER
[FINEST] [0.1 fd=208 cfd=0] pxy_conn_init: ENTER
[FINEST] [0.1 fd=208 cfd=0] pxy_thr_attach: Adding conn
[FINER] [0.1 fd=208 cfd=0] check_fd_usage: descriptor_table_size=1024, dtablecount=0, reserve=10
[FINEST] [0.1 fd=208 cfd=0] pxy_conn_init: srcaddr= [192.168.150.201]:51056
[FINEST] [0.1 fd=208 cfd=0] protossl_fd_readcb: ENTER
ClientHello parser: parsing buffer of sz 0
ClientHello parser: candidate at offset 0
ClientHello parser: byte 0: 90
ClientHello parser: ===> No match: rv 1, *clienthello NULL
Peeking did not yield a (truncated) ClientHello message, aborting connection
[FINE] [0.1 fd=208 cfd=0] protossl_fd_readcb: Peeking did not yield a (truncated) ClientHello message, aborting connection
[FINEST] [0.1 fd=208 cfd=0] pxy_conn_ctx_free: ENTER
[FINEST] [0.1 fd=208 cfd=0] pxy_thr_detach: Removing conn
[FINEST] [0.1 fd=208 cfd=0] pxy_thr_detach: Cannot find conn in thr conns, empty

I am trying to get a hold of the code, but I am wondering why sslproxy defers connection setup. Can we not just initiate a TCP connection from the proxy to the server as soon as the proxy receives a TCP connection from the client?

[symexec]

To establish the initial TCP connection, I have added the pxy_conn_connect() call:

/* for SSL, defer dst connection setup to initial_readcb */
pxy_conn_connect(ctx);
ctx->ev = event_new(ctx->thr->evbase, ctx->fd, EV_READ, protossl_fd_readcb, ctx);

Now the protocol proceeds, but basically both the server and the proxy send ClientHello to each other, so rest of the communication falls apart. Where is the code to send ClientHello in sslproxy ? May be I can comment that out to prevent the ClientHello in packet#4 and see how it goes.

The following traffic is captured on the server.

[sonertari]

I don't want to discourage you, but I think it wouldn't be easy to reverse the roles of client and server.
(For example, sslsplit/sslproxy expects SNI from client before connecting to server too.)

[symexec]

I can imagine and probably I'll end up giving up :-( Still I want to take a last shot, even if something quick and dirty works just for my use-case. Can you please point me to the part of the code that sends the ClientHello? I cannot readily spot it.

[sonertari]

It's more complicated than that.