Sudden notary hanging

[ofek]

Example pipeline running rcodesign notary-submit: https://github.com/pypa/hatch/actions/runs/8775583147/job/24078126572

As you can see in the beginning the binaries are signed quite quickly but the package notarization below times out. One interesting thing I'm seeing is the following debug output:

[2024-04-21T21:41:13Z DEBUG reqwest::connect] starting new connection: https://appstoreconnect.apple.com/
[2024-04-21T21:41:13Z DEBUG rustls::client::hs] No cached session for DnsName("appstoreconnect.apple.com")
[2024-04-21T21:41:13Z DEBUG rustls::client::hs] Not resuming any session
[2024-04-21T21:41:13Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_256_GCM_SHA384
[2024-04-21T21:41:13Z DEBUG rustls::client::tls13] Not resuming
[2024-04-21T21:41:13Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck, Unknown(UnknownExtension *** typ: EllipticCurves, payload: 00040017001d ***), Protocols([ProtocolName(6832)])]
[2024-04-21T21:41:13Z DEBUG rustls::client::hs] ALPN protocol is Some(b"h2")

note the ALPN protocol is Some(b"h2") whereas in the step that succeeds it is ALPN protocol is Some(b"http/1.1"):

[2024-04-21T21:24:18Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]
[2024-04-21T21:24:18Z DEBUG rustls::client::hs] ALPN protocol is Some(b"http/1.1")