panic in addTLS in cgo code

[jmgvfr04]

Hi, has anyone seen a stack like this from a go panic? This is using version 1.22.5-1

SIGSEGV: segmentation violation
PC=0x7faddcc4acb1 m=5 sigcode=1 addr=0x40
signal arrived during cgo execution

goroutine 11437845 gp=0xc0005c41c0 m=5 mp=0xc000100008 [syscall]:
runtime.cgocall(0xc1d5c0, 0xc001327858)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/runtime/cgocall.go:157 +0x4b fp=0xc001327830 sp=0xc0013277f8 pc=0x41b7cb
vendor/github.com/golang-fips/openssl/v2._Cfunc_go_openssl_EVP_PKEY_derive(0x7fadc8001400, 0xc000f3c0f0, 0xc002bb0010)
        _cgo_gotypes.go:1539 +0x4b fp=0xc001327858 sp=0xc001327830 pc=0x5a986b
vendor/github.com/golang-fips/openssl/v2.ExtractHKDF.func6(0xc000f3c030?, {0xc000f3c0f0?, 0x0?, 0xc000f3c030?}, 0xc002bb0010)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:140 +0x67 fp=0xc001327898 sp=0xc001327858 pc=0x5bf8e7
vendor/github.com/golang-fips/openssl/v2.ExtractHKDF(0x60067b?, {0xc000f3c030, 0x30, 0x30}, {0x0, 0x0, 0x0})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:140 +0x1a9 fp=0xc0013278f0 sp=0xc001327898 pc=0x5bf749
crypto/internal/backend.ExtractHKDF(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/internal/backend/openssl_linux.go:261
crypto/tls.(*cipherSuiteTLS13).extract(0x19e71c0?, {0x0?, 0xc0013279b8?, 0x41b825?}, {0x0?, 0x0?, 0x0?})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/key_schedule.go:93 +0x145 fp=0xc001327970 sp=0xc0013278f0 pc=0x68dba5
crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys(0xc001327bd0)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:384 +0xd3 fp=0xc001327ac0 sp=0xc001327970 pc=0x674193
crypto/tls.(*clientHandshakeStateTLS13).handshake(0xc001327bd0)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:86 +0x274 fp=0xc001327b00 sp=0xc001327ac0 pc=0x6727f4
crypto/tls.(*Conn).clientHandshake(0xc00144a008, {0x12e9c30, 0xc0013dbb30})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client.go:265 +0x594 fp=0xc001327d30 sp=0xc001327b00 pc=0x66d034
crypto/tls.(*Conn).clientHandshake-fm({0x12e9c30?, 0xc0013dbb30?})
        <autogenerated>:1 +0x33 fp=0xc001327d58 sp=0xc001327d30 pc=0x693633
crypto/tls.(*Conn).handshakeContext(0xc00144a008, {0x12e9ca0, 0xc002184700})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/conn.go:1553 +0x3cb fp=0xc001327f70 sp=0xc001327d58 pc=0x66aa6b
crypto/tls.(*Conn).HandshakeContext(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/conn.go:1493
net/http.(*persistConn).addTLS.func2()
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/net/http/transport.go:1573 +0x6e fp=0xc001327fe0 sp=0xc001327f70 pc=0x6f32ce
runtime.goexit({})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc001327fe8 sp=0xc001327fe0 pc=0x486be1
created by net/http.(*persistConn).addTLS in goroutine 11437527
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/net/http/transport.go:1569 +0x309

there is one other goroutine in cgo code in the panic log, not sure if that is relevant:

goroutine 11437828 gp=0xc000231180 m=nil [runnable]:
runtime.cgocall(0xc1db90, 0xc0011cf718)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/runtime/cgocall.go:157 +0x4b fp=0xc0011cf6f0 sp=0xc0011cf6b8 pc=0x41b7cb
vendor/github.com/golang-fips/openssl/v2._Cfunc_go_openssl_EVP_MD_CTX_new()
        _cgo_gotypes.go:1263 +0x48 fp=0xc0011cf718 sp=0xc0011cf6f0 pc=0x5a8948
vendor/github.com/golang-fips/openssl/v2.newEvpHash(0x6, 0x30, 0x80)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hash.go:134 +0x6e fp=0xc0011cf770 sp=0xc0011cf718 pc=0x5bbc6e
vendor/github.com/golang-fips/openssl/v2.NewSHA384(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hash.go:541
crypto/internal/backend.NewSHA384(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/internal/backend/openssl_linux.go:139
crypto/sha512.New384()
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/sha512/sha512.go:234 +0x25 fp=0xc0011cf7a0 sp=0xc0011cf770 pc=0x60a765
crypto.Hash.New(0x5b3d29?)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/crypto.go:131 +0x3d fp=0xc0011cf7e8 sp=0xc0011cf7a0 pc=0x5a30bd
crypto.Hash.New-fm()
        <autogenerated>:1 +0x25 fp=0xc0011cf800 sp=0xc0011cf7e8 pc=0x6936a5
vendor/github.com/golang-fips/openssl/v2.newHKDF(0xc001587140?, 0x1)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:24 +0x92 fp=0xc0011cf898 sp=0xc0011cf800 pc=0x5be8b2
vendor/github.com/golang-fips/openssl/v2.ExtractHKDF(0x60067b?, {0xc001587140, 0x30, 0x30}, {0x0, 0x0, 0x0})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:109 +0x45 fp=0xc0011cf8f0 sp=0xc0011cf898 pc=0x5bf5e5
crypto/internal/backend.ExtractHKDF(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/internal/backend/openssl_linux.go:261
crypto/tls.(*cipherSuiteTLS13).extract(0x19e71c0?, {0x0?, 0xc0011cf9b8?, 0x41b825?}, {0x0?, 0x0?, 0x0?})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/key_schedule.go:93 +0x145 fp=0xc0011cf970 sp=0xc0011cf8f0 pc=0x68dba5
crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys(0xc0011cfbd0)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:384 +0xd3 fp=0xc0011cfac0 sp=0xc0011cf970 pc=0x674193
crypto/tls.(*clientHandshakeStateTLS13).handshake(0xc0011cfbd0)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:86 +0x274 fp=0xc0011cfb00 sp=0xc0011cfac0 pc=0x6727f4
crypto/tls.(*Conn).clientHandshake(0xc000a2ca88, {0x12e9c30, 0xc0013652c0})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client.go:265 +0x594 fp=0xc0011cfd30 sp=0xc0011cfb00 pc=0x66d034
crypto/tls.(*Conn).clientHandshake-fm({0x12e9c30?, 0xc0013652c0?})
        <autogenerated>:1 +0x33 fp=0xc0011cfd58 sp=0xc0011cfd30 pc=0x693633
crypto/tls.(*Conn).handshakeContext(0xc000a2ca88, {0x12e9ca0, 0xc0000ec700})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/conn.go:1553 +0x3cb fp=0xc0011cff70 sp=0xc0011cfd58 pc=0x66aa6b
crypto/tls.(*Conn).HandshakeContext(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/conn.go:1493
net/http.(*persistConn).addTLS.func2()
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/net/http/transport.go:1573 +0x6e fp=0xc0011cffe0 sp=0xc0011cff70 pc=0x6f32ce
runtime.goexit({})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0011cffe8 sp=0xc0011cffe0 pc=0x486be1
created by net/http.(*persistConn).addTLS in goroutine 11437448
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/net/http/transport.go:1569 +0x309

[qmuntal]

Thanks for reporting. I have some questions:

• Which Linux distro and OpenSSL version are you using?
• Is this panic sporadic? How often it happens?
• Can you provide a reproducer?

[jmgvfr04]

Thanks for the response! This happened once on SLES. I need to check on the openssl version.
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

[jmgvfr04]

In this case GO_OPENSSL_VERSION_OVERRIDE is not set so I believe it is using the standard version on the system:

ls -l /usr/lib64/libcrypto.so.1.1

-rwxr-xr-x 1 root root 3389800 May 17 2023 /usr/lib64/libcrypto.so.1.1

[jmgvfr04]

As I dig into it I will try to come up with a reproducer but I am not hopeful that I will be able to do so. Thanks

[dagood]

A note for repro attempts (on our side): an easy way to start that probably has the same OpenSSL as SLES 15.5:

Start with a opensuse/leap:15.5 Docker container and then:

zypper install -y wget tar gzip git gcc
wget https://download.visualstudio.microsoft.com/download/pr/766eefd8-51c7-431c-8b58-5136273eced8/d6f0ed417acc7881cb620a7c1bdd0358/go1.22.5-20240702.3.linux-amd64.tar.gz
tar -xf go1.22.5-20240702.3.linux-amd64.tar.gz