[Bug]: Invalid instruction exception in sp_2048_get_from_table_32

[vaintroub]

Contact Details

[email protected]

Version

5.7.0

Description

This happens in MariaDB, which uses WolfSSL 5.7.0 (compiles as submodule), using SP_MATH_ALL with assembly support.

Most recent version of MariaDB would call wolfSSL_EVP_PKEY_keygen to generate 4096 bit large private key, to then generate an "ephemeral" server SSL certificate, if user does not provide own.

So far it worked well for us, however there is a recent bug report MDEV-34372 about invalid instruction exception, and attached crashdump confirms that instruction comes from AVX instruction set.

wolfssl\wolfcrypt\src\sp_x86_64_asm.asm, line 13172

sp_2048_get_from_table_32 PROC
        sub	rsp, 128
        vmovdqu	OWORD PTR [rsp], xmm6 //<--- HERE

Mostly likely, the code did not check CPUID before using it.

this is how we build it

here is generated user_settings.h

#ifndef WOLFSSL_USER_SETTINGS_H
#define WOLFSSL_USER_SETTINGS_H

#define HAVE_CRL
#define WOLFSSL_HAVE_ERROR_QUEUE
#define WOLFSSL_MYSQL_COMPATIBLE
#define HAVE_ECC
#define ECC_TIMING_RESISTANT
#define HAVE_HASHDRBG
#define WOLFSSL_AES_DIRECT
#define WOLFSSL_SHA384
#define WOLFSSL_SHA512
#define WOLFSSL_SHA224
#define SESSION_CERT
#define KEEP_OUR_CERT
#define WOLFSSL_STATIC_RSA
#define WOLFSSL_USER_IO
#define WC_RSA_BLINDING
#define HAVE_TLS_EXTENSIONS
#define HAVE_AES_ECB
#define HAVE_AESGCM
#define HAVE_CHACHA
#define HAVE_POLY1305
#define HAVE_THREAD_LS
#define WOLFSSL_AES_COUNTER
#define NO_WOLFSSL_STUB
#define OPENSSL_ALL
#define WOLFSSL_ALLOW_TLSV10
#define NO_OLD_TIMEVAL_NAME
#define HAVE_SECURE_RENEGOTIATION
#define HAVE_EXTENDED_MASTER
/*
  Following is workaround about a WolfSSL 5.6.6 bug.
  The bug is about undefined sessionCtxSz during compilation.
*/
#define WOLFSSL_SESSION_ID_CTX
#define WOLFSSL_KEY_GEN
#define WOLFSSL_CERT_GEN

/* TLSv1.3 definitions (all needed to build) */
#define WOLFSSL_TLS13
#define HAVE_HKDF
#define HAVE_TLS_EXTENSIONS
#define HAVE_SUPPORTED_CURVES
#define HAVE_FFDHE_2048
#define WC_RSA_PSS
/* End of TLSv1.3 defines */

/* Features we exclude */
#define NO_DSA
#define NO_HC128
#define NO_MD4
#define NO_PSK
#define NO_RABBIT
#define NO_RC4

#define RSA_MAX_SIZE 8192
#define WOLFSSL_SP_MATH_ALL
#define WOLFSSL_HAVE_SP_RSA
#ifndef WOLFSSL_SP_4096
#define WOLFSSL_SP_4096
#endif

#define WOLFSSL_AESNI
#define HAVE_INTEL_RDSEED
#define HAVE_INTEL_RDRAND
/* #undef USE_INTEL_SPEEDUP */
#define WOLFSSL_X86_64_BUILD
#define WOLFSSL_SP_X86_64
#define WOLFSSL_SP_X86_64_ASM

#endif  /* WOLFSSL_USER_SETTINGS_H */

Reproduction steps

I did not reproduce myself, so the dump and the log is only thing I have, filing on behalf of our user
It would be something like that

1. Grab an Intel Celeron or Pentium with Windows 10 (possibly, AVX can be switched off using other means, I do not know)
2. Download MariaDB 11.4.2 MSI from https://mariadb.org/download/
3. Try to install (chose create a server instance)

Relevant log output

From the mentioned MariaDB bug, the error log contains following
---------------------------------------------------------------
240612  2:02:04 [ERROR] mysqld got exception 0xc000001d ;
Sorry, we probably made a mistake, and this is a bug.
<lots of irrelevant stuff>
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
server.dll!sp_2048_get_from_table_32()[sp_x86_64_asm.asm:13172]
---------------------------------------------

[SparkiDev]

Hi @vaintroub

I've put up a PR to fix this: #7666

Let us know if this fixes your issues!

Thanks,
Sean

[vaintroub]

I asked to test the fix in MariaDB bug report, hopefully there will be some feedback. Myself, I do not have hardware that old

[SparkiDev]

Thanks @vaintroub.
The problem was never noticed by us as we don't hardware 'that old' either!

[vaintroub]

So, I found a way to reproduce it, apparently just install of fresh Windows 10 to VirtualBox does it for me. AVX/AVX2, and couple of other instructions (FMA) are unsupported.

Bad news, there is still illegal instruction exception, on the same line as originally reported , line 13172 here
sp_x86_64_asm.asm

IFNDEF WC_NO_CACHE_RESISTANT
_text SEGMENT READONLY PARA
sp_2048_get_from_table_32 PROC
        sub	rsp, 128
        vmovdqu	OWORD PTR [rsp], xmm6
        vmovdqu	OWORD PTR [rsp+16], xmm7
        vmovdqu	OWORD PTR [rsp+32], xmm8

Maybe, those should be movdqu instructions, not vmovdqu (same at the end of the function) ??

Here is the test program I'm using

#include <user_settings.h>
#include <openssl/evp.h>
#include <openssl/dh.h>
#include <openssl/bn.h>
#include <openssl/x509.h>

 
static EVP_PKEY *vio_keygen()
{
  EVP_PKEY_CTX *ctx;
  EVP_PKEY *pkey= NULL;

  if (!(ctx= EVP_PKEY_CTX_new_id(EVP_PKEY_RSA, NULL)))
    return NULL;

  if (EVP_PKEY_keygen_init(ctx) <= 0)
    goto end;

  if (EVP_PKEY_CTX_set_rsa_keygen_bits(ctx,4096) <= 0)
    goto end;

  if (EVP_PKEY_keygen(ctx, &pkey) <= 0)
    pkey= NULL; /* just in case */

end:
  EVP_PKEY_CTX_free(ctx);
  return pkey;
}

static X509 *vio_gencert(EVP_PKEY *pkey)
{
  X509 *x;
  X509_NAME *name;

  if (!(x= X509_new()))
    goto err;

  if (!X509_set_version(x, 2))
    goto err;
  if (!(name= X509_get_subject_name(x)))
    goto err;
  if (!X509_NAME_add_entry_by_txt(name, "CN", MBSTRING_ASC, (const unsigned char *)"x",1,-1,0))
    goto err;
  if (!X509_set_issuer_name(x, name))
    goto err;
  if (!X509_gmtime_adj(X509_get_notBefore(x), 0))
    goto err;
  if (!X509_gmtime_adj(X509_get_notAfter(x), 60 * 60 * 24 * 365 * 10))
    goto err;
  if (!X509_set_pubkey(x, pkey))
    goto err;
  if (!X509_sign(x, pkey, EVP_sha256()))
    goto err;

  return x;

err:
  X509_free(x);
  return NULL;
}

int main()
{
  printf("Hello\n");
  unsigned long long start,end,freq;
  QueryPerformanceFrequency((LARGE_INTEGER*)&freq);
  QueryPerformanceCounter((LARGE_INTEGER*)&start);
  auto key= vio_keygen();
  if (!key)
  {
    printf("Failed to generate key\n");
    return 1;
  }
  QueryPerformanceCounter((LARGE_INTEGER*)&end);
  printf("Keygen took %f seconds\n", (end-start)/(double)freq);
  auto cert= vio_gencert(key);
  if (!cert)
  {
    printf("Failed to generate cert\n");
    return 1;
  }
  printf("Success\n");
  return 0;
}

Stacktrace just before the crash:

1      wolfssl-test.exe!sp_2048_get_from_table_32() Line 13171 
2      wolfssl-test.exe!sp_2048_mod_exp_32(unsigned __int64 * r, const unsigned __int64 * a, const unsigned __int64 * e, int bits, const unsigned __int64 * m, int reduceA) Line 1572 
3      wolfssl-test.exe!sp_ModExp_2048(const sp_int * base, const sp_int * exp, const sp_int * mod, sp_int * res) Line 2526 
4      wolfssl-test.exe!sp_exptmod_ex(const sp_int * b, const sp_int * e, int digits, const sp_int * m, sp_int * r) Line 13753 
5      wolfssl-test.exe!sp_exptmod(const sp_int * b, const sp_int * e, const sp_int * m, sp_int * r) Line 13860 
6      wolfssl-test.exe!sp_prime_miller_rabin(const sp_int * a, sp_int * b, int * result, sp_int * n1, sp_int * r) Line 18792 
7      wolfssl-test.exe!_sp_prime_random_trials(const sp_int * a, int trials, int * result, WC_RNG * rng) Line 19180 
8      wolfssl-test.exe!sp_prime_is_prime_ex(const sp_int * a, int trials, int * result, WC_RNG * rng) Line 19263 
9      wolfssl-test.exe!_CheckProbablePrime(sp_int * p, sp_int * q, sp_int * e, int nlen, int * isPrime, WC_RNG * rng) Line 4568 
10     wolfssl-test.exe!wc_MakeRsaKey(RsaKey * key, int size, long e, WC_RNG * rng) Line 4868 
11     wolfssl-test.exe!wolfssl_rsa_generate_key_native(WOLFSSL_RSA * rsa, int bits, WOLFSSL_BIGNUM * e, void * cb) Line 3146 
12     wolfssl-test.exe!wolfSSL_RSA_generate_key(int bits, unsigned long e, void(*)(int, int, void *) cb, void * data) Line 3239 
13     wolfssl-test.exe!wolfSSL_EVP_PKEY_keygen(WOLFSSL_EVP_PKEY_CTX * ctx, WOLFSSL_EVP_PKEY * * ppkey) Line 3363 
14     wolfssl-test.exe!vio_keygen() Line 22 
15     wolfssl-test.exe!main() Line 68 

[SparkiDev]

Hi @vaintroub,

I've put up another pull request that should fix this issue now.
There were other functions that I targeted for the same fix and for some reason assumed get_from_table was only called with AVX1.

Let us know if this does fix the issue.

Thanks,
Sean

[vaintroub]

Perfect, this patch seems to have fixed the issue. thank you!

[SparkiDev]

Great to hear!
Thanks for testing!

Sean