encode() raises an RSA lib error when using a small RSA PSS key

[BramSrna]

When calling JWT.encode() using an RSA PSS key of length 1024, exponent of 3, and digest of 512, the following error is raised:

/Users/bsrna/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/jwt-2.9.3/lib/jwt/jwa/ps.rb:18:in "sign_pss": RSA lib (OpenSSL::PKey::RSAError)
from /Users/bsrna/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/jwt-2.9.3/lib/jwt/jwa/ps.rb:18:in "sign"
from /Users/bsrna/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/jwt-2.9.3/lib/jwt/encode.rb:48:in "signature"
from /Users/bsrna/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/jwt-2.9.3/lib/jwt/encode.rb:58:in "encode_signature"
from /Users/bsrna/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/jwt-2.9.3/lib/jwt/encode.rb:32:in "encoded_signature"
from /Users/bsrna/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/jwt-2.9.3/lib/jwt/encode.rb:18:in "segments"
from /Users/bsrna/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/jwt-2.9.3/lib/jwt.rb:30:in "encode"

This is the code block used for testing it:

require 'openssl'
require 'jwt'

key = OpenSSL::PKey::RSA.new(1024, 3)
key.sign_pss("SHA-512", "test", salt_length: 62, mgf1_hash: "SHA-512")
JWT.encode("test", key, 'PS512')

It was able to run key.sign_pss successfully, but JWT.encode hit the error.

The installed versions are jwt 2.9.3 and openssl 3.2.

[BramSrna]

When using :digest for the salt_length, this error occurs with digest 512 on smaller keys. I think :digest sets the salt length to 64 but the actual acceptable maximum from sign_pss is 62.

[BramSrna]

It looks like the issue is the key size. Using a key size of 2048 fixes the issue. Changing the exponent used for the key results in the same error if the key length stays as is.