Unable to Load Certain EC Keys in Botan 3.6+

[benj-zen]

When generating EC keys prior to version 3.6, the private key byte size is not stretched to match the byte size of the curve order. In Botan 3.6+, EC keys will not be loaded if these sizes do not match:

botan/src/lib/pubkey/ec_group/ec_inner_data.cpp

Lines 245 to 248 in 440b645

	std::unique_ptr<EC_Scalar_Data> EC_Group_Data::scalar_deserialize(std::span<const uint8_t> bytes) const {
	if(bytes.size() != m_order_bytes) {
	return nullptr;
	}

This issue occurs 50% of the time when generating keys for secp521r1, as 521 mod 8 = 1. When generating an example key with Botan 3.5 or below, which has a byte size of 65:

$ botan keygen --algo=ECDH --params=secp521r1
-----BEGIN PRIVATE KEY-----
MIHrAgEAMA4GBSuBBAEMBgUrgQQAIwSB1TCB0gIBAQRBrUnLx/qoPTdu7xSEZNnO
QHODpOgVgI60AOQjfK7oj45U9J1gG7G+k8hJcmVis2Dys64Ggqw2hhYgNYuDsBDw
EX2hgYkDgYYABAF/MiwGdehgdlNzFWRQywQtcUy4QHXBlREF1q7/ecclXv1xzFAW
ZvgUKGe9r1ox7piGSP7gteS4PJ4RE17lqNmgIAACl9dvTRMGxMbBgfWXLctYtAhv
xS0wery1wQvskt9+FjwPiHSXi4rmjLDM6vcyUliITumR4ArULaowUm5OGvD/Lw==
-----END PRIVATE KEY-----

Attempting to load the key in Botan 3.6 results in an error:

$ botan pkcs8 test.pem
Error: EC_Scalar::from_bytes is not a valid scalar value

[randombit]

Interestingly this encoding bug was fixed in #4110 but what I failed to realize at the time is, thinking that the bug had been introduced in #4056 and had never been in a release. But in fact we were always encoding incorrectly prior to #4056 as well

.encode(BigInt::encode_1363(m_private_key, m_private_key.bytes()), ASN1_Type::OctetString)

Using the size of the key vs using the size of the group order as it should have been. :/

[randombit]

Are you sure about the version timeline? As best I can tell the encoding was fixed in 3.5.0.

[benj-zen]

Oh, I see. Would still be great if there would be an API option to somehow import the old keys in 3.6+.
I'm pretty sure about the versions. I compiled both versions before submitting the issue, generated the key in 3.5.0 and tried to load in 3.6.0.

[randombit]

Would still be great if there would be an API option to somehow import the old keys in 3.6+.

We just have to support this in the default codepath. It might be a different story had this been done correctly the whole time, but it wasn't. Can you try the patch in #4541

[benj-zen]

I thought that throwing an error when loading a short private key via the default function may be desirable, hence my comment about the API option. However, #4541 works perfectly. Thanks!