Remote Key signing using Aws KMS (Consensys#837)

signing/secp256k1/aws

AwsKmsSigner.java: Implements Signer interface. For each kms configuration metadata file, an instance of this class will be registered.
AwsKmsSignerFactory.java: Create an instance of AwsKmsSigner based on AwsKmsMetadata. The instance of this class is initialized in Eth1Runner and FilecoinRunner. This class utilizes CachedAwsKmsClientFactory.
AwsKmsClient.java: Wraps Aws Kms Client library. Exposes sign and get ECPublicKey. Used by AwsKmsSigner
CachedAwsKmsClientFactory.java: Factory class for providing cached instances of AwsKmsClient. The AwsKmsClientKey.java is used as a key for cache entry. The credentials and region are used for equals and hashcode. It is anticipated that same credentials/region will be specified in AwsKmsMetadata yaml files to perform various operations of multiple keys.
signing/secp256k1/util

Eth1SignatureUtil.java - Refactored signature calculation for internal usage i.e. R, S and V. Azure return P1363Encoded signature while AWS returns ANS.1/DER encoded signature.
common/config
AwsAuthenticationMode.java is moved into common as this file is common both for Aws Secrets Manager and Aws Kms Client. In addition, AwsCredentials.java is also created in common. This file is currently used by Aws Kms Client, however, in future, it should also be used by Aws Secrets Manager related classes.

signing/config

AwsCredentialsProviderFactory.java: Factory class that return AWS library's AwsCredentialsProvider from AwsAuthenticationMode and AwsCredentials (which are derived from metadata deserializer).
signing/config/metadata

AwsKmsMetadata.java - Represents new configuration type aws-kms. Supports only SECP256K1.
AwsKmsMetadataDeserializer.java is responsible to convert yaml -> AwsKmsMetadata during metadata files loading. It performs validation of required fields. Very similar to existingAwsKeySigningMetadataDeserializer.java.

core

Eth1Runner and FilecoinRunner are modified to construct AwsKmsSignerFactory with awsKmsClientCacheSize and applySha3Hash flag.

commandline/subcommands

Eth1SubCommand.java and FilecoinSubCommand.java are modified to introduce cli option --aws-kms-client-cache-size which is used to construct AwsKmsSignerFactory in the runner classes mentioned above.

CHANGELOG.md

@@ -6,6 +6,7 @@
 - Eth2 Azure command line option --azure-secrets-tags is now deprecated and is replaced with --azure-tags. The --azure-secrets-tags option will be removed in a future release.
 
 ### Features Added
+- Add support for SECP256K1 remote signing using AWS Key Management Service. [#501](https://github.com/Consensys/web3signer/issues/501)
 - Azure bulk mode support for loading multiline (`\n` delimited, up to 200) keys per secret.
 - Hashicorp connection properties can now override http protocol to HTTP/1.1 from the default of HTTP/2. [#817](https://github.com/ConsenSys/web3signer/pull/817)
 - Add --key-config-path as preferred alias to --key-store-path [#826](https://github.com/Consensys/web3signer/pull/826)

acceptance-tests/build.gradle

@@ -25,6 +25,7 @@ dependencies {
   testImplementation project(":core")
   testImplementation project(":signing")
   testImplementation project(":commandline")
+  testImplementation project(":common")
 
   testImplementation 'org.apache.tuweni:tuweni-bytes'
   testImplementation 'org.apache.tuweni:tuweni-units'
@@ -84,6 +85,7 @@ dependencies {
 
   implementation 'software.amazon.awssdk:auth'
   implementation 'software.amazon.awssdk:secretsmanager'
+  implementation 'software.amazon.awssdk:kms'
 }
 
 test.enabled = false

acceptance-tests/src/test/java/tech/pegasys/web3signer/dsl/utils/MetadataFileHelpers.java

@@ -25,12 +25,16 @@
 import tech.pegasys.signers.bls.keystore.model.Pbkdf2Param;
 import tech.pegasys.signers.bls.keystore.model.SCryptParam;
 import tech.pegasys.teku.bls.BLSKeyPair;
+import tech.pegasys.web3signer.common.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.dsl.HashicorpSigningParams;
 import tech.pegasys.web3signer.keystore.hashicorp.dsl.certificates.CertificateHelpers;
 import tech.pegasys.web3signer.signing.KeyType;
+import tech.pegasys.web3signer.signing.config.metadata.AwsKmsMetadata;
+import tech.pegasys.web3signer.signing.config.metadata.AwsKmsMetadataDeserializer;
 
 import java.io.IOException;
 import java.io.Serializable;
+import java.net.URI;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.security.SecureRandom;
@@ -262,6 +266,37 @@ public void createAwsYamlFileAt(
     }
   }
 
+  public void createAwsKmsYamlFileAt(
+      final Path metadataFilePath,
+      final String awsRegion,
+      final String accessKeyId,
+      final String secretAccessKey,
+      final Optional<String> sessionToken,
+      final Optional<URI> endpointOverride,
+      final String kmsKeyId) {
+    try {
+      final Map<String, String> signingMetadata = new HashMap<>();
+
+      signingMetadata.put("type", AwsKmsMetadata.TYPE);
+      signingMetadata.put(
+          AwsKmsMetadataDeserializer.AUTH_MODE, AwsAuthenticationMode.SPECIFIED.toString());
+      signingMetadata.put(AwsKmsMetadataDeserializer.REGION, awsRegion);
+      signingMetadata.put(AwsKmsMetadataDeserializer.ACCESS_KEY_ID, accessKeyId);
+      signingMetadata.put(AwsKmsMetadataDeserializer.SECRET_ACCESS_KEY, secretAccessKey);
+      sessionToken.ifPresent(
+          token -> signingMetadata.put(AwsKmsMetadataDeserializer.SESSION_TOKEN, token));
+      endpointOverride.ifPresent(
+          endpoint ->
+              signingMetadata.put(
+                  AwsKmsMetadataDeserializer.ENDPOINT_OVERRIDE, endpoint.toString()));
+      signingMetadata.put(AwsKmsMetadataDeserializer.KMS_KEY_ID, kmsKeyId);
+
+      createYamlFile(metadataFilePath, signingMetadata);
+    } catch (final Exception e) {
+      throw new RuntimeException("Unable to construct aws-kms yaml file", e);
+    }
+  }
+
   private void createPasswordFile(final Path passwordFilePath, final String password) {
     try {
       Files.writeString(passwordFilePath, password);

acceptance-tests/src/test/java/tech/pegasys/web3signer/tests/bulkloading/AwsSecretsManagerAcceptanceTest.java

@@ -18,9 +18,9 @@
 
 import tech.pegasys.teku.bls.BLSKeyPair;
 import tech.pegasys.web3signer.AwsSecretsManagerUtil;
+import tech.pegasys.web3signer.common.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.dsl.signer.SignerConfigurationBuilder;
 import tech.pegasys.web3signer.signing.KeyType;
-import tech.pegasys.web3signer.signing.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.signing.config.AwsSecretsManagerParameters;
 import tech.pegasys.web3signer.signing.config.AwsSecretsManagerParametersBuilder;
 import tech.pegasys.web3signer.tests.AcceptanceTestBase;

acceptance-tests/src/test/java/tech/pegasys/web3signer/tests/bulkloading/AwsSecretsManagerMultiValueAcceptanceTest.java

@@ -18,9 +18,9 @@
 import tech.pegasys.teku.bls.BLSPublicKey;
 import tech.pegasys.web3signer.AwsSecretsManagerUtil;
 import tech.pegasys.web3signer.BLSTestUtil;
+import tech.pegasys.web3signer.common.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.dsl.signer.SignerConfigurationBuilder;
 import tech.pegasys.web3signer.signing.KeyType;
-import tech.pegasys.web3signer.signing.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.signing.config.AwsSecretsManagerParameters;
 import tech.pegasys.web3signer.signing.config.AwsSecretsManagerParametersBuilder;
 import tech.pegasys.web3signer.tests.AcceptanceTestBase;

acceptance-tests/src/test/java/tech/pegasys/web3signer/tests/bulkloading/AwsSecretsManagerPerformanceAcceptanceTest.java

@@ -16,9 +16,9 @@
 
 import tech.pegasys.teku.bls.BLSKeyPair;
 import tech.pegasys.web3signer.AwsSecretsManagerUtil;
+import tech.pegasys.web3signer.common.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.dsl.signer.SignerConfigurationBuilder;
 import tech.pegasys.web3signer.signing.KeyType;
-import tech.pegasys.web3signer.signing.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.signing.config.AwsSecretsManagerParameters;
 import tech.pegasys.web3signer.signing.config.AwsSecretsManagerParametersBuilder;
 import tech.pegasys.web3signer.tests.AcceptanceTestBase;

acceptance-tests/src/test/java/tech/pegasys/web3signer/tests/signing/SecpSigningAcceptanceTest.java

@@ -17,27 +17,40 @@
 import static org.assertj.core.api.Assertions.assertThat;
 import static org.web3j.crypto.Sign.signedMessageToKey;
 
+import tech.pegasys.web3signer.common.config.AwsAuthenticationMode;
+import tech.pegasys.web3signer.common.config.AwsCredentials;
 import tech.pegasys.web3signer.dsl.HashicorpSigningParams;
 import tech.pegasys.web3signer.dsl.utils.MetadataFileHelpers;
 import tech.pegasys.web3signer.keystore.hashicorp.dsl.HashicorpNode;
 import tech.pegasys.web3signer.signing.KeyType;
+import tech.pegasys.web3signer.signing.config.AwsCredentialsProviderFactory;
 import tech.pegasys.web3signer.signing.secp256k1.EthPublicKeyUtils;
+import tech.pegasys.web3signer.signing.secp256k1.aws.AwsKmsClient;
+import tech.pegasys.web3signer.signing.secp256k1.aws.CachedAwsKmsClientFactory;
 
 import java.io.File;
 import java.math.BigInteger;
+import java.net.URI;
 import java.net.URISyntaxException;
 import java.nio.file.Path;
 import java.security.SignatureException;
 import java.security.interfaces.ECPublicKey;
+import java.util.Map;
 import java.util.Optional;
 
+import com.google.common.collect.Maps;
 import com.google.common.io.Resources;
 import io.restassured.response.Response;
 import org.apache.tuweni.bytes.Bytes;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
 import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariables;
 import org.web3j.crypto.Sign.SignatureData;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.services.kms.model.CreateKeyRequest;
+import software.amazon.awssdk.services.kms.model.KeySpec;
+import software.amazon.awssdk.services.kms.model.KeyUsageType;
+import software.amazon.awssdk.services.kms.model.ScheduleKeyDeletionRequest;
 
 public class SecpSigningAcceptanceTest extends SigningAcceptanceTestBase {
 
@@ -110,6 +123,57 @@ public void signDataWithKeyInAzure() {
     signAndVerifySignature(AZURE_PUBLIC_KEY_HEX_STRING);
   }
 
+  @Test
+  @EnabledIfEnvironmentVariables({
+    @EnabledIfEnvironmentVariable(
+        named = "RW_AWS_ACCESS_KEY_ID",
+        matches = ".*",
+        disabledReason = "RW_AWS_ACCESS_KEY_ID env variable is required"),
+    @EnabledIfEnvironmentVariable(
+        named = "RW_AWS_SECRET_ACCESS_KEY",
+        matches = ".*",
+        disabledReason = "RW_AWS_SECRET_ACCESS_KEY env variable is required"),
+    @EnabledIfEnvironmentVariable(
+        named = "AWS_ACCESS_KEY_ID",
+        matches = ".*",
+        disabledReason = "AWS_ACCESS_KEY_ID env variable is required"),
+    @EnabledIfEnvironmentVariable(
+        named = "AWS_SECRET_ACCESS_KEY",
+        matches = ".*",
+        disabledReason = "AWS_SECRET_ACCESS_KEY env variable is required"),
+  })
+  public void remoteSignWithAwsKMS() {
+    final String roAwsAccessKeyId = System.getenv("AWS_ACCESS_KEY_ID");
+    final String roAwsSecretAccessKey = System.getenv("AWS_SECRET_ACCESS_KEY");
+    final Optional<String> awsSessionToken =
+        Optional.ofNullable(System.getenv("AWS_SESSION_TOKEN"));
+    // default region to us-east-2 if environment variable is not defined.
+    final String region = Optional.ofNullable(System.getenv("AWS_REGION")).orElse("us-east-2");
+    // can be pointed to localstack
+    final Optional<URI> awsEndpointOverride =
+        Optional.ofNullable(System.getenv("AWS_ENDPOINT_OVERRIDE")).map(URI::create);
+
+    final Map.Entry<String, ECPublicKey> remoteAWSKMSKey = createRemoteAWSKMSKey();
+    final String awsKeyId = remoteAWSKMSKey.getKey();
+    final ECPublicKey ecPublicKey = remoteAWSKMSKey.getValue();
+
+    try {
+      METADATA_FILE_HELPERS.createAwsKmsYamlFileAt(
+          testDirectory.resolve("aws_kms_test.yaml"),
+          region,
+          roAwsAccessKeyId,
+          roAwsSecretAccessKey,
+          awsSessionToken,
+          awsEndpointOverride,
+          awsKeyId);
+
+      signAndVerifySignature(EthPublicKeyUtils.toHexString(ecPublicKey));
+
+    } finally {
+      markAwsKeyForDeletion(region, awsEndpointOverride, awsKeyId);
+    }
+  }
+
   private void signAndVerifySignature() {
     signAndVerifySignature(PUBLIC_KEY_HEX_STRING);
   }
@@ -141,4 +205,53 @@ private BigInteger recoverPublicKey(final SignatureData signature) {
       throw new IllegalStateException("signature cannot be recovered", e);
     }
   }
+
+  private static Map.Entry<String, ECPublicKey> createRemoteAWSKMSKey() {
+    final String region = Optional.ofNullable(System.getenv("AWS_REGION")).orElse("us-east-2");
+    final Optional<URI> awsEndpointOverride =
+        System.getenv("AWS_ENDPOINT_OVERRIDE") != null
+            ? Optional.of(URI.create(System.getenv("AWS_ENDPOINT_OVERRIDE")))
+            : Optional.empty();
+
+    final AwsCredentialsProvider rwAwsCredentialsProvider =
+        AwsCredentialsProviderFactory.createAwsCredentialsProvider(
+            AwsAuthenticationMode.SPECIFIED, Optional.of(getAwsCredentialsFromEnvVar()));
+    final AwsKmsClient rwKmsClient =
+        new CachedAwsKmsClientFactory(1)
+            .createKmsClient(rwAwsCredentialsProvider, region, awsEndpointOverride);
+    // create a test key
+    final CreateKeyRequest web3SignerTestingKey =
+        CreateKeyRequest.builder()
+            .keySpec(KeySpec.ECC_SECG_P256_K1)
+            .description("Web3Signer Testing Key")
+            .keyUsage(KeyUsageType.SIGN_VERIFY)
+            .build();
+
+    final String testKeyId = rwKmsClient.createKey(web3SignerTestingKey);
+    final ECPublicKey ecPublicKey = rwKmsClient.getECPublicKey(testKeyId);
+    return Maps.immutableEntry(testKeyId, ecPublicKey);
+  }
+
+  private static void markAwsKeyForDeletion(
+      String region, Optional<URI> awsEndpointOverride, String awsKeyId) {
+    // mark aws key for deletion
+    ScheduleKeyDeletionRequest deletionRequest =
+        ScheduleKeyDeletionRequest.builder().keyId(awsKeyId).pendingWindowInDays(7).build();
+    final AwsCredentialsProvider rwAwsCredentialsProvider =
+        AwsCredentialsProviderFactory.createAwsCredentialsProvider(
+            AwsAuthenticationMode.SPECIFIED, Optional.of(getAwsCredentialsFromEnvVar()));
+
+    final AwsKmsClient rwKmsClient =
+        new CachedAwsKmsClientFactory(1)
+            .createKmsClient(rwAwsCredentialsProvider, region, awsEndpointOverride);
+    rwKmsClient.scheduleKeyDeletion(deletionRequest);
+  }
+
+  private static AwsCredentials getAwsCredentialsFromEnvVar() {
+    return AwsCredentials.builder()
+        .withAccessKeyId(System.getenv("RW_AWS_ACCESS_KEY_ID"))
+        .withSecretAccessKey(System.getenv("RW_AWS_SECRET_ACCESS_KEY"))
+        .withSessionToken(System.getenv("AWS_SESSION_TOKEN"))
+        .build();
+  }
 }

commandline/src/main/java/tech/pegasys/web3signer/commandline/PicoCliAwsSecretsManagerParameters.java

@@ -12,7 +12,7 @@
  */
 package tech.pegasys.web3signer.commandline;
 
-import tech.pegasys.web3signer.signing.config.AwsAuthenticationMode;
+import tech.pegasys.web3signer.common.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.signing.config.AwsSecretsManagerParameters;
 
 import java.net.URI;

commandline/src/main/java/tech/pegasys/web3signer/commandline/subcommands/Eth1SubCommand.java

@@ -140,6 +140,8 @@ public void setDownstreamHttpPath(final String path) {
       arity = "1")
   private String httpProxyPassword = null;
 
+  private long awsKmsClientCacheSize = 1;
+
   @CommandLine.Mixin private PicoCliClientTlsOptions clientTlsOptions;
 
   @CommandLine.Mixin private PicoCliEth1AzureKeyVaultParameters azureKeyVaultParameters;
@@ -213,4 +215,23 @@ public ChainIdProvider getChainId() {
   public AzureKeyVaultParameters getAzureKeyVaultConfig() {
     return azureKeyVaultParameters;
   }
+
+  @CommandLine.Option(
+      names = {"--aws-kms-client-cache-size"},
+      paramLabel = "<LONG>",
+      defaultValue = "1",
+      description =
+          "AWS Kms Client cache size. Should be set based on different set of credentials and region (default: ${DEFAULT-VALUE})")
+  public void setAwsKmsClientCacheSize(long awsKmsClientCacheSize) {
+    if (awsKmsClientCacheSize < 1) {
+      throw new CommandLine.ParameterException(
+          spec.commandLine(), "--aws-kms-client-cache-size must be positive");
+    }
+    this.awsKmsClientCacheSize = awsKmsClientCacheSize;
+  }
+
+  @Override
+  public long getAwsKmsClientCacheSize() {
+    return awsKmsClientCacheSize;
+  }
 }

commandline/src/main/java/tech/pegasys/web3signer/commandline/subcommands/Eth2SubCommand.java

@@ -29,9 +29,9 @@
 import tech.pegasys.web3signer.commandline.PicoCliEth2AzureKeyVaultParameters;
 import tech.pegasys.web3signer.commandline.PicoCliSlashingProtectionParameters;
 import tech.pegasys.web3signer.commandline.config.PicoKeystoresParameters;
+import tech.pegasys.web3signer.common.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.core.Eth2Runner;
 import tech.pegasys.web3signer.core.Runner;
-import tech.pegasys.web3signer.signing.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.signing.config.KeystoresParameters;
 import tech.pegasys.web3signer.slashingprotection.SlashingProtectionParameters;
 

commandline/src/main/java/tech/pegasys/web3signer/commandline/subcommands/FilecoinSubCommand.java

@@ -16,6 +16,7 @@
 import tech.pegasys.web3signer.core.Runner;
 import tech.pegasys.web3signer.signing.filecoin.FilecoinNetwork;
 
+import picocli.CommandLine;
 import picocli.CommandLine.Command;
 import picocli.CommandLine.HelpCommand;
 import picocli.CommandLine.Option;
@@ -28,6 +29,7 @@
 public class FilecoinSubCommand extends ModeSubCommand {
 
   public static final String COMMAND_NAME = "filecoin";
+  @CommandLine.Spec private CommandLine.Model.CommandSpec spec; // injected by picocli
 
   @Option(
       names = {"--network"},
@@ -36,9 +38,11 @@ public class FilecoinSubCommand extends ModeSubCommand {
       arity = "1")
   private final FilecoinNetwork network = FilecoinNetwork.MAINNET;
 
+  private long awsKmsClientCacheSize = 1;
+
   @Override
   public Runner createRunner() {
-    return new FilecoinRunner(config, network);
+    return new FilecoinRunner(config, network, awsKmsClientCacheSize);
   }
 
   @Override
@@ -50,4 +54,18 @@ public String getCommandName() {
   protected void validateArgs() {
     // no special validation required
   }
+
+  @CommandLine.Option(
+      names = {"--aws-kms-client-cache-size"},
+      paramLabel = "<LONG>",
+      defaultValue = "1",
+      description =
+          "AWS Kms Client cache size. Should be set based on different set of credentials and region (default: ${DEFAULT-VALUE})")
+  public void setAwsKmsClientCacheSize(long awsKmsClientCacheSize) {
+    if (awsKmsClientCacheSize < 1) {
+      throw new CommandLine.ParameterException(
+          spec.commandLine(), "--aws-kms-client-cache-size must be positive");
+    }
+    this.awsKmsClientCacheSize = awsKmsClientCacheSize;
+  }
 }

commandline/src/test/java/tech/pegasys/web3signer/commandline/CommandlineParserTest.java

@@ -25,11 +25,11 @@
 import static tech.pegasys.web3signer.commandline.PicoCliAwsSecretsManagerParameters.AWS_SECRETS_TAG_VALUES_FILTER_OPTION;
 
 import tech.pegasys.web3signer.commandline.subcommands.Eth2SubCommand;
+import tech.pegasys.web3signer.common.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.core.Context;
 import tech.pegasys.web3signer.core.Runner;
 import tech.pegasys.web3signer.core.config.BaseConfig;
 import tech.pegasys.web3signer.signing.ArtifactSignerProvider;
-import tech.pegasys.web3signer.signing.config.AwsAuthenticationMode;
 import tech.pegasys.web3signer.signing.config.DefaultArtifactSignerProvider;
 
 import java.io.PrintWriter;

signing/src/main/java/tech/pegasys/web3signer/signing/config/AwsAuthenticationMode.java → common/src/main/java/tech/pegasys/web3signer/common/config/AwsAuthenticationMode.java

@@ -10,7 +10,7 @@
  * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
  * specific language governing permissions and limitations under the License.
  */
-package tech.pegasys.web3signer.signing.config;
+package tech.pegasys.web3signer.common.config;
 
 public enum AwsAuthenticationMode {
   ENVIRONMENT,