feat: support enabled status for kmp keys/certs #1874

duffney · 2024-10-17T19:01:54Z

Description

What this PR does / why we need it:

This PR adds an Enabled field to the KMPMapyKey struct which allows the provider to store the status of certificates and keys being pulled from the provider into Ratify's KMP. Which is needed to support multiple versions of certificates and keys being stored in the KMP. See discussions and the design doc, here, for more details.

Which issue(s) this PR fixes (optional, using `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when the PR gets merged):

Fixes:

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Please also list any relevant details for your test configuration

Deployed to AKS for Manual testing

> k describe keymanagementprovider akv
Name:         akv
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  config.ratify.deislabs.io/v1beta1
Kind:         KeyManagementProvider
Metadata:
  Creation Timestamp:  2024-10-17T14:56:38Z
  Generation:          1
  Resource Version:    5614657
  UID:                 dbb53be3-1dab-4ad5-be79-f565c3269697
Spec:
  Parameters:
    Certificates:
      Name:     ratify
      Version:  5fe70628ce34425ca1396e88c509d355
      Name:     ratify
      Version:  0ff373a9259c4578a247cfd7861a8805
    Client ID:  38f733d4-05f9-41b3-b8e1-896376ba52d6
    Keys:
      Name:          issue1751
    Tenant ID:       b4c72be8-cae1-4584-be77-62b1e94ad0dc
    Vault URI:       https://example.vault.azure.net/
  Refresh Interval:  1m
  Type:              azurekeyvault
Status:
  Issuccess:        true
  Lastfetchedtime:  2024-10-17T18:56:50Z
  Properties:
    Certificates:
      Enabled:         false
      Last Refreshed:  2024-10-17T18:56:50Z
      Name:            ratify
      Version:         5fe70628ce34425ca1396e88c509d355
      Enabled:         true
      Last Refreshed:  2024-10-17T18:56:50Z
      Name:            ratify
      Version:         0ff373a9259c4578a247cfd7861a8805
    Keys:
      Enabled:         false
      Last Refreshed:  2024-10-17T18:56:50Z
      Name:            issue1751
      Version:         
Events:                <none>

> k logs deployment/ratify -n gatekeeper-system
time=2024-10-17T18:56:50.402944925Z level=info msg=reconciling cluster key management provider 'akv'
time=2024-10-17T18:56:50.403102625Z level=debug msg=vaultURI https://example.vault.azure.net/ component-type=keyManagementProvider go.version=go1.22.8
time=2024-10-17T18:56:50.495968077Z level=debug msg=fetching secret from key vault, certName ratify,  keyvault https://example.vault.azure.net/ component-type=keyManagementProvider go.version=go1.22.8
time=2024-10-17T18:56:50.525906097Z level=debug msg=fetching secret from key vault, certName ratify,  keyvault https://example.vault.azure.net/ component-type=keyManagementProvider go.version=go1.22.8
time=2024-10-17T18:56:50.578216358Z level=debug msg=azurekeyvault certprovider getCertsFromSecretBundle: 1 certificates parsed, Certificate 'ratify', version '0ff373a9259c4578a247cfd7861a8805' component-type=keyManagementProvider go.version=go1.22.8
time=2024-10-17T18:56:50.578267858Z level=debug msg=fetching key from key vault, keyName issue1751,  keyvault https://example.vault.azure.net/ component-type=keyManagementProvider go.version=go1.22.8
time=2024-10-17T18:56:50.595643612Z level=info msg=2 certificate(s) & 1 key(s) fetched for key management provider akv
time=2024-10-17T18:56:50.595670912Z level=info msg=Reconciled KeyManagementProviderintervalDuration1m0s

Unit test currently do not fully test the GetCertificate and GetKeys methods of KMP providers, but can be added if desired to the PR or a follow up PR. Also waiting to see if the azidentiy work will be done before adding unit tests because that will likely change the client that will need to be mocked to fully tests the methods.

Checklist:

Does the affected code have corresponding tests?
Are the changes documented, not just with inline documentation, but also with conceptual documentation such as an overview of a new feature, or task-based documentation like a tutorial? Consider if this change should be announced on your project blog.
Does this introduce breaking changes that would require an announcement or bumping the major version?
Do all new files have appropriate license header?

Post Merge Requirements

MAINTAINERS: manually trigger the "Publish Package" workflow after merging any PR that indicates Helm Chart Change

codecov · 2024-10-17T19:06:14Z

Codecov Report

Attention: Patch coverage is 19.51220% with 33 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...kg/keymanagementprovider/azurekeyvault/provider.go	19.51%	33 Missing ⚠️

Files with missing lines	Coverage Δ
pkg/keymanagementprovider/keymanagementprovider.go	`92.94% <ø> (ø)`
...kg/keymanagementprovider/azurekeyvault/provider.go	`65.74% <19.51%> (-9.14%)`	⬇️

akashsinghal · 2024-10-21T18:17:48Z

pkg/keymanagementprovider/azurekeyvault/provider.go

+			fmt.Printf("debug: certificate %s version %s is disabled.", keyVaultCert.Name, keyVaultCert.Version)
+
+			isEnabled := "false"
+			startTime := time.Now()


The reported metric is for the duration of the keyvault fetch operation. We should move the metric reporting to be isolated to that operation above.

Ah, I see that now. Now that there are two calls to AKV I can define the startTime sooner to capture the first call then reset the value for the second call.

akashsinghal · 2024-10-21T18:19:45Z

pkg/keymanagementprovider/azurekeyvault/provider.go

+			// GetSecret is required so we can fetch the entire cert chain. See issue https://github.com/ratify-project/ratify/issues/695 for details
+			isEnabled := "true"
+			startTime := time.Now()
+			secretBundle, err := s.kvClient.GetSecret(ctx, s.vaultURI, keyVaultCert.Name, keyVaultCert.Version)


Are we making two calls now to Keyvault? One for an initial GetCertificates and then another for a GetSecret?

That's correct. Instead of attempting to trap a specific error from GetSecrets I opted to use GetCertificates to get the enabled status of the certificate to determine how to add the entry to the KMP.

Oh ok. I see. I'm concerned with the network overhead and how that will affect upstream throttling quotas for AKV. What is the value of doing the existence check first and then calling GetSecret. Is it primarily so the disabled certificate content is not returned as part of the response from KeyVault?

That's correct, with GetCertificate I can check the status of enabled without returning any of the secret data. It also avoids having to trap a particular error from AKV when the cert is disabled when GetSecret is called.

That said, I'm open to suggestions. :)

akashsinghal · 2024-10-21T18:20:34Z

pkg/keymanagementprovider/azurekeyvault/provider.go

+		if !*certBundle.Attributes.Enabled {
+			fmt.Printf("debug: certificate %s version %s is disabled.", keyVaultCert.Name, keyVaultCert.Version)
+
+			isEnabled := "false"


why are these typed as strings and not boolean?

No good reason, the KMPMapyKey field can and probably should be a bool. That's left over from when I was testing updating the status which requires a string to be passed to getStatusProperty. IMO, bool makes more sense and I'll start working on that change for the map key.

akashsinghal · 2024-10-21T18:33:07Z

pkg/keymanagementprovider/azurekeyvault/provider.go

+			certProperty := getStatusProperty(keyVaultCert.Name, keyVaultCert.Version, isEnabled, lastRefreshed)
+			certsStatus = append(certsStatus, certProperty)
+			certMapKey := keymanagementprovider.KMPMapKey{Name: keyVaultCert.Name, Version: keyVaultCert.Version, Enabled: isEnabled}
+			certsMap[certMapKey] = []*x509.Certificate{} // empty cert chain


where is the certificate value (without chain) set in certsMap?

When the certificate is disabled, the certificate isn't stored at all. My thinking was that this would prevent the FlattenKMPMap function from including any disabled certificates. Self-signed certs for example could be used to verify if that cert was passed to the verifier because it doesn't have any cert chain beyond itself. That said, now that there is an enabled field it's possible to update the Flatten function to look for only enabled certs. Your comment does bring up a good question, what value should be used for disabled certs?

QQ: by certificate value, you're meaning the cert in x509 format?

I didn't realize this was for the disabled case. My bad. I should've looked at the if block condition above. Can you remind me if we plan to show if a certificate is disabled at all in status? I'm trying to understand if the operation on L165 is to clear map value but still have the key available to signify it is disabled.

Not a problem. :) And yes, the status is now shown in the KMP status. Here's an example:

Certificates: Enabled: true Last Refreshed: 2024-10-21T20:27:23Z Name: ratify Version: 0ff373a9259c4578a247cfd7861a8804

L165 is meant to clear the value so the disabled certificate data so it's not stored in the KMP but the key remains. The key values are then used to populate the status.

Technically, speaking I don't think there is any reason the key need to remain in the map either, other than to match the status that's being reflected to the user.

akashsinghal · 2024-10-21T18:33:42Z

pkg/keymanagementprovider/azurekeyvault/provider.go

+			certsMap[certMapKey] = []*x509.Certificate{} // empty cert chain
+		} else {
+			// GetSecret is required so we can fetch the entire cert chain. See issue https://github.com/ratify-project/ratify/issues/695 for details
+			isEnabled := "true"


general comment: if we decide to use strings instead of boolean here, I'd recommend defining constants for "true" and "false"

Signed-off-by: Joshua Duffney <[email protected]>

…vider Signed-off-by: Joshua Duffney <[email protected]>

duffney requested review from akashsinghal, binbin-li, jimmyraywv, luisdlp, susanshi and toddysm as code owners October 17, 2024 19:01

akashsinghal reviewed Oct 21, 2024

View reviewed changes

duffney added 3 commits October 21, 2024 13:44

mod: support enabled status for kmp keys/certs

440de8d

Signed-off-by: Joshua Duffney <[email protected]>

fix: remove trailing newline

787420c

Signed-off-by: Joshua Duffney <[email protected]>

mod: update empty latest string with latest version from kid

3ffa53e

Signed-off-by: Joshua Duffney <[email protected]>

duffney force-pushed the issue-1751/enabled-status branch from 38091b6 to 3ffa53e Compare October 21, 2024 18:44

mod: set KMPMapKey Enabled to bool & fix metrics reporting in akv pro…

ee5d257

…vider Signed-off-by: Joshua Duffney <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support enabled status for kmp keys/certs #1874

feat: support enabled status for kmp keys/certs #1874

duffney commented Oct 17, 2024

codecov bot commented Oct 17, 2024 •

edited

Loading

akashsinghal Oct 21, 2024

duffney Oct 21, 2024

akashsinghal Oct 21, 2024

duffney Oct 21, 2024 •

edited

Loading

akashsinghal Oct 21, 2024

duffney Oct 21, 2024

akashsinghal Oct 21, 2024

duffney Oct 21, 2024

akashsinghal Oct 21, 2024

duffney Oct 21, 2024 •

edited

Loading

akashsinghal Oct 21, 2024

duffney Oct 21, 2024

akashsinghal Oct 21, 2024

feat: support enabled status for kmp keys/certs #1874

Are you sure you want to change the base?

feat: support enabled status for kmp keys/certs #1874

Conversation

duffney commented Oct 17, 2024

Description

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):

Type of change

How Has This Been Tested?

Checklist:

Post Merge Requirements

codecov bot commented Oct 17, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duffney Oct 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duffney Oct 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Which issue(s) this PR fixes (optional, using `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when the PR gets merged):

codecov bot commented Oct 17, 2024 •

edited

Loading

duffney Oct 21, 2024 •

edited

Loading

duffney Oct 21, 2024 •

edited

Loading