Monitor API

[calvinrp]

Background

Monitors in Certificate Transparency "watch logs and check that they behave correctly". They verify things that may be impossible or impractical for every client to verify for themselves.

Introduction

In Package Transparency, monitors watch different logs and check different things than in Certificate Transparency but abstractly fill an analogous role.

In the protocol as discussed to this point, monitors are expected to verify two distinct things:

1. That the log and the map in a checkpoint are consistent with each other (i.e. that applying every update in the log produces the map);
2. That no fork attack has occurred or that if there is one we are using a checkpoint on the "correct" branch of the log history;

Because the state of the registry could theoretically diverge into two valid histories in the event of a compromised or malicious registry performing a fork attack, checking in with monitors provides a way for everyone to know they are working from the same history.

Proposal

1. New endpoint for registries and monitors:
   POST /v1/verify/checkpoint
   Request body is the signed timestamped checkpoint envelope to be verified. Request header that specifies the the registry domain of the checkpoint. Response is success status code or error status code with an error messsage.
2. New endpoint for registries:
   GET /v1/fetch/ledger/{startingRegistryIndex}
   Response is the series of [[RegistryIndex, LogId, RecordId]]. Or this could be endpoint that provides URLs to binary blobs which support HTTP Range header requests? However, I'm unsure if this would be hash algo independent.

Do we require a registry API endpoint to fetch the series of RegistryLen values corresponding to the logLength for every registry checkpoint?

The Warg CLI client can set monitors for a registry. After a new checkpoint is fetched, the client will send a request to a monitor for independent checkpoint verification.

Just a quick thank you to @esoterra for helping with intro and background.

[lann]

Response is the series of [[RegistryIndex, LogId, RecordId]]. Or this could be endpoint that provides URLs to binary blobs which support HTTP Range header requests? However, I'm unsure if this would be hash algo independent.

The API endpoint could just return metadata and location(s) for the binary data, maybe something along the lines of:

{
  "hash_algorithm": "sha256",
  "sources": [
    {
      "content_type": "application/vnd.bytecodealliance.registry.ledger.packed", // or something
      "start_index": 0,
      "end_index": 10000,
      "urls": ["https://my.blob.store/ledger/abcdefg/0-10000.bin"],
    },
    {
      "content_type": "application/vnd.bytecodealliance.registry.ledger.packed",
      "start_index": 10001,
      // no end_index; still being "filled"
      "urls": ["https://my.blob.store/ledger/abcdefg/10001-latest.bin"],
    },
  ]
}

[calvinrp]

I'm a fan of an implementation of this nature. Can a registry transition to a new hash algorithm?

[lann]

Can a registry transition to a new hash algorithm?

We've discussed this in the past and my vague belief is that it should be possible with protocol extensions, though I'm not super confident in that. In any case, the details would depend on why a new hash algorithm is needed; there might be a different answer if sha256 was shown to be vulnerable to a preimage attack than just a collision attack; a collision attack might lead you to want to leave history alone and just migrate new log entries to another algo, while a (less likely!) preimage attack would probably force you to rewrite all of history since all content digests would be vulnerable.

[esoterra]

To echo that sentiment, we've designed in information that we expect will make such a transition more feasible though we don't have an exact plan for how such a transition would work. Annotating all cryptographic data with the algorithm used is part of that attempt to have some amount of "cryptographic agility".

[lann]

POST /v1/verify/checkpoint

Another useful thing a registry can do here (probably a "should" but not "must"): if an invalid checkpoint is received with a valid signature, it should complain loudly, e.g. by sending an email to some list.

[lann]

Do we require a registry API endpoint to fetch the series of RegistryLen values corresponding to the logLength for every registry checkpoint?

My hunch is that this isn't necessary. A monitor will already need to have replayed history and reconstructed all possible verifiable maps in order to validate correct behavior; it can either store every valid checkpoint or - if necessary - store snapshots of the log/map state every e.g. 1000 updates and replay history from the closest snapshot.

[esoterra]

They will have reconstructed all possible maps, but to avoid needing to do dynamic partial replay, they'll have to store every the log/map root for ever increment not just increments where checkpoints were actually taken.

For high-throughput registries that take checkpoints only every 10 to 100 updates, knowing where checkpoints were actually taken can give a 10 to 100 time storage reduction while avoiding needing to recompute things on the fly which can get expensive.

[lann]

Maybe there is a way to split the difference here by allowing a monitor to give an indeterminate answer: "I don't have any information about that point in time". I still think it is reasonable for "big, beefy" monitors to validate any possible checkpoint (by storing them all or by dynamically recomputing from a map snapshot), which would allow registries to drop old checkpoints. At the same time, this would let a "lightweight" monitor only store those checkpoints it has actually seen, or even drop "very old" checkpoints itself.

[calvinrp]

If a checkpoint is newer than the monitor has last processed, how should the monitor respond? It is realistic to expect some sort of delay in processing. Doubt we want to block and wait.

Not sure how we want to handle that. Client could check back with the monitor.

[lann]

There could be multiple flavors of "I don't have that information...":

• "...because I never saw that checkpoint."
• "...because I throw away checkpoints that old."
• "...because I haven't synced recently enough."

Doubt we want to block and wait.

Not synchronously, but another possible response could be: "I don't have that information yet but I am now actively processing it; check back in X seconds."

[calvinrp]

Proposed API Changes:

For Monitors & Registries:

Warg-Registry HTTP Header

For all API endpoints, if the Warg-Registry header is present in the request, the response will be as if the registry at the domain name specified in the header value was the one responding.

The response will include the same Warg-Registry header and value if the server is responding as that registry. If the response header is omitted, the client should assume that the server ignored the request header.

POST /v1/verify/checkpoint

Request body is the signed timestamped checkpoint envelope to be verified. The Warg-Registry specifies the domain name of the registry checkpoint that is being verified. Response is success status code or error status code with an error messsage.

Example:
Request Header Warg-Registry: registry.example.com
POST Body:

{
  "contents": {
    "logLength": 42,
    "logRoot": "sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c",
    "mapRoot": "sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c",
    "timestamp": 1692035502
  },
  "keyId": "sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c",
  "signature": "ecdsa-p256:MEUCIQCzWZBW6ux9LecP66Y+hjmLZTP/hZVz7puzlPTXcRT2wwIgQZO7nxP0nugtw18MwHZ26ROFWcJmgCtKOguK031Y1D0="
}

For Registries:

GET /v1/ledger

Response is a JSON:

{
  "hashAlgorithm": "sha256",
  "sources": [
    {
      "firstRegistryIndex": 0,
      "lastRegistryIndex": 99999,
      "url": "https://someurl.com/file-1",
      "acceptRanges": true,
      "contentType": "application/vnd.bytecodealliance.registry.ledger.packed"
    },
    {
      "firstRegistryIndex": 100000,
      "lastRegistryIndex": 149999,
      "url": "https://someurl.com/file-2",
      "acceptRanges": true,
      "contentType": "application/vnd.bytecodealliance.registry.ledger.packed"
    }
  ]
}

Each source is an HTTP GET that returns a binary file with Vec<(LogId, RecordId)> with both LogId and RecordId serialized according to the binary representation of the hashAlgorithm and the length of the Vec in the file equal to (lastRegistryIndex - firstRegistryIndex + 1). The firstRegistryIndex and lastRegistryIndex are inclusive ranges.

In the case of the example, this would be a repeating sequence of 64 bytes (32 bytes for each the LogId and RecordId). The sources array and the url data is in ascending order without gaps. acceptRanges field is optional for accepting HTTP Range header.

[lann]

For the checkpoints field, a list of registry log lengths is provided for recent checkpoints that monitors are expected to verify.

What is the intent behind this? Is it "these are the only checkpoints I have"?

[lann]

Do we need the wrapping "ledger": [...]" now? I think given that we don't have a design for migrating ledgers yet we'd want to specify "there must be exactly one entry in the ledger array" and at that point I think there are alternative future designs that don't require the extra structure today, like a "previousLedgers": [...] entry.

[lann]

Warg-Registry HTTP Header

• I think under the current design we should probably specify that the server must reflect the header back in the response (which the client should verify). Otherwise it would be very easy for a server to ignore the header and potentially accidentally return incorrect data.
• Should we allow redirect responses?

[calvinrp]

I'm OK with that. Updated.

[lann]

The sources array and the url data is in ascending order without gaps.

An alternative way to frame this might be that clients should search the array in order looking for an entry that includes (some of) what they want which would give a little more flexibility if we want to allow other content types or transports in the future.

To that point, should we include an e.g. "contentType": "application/vnd.bytecodealliance.registry.ledger.packed" entry?

[calvinrp]

Warg-Registry HTTP Header

• I think under the current design we should probably specify that the server must reflect the header back in the response (which the client should verify). Otherwise it would be very easy for a server to ignore the header and potentially accidentally return incorrect data.
• Should we allow redirect responses?

• I like that solve to that problem, providing the Warg-Registry header in the response to confirm the registry subject. Updated with that change;
• I'm OK with a redirect response option. But is it helpful if the client can directly talk to the source registry?

[lann]

But is it helpful if the client can directly talk to the source registry?

Uhhh....nope! 😅

[calvinrp]

For the checkpoints field, a list of registry log lengths is provided for recent checkpoints that monitors are expected to verify.

What is the intent behind this? Is it "these are the only checkpoints I have"?

This is based upon the assumption that only recent checkpoints are sent to a monitor for verify requests. Not sure about the value of verifying old checkpoints, but let me know if you feel otherwise.

Monitors respond to client requests to verify a registry's TimestampedCheckpoint. The client CLI would fetch the latest registry checkpoint and then immediately request monitors to verify that checkpoint. So, timing-wise, there would only be a few relevant checkpoints that should be expected for verification requests, at any time. Maybe some longer delay for those caching / mirroring.

Since registries don't necessarily produce a checkpoint after every logLength increment, it is helpful (less work for monitors) to know which logLength checkpoints were issued. But we only need the most recent logLengths that a monitor could conceivably receive a verification request for.

Thoughts?

[calvinrp]

Do we need the wrapping "ledger": [...]" now? I think given that we don't have a design for migrating ledgers yet we'd want to specify "there must be exactly one entry in the ledger array" and at that point I think there are alternative future designs that don't require the extra structure today, like a "previousLedgers": [...] entry.

This structure with "ledger": [...]" was intended not for multiple ledgers but for flexibility on having ledger segments with different hashAlgorithms. Does that make sense?

[lann]

OK sure, replace "ledgers" with "ledger chunks" or whatever we would call those things; I don't think that changes the argument.

[calvinrp]

Renamed the field ledgerChunks in the example. If we decide to drop the other field (checkpoints) in this object and are OK with an Array instead of an Object, we could have the Array value of ledgerChunks as the entire JSON response. Just to confirm, that is what you are thinking?

[lann]

I was thinking of flattening the structure:

{
  "hashAlgorithm": "sha256",
  "sources": [ ... ],
  "checkpoints": [ ... ]
}

with the idea that a hypothetical future hash migration feature could work as something like:

{
  "hashAlgorithm": "sha-4",
  "sources": [ ... ],
  "checkpoints": [ ... ],
  "oldLedgers": [
    {"hashAlgorithm": "sha256", "sources": ...}
  ]
}

This leaves the door open to this style of hash migration but I'm not sure how likely it is that we would want to do it this way; as discussed above the mechanism would probably depend on exactly why the hash migration was needed.

[calvinrp]

Ah, I'm cool with that. Simpler. Updated the proposal above.

[calvinrp]

The sources array and the url data is in ascending order without gaps.

An alternative way to frame this might be that clients should search the array in order looking for an entry that includes (some of) what they want which would give a little more flexibility if we want to allow other content types or transports in the future.

To that point, should we include an e.g. "contentType": "application/vnd.bytecodealliance.registry.ledger.packed" entry?

I'm OK with that. Updated.