From c5ed21f54fc9b0b13d24e3b41ab85eabd1235dd3 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Sun, 10 Sep 2023 23:11:24 -0700 Subject: [PATCH 01/18] Create Message Backup Requests XIP --- XIPs/xip-28-message-backup-request-system.md | 362 +++++++++++++++++++ 1 file changed, 362 insertions(+) create mode 100644 XIPs/xip-28-message-backup-request-system.md diff --git a/XIPs/xip-28-message-backup-request-system.md b/XIPs/xip-28-message-backup-request-system.md new file mode 100644 index 0000000..19e7e0b --- /dev/null +++ b/XIPs/xip-28-message-backup-request-system.md @@ -0,0 +1,362 @@ +--- +xip: 29 +title: Message backup request system +description: A system for requesting and receiving message backups between V3 installations +author: Nick Molnar +status: Draft +type: Standards Track +category: XRC +created: 2023-09-10 +--- + +## Abstract + +A proposed system for transferring backups of message history between installations as part of XMTP V3. + +## Motivation + +The current [XMTP V3 Protocol Specification](https://github.com/xmtp/libxmtp) allows new installations to be created and receive all messages sent to a blockchain account after it's creation. It does _not_ specify how messages sent before the installation was created can be loaded into the new installation. Having conversation history sync between devices is a popular feature of V1 and V2 of the XMTP protocol and there is clear demand from developers for this to be possible as part of V3. + +There are a number of cases where message history synchronization is desirable: + +- User installs App A and starts engaging in conversations. Later, they install App B on the same device and want access to their existing conversations and messages. +- User installs App A and starts engaging in conversations. Later, they install App B (or another copy of App A) on a second device and desire access to their existing conversations and messages. +- User installs App A and starts engaging in conversations. They lose access to the device App A was installed on and purchase a new device. They then need to restore their previous conversations and messages to the new device. +- User installs App A and starts engaging in conversations. App A is later compromised, and they need to migrate their messages and conversations to a different application. + +It's also worth noting that there are many use-cases for XMTP where message history synchronization is not necessary or desirable. + +- A bot that replies to messages from users. Most bots don't care about messages sent before they were created. +- A service that sends notifications to a list of subscribers via XMTP. +- An application that allows users to talk to customer support only needs access to the newly created conversation with the support agent. +- A marketplace application that wants to allow buyers and sellers to communicate about a particular item. These would only be new conversations and may not care about pre-existing messages. + +### Design goals + +1. Message history synchronization should not compromise the Forward Secrecy or Post Compromise Security properties of XMTP V3. We expect the majority of consumer applications using XMTP to have some form of message history syncing enabled. +1. Message history synchronization should be opt-in for developers. +1. It should be simple for developers to implement message history synchronization correctly. Implementation work should be focused on application-specific UX choices, with the SDK responsible for core transport and security decisions. +1. Compatibility between applications is expected. Applications should be able to share history irrespective of operating system, SDK version, or device type (web vs mobile vs server). +1. Even if developers of some popular applications choose to not support message history synchronization, it should be simple for users to opt themselves in + +## Specification + +This specification offers the user two modes of message backup, which accomodate different use-cases: Remote Message Backups and Backup Account Files. Remote Message Backups are the lowest friction solution, provided the blockchain account has access to another existing installation. Backup Account Files are an emergency solution for cases where the user has lost access to all of their XMTP applications (for example, if they lost the only device they have used to connect to XMTP). + +There are three types of actors in this specification: Backup Requesters, Message Backup Providers, and Backup Storage Providers. Backup requesters are XMTP Clients that are missing message history and would like to receive a backup. Message Backup Providers are XMTP Clients that have a message history and are capable of sending it to a requester. These first two types of actors are XMTP Clients, and the implementation required to fulfill the responsibilities of both roles would be built into `libxmtp`. The Backup Storage Provider is a remote service responsible for temporarily storing backup files, and is only needed for Remote Message Backups. + +### Remote Message Backups + +Remote message backups work by Backup Requesters sending a special message type (`MessageHistoryBackupRequest`) across the XMTP network to all other installations signed by the same blockchain account as the sender (Message Backup Providers). Upon receipt of these messages, Message Backup Providers should display a prompt to the user asking whether they consent to share their message history with the requesting installation. Upon approval, the application will convert their local database into a standard Message History Backup File and upload it to the Backup Storage Provider specified in the `MessageHistoryBackupRequest`. + +#### Flow for Backup Requesters + +1. Get a list of all other installations associated with their blockchain account. Ignore any installations that have been revoked +2. Obtain a one-time upload URL from the Backup Storage Provider for each installation +3. Generate a `MessageHistoryBackupRequest` for each installation (see below for Protobuf spec) and store in the local database +4. Send the `MessageHistoryBackupRequest` messages to the standard message topic for each installation +5. Wait for a response from any of the Message Backup Providers +6. For each `MessageHistoryBackupResponse` + 6a. Ensure there is a `MessageHistoryBackupRequest` with a matching `requestId` stored in the database. If not, ignore + 6b. Ensure the `backupUrl` is on the same host as the requested `backupStorageProviderUploadUrl`. If not, ignore + 6c. Download the file from the `backupUrl` and decrypt using the credentials provided in the `MessageHistoryBackupResponse`. If the hash of the downloaded file does not match the hash in the `MessageHistoryBackupResponse`, abort. + 6d. Load each message into the local database, ignoring any duplicate messages + 6e. Delete the `MessageHistoryBackupResponse` and all associated credentials + 6f. Set the status of the `MessageHistoryBackupRequest` to `Applied` + +#### Flow for Message Backup Providers + +1. Receive a `MessageHistoryBackupRequest` as part of normal message receiving flow +1. Retrieve the contact for the installation that sent the message, and ensure that it has not been revoked. If revoked, ignore. +1. Ensure that the installation that sent the message has a contact signed by the same blockchain account as the current user. If not, ignore. +1. Convert all messages in the local database into a Message Backup File (maybe we want to chunk here?) +1. Generate ephemeral encryption key, salt, and nonce. Encrypt the file using these keys +1. Upload the file to the `backupStorageProviderUploadUrl` from the request +1. Reply to the message with a `MessageHistoryBackupResponse` containing the encryption details, the hash of the backup file, and the `backupUrl` provided by the Backup Storage Provider +1. Delete the `MessageHistoryBackupRequest`, the local database dump, and all the encryption keys. + +#### End-to-end flow + +```mermaid +sequenceDiagram + Participant B as Backup Requester + Participant S as Backup Storage Provider + Participant N as XMTP Network + Participant M as Message Backup Provider + B->>S: Request a backup storage upload URL + S-->>B: Receive one time upload URL + B->>N: Send a MessageHistoryBackupRequest to each potential Message Backup Provider + M->>N: Check for MessageHistoryBackupRequest + N-->>M: Receive MessageHistoryBackupRequest + M->>S: Upload backup file + M->>N: Send MessageHistoryBackupResponse + B->>N: Check for MessageHistoryBackupResponse + N-->>B: Receive response + B->>S: Request file from URL in response + S-->>B: Receive file + B-->>B: Decrypt and load file into database +``` + +### Backup Account Files + +Backup account files are designed as a solution for users who have lost access to all of their XMTP applications. To create a Backup Account File, an application will generate a new set of XMTP keys and register the installation on the XMTP network. The installation contact published to the network will be set with a type of `backup`, which instructs all well-behaved clients to disregard any messages from this installation other than `MessageHistoryBackupResponse`s. This means that a compromised backup installation cannot be used to send messages on the user's behalf, which makes these files a less valuable target for attack. + +Once a backup account's contact has been published to the network, their installation messaging topic will receive a copy of all new messages sent to the associated blockchain account. + +The newly generated XMTP keys are encrypted and downloaded to the user's device. They may be encrypted by either a passphrase or a wallet signature. Users or applications may decide to store the Backup Account File in a private cloud storage provider (iCloud, Google Drive, etc). Or the file could simply be stored on-device with the risk of loss if the device was inaccessible. + +It is recommended - but not required - that applications provide an easy method to generate a Backup Account File. + +We may want to consider making backup client's contact bundle signed by the identity key of the installation that created it instead of a wallet. This removes the need for an additional wallet signature. Ownership could still be determined by following the chain of signatures back to the original wallet. This would make revocation more complicated - we would want to revoke the backup account alongside the parent account - but should be possible. We would have to be careful to ensure that backups could still be restored even if the main account was compromised. + +#### Restoring from a Backup Account File directly + +A client application can allow a user to restore from a Backup Account File directly, using the following steps: + +1. Load the Backup Account File from disk +1. Create a temporary client instance with a new location for the database +1. Call `receive` on the temporary client instance to download all messages stored on the network. +1. Convert all messages into a Message Backup File +1. Import the Message Backup File into the main account's database +1. Delete the database associated with the temporary client instance and the Message Backup File + +While some applications may choose to support Backup Account Files directly, we can also offer means to convert Backup Account Files into Remote Message Backups. This allows developers to support all possible backup scenarios simply by supporting the more common Remote Message Backups. + +#### Converting a Backup Account File to a Remote Message Backup + +XMTP Labs should create a simple web application to convert Backup Account Files to Remote Message Backups. In this application a user could import their Backup Account File (with the file never leaving their machine), and create a temporary client instance with that backup file. The user would be presented with any outstanding `MessageHistoryBackupRequests`, and select any backup requests they would like to fulfill. The client would download all unread messages from the network and proceed with the regular Message Backup Provider flow for Remote Message Backups. All data would be cleared from the client as soon as the operation was completed. + +This same web application could be used to create Backup Account Files for cases where a user's preferred XMTP application does not support creating Backup Account Files itself. This means every user on the XMTP network is able to create a Backup Account File if desired. + +### Backup Storage Provider + +A Backup Storage Provider is a simple HTTP service with two endpoints. Anyone can implement a Backup Storage Provider. It is up to the Backup Requester to choose the Backup Service Provider for their application. + +Backup Storage Providers only need to implement three APIs: + +`POST /backups`: +Example response: + +```json +{ + "uploadUrl": "https://backupproviderdomain.com/backups/some-long-unguessable-upload-id" +} +``` + +`POST /backups/$UPLOAD_ID`: +Example response: + +```json +{ + "downloadUrl": "https://backupproviderdomain.com/backups/some-long-unguessable-download-id" +} +``` + +`GET /backups/$DOWNLOAD_ID`: +Returns the uploaded file matching the ID + +It would be the responsibility of the Backup Storage Provider to authenticate requests to `/backups` and mitigate abuse. Uploaded files would only need to be stored for maybe 72 hours before they could be safely purged, as backups are meant to be temporary storage. We could also just delete the file after it had been downloaded once. + +XMTP Labs would provide a reference implementation of a Backup Storage Provider. + +I am also proposing that XMTP Labs runs a Backup Storage Provider as a free public good for the next 2 years, at which point this functionality becomes a part of an ecosystem of third party gateway service providers. + +### Changes to `libxmtp` + +```rust +impl Client { + ... + /** + Methods for Backup Requesters + **/ + pub fn requesRemotetMessageHistoryBackup(&self) -> Result, ClientError> { + // Create and send a MessageHistoryBackupRequest to all other installations associated with the current blockchain account + ... + } + + pub fn getRemoteMessageHistoryBackupRequestStatus(&self, requestId: String) -> Result { + // Get the status of a pending backup request + ... + } + + pub fn applyRemoteMessageHistoryBackup(&self, requestId: String) -> Result<(), BackupApplyError> { + // Applies the following steps: + // 1. Look for the matching MessageHistoryBackupResponse and MessageHistoryBackupRequest in the database + // 2. If either are not found, return `ResponseNotFound` + // 3. Download the backup from the URI specified in the MessageHistoryBackupResponse + // 4. Decrypt the backup using the keys specified in the MessageHistoryBackupResponse + // 5. Run applyMessageHistoryBackupFromFile with the decrypted file + // 6. Delete the MessageHistoryBackupResponse from the local database, removing all sensitive key material + // 7. Set the MessageHistoryBackupRequest status to Applied + ... + } + + pub fn applyMessageHistoryBackupFromFile(&self, file: std::fs::File) -> Result<(), BackupApplyError> { + // Read from the file line by line and add each message to the database. If message already exists, skip it and move on to the next line + } + + /** + Methods for Message Backup Providers + **/ + pub fn listInboundBackupRequests(&self) -> Result, ClientError> { + // Return all pending backup requests. + ... + } + + pub fn respondToBackupRequest(&self, requestId: String, approve: bool) -> Result<(), BackupCreateError> { + // Perform the following steps: + // 1. If approve is false, delete the MessageHistoryBackupRequest from the local database and return + // 2. Dump all messages from the database into a correctly encoded MessageHistoryBackupFile format + // 3. Generate a random encryption key, nonce, and salt + // 4. Encrypt the file using the encryption parameters from step 2. + // 5. Upload the file to the provided `backupStorageProviderUploadUrl` + // 6. Send a message over the XMTP network containing the encryption parameters and the `backupUrl` returned from the backup storage provider + // 7. Delete the backup request from the database and return + ... + } + + /** + Methods for Backup Account Files + **/ + pub fn createBackupAccountFile(&self, path: String) -> Result<(), BuckupCreateError> { + // Create a file with a new account's private keys and return the contents of the file + ... + } + + pub fn importFromBackupAccountFile(&self, path: String) -> Result<(), BackupApplyError> { + // Load the file from the path, spin up a temporary client with the account keys, download all messages, and create a new Message Backup File + // Technically we could simplify this to just load the messages directly into the database without the intermediate file, but I like the idea of reducing surface area and re-using as much code as possible from the remote option + ... + } +} + +pub struct MessageHistoryBackupRequest { + pub requestId: String, + pub verificationPin: i16, // A four digit PIN that can be displayed in both the Backup Requester app and the Backup Provider app to ensure the user is responding to the correct backup request + pub backupStorageProviderUploadUrl: String, + pub status: BackupRequestStatus + + ... // a bunch of likely private fields with encryption information +} + +pub struct MessageHistoryBackupResponse { + pub requestId: String, + pub verificationPin: i16, + backupUrl: String, + encryptionKey: Vec, + nonce: Vec, + salt: Vec, + backupFileHash: Vec, + expirationTimeNs: u8 +} + +pub enum BackupRequestStatus { + Pending, + Expired, + Applied, + Failed +} + +pub enum BackupApplyError { + ResponseNotFound, + DecryptionFailed, + ValidationFailed, +} + +pub enum BackupCreateError { + RequestNotFound, + InvalidRequest, + NetworkError, + EncryptionError +} +``` + +## Rationale + +There are many other potential solutions to message portability listed below. I am proposing this particular solution because it solves the key user problems stated above with the minimal amount of developer friction. Many of the alternatives require substantial cooperation from application developers, who may care more about shipping new features than supporting competing applications. Developers have strong incentives to receive backups, but limited incentives to provide them. + +### Alternatives considered + +#### Use the XMTP Network as a Backup Storage Provider + +Instead of storing backups in a remotely hosted file and serving via HTTP, messages would be divided into chunks and stored on the XMTP network on a special topic only known to the participants in the backup message exchange. + +##### XMTP Network Advantages + +- No new infrastructure required. Everything stays on the XMTP network + +##### XMTP Network Disadvantages + +- The XMTP network was not designed for storage of large files. The network currently has a 1mb per message limit, so backups would be divided across potentially hundreds of chunks. +- We would want to handle expiration/deletion of these large backup files to reduce network storage needs +- Costly as the network decentralizes and fees are added to the network. Given that alternative storage providers can be used in the current proposal, the benefits to decentralization are limited. + +I could get on board with this proposal if others felt strongly that we shouldn't create new infrastructure. While backups are a bit of an odd fit for the XMTP transport network, it would work in the short term and would be easier to bootstrap. + +#### Use a QR code instead of an XMTP message + +Following in the footsteps of Signal, message history synchronization could be achieved as a synchronous exchange of data between clients. + +A QR code displayed in the Message Backup Provider, could establish an encrypted channel with the Backup Requester. This model follows a flow similar to WalletConnect. + +Messages would be encrypted using either an [N](https://noiseexplorer.com/patterns/X/) or [X](https://noiseexplorer.com/patterns/X/) pattern Noise handshake, with the bootstrap information encoded in the QR for out-of-band transmission. This channel would be private and ephemeral, offering the best protection for backups. + +##### QR Code Advantages + +- Does not require any additional installations to be added to the user's account +- Could be combined with the Backup Account File approach if desired, to handle cases where the Message Backup Provider is inaccessible + +##### QR Code Disadvantages + +- Forces the process to be synchronous. +- All participating applications would have to be able to handle the QR code flow. This requires more complicated front-end designs that every application needs to implement, access to a camera, and would require a very different flow for server-side use-cases. +- Would require mobile application developers to handle XMTP deep links for app -> app transfer. This gets complicated when there are multiple potential applications as backup providers. The Backup Requester would need to be aware of all possible Backup Provider apps and the user would have to select the right one. +- Would need to be rethought for webapp -> server transfer +- Would need to be rethought for webapp -> webapp transfers where QR code scanning and deep-linking are not possible. Maybe something closer to oAuth. + +#### Trusted Backup Service + +This approach is similar to the Backup Account File, but the backup account would be run on a server and listen for new messages continuously. Messages could be added to an archive in real-time, and backups could be served to any application that provided appropriate authentication over HTTP or XMTP protocols. + +##### Trusted Service Advantages + +- Continuous real-time backup of all messages after the service is created. Backups would continue even if user's devices were lost or compromised. +- No need to involve any other applications in backup process. User would have complete control over where their backups were stored +- Users could self-host trusted services + +##### Trusted Service Disadvantages + +- The service provider could read all messages as they come off the network. Even if they were encrypted at rest, there is significant risk of compromise. +- More costly to run than the Backup Account File, since it requires server resources to be continuously polling/streaming from the network. +- Multi-tenant providers could be legally compelled to provide access to user's messaging history or otherwise spy on users. +- Self hosting with a high level of security would be a high bar and might be prohibitive for all but the most committed users. +- Nothing is stopping users from doing this under any scenario. The trusted server would just be a client. If there is demand for this, someone will build it regardless. + +## Backward compatibility + +Given that V3 is brand new, there is no risk of backwards incompatible changes with the initial release of this feature. + +Caution should be taken in the design of the Message Backup File format to ensure that it is flexible and compatible with other potential changes. Any fields that may have changing data formats should be self-describing, and allow for versioning in the future. + +## Reference implementation + +TODO + +## Security considerations + +### Risks and drawbacks to Remote Message Backups + +- Developers of the Message Backup Provider could implement the consent flow in a way that confuses users into accepting backup requests from malicious applications. PIN verification may not be implemnted across all providers +- Bac + +### Risks and drawbacks to Backup Account Files + +- Because the account stored in a Backup Account File never sends messages, all messages sent to the account will be encrypted using a one time prekey with no ratcheting. A single ratchet step could happen each time a backup was created, although this would require the user to replace their Backup Acount File with a new one containing the current ratchet state +- Accounts in the Backup Account File will only be online during backup restoration. This means there are few opportunities to generate new one time prekeys. This may be mitigated by creating a very large number of one-time prekeys as part of creation. +- This approach relies on the XMTP network allowing messages to remain in pre-delivery storage indefinitely. While that works today, in future iterations of the network this may be cost prohibitive. +- Applications creating Backup Account Files would need to handle interacting with the device filesystem or cloud storage providers. For mobile applications this may require extra permissions. +- If Backup Account Files were accessed, and it's encryption keys were compromised, the attacker would have access to all historical messages + +## Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From 6c82eff181aa37493a38476dd88c8450fcf40401 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Sun, 10 Sep 2023 23:12:20 -0700 Subject: [PATCH 02/18] Rename to 31 --- ...equest-system.md => xip-31-message-backup-request-system.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename XIPs/{xip-28-message-backup-request-system.md => xip-31-message-backup-request-system.md} (99%) diff --git a/XIPs/xip-28-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md similarity index 99% rename from XIPs/xip-28-message-backup-request-system.md rename to XIPs/xip-31-message-backup-request-system.md index 19e7e0b..56e63f9 100644 --- a/XIPs/xip-28-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -1,5 +1,5 @@ --- -xip: 29 +xip: 31 title: Message backup request system description: A system for requesting and receiving message backups between V3 installations author: Nick Molnar From 1f281eaf44e1dc7792438b563617becf70df71a4 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Sun, 10 Sep 2023 23:34:30 -0700 Subject: [PATCH 03/18] Minor edits --- XIPs/xip-31-message-backup-request-system.md | 29 ++++++++++---------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index 56e63f9..a7f698e 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -15,7 +15,7 @@ A proposed system for transferring backups of message history between installati ## Motivation -The current [XMTP V3 Protocol Specification](https://github.com/xmtp/libxmtp) allows new installations to be created and receive all messages sent to a blockchain account after it's creation. It does _not_ specify how messages sent before the installation was created can be loaded into the new installation. Having conversation history sync between devices is a popular feature of V1 and V2 of the XMTP protocol and there is clear demand from developers for this to be possible as part of V3. +The current [XMTP V3 Protocol Specification](https://github.com/xmtp/libxmtp) creates a new XMTP identity for each installation associated with a blockchain account, instead of sharing a single set of keys across all installations like V1/V2. New installations will receive all messages sent to a blockchain account after they are created, but V3 does _not_ specify how to receive messages sent before installation creation. Having conversation history sync between devices is a popular feature of V1 and V2 of the XMTP protocol and there is clear demand from developers for this to be possible as part of V3. There are a number of cases where message history synchronization is desirable: @@ -26,7 +26,7 @@ There are a number of cases where message history synchronization is desirable: It's also worth noting that there are many use-cases for XMTP where message history synchronization is not necessary or desirable. -- A bot that replies to messages from users. Most bots don't care about messages sent before they were created. +- A bot that replies to messages from users. Many bots don't care about messages sent before they were created. - A service that sends notifications to a list of subscribers via XMTP. - An application that allows users to talk to customer support only needs access to the newly created conversation with the support agent. - A marketplace application that wants to allow buyers and sellers to communicate about a particular item. These would only be new conversations and may not care about pre-existing messages. @@ -37,7 +37,7 @@ It's also worth noting that there are many use-cases for XMTP where message hist 1. Message history synchronization should be opt-in for developers. 1. It should be simple for developers to implement message history synchronization correctly. Implementation work should be focused on application-specific UX choices, with the SDK responsible for core transport and security decisions. 1. Compatibility between applications is expected. Applications should be able to share history irrespective of operating system, SDK version, or device type (web vs mobile vs server). -1. Even if developers of some popular applications choose to not support message history synchronization, it should be simple for users to opt themselves in +1. Even if developers of some popular applications choose to not support message history synchronization, it should be simple for users to opt themselves in. ## Specification @@ -49,12 +49,14 @@ There are three types of actors in this specification: Backup Requesters, Messag Remote message backups work by Backup Requesters sending a special message type (`MessageHistoryBackupRequest`) across the XMTP network to all other installations signed by the same blockchain account as the sender (Message Backup Providers). Upon receipt of these messages, Message Backup Providers should display a prompt to the user asking whether they consent to share their message history with the requesting installation. Upon approval, the application will convert their local database into a standard Message History Backup File and upload it to the Backup Storage Provider specified in the `MessageHistoryBackupRequest`. +For mobile applications already handling push notifications, `MessageHistoryBackupRequest`s would become a special case of push notification handling. + #### Flow for Backup Requesters 1. Get a list of all other installations associated with their blockchain account. Ignore any installations that have been revoked 2. Obtain a one-time upload URL from the Backup Storage Provider for each installation 3. Generate a `MessageHistoryBackupRequest` for each installation (see below for Protobuf spec) and store in the local database -4. Send the `MessageHistoryBackupRequest` messages to the standard message topic for each installation +4. Send the `MessageHistoryBackupRequest` messages to the normal inbound messaging topic for each installation 5. Wait for a response from any of the Message Backup Providers 6. For each `MessageHistoryBackupResponse` 6a. Ensure there is a `MessageHistoryBackupRequest` with a matching `requestId` stored in the database. If not, ignore @@ -130,9 +132,9 @@ This same web application could be used to create Backup Account Files for cases ### Backup Storage Provider -A Backup Storage Provider is a simple HTTP service with two endpoints. Anyone can implement a Backup Storage Provider. It is up to the Backup Requester to choose the Backup Service Provider for their application. +A Backup Storage Provider is a simple HTTP service with three endpoints. Anyone can implement a Backup Storage Provider. It is up to the Backup Requester application to choose the Backup Service Provider for their application. -Backup Storage Providers only need to implement three APIs: +These are the required APIs for a minimal Backup Storage Provider: `POST /backups`: Example response: @@ -288,7 +290,6 @@ Instead of storing backups in a remotely hosted file and serving via HTTP, messa ##### XMTP Network Disadvantages - The XMTP network was not designed for storage of large files. The network currently has a 1mb per message limit, so backups would be divided across potentially hundreds of chunks. -- We would want to handle expiration/deletion of these large backup files to reduce network storage needs - Costly as the network decentralizes and fees are added to the network. Given that alternative storage providers can be used in the current proposal, the benefits to decentralization are limited. I could get on board with this proposal if others felt strongly that we shouldn't create new infrastructure. While backups are a bit of an odd fit for the XMTP transport network, it would work in the short term and would be easier to bootstrap. @@ -297,7 +298,7 @@ I could get on board with this proposal if others felt strongly that we shouldn' Following in the footsteps of Signal, message history synchronization could be achieved as a synchronous exchange of data between clients. -A QR code displayed in the Message Backup Provider, could establish an encrypted channel with the Backup Requester. This model follows a flow similar to WalletConnect. +A QR code displayed in the Message Backup Provider, could establish an encrypted channel with the Backup Requester. This model follows a flow similar to WalletConnect. Deep links would be used for app -> app transfers. Messages would be encrypted using either an [N](https://noiseexplorer.com/patterns/X/) or [X](https://noiseexplorer.com/patterns/X/) pattern Noise handshake, with the bootstrap information encoded in the QR for out-of-band transmission. This channel would be private and ephemeral, offering the best protection for backups. @@ -308,8 +309,8 @@ Messages would be encrypted using either an [N](https://noiseexplorer.com/patter ##### QR Code Disadvantages -- Forces the process to be synchronous. -- All participating applications would have to be able to handle the QR code flow. This requires more complicated front-end designs that every application needs to implement, access to a camera, and would require a very different flow for server-side use-cases. +- QR codes are most appropriate for cases where there isn't an authenticated and private communication channel between parties. That isn't the problem here. We already have a solution for authenticated and secure messaging between two installations as part of V3. +- All participating applications would have to be able to handle the QR code flow. This requires more complicated front-end designs that every application needs to implement, as well as access to a camera in some cases. This could be enough to discourage developers from implementing their side of the flow. - Would require mobile application developers to handle XMTP deep links for app -> app transfer. This gets complicated when there are multiple potential applications as backup providers. The Backup Requester would need to be aware of all possible Backup Provider apps and the user would have to select the right one. - Would need to be rethought for webapp -> server transfer - Would need to be rethought for webapp -> webapp transfers where QR code scanning and deep-linking are not possible. Maybe something closer to oAuth. @@ -320,7 +321,7 @@ This approach is similar to the Backup Account File, but the backup account woul ##### Trusted Service Advantages -- Continuous real-time backup of all messages after the service is created. Backups would continue even if user's devices were lost or compromised. +- Continuous real-time backup of all messages after the service is created. Backups would continue even if all of a user's devices were lost or compromised. - No need to involve any other applications in backup process. User would have complete control over where their backups were stored - Users could self-host trusted services @@ -336,7 +337,7 @@ This approach is similar to the Backup Account File, but the backup account woul Given that V3 is brand new, there is no risk of backwards incompatible changes with the initial release of this feature. -Caution should be taken in the design of the Message Backup File format to ensure that it is flexible and compatible with other potential changes. Any fields that may have changing data formats should be self-describing, and allow for versioning in the future. +Caution should be taken in the design of the Message Backup File format to ensure that it is flexible and compatible with other potential changes. Any fields that may have changing data formats should be self-describing and versioned for backwards/forwards compatibility. ## Reference implementation @@ -347,11 +348,11 @@ TODO ### Risks and drawbacks to Remote Message Backups - Developers of the Message Backup Provider could implement the consent flow in a way that confuses users into accepting backup requests from malicious applications. PIN verification may not be implemnted across all providers -- Bac +- Bugs in the validation of requests in Message Backup Providers would be very very bad. This would need to be very well tested code, since a compromise here would effectively be a 0-click exploit of someone's XMTP account. ### Risks and drawbacks to Backup Account Files -- Because the account stored in a Backup Account File never sends messages, all messages sent to the account will be encrypted using a one time prekey with no ratcheting. A single ratchet step could happen each time a backup was created, although this would require the user to replace their Backup Acount File with a new one containing the current ratchet state +- Because the account stored in a Backup Account File never sends messages, all messages sent to the account will be encrypted using a one time prekey with no ratcheting. A single ratchet step could happen each time a backup was created since the account is briefly online, although this would require the user to replace their Backup Acount File with a new one containing the current ratchet state. - Accounts in the Backup Account File will only be online during backup restoration. This means there are few opportunities to generate new one time prekeys. This may be mitigated by creating a very large number of one-time prekeys as part of creation. - This approach relies on the XMTP network allowing messages to remain in pre-delivery storage indefinitely. While that works today, in future iterations of the network this may be cost prohibitive. - Applications creating Backup Account Files would need to handle interacting with the device filesystem or cloud storage providers. For mobile applications this may require extra permissions. From 6d347e05763404ff6171a33ab5939fd0383c3a1d Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Sun, 10 Sep 2023 23:40:22 -0700 Subject: [PATCH 04/18] Add spaces --- XIPs/xip-31-message-backup-request-system.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index a7f698e..db41c5c 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -59,11 +59,17 @@ For mobile applications already handling push notifications, `MessageHistoryBack 4. Send the `MessageHistoryBackupRequest` messages to the normal inbound messaging topic for each installation 5. Wait for a response from any of the Message Backup Providers 6. For each `MessageHistoryBackupResponse` + 6a. Ensure there is a `MessageHistoryBackupRequest` with a matching `requestId` stored in the database. If not, ignore + 6b. Ensure the `backupUrl` is on the same host as the requested `backupStorageProviderUploadUrl`. If not, ignore + 6c. Download the file from the `backupUrl` and decrypt using the credentials provided in the `MessageHistoryBackupResponse`. If the hash of the downloaded file does not match the hash in the `MessageHistoryBackupResponse`, abort. + 6d. Load each message into the local database, ignoring any duplicate messages + 6e. Delete the `MessageHistoryBackupResponse` and all associated credentials + 6f. Set the status of the `MessageHistoryBackupRequest` to `Applied` #### Flow for Message Backup Providers From 1728354618ba19969235b86fd3497b332dd8ee57 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 08:01:27 -0700 Subject: [PATCH 05/18] Add some more notes --- XIPs/xip-31-message-backup-request-system.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index db41c5c..d586c52 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -163,7 +163,7 @@ Example response: `GET /backups/$DOWNLOAD_ID`: Returns the uploaded file matching the ID -It would be the responsibility of the Backup Storage Provider to authenticate requests to `/backups` and mitigate abuse. Uploaded files would only need to be stored for maybe 72 hours before they could be safely purged, as backups are meant to be temporary storage. We could also just delete the file after it had been downloaded once. +It would be the responsibility of the Backup Storage Provider to authenticate requests to `/backups` and mitigate abuse.Uploaded files would only need to be stored for maybe 72 hours before they could be safely purged, as backups are meant to be temporary storage. We could also just delete the file after it had been downloaded once. XMTP Labs would provide a reference implementation of a Backup Storage Provider. @@ -243,8 +243,7 @@ pub struct MessageHistoryBackupRequest { pub verificationPin: i16, // A four digit PIN that can be displayed in both the Backup Requester app and the Backup Provider app to ensure the user is responding to the correct backup request pub backupStorageProviderUploadUrl: String, pub status: BackupRequestStatus - - ... // a bunch of likely private fields with encryption information + ... } pub struct MessageHistoryBackupResponse { @@ -354,6 +353,7 @@ TODO ### Risks and drawbacks to Remote Message Backups - Developers of the Message Backup Provider could implement the consent flow in a way that confuses users into accepting backup requests from malicious applications. PIN verification may not be implemnted across all providers +- A malicious Message Backup Provider could choose to censor messages in the backup, or add spoofed messages, since the messages in the backup are unauthenticated. - Bugs in the validation of requests in Message Backup Providers would be very very bad. This would need to be very well tested code, since a compromise here would effectively be a 0-click exploit of someone's XMTP account. ### Risks and drawbacks to Backup Account Files @@ -364,6 +364,10 @@ TODO - Applications creating Backup Account Files would need to handle interacting with the device filesystem or cloud storage providers. For mobile applications this may require extra permissions. - If Backup Account Files were accessed, and it's encryption keys were compromised, the attacker would have access to all historical messages +### Risks and drawbacks to Backup Storage Providers + +- Offering free storage on the internet has potential for abuse. Requests to this service should be rate limited or otherwise protected against attack. If we were less concerned about abuse potential, we could remove the need for pre-generation of upload URLs, which would remove the need for the Backup Requester to contact the Backup Storage Provider before creating a `MessageHistoryBackupRequest` + ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From b888a13e4f2ba01e0061718efc4692b7e41678d9 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 08:28:57 -0700 Subject: [PATCH 06/18] Add proto formats --- XIPs/xip-31-message-backup-request-system.md | 88 ++++++++++++++++++++ 1 file changed, 88 insertions(+) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index d586c52..d446c19 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -43,8 +43,34 @@ It's also worth noting that there are many use-cases for XMTP where message hist This specification offers the user two modes of message backup, which accomodate different use-cases: Remote Message Backups and Backup Account Files. Remote Message Backups are the lowest friction solution, provided the blockchain account has access to another existing installation. Backup Account Files are an emergency solution for cases where the user has lost access to all of their XMTP applications (for example, if they lost the only device they have used to connect to XMTP). +### Actors + There are three types of actors in this specification: Backup Requesters, Message Backup Providers, and Backup Storage Providers. Backup requesters are XMTP Clients that are missing message history and would like to receive a backup. Message Backup Providers are XMTP Clients that have a message history and are capable of sending it to a requester. These first two types of actors are XMTP Clients, and the implementation required to fulfill the responsibilities of both roles would be built into `libxmtp`. The Backup Storage Provider is a remote service responsible for temporarily storing backup files, and is only needed for Remote Message Backups. +```mermaid +classDiagram +class R["Backup Requester"]{ + The new installation requesting missing messages + Responsible for selecting a Backup Storage Provider and obtaining authorization + Creates MessageHistoryBackupRequests + Reads MessageHistoryBackupResponses +} + +class M["Message Backup Provider"]{ + An existing installation that has access to message history + Reads MessageHistoryBackupRequests + Creates MessageHistoryBackupResponses + Uploads Message Backup Files to the pre-selected storage provider + Responsible for generating one-time encryption keys +} + +class S["Backup Storage Provider"]{ + A cloud service with durable storage + Stores encrypted backups for a few days + Responsible for authenticating requests to reduce abuse potential +} +``` + ### Remote Message Backups Remote message backups work by Backup Requesters sending a special message type (`MessageHistoryBackupRequest`) across the XMTP network to all other installations signed by the same blockchain account as the sender (Message Backup Providers). Upon receipt of these messages, Message Backup Providers should display a prompt to the user asking whether they consent to share their message history with the requesting installation. Upon approval, the application will convert their local database into a standard Message History Backup File and upload it to the Backup Storage Provider specified in the `MessageHistoryBackupRequest`. @@ -169,6 +195,68 @@ XMTP Labs would provide a reference implementation of a Backup Storage Provider. I am also proposing that XMTP Labs runs a Backup Storage Provider as a free public good for the next 2 years, at which point this functionality becomes a part of an ecosystem of third party gateway service providers. +### Changes to the v3 protocol buffers + +#### [Mesage Protos](https://github.com/xmtp/proto/blob/xmtpv3/proto/v3/message_contents/message.proto) + +```proto3 +// The decrypted message contents of any message on the installation's messaging topic +message PadlockMessagePayload { + EdDsaSignature header_signature = 1; + oneof contents { + DirectMessage direct_message = 2; + MessageHistoryBackupRequest message_history_backup_request = 3; + MessageHistoryBackupResponse message_history_backup_response = 4; + } +} + +// The decrypted contents of a MessageHistoryBackupRequest +message MessageHistoryBackupRequest { + string request_id = 1; + int32 verification_pin = 2; + string upload_url = 3; +} + +message MessageHistoryBackupResponse { + string request_id = 1; + string backup_url = 2; + // TBD on exact parameter definitions. Want to have flexibilty to change key types later + EncryptionParameters encryption_parameters = 3; + bytes backup_file_hash = 4; + int64 expiration_time_ns = 5; +} + +message DirectMessage { + string convo_id = 1; + bytes content_bytes = 2; +} +``` + +#### [Contact protos](https://github.com/xmtp/proto/blob/xmtpv3/proto/v3/message_contents/public_key.proto) + +```proto3 +enum ContactRole { + UNKNOWN_CONTACT_ROLE = 0; + FULL = 1; + BACKUP = 2; +} + +message VmacInstallationPublicKeyBundleV1 { + VmacAccountLinkedKey identity_key = 1; + VmacInstallationLinkedKey fallback_key = 2; + // NEW FIELD + ContactRole contact_role = 3; +} + +// A wrapper for versions of the installation contact bundle to allow +// upgradeability +message InstallationContactBundle { + oneof version { + VmacInstallationPublicKeyBundleV1 v1 = 1; + } +} +``` + ### Changes to `libxmtp` ```rust From 05a0461802c7523dd76f38dc8d4548200a3f4764 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 08:30:15 -0700 Subject: [PATCH 07/18] Rename to proto --- XIPs/xip-31-message-backup-request-system.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index d446c19..da93b8f 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -199,7 +199,7 @@ I am also proposing that XMTP Labs runs a Backup Storage Provider as a free publ #### [Mesage Protos](https://github.com/xmtp/proto/blob/xmtpv3/proto/v3/message_contents/message.proto) -```proto3 +```proto // The decrypted message contents of any message on the installation's messaging topic message PadlockMessagePayload { EdDsaSignature header_signature = 1; @@ -234,7 +234,7 @@ message DirectMessage { #### [Contact protos](https://github.com/xmtp/proto/blob/xmtpv3/proto/v3/message_contents/public_key.proto) -```proto3 +```proto enum ContactRole { UNKNOWN_CONTACT_ROLE = 0; FULL = 1; From 926bfb54b64d5403343a0c2306bddd31c2da9c88 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 08:30:57 -0700 Subject: [PATCH 08/18] Rename language to protobuf --- XIPs/xip-31-message-backup-request-system.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index da93b8f..8842be7 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -199,7 +199,7 @@ I am also proposing that XMTP Labs runs a Backup Storage Provider as a free publ #### [Mesage Protos](https://github.com/xmtp/proto/blob/xmtpv3/proto/v3/message_contents/message.proto) -```proto +```protobuf // The decrypted message contents of any message on the installation's messaging topic message PadlockMessagePayload { EdDsaSignature header_signature = 1; @@ -234,7 +234,7 @@ message DirectMessage { #### [Contact protos](https://github.com/xmtp/proto/blob/xmtpv3/proto/v3/message_contents/public_key.proto) -```proto +```protobuf enum ContactRole { UNKNOWN_CONTACT_ROLE = 0; FULL = 1; From c4c4bd9def18a84c69c8433ddf205c839f955454 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 08:32:18 -0700 Subject: [PATCH 09/18] Add syntax --- XIPs/xip-31-message-backup-request-system.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index 8842be7..482d5e0 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -199,7 +199,9 @@ I am also proposing that XMTP Labs runs a Backup Storage Provider as a free publ #### [Mesage Protos](https://github.com/xmtp/proto/blob/xmtpv3/proto/v3/message_contents/message.proto) -```protobuf +```proto +syntax = "proto3"; + // The decrypted message contents of any message on the installation's messaging topic message PadlockMessagePayload { EdDsaSignature header_signature = 1; @@ -234,7 +236,8 @@ message DirectMessage { #### [Contact protos](https://github.com/xmtp/proto/blob/xmtpv3/proto/v3/message_contents/public_key.proto) -```protobuf +```proto +syntax = "proto3"; enum ContactRole { UNKNOWN_CONTACT_ROLE = 0; FULL = 1; From e31f3b8969f82f84229adf1f40aea04e535f9317 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 08:34:17 -0700 Subject: [PATCH 10/18] Clean up code sample --- XIPs/xip-31-message-backup-request-system.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index 482d5e0..943d1a2 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -200,8 +200,6 @@ I am also proposing that XMTP Labs runs a Backup Storage Provider as a free publ #### [Mesage Protos](https://github.com/xmtp/proto/blob/xmtpv3/proto/v3/message_contents/message.proto) ```proto -syntax = "proto3"; - // The decrypted message contents of any message on the installation's messaging topic message PadlockMessagePayload { EdDsaSignature header_signature = 1; @@ -237,7 +235,7 @@ message DirectMessage { #### [Contact protos](https://github.com/xmtp/proto/blob/xmtpv3/proto/v3/message_contents/public_key.proto) ```proto -syntax = "proto3"; + enum ContactRole { UNKNOWN_CONTACT_ROLE = 0; FULL = 1; @@ -268,7 +266,7 @@ impl Client { /** Methods for Backup Requesters **/ - pub fn requesRemotetMessageHistoryBackup(&self) -> Result, ClientError> { + pub fn requestRemoteMessageHistoryBackup(&self) -> Result, ClientError> { // Create and send a MessageHistoryBackupRequest to all other installations associated with the current blockchain account ... } @@ -284,13 +282,13 @@ impl Client { // 2. If either are not found, return `ResponseNotFound` // 3. Download the backup from the URI specified in the MessageHistoryBackupResponse // 4. Decrypt the backup using the keys specified in the MessageHistoryBackupResponse - // 5. Run applyMessageHistoryBackupFromFile with the decrypted file + // 5. Run loadMessageBackupFile with the decrypted file // 6. Delete the MessageHistoryBackupResponse from the local database, removing all sensitive key material // 7. Set the MessageHistoryBackupRequest status to Applied ... } - pub fn applyMessageHistoryBackupFromFile(&self, file: std::fs::File) -> Result<(), BackupApplyError> { + pub fn loadMessageBackupFile(&self, file: std::fs::File) -> Result<(), BackupApplyError> { // Read from the file line by line and add each message to the database. If message already exists, skip it and move on to the next line } From 7dcbc88552b96b662a3d7ec6fdcf5b9704ce1c18 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 08:51:42 -0700 Subject: [PATCH 11/18] Copy edits --- XIPs/xip-31-message-backup-request-system.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index 943d1a2..7f77647 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -15,7 +15,7 @@ A proposed system for transferring backups of message history between installati ## Motivation -The current [XMTP V3 Protocol Specification](https://github.com/xmtp/libxmtp) creates a new XMTP identity for each installation associated with a blockchain account, instead of sharing a single set of keys across all installations like V1/V2. New installations will receive all messages sent to a blockchain account after they are created, but V3 does _not_ specify how to receive messages sent before installation creation. Having conversation history sync between devices is a popular feature of V1 and V2 of the XMTP protocol and there is clear demand from developers for this to be possible as part of V3. +The [XMTP V3 Protocol Specification](https://github.com/xmtp/libxmtp) creates a new XMTP identity for each installation associated with a blockchain account, instead of sharing a single set of keys across all installations like V1/V2. New installations will receive all messages sent to a blockchain account after they are created, but V3 does _not_ specify how to receive messages sent before installation creation. Having conversation history sync between devices is a popular feature of V1 and V2 of the XMTP protocol and there is clear demand from developers for this to be possible as part of V3. There are a number of cases where message history synchronization is desirable: @@ -37,7 +37,7 @@ It's also worth noting that there are many use-cases for XMTP where message hist 1. Message history synchronization should be opt-in for developers. 1. It should be simple for developers to implement message history synchronization correctly. Implementation work should be focused on application-specific UX choices, with the SDK responsible for core transport and security decisions. 1. Compatibility between applications is expected. Applications should be able to share history irrespective of operating system, SDK version, or device type (web vs mobile vs server). -1. Even if developers of some popular applications choose to not support message history synchronization, it should be simple for users to opt themselves in. +1. Even if developers of some popular applications choose to not support message history synchronization, it should be simple for users to opt enable themselves. ## Specification From 9333ddf0617551787523be3c2160232cd96fad2b Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 08:57:26 -0700 Subject: [PATCH 12/18] Clarify PIN usage --- XIPs/xip-31-message-backup-request-system.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index 7f77647..a9ce874 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -73,7 +73,9 @@ class S["Backup Storage Provider"]{ ### Remote Message Backups -Remote message backups work by Backup Requesters sending a special message type (`MessageHistoryBackupRequest`) across the XMTP network to all other installations signed by the same blockchain account as the sender (Message Backup Providers). Upon receipt of these messages, Message Backup Providers should display a prompt to the user asking whether they consent to share their message history with the requesting installation. Upon approval, the application will convert their local database into a standard Message History Backup File and upload it to the Backup Storage Provider specified in the `MessageHistoryBackupRequest`. +Remote message backups work by Backup Requesters sending a special message type (`MessageHistoryBackupRequest`) across the XMTP network to all other installations signed by the same blockchain account as the sender (Message Backup Providers). Upon receipt of these messages, Message Backup Providers should display a prompt to the user asking whether they consent to share their message history with the requesting installation. The prompt should display the Verification PIN from the request - which will also be displayed in the Backup Requester's app - to ensure that the user is approving the correct request. + +Upon approval, the application will convert their local database into a standard Message History Backup File, encrypt it, and upload it to the Backup Storage Provider specified in the `MessageHistoryBackupRequest`. For mobile applications already handling push notifications, `MessageHistoryBackupRequest`s would become a special case of push notification handling. From 1a57f3f7ec691eb4f957db4eb89023ecbae6524f Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 08:58:58 -0700 Subject: [PATCH 13/18] Clarify user entering PIN --- XIPs/xip-31-message-backup-request-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index a9ce874..4ba06aa 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -73,7 +73,7 @@ class S["Backup Storage Provider"]{ ### Remote Message Backups -Remote message backups work by Backup Requesters sending a special message type (`MessageHistoryBackupRequest`) across the XMTP network to all other installations signed by the same blockchain account as the sender (Message Backup Providers). Upon receipt of these messages, Message Backup Providers should display a prompt to the user asking whether they consent to share their message history with the requesting installation. The prompt should display the Verification PIN from the request - which will also be displayed in the Backup Requester's app - to ensure that the user is approving the correct request. +Remote message backups work by Backup Requesters sending a special message type (`MessageHistoryBackupRequest`) across the XMTP network to all other installations signed by the same blockchain account as the sender (Message Backup Providers). Upon receipt of these messages, Message Backup Providers should display a prompt to the user asking whether they consent to share their message history with the requesting installation. The prompt should display the Verification PIN from the request - which will also be displayed in the Backup Requester's app - to ensure that the user is approving the correct request. Some apps may choose to force the user to enter the PIN, rather than display it, as an additional layer of security. Upon approval, the application will convert their local database into a standard Message History Backup File, encrypt it, and upload it to the Backup Storage Provider specified in the `MessageHistoryBackupRequest`. From 9611d0c4c7ccd73ea51d747400898f28eac2e066 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 08:59:36 -0700 Subject: [PATCH 14/18] Fix typo --- XIPs/xip-31-message-backup-request-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index 4ba06aa..6ba2e41 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -443,7 +443,7 @@ TODO ### Risks and drawbacks to Remote Message Backups -- Developers of the Message Backup Provider could implement the consent flow in a way that confuses users into accepting backup requests from malicious applications. PIN verification may not be implemnted across all providers +- Developers of the Message Backup Provider could implement the consent flow in a way that confuses users into accepting backup requests from malicious applications. PIN verification may not be implemented across all providers - A malicious Message Backup Provider could choose to censor messages in the backup, or add spoofed messages, since the messages in the backup are unauthenticated. - Bugs in the validation of requests in Message Backup Providers would be very very bad. This would need to be very well tested code, since a compromise here would effectively be a 0-click exploit of someone's XMTP account. From 222b04c00729d1a2c5907b36468b836fbbb02fdf Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 09:10:02 -0700 Subject: [PATCH 15/18] Specify upload format --- XIPs/xip-31-message-backup-request-system.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index 6ba2e41..4452b6d 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -180,6 +180,8 @@ Example response: ``` `POST /backups/$UPLOAD_ID`: +Request body would contain the file as multipart/form-data. + Example response: ```json From c30a8df7318bc3ea1339450c3833492f77f67a59 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 09:31:25 -0700 Subject: [PATCH 16/18] Add advantage to QR code approach --- XIPs/xip-31-message-backup-request-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index 4452b6d..187aee2 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -403,7 +403,7 @@ Messages would be encrypted using either an [N](https://noiseexplorer.com/patter ##### QR Code Advantages - Does not require any additional installations to be added to the user's account -- Could be combined with the Backup Account File approach if desired, to handle cases where the Message Backup Provider is inaccessible +- Ensures the Backup Requester and Message Backup Provider are in the same room (QR Code) or on the same device (deep link) ##### QR Code Disadvantages From 47523903f3272eac7d540005ce828cbc129028b6 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Mon, 11 Sep 2023 18:16:04 -0700 Subject: [PATCH 17/18] Update XIPs/xip-31-message-backup-request-system.md Co-authored-by: Richard Hua --- XIPs/xip-31-message-backup-request-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index 187aee2..bace391 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -15,7 +15,7 @@ A proposed system for transferring backups of message history between installati ## Motivation -The [XMTP V3 Protocol Specification](https://github.com/xmtp/libxmtp) creates a new XMTP identity for each installation associated with a blockchain account, instead of sharing a single set of keys across all installations like V1/V2. New installations will receive all messages sent to a blockchain account after they are created, but V3 does _not_ specify how to receive messages sent before installation creation. Having conversation history sync between devices is a popular feature of V1 and V2 of the XMTP protocol and there is clear demand from developers for this to be possible as part of V3. +The [XMTP V3 Protocol Specification](https://github.com/xmtp/libxmtp/blob/main/TECHNICAL_OVERVIEW.md) creates a new XMTP identity for each installation associated with a blockchain account, instead of sharing a single set of keys across all installations like V1/V2. New installations will receive all messages sent to a blockchain account after they are created, but V3 does _not_ specify how to receive messages sent before installation creation. Having conversation history sync between devices is a popular feature of V1 and V2 of the XMTP protocol and there is clear demand from developers for this to be possible as part of V3. There are a number of cases where message history synchronization is desirable: From ea90e71a3d8e812fee3150be8f9ef7420c222e29 Mon Sep 17 00:00:00 2001 From: Nicholas Molnar <65710+neekolas@users.noreply.github.com> Date: Tue, 12 Sep 2023 14:22:05 -0700 Subject: [PATCH 18/18] Simplify backup provider --- XIPs/xip-31-message-backup-request-system.md | 38 +++++++------------- 1 file changed, 13 insertions(+), 25 deletions(-) diff --git a/XIPs/xip-31-message-backup-request-system.md b/XIPs/xip-31-message-backup-request-system.md index bace391..984b504 100644 --- a/XIPs/xip-31-message-backup-request-system.md +++ b/XIPs/xip-31-message-backup-request-system.md @@ -67,7 +67,7 @@ class M["Message Backup Provider"]{ class S["Backup Storage Provider"]{ A cloud service with durable storage Stores encrypted backups for a few days - Responsible for authenticating requests to reduce abuse potential + Responsible for authenticating requests to reduce abuse } ``` @@ -82,23 +82,22 @@ For mobile applications already handling push notifications, `MessageHistoryBack #### Flow for Backup Requesters 1. Get a list of all other installations associated with their blockchain account. Ignore any installations that have been revoked -2. Obtain a one-time upload URL from the Backup Storage Provider for each installation -3. Generate a `MessageHistoryBackupRequest` for each installation (see below for Protobuf spec) and store in the local database -4. Send the `MessageHistoryBackupRequest` messages to the normal inbound messaging topic for each installation -5. Wait for a response from any of the Message Backup Providers -6. For each `MessageHistoryBackupResponse` +2. Generate a `MessageHistoryBackupRequest` for each installation (see below for Protobuf spec) and store in the local database +3. Send the `MessageHistoryBackupRequest` messages to the normal inbound messaging topic for each installation +4. Wait for a response from any of the Message Backup Providers +5. For each `MessageHistoryBackupResponse` - 6a. Ensure there is a `MessageHistoryBackupRequest` with a matching `requestId` stored in the database. If not, ignore + 5a. Ensure there is a `MessageHistoryBackupRequest` with a matching `requestId` stored in the database. If not, ignore - 6b. Ensure the `backupUrl` is on the same host as the requested `backupStorageProviderUploadUrl`. If not, ignore + 5b. Ensure the `backupUrl` is on the same host as the requested `backupStorageProviderUploadUrl`. If not, ignore - 6c. Download the file from the `backupUrl` and decrypt using the credentials provided in the `MessageHistoryBackupResponse`. If the hash of the downloaded file does not match the hash in the `MessageHistoryBackupResponse`, abort. + 5c. Download the file from the `backupUrl` and decrypt using the credentials provided in the `MessageHistoryBackupResponse`. If the hash of the downloaded file does not match the hash in the `MessageHistoryBackupResponse`, abort. - 6d. Load each message into the local database, ignoring any duplicate messages + 5d. Load each message into the local database, ignoring any duplicate messages - 6e. Delete the `MessageHistoryBackupResponse` and all associated credentials + 5e. Delete the `MessageHistoryBackupResponse` and all associated credentials - 6f. Set the status of the `MessageHistoryBackupRequest` to `Applied` + 5f. Set the status of the `MessageHistoryBackupRequest` to `Applied` #### Flow for Message Backup Providers @@ -119,8 +118,6 @@ sequenceDiagram Participant S as Backup Storage Provider Participant N as XMTP Network Participant M as Message Backup Provider - B->>S: Request a backup storage upload URL - S-->>B: Receive one time upload URL B->>N: Send a MessageHistoryBackupRequest to each potential Message Backup Provider M->>N: Check for MessageHistoryBackupRequest N-->>M: Receive MessageHistoryBackupRequest @@ -166,20 +163,11 @@ This same web application could be used to create Backup Account Files for cases ### Backup Storage Provider -A Backup Storage Provider is a simple HTTP service with three endpoints. Anyone can implement a Backup Storage Provider. It is up to the Backup Requester application to choose the Backup Service Provider for their application. +A Backup Storage Provider is a simple HTTP service with two endpoints. Anyone can implement a Backup Storage Provider. It is up to the Backup Requester application to choose the Backup Service Provider for their application. These are the required APIs for a minimal Backup Storage Provider: `POST /backups`: -Example response: - -```json -{ - "uploadUrl": "https://backupproviderdomain.com/backups/some-long-unguessable-upload-id" -} -``` - -`POST /backups/$UPLOAD_ID`: Request body would contain the file as multipart/form-data. Example response: @@ -459,7 +447,7 @@ TODO ### Risks and drawbacks to Backup Storage Providers -- Offering free storage on the internet has potential for abuse. Requests to this service should be rate limited or otherwise protected against attack. If we were less concerned about abuse potential, we could remove the need for pre-generation of upload URLs, which would remove the need for the Backup Requester to contact the Backup Storage Provider before creating a `MessageHistoryBackupRequest` +- Offering free storage on the internet has potential for abuse. Requests to this service should be rate limited or otherwise protected against attack. ## Copyright