KDD for Incremental Archive/snapshot/backup solution

[CarlosNZ]

Initial issue with stray thoughts: #1042

Background

The system snapshots (particularly Fiji) are getting increasingly large and hard to work with. The Fiji snapshot is now up to 1GB and they've only had the system running for a couple of months! This causes several problems:

• Waste of space in backups (e.g. Dropbox). We currently export the whole system each backup, so we're ending up with multiple copies of the same files every day
• Downloading a system snapshot is slow
• Uploading a snapshot is even slower, and it often exceeds the upload size of the nginx server, resulting in having to copy the file using some roundabout way.

So we need some kind of "incremental" system which allows us not to duplicate the files when backing up, and download/upload smaller snapshots if we already have most of the files downloaded/uploaded.

One thing we have going for us is that files don't change -- once they're in the system, they don't get updated, so we can use this to our advantage to make this process a bit simpler. (We'll need to be careful that we strictly adhere to this -- we've been known to swap out Carbone doc files in-place and re-upload a snapshot, but this won't work with this new, proposed solution.)

Requirements

• We should be able to download a snapshot that is a reasonable size (i.e. most files are not included), but when we restore it, the older files (that we didn't download) should be preserved.
• Each backup snapshot should only include files that have been added since the last backup. However, when restoring from a backup, we should be able to get back all the files needed, not just the ones included directly in this backup.
• We should be able to download archives separately, and specify whether it's all the archive or just recent changes
• We need to be able to re-build an archive using multiple partial archives (as per previous point)

Terminology

Let's get some definitions established so the rest of this makes sense:

• files folder: the current system "files", currently in /files with all files recorded in the "files" table of the database
• archive folder: a proposed additional system folder that sits within "files", where files older than a certain age are moved to so they can be kept out of snapshot/backup exports (if desired). See below for specs.
• archive snapshot: similar to a normal "snapshot", but for the archive folder (or part thereof). We would add an additional subfolder in the system "snapshots" folder, called "archives".

Proposal

The following assumes prior understanding of how files, snapshots and backups currently work in Conforma.

We establish a new subfolder called "__ARCHIVE" within the system files folder where we move files that can be "archived", i.e. older than a certain age.

• The reason it needs to be within files rather than alongside it is that the database currently stores file paths relative to the system files folder, so it would be messy to have to refer to files outside this
• The minimum age (for archiving) would be configurable, but should be at least a week or two. (It needs to be long enough that we don't archive files scheduled for deletion, which only happens once a day by default, but is also configurable)

We have an "Archive" script that runs on a schedule (probably once or twice a week -- configurable), which would move any files meeting the archiving criteria into a new dated subfolder within the archive folder, e.g.

- __ARCHIVE
  |
  - 2023-05-05_23-00-00 / 
  - 2023-05-13_23-00-00 /
  - etc...

Within each timestamped subfolder, we'd preserve the existing directory structure of any files moved into it.

That's the basic idea, and from here we can define (or update) procedures for the following operations that would complete this overall solution:

• Archive script (as above, but more detail)
• Take snapshot (extension of the existing "take snapshot" function/endpoint)
• Use snapshot (extension of the existing "use snapshot" function/endpoint)
• Take Archive snapshot (a new function/endpoint similar to take snapshot, but for full or partial archives)
• Use Archive snapshot (as above)
• Download/Upload Archive
• Backup script

Procedures

Archive script

• Files older than XXX days moved to archive
• Every time script runs, makes a new subfolder in archive:
  • Folder name YYYY-MM-DD_HH-MM-SS
• Subfolder contains “info.json” file with the following:

  {
    "timestamp": ISO string or Unix Epoch number
    "previousFolder": name of last subfolder that was archived to
    "previousUID": UID of last subfolder
    "UID": new UID
  }
  

• Having references to the previous folder and UID means we can reconstruct an entire archive working backwards from a current UID, and use this to preserve the integrity and verify completion of archives
• Archive root folder also maintains a “archive.json” file with all the included subfolders, indexed by UID, I.e.

  {
     "73-7AbQwwToQJcVo-wjBE": { "timestamp": <timestamp>, "archiveFolder": <name-of-subfolder> },
     ...
  }
  

  This can be used make subsequent scans and searches quicker.
• When script runs:
  • find most recent subfolder (based on timestamp in info.json)
  • verify integrity of archive by ensuring chain of UIDs all exist (back to the subfolder with null for “previous” properties)
• Move older files to new subfolder (preserving structure)
• Update root “archive.json” file
• Update database with new paths
• Mark files as "archived" (new boolean field) in database -- this is so subsequent script executions can tell which files have already been archived.
• Add latest subfolder and UID to “system_info” table in Database (so system can always know if it has up-to-date archive)

Take snapshot

• 3 options:
  • No archive included (just current files)
  • Full archive included
  • Partial archive included (provide a UID, and it’ll grab everything subsequent)
• Snapshot’s own “info.json” should include the UID of the latest archive subfolder

Use snapshot

• Check UID in snapshot and then verify the entire Archive UID chain can be constructed from any of the following sources:
  • existing system archive folder
  • within this snapshot folder
  • in “Snapshots/archives” folder (having been uploaded separately in and archive snapshot)
  • in "Backups" folder
• If not, don’t proceed, show/return error
• If so, load snapshot and restore archive from the above sources
• If snapshot has no Archive UID, restore as normal (as it must be from a snapshot prior to this system)

Take Archive snapshot

• Similar to “Take snapshot” — save a copy of the “Archive” in “snapshots/archives” folder — name contains date string and UID of most recent subfolder
• Can specify either ALL or Partial (from a specific UID onwards)
• root “archive.json” should include full list not just the folders contained in this archive snapshot

Use Archive snapshot

• Similar to “Use snapshot” but for archive exports — restore archive to current system
• Will only proceed if a complete archive can be rebuilt, using the contents of this archive snapshot and the current system archive
• Will show a warning (before restore) if current record in “system_info” doesn’t correspond to archive latest record. However, we do allow to proceed as long as the latest UID is in system_info somewhere (i.e not the most recent), since it might be the case that we’re restoring an archive from multiple incremental snapshots. But we would reject any archives that are not in the system_info at all as they'd be from a completely different system.

Download archive

• Download .zip of above saved “archive snapshot” as per snapshot downloads

Upload archive

• Upload .zip of archive snapshot as per snapshot uploads

Backup script

• Check UID of most recent Archive file(zip) in backup folder
• Take partial archive snapshot from that UID to current latest UID and export to backups folder (using partial "Take archive snapshot" method above)
• Take snapshot (with no archive included) as per normal

Other considerations

• This proposed solution aims to capture all the possible use cases based on our way of working with configurations and backups that we've established over the last year or two. It would be good to talk through some possible scenarios to see if we can come up with any that aren't possible with this solution.
• This solution doesn't provide any validation of the integrity of the archive folder contents. So if someone manually removes or changes a file in an archive and restores it, we wouldn't know. But this is possible with the current "files" in a snapshot anyway. I did consider making each UID a hash/checksum of its contents, but the performance overhead to verify an entire archive would be unnecessarily burdensome, and there's no point in such a strict check if the files themselves are still vulnerable to meddling. If we did want to go down this path, then a better way would be to store a hash of each file in the database, then verify each one as part of the existing "file cleanup" script. The UIDs here are just to allow for checking that snapshots can be matched with the appropriate archives, and to allow full archives to be reconstructed from several partial ones.

[jmbrunskill]

Would a tool that automatically manages snapshots be useful here. E.g. https://kopia.io/ rather than writing our own custom logic?

[CarlosNZ]

Would a tool that automatically manages snapshots be useful here. E.g. https://kopia.io/ rather than writing our own custom logic?

Maybe, but we already have a pretty solid snapshot system (full database dump, and all files, etc.), so whatever we do needs to be an extension of that.

Primarily, our snapshots are used for moving data between systems (local, demo server, live server, etc.), not backups. We use it for backups, but that's only because it was easy to co-opt what we already had to take scheduled snapshots and call them "backups".

I'd need to have a closer look at third-party options to see if it can be made to fit our needs. And maybe talk it through with someone who might have some insight.

[jmbrunskill]

I'm planning to try out kopia and happy to have a discussion with you if you'd like? I don't understand the use cases perfectly but it doesn't sound like a perfect fit. Building our own system to match UUIDs etc sounds like it could be a lot of work so just thinking it would be good to leverage an existing solution if it did fit :)

Kopia (for example) uses content addressable hashing to de-dup files so in theory it would save a lot of space over a full system dump, but would probably require re-doing the encrypted zip backups I think.

[CarlosNZ]

Couple of additional use cases mentioned in Telegram chat. They're kind of implied by the above "requirements", but have a bit more specificity:

• I download a full snapshot on Monday with all the archived files
• On Friday I download another snapshot but this time just without the archive.
• I need to easily see what the difference is and download a separate archive snapshot with just the archived files I don't have locally. (In my solution you would just provide the ID of the most recent chunk you have locally, and then download a partial archive)

2. Backups are encrypted zip files of the whole system, so it's not feasible to inspect the contents of the zip when working out what to backup. So the backup folder needs a record of what files have been archived that exists outside each zipped snapshot. (And when restoring from a backup, it needs to know which snapshots contain which files)

3. If I upload a snapshot that contains a subset (or none) of the files, the system needs to know where to find the missing ones. If uploading to the same system you downloaded from, those files should already be in the system. But there may be more there than is needed for current snapshot, so the restore process needs to figure out exactly which files are required and where to get them from (and what to throw away).
   (The last part of that could already be achieved with our existing file cleanup script -- it deletes any files that aren't logged in the database)

[CarlosNZ]

Having had a little play around with some of these ideas, I think the whole chain-of-UIDs spec is a little overkill, as the archive root archive.json file will always contain the full list of archive subfolders required for a given snapshot. However, the broad principle still applies, as we still need UIDs for each subfolder.

Basically, the archive.json file always contains the full list of required archive folders for the snapshot/system that's its contained within. However, the subfolders that it refers to might not be present in the snapshot.

The info.json file within each subfolder is what indicates the presence of a given subfolder, and it's what we search for when trying to re-construct a snapshot.

So there probably isn't really a need to each info.json to have the UID of the previous folder -- we'll always use the root archive.json to describe a given snapshot.