borg2: enhance compact stats

[ThomasWaldmann]

When building a ChunkIndex it currently starts from refcount=0 and then sets refcount=MAX_VALUE if a chunk is used.

That's how most of borg2 works now: it doesn't do refcounting anymore, just a boolean "do we have chunk X".

For better deduplication stats in borg compact, we could deviate from that in just borg compact and do precise refcounting without any additional effort.

Before persisting the ChunkIndex, we then need to set refcounts to MAX_VALUE, similar as we clean up the size values.

To consider:

• what do we win?
• we have the total size also in the archive metadata and can sum up all of these.
• do we want to add per-directory stats / analytics? maybe even 2-pass stuff?

[ThomasWaldmann]

Comment about what's interesting for practical usage: #122 (comment)

[awgcooper]

This would definitely be useful: #122 (comment)

Question: if compression and/or obfuscation is enabled, would the size stats be given for the native file, pre-compression etc?

[awgcooper]

Something else, not sure if it relates specifically to this: let's say I have a specific file backed up. I know this because it appears in a list contents of the most recent archive. Let's say I wanted to eliminate this file from the whole repo, how would I do that. Would I simply delete the first instance of it being backup up and by doing so that would automatically eliminate all dedups? If so, how would I find it? Fusermout?

[ThomasWaldmann]

@awgcooper No, it does not work like that.

But you can use borg recreate to rewrite all the archives that contain the unwanted file (or the directory). Just be very careful with that and first use --dry-run --list to see if it does what you want.

[ThomasWaldmann]

About "what do we win?" (see top post):

I guess the only thing would be the "deduplication factor", computed as:

DF = total_deduplicated_size_uncompressed / total_undeduplicated_size_uncompressed

The first value is just the sum of all plaintext chunk sizes.
The second value is the sum of the total archive sizes of all archives.

To do that in a memory efficient way together with the already present stats (which need the compressed chunk sizes), we need to store the plaintext size AND the compressed size into the in-memory ChunkIndex we build.

So, in the end, we could show deduplication and compression factors.