Backup-restore throughput is awful

[cemeyer]

I have a gigabit internet connection, a fast multi-core CPU, and a fast local NVMe disk.

Yet tarsnap -rf xxx | pv | tar -xf - shows tarsnap is only able to download around 50 kB/s. What gives? At this rate it'll take 6 days to download a 26 GB backup set.

[gperciva]

Yes, this is a known problem with Tarsnap. The extract performance is currently latency-bound by the network connection to S3. @cperciva has a plan for addressing it, but at the moment we are not announcing any estimated time for this improvement.

This can be mitigated somewhat by doing parallel extracts:
http://www.tarsnap.com/improve-speed.html#faster-restore

[cemeyer]

Thanks, the context helps a little. It's unfortunate the throughput is so limited in the product tool.

I've gone ahead and forked the redsnapper tool to remove the dependency on a rubygem that isn't in FreeBSD ("thread"). It seems to be parallelizing successfully, although it uses some awful O(N log N + N · M log M) algorithm to balance files across workers, which took quite a long time to run between the -tf and -xf portions of the operation. (Obviously it doesn't scale well to backups with large numbers of individual files.)

Edit: with about 500 workers it seems I can pull 80-140 Mbps out of EC2 even with relatively small files.

[cemeyer]

Sent my redsnapper changes upstream in case they're helpful to others: https://github.com/directededge/redsnapper/pulls . I suggest --jobs 1000 if you've got the RAM and ulimit for it.

Edit: my fork is here: https://github.com/cemeyer/redsnapper

[cperciva]

I don't think --jobs 1000 is what you want -- because the Tarsnap service won't (currently) allow you to have that many connections open at once. I don't recommend going past --jobs 100.

[cemeyer]

Well, I'd appreciate it if you'd solve the restore performance problem in the tarsnap client itself, but failing that, please get out of the way and lift the arbitrary connections restriction.

EC2 has big pipes, S3 has big pipes, and I have a fast computer and fast internet connection, and even with 100+ parallel jobs, redsnapper+tarsnap can only use about 16% of my available internet bandwidth.

[jrnewton]

Any update about whether this issue will be addressed in tarsnap?

[gperciva]

At the moment we can't provide an estimated time of completion, sorry.

[greghuc]

@cperciva and @gperciva. Any progress on improving tarsnap backup-restore throughput, as per this ticket from Nov 2018?

I thought I'd ask, as I've been evaluating tarsnap for production use. The design philosophy is great, the cli tool is nice to use, but I keep seeing recent concerns re restore speed:

• This current Github ticket from Nov 2018
• This tarsnap-users thread from May 2021, where a French user notes the slowness of extracting a 100GB file.
• This hacker news thread from Sept 2020, where user "regecks" talks about the "glacial" slowness of tarsnap restore, resulting in migration to borgbase.com.

As per Tarsnap "improve speed" tips, I've been experimenting with restoring named files in parallel, using this gist as a basis. Am currently running tests from my UK computer, but will also try from US cloud to be closer to the data.

Many thanks

Greg

[greghuc]

Brief update re Tarsnap performance test. Restore performance for 17 git repos totalling 2.7G, using 100 tarsnap clients in parallel with xargs (as per #333 (comment)):

• UK computer = 57m 12s
• Heroku US dyno = 8m 46s

That's a 6.5x speed up.