Is low memory operation in scope for gitoxide?

[tanriol]

Recently I wanted to mirror several big git reporisories by cloning them (as bare) and then periodically fetching updates on a low-end NAS. Turned out neither git nor gitoxide can currently be used to clone something like https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ while having just 512MB RAM.

I've failed to find whether being usable in low-memory environments is a goal for gitoxide. Would it make sense to try adding support for such use cases or would trying to support these without degrading performance in the normal case be unwanted complexity for gitoxide that'd be really unlikely to be accepted?

[Byron]

Thanks for sharing, that's a very interesting comparison and I am somewhat glad that gitoxide isn't worse than Git here.

However, it's also not a project goal to support smaller systems, particularly not 32bit systems which will have more trouble than with Git. Most 'small' systems nowadays are 64 bit, so the greatest obstacle towards completing a task will be swapping.

When you say that cloning the linux kernel with 512MB RAM didn't work, what does that mean exactly? OOM kill? Or excruciating swapping? If the case is the latter, I would have expected it to finish eventually.

Normally, I'd say that there is no way that this is made to work with the current algorithm - it's already mindful about memory and should be a bit better than Git, but it still has to hold millions of objects in memory all at once, which is where the high demands for the linux kernel are coming from.

However, there is an issue whose resolution basically requires rewriting the way packs are resolved. This algorithm will be more compatible, but also will be less parallel. While at it, I don't think it would hurt to make it possible to work in low-memory situations. To make that work, the only thing I can imagine that it crates some on-disk memory-mappable data-structure that first has to grow while streaming the pack, to later be traversed. Interestingly I'd think that if the disk is moderately fast, that streaming of data shouldn't be a bottleneck neither for reading nor for writing, if done cleverly, (or if a sqlite database is used ;)).

That's a long-winded way of saying: A fix would be welcome if it would also fix #1025 at the same time.

[Byron]

That's a great hint!

Even though I don't think reserve() is used incorrectly (i.e. used as if it was absolute, even though it's relative, i.e. additional items), I definitely found plenty of instances where potentially large vectors are instantiated with with_capacity(), which will typically over-allocate. What's meant in this case is reserve_exact().

I will change it and see if this has a noticeable impact, and wouldn't be surprised if it does.
Will post here once I have the data.

[tanriol]

Maybe using a different allocator could help?

In some cases maybe. In the gix slowdown and connection cutoff case - not really, at that time most memory consumption happens in just two arrays (64 bytes per entry on 64-bit, 52 I think on 32-bit) that are accessed by binary search in a mostly random pattern I think.

use a faster SSD

In my case it's a NAS without SSD slots, it's swapping to HDD, so swap-in has really high latency there.

Going to look into the RAM consumption and access patterns of these. Looks to me like it should be possible to change access patterns even without reducing peak RAM, but not certain yet.

[tanriol]

Even though I don't think reserve() is used incorrectly (i.e. used as if it was absolute, even though it's relative, i.e. additional items), I definitely found plenty of instances where potentially large vectors are instantiated with with_capacity(), which will typically over-allocate.

I think that may be actually fine - it'd increase address space consumption, but not RAM consumption as untouched pages won't ever be allocated. Well, unless you have overcommit disabled.

[NobodyXu]

In my case it's a NAS without SSD slots, it's swapping to HDD, so swap-in has really high latency there.

@tanriol I think you could try zswap/zram, it's designed for situations like this.

It will compress the page using whatever algorithm you pick instead of immediately swapping it out to a disk, and that could be beneficial if gix memory have good compression ratio, and only swap out if compressing the page isn't enough.

[Byron]

Right, I keep forgetting that modern OS's are very capable!

The PR here experiments with this, and comes up with nothing, no change at all. To add insult to injury, I am unable to profile memory consumption on MacOS due to code-signature requirements that I apparently can't fulfil.

Maybe you could share the tooling you used so I could run that in a Linux VM?