LFS threshold

[Brilator]

Currently a lot of users run into issues due to the DataHUB lfs threshold.

This issue pops up repeatedly and requires discussion on multiple tools (ARC Commander, ARCitect, e.g.) and services (DataHUB) and possibly a discussion on what a useful LFS threshold would be. Hence the higher-level discussion here.

remote: fatal: pack exceeds maximum allowed size (100.00 MiB)
POST git-receive-pack (chunked)
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500
Writing objects: 100% (728/728), 5.32 GiB | 45.30 MiB/s, done.
Total 728 (delta 3), reused 0 (delta 0), pack-reused 0
error: remote unpack failed: index-pack abnormal exit

nfdi4plants/ARCitect#264

@SabrinaZander @Hannah-Doerpholz @j-bauer @JonasLukasczyk @TetraW @HLWeil

[j-bauer]

The way I see it: this limit seems to have created quite a lot of problems and frustration with the users. We might consider disabling this for now until we have a proper feedback/tool integration for such a limit.

The problem we tried to solve would still be there however: users pushing large files in a non-LFS part of the ARC. We already have a few ARCs were that is the case and thus we do need some way to a) detect such ARCs, b) find a way to (ideally automatically) rebase those ARCs so that the files are moved to the LFS storage.

But I think, I would really consider disabling the limit for now since both the integration with the tools and the user documentation/teaching isn't quite there yet.

[Brilator]

Really not sure, what's less of a headache: Teaching / implementing proper LFS now or trying to clean history of non-lfs-clogged ARCs.

[j-bauer]

Not sure either but we do have to cleanup some already clogged ARCs anyway.

At least by disabling it, the users wouldn't get frustrated/confronted with the problems.

But yeah, not sure which approach would be the best overall.

[Hannah-Doerpholz]

In our institute at least we are trying to teach people to correctly track large files. However, sometimes there are also a lot of small output files and it would probably make sense to still track those as well since they can add up to more than 100Mb easily (at least for our data). Would it not make sense to also track those even if they are small individually? In particular, they are usually not assay data files, but rather workflow results in the runs folders. Perhaps, tracking those files by default might be something to look into?

[kMutagene]

I just had a similar discussion with @ZimmerD, I think this highlights that we need a way to correctly distribute information like this inside the consortium. This change should have been first discussed with (or announced to) the devs of the tools so they can implement the threshold, while giving data stewards the info and potential release window so they can prepare users/facilities in advance.

[Brilator]

But I think, I would really consider disabling the limit for now since both the integration with the tools and the user documentation/teaching isn't quite there yet.

Ok, there have been quite some user requests this week. I'd appreciate, if we could unset the limit for a while. Hoping that we can clean up properly afterwards.

[TetraW]

I have deactivated the limit for now.

[Brilator]

Ok, now that it's deactivated, it turns out that this was not the (only) issue.

Some users also report this or a similar error, which is apparently a network issue?

error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500
send-pack: unexpected disconnect while reading sideband packet
Writing objects: 100% (105/105), 5.65 MiB | 9.16 MiB/s, done.
Total 105 (delta 23), reused 0 (delta 0), pack-reused 0
fatal: the remote end hung up unexpectedly

[j-bauer]

Might have been when the DataHUB was briefly unreachable due to the cloud issue? If possible, please provide timestamps to help us find something in the logs.

[j-bauer]

FYI: I've debugged with the user in question and it seems to be an ARCitect problem as the person could push using git directly. I asked him to create an issue in the ARCitect repo.

Update: the user just retried with the ARCitect (after updating it apparently) and it seems to work with the ARCitect aswell now. So the issue is solved for that particular user for now.

[Brilator]

While for some users, the LFS limit was a "correct" error - some ARCs were now again uploaded with large files not in LFS. For reasons I cannot reproduce. I assume there's been a mix of manual work on files + ARC commander + pure git + ARCitect.

[kMutagene]

We had a call about this topic, and here are the main results:

• LFS tracking should generally be handled by our client tools (e.g. ARCitect, ARCCommander)
  • There should be a concentrated effort on figuring out sensible defaults, may be a topic for data steward cycle
• Server-side restrictions should be implemented carefully and produce usable error messages that can be either used by the client tools to suggest solutions, or read and understood by git black-belts that use git directly to communicate with a data hub

Related ARCitect issue: nfdi4plants/ARCitect#264

Related ARCCommander issue: nfdi4plants/ARCCommander#245

I think both tools should try to align on the functionality.

cc @HLWeil @JonasLukasczyk @Brilator @j-bauer @TetraW

[Brilator]

Just for completeness: Limit of push-size overall (also 100MB?) was another issue last week in addition to the 100MB lfs-threshold.

[HLWeil]

Just for clarity, @Brilator: You're writing lfs-threshold (also in the header), but don't you mean non-lfs-threshold?

[Brilator]

I mean that files over a certain size should be required to be tracked by LFS

[HLWeil]

As for solutions: Is it possible to check whether a file is tracked by LFS in script? If so, we could include this as a validation case for the specification validation.

By this the users would not be hard-blocked but encouraged to e.g. ask a Data Steward for help.

@kMutagene @JonasLukasczyk @Brilator @j-bauer @TetraW

[Brilator]

Yes, there's a way via git-lfs. See

https://nfdi4plants.github.io/nfdi4plants.knowledgebase/git/git-troubleshooting/#check-the-status-of-lfs-tracked-files

[Brilator]

The original issue has since been "solved" or at least understood see #6.
I don't think it requires specific validation for LFS.
I would however suggest to have a DataHUB-site LFS threshold (e.g. file ≥ 20MB -> LFS). Should easily be implemented in DataHUB. And then return a warning, if files were not tracked properly. This would make sure, that people can still use pure git, but get a warning for large files.

[Hannah-Doerpholz]

Not sure if this is still relevant, but I have multiple scripts for tracking files according to my needs. For size it is this one:

# track all files in the ARC which are above 10 MB in size
find . -path './.git' -prune -o -type f -size +10M -print | while read -r file; do
  git lfs track "$file"
done

Maybe this or something like this could be helpful? I also have a script to track everything in runs based on file endings (because I don't want my yaml files to be tracked)

[Brilator]

Please contribute this to the git trouble shooting or other suitable place in the knowledge base.
This might help some people!