Upgrade huggingface_hub to v0.27.x in dataset initializer v2

[astefanutti]

What this PR does / why we need it:

This upgrades huggingface_hub in the dataset initializer container image to v0.27.1 and includes huggingface/huggingface_hub#2333 that fixes #2378.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

Fixes #2378

Checklist:

• Docs included if any changes are user facing

[coveralls]

Pull Request Test Coverage Report for Build 12675989950

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 12656204551:	0.0%
Covered Lines:	85
Relevant Lines:	85

---

💛 - Coveralls

[andreyvelich]

Thank you for this fix @astefanutti!

[andreyvelich]

This bug is similar to: #2367

@astefanutti @Electronic-Waste @kubeflow/wg-training-leads @helenxie-bit @akshaychitneni @shravan-achar @seanlaii @deepanker13 @saileshd1402 what do we think about relaxing the huggingface_hub dependency for the initializer in V2 ?
Since we don't do serialization between the client and server, I am not sure if we should have strict version for the huggingface_hub.

I can see that datasets are using huggingface-hub>=0.24.0

[astefanutti]

Un-constraining the version would certainly work. That being said, non-reproducible builds can sometime make regressions difficult to track down as the upgrades are pulled implicitly. So without going as far as locking the dependency tree with Pipfile.lock for example, I find constraining the versions in the requirements a good compromise.

[andreyvelich]

Do we know if HuggingFace community cherry-pick fixes to the previous minor releases of huggingface_hub?
If yes, we can always do: huggingface-hub>=0.27.0,<0.28.

[astefanutti]

It doesn't seem like fixes are cherry-picked in Z stream, but the fix for that particular issue is already present in previous minor releases, so we can use huggingface-hub>=0.27.0,<0.28 in any case and get patches.
I've updated it according to your suggestion.

[andreyvelich]

Yeah, let's try to use this for now.

[andreyvelich]

Thanks for the fix @astefanutti!
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment