Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: initial commit for erofs support #25

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

rchincha
Copy link

@rchincha rchincha commented Nov 9, 2024

A second attempt, converging faster.

pkg/fs <- main interface .. everything calls into here, stacker, atomfs, atomfs-snapshotter
pkg/squashfs
pkg/erofs
pkg/common <- common fs routines .. should probably become "internal"
pkg/verity <- verity stuff
pkg/molecule
pkg/log
pkg/mount

@rchincha rchincha force-pushed the ero branch 5 times, most recently from d0480ec to 4f4ab76 Compare November 10, 2024 07:36
@rchincha
Copy link
Author

project-stacker/stacker#626

^ have a stacker PR that builds against this PR.

@rchincha rchincha changed the title feat: initial commit feat: initial commit for erofs support Nov 10, 2024
pkg/common/exclude.go Outdated Show resolved Hide resolved
pkg/common/mount.go Outdated Show resolved Hide resolved
pkg/common/mount.go Outdated Show resolved Hide resolved
pkg/common/mount.go Show resolved Hide resolved
pkg/erofs/erofs.go Outdated Show resolved Hide resolved
pkg/erofs/superblock.go Outdated Show resolved Hide resolved
pkg/erofs/verity.go Show resolved Hide resolved
pkg/erofs/verity.go Outdated Show resolved Hide resolved
pkg/fs/fs.go Outdated Show resolved Hide resolved
pkg/verity/verity.go Show resolved Hide resolved
@rchincha
Copy link
Author

rchincha commented Nov 13, 2024

not ok 1 guestmount works ignoring verity in 130ms                                                                                                                        # (in test file unpriv-guestmount.bats, line 20)                                                                                                                          #   `lxc-usernsexec -s <<EOF' failed                                                                                                                                      # + export ATOMFS_TEST_RUN_DIR=/tmp/bats-run-Hh1tGT/suite/run/atomfs                                                                                                      # + export PERSIST_DIR=/tmp/bats-run-Hh1tGT/test/1/persist-dir
# + mkdir -p /tmp/bats-run-Hh1tGT/test/1/persist-dir                                                                                                                      # + readlink /proc/self/ns/mnt                                                                                                                                            # + cut -c 6-15                                                                                                                                                           # + export INNER_MNTNSNAME=4026533012                                                                                                                                     # + set +e                                                                                                                                                                # + atomfs --debug mount --persist=/tmp/bats-run-Hh1tGT/test/1/persist-dir /tmp/bats-run-Hh1tGT/suite/oci:test-squashfs /tmp/bats-run-Hh1tGT/test/1/testmountpoint
# 2024/11/13 03:39:43 debug -> ws.recurse             digest=sha256:e1f7649d181d720b424c96aa8911f00fb16a9f95be1c7526caef859008de235e
# 2024/11/13 03:39:43 debug <- ws.recurse             digest=sha256:e1f7649d181d720b424c96aa8911f00fb16a9f95be1c7526caef859008de235e                                      # 2024/11/13 03:39:43 debug casext.ResolveReference(test-squashfs) got these descriptors refs=[{[{application/vnd.oci.image.manifest.v1+json sha256:e1f7649d181d720b424c96aa8911f00fb16a9f95be1c7526caef859008de235e 1226 [] map[org.opencontainers.image.ref.name:test-squashfs] [] <nil> }]}]                                                     # 2024/11/13 03:39:43  info Failure detected: cleaning up "/tmp/bats-run-Hh1tGT/suite/run/atomfs/meta/4026533012/tmp-bats-run-Hh1tGT-test-1-testmountpoint"
# Error: won't guestmount an image with verity data without --allow-missing-verity 

Just one bats test is failing

@rchincha rchincha force-pushed the ero branch 3 times, most recently from 1b6bc44 to 5940164 Compare November 13, 2024 03:53
@raharper
Copy link
Contributor

Thanks for the update. I think I'm still missing something. I see squashfsFile or squashFile passed around in the common and per-fs code... are we really passing in a path or sha256sum of a squashfile?

pkg/common/mount.go Outdated Show resolved Hide resolved
pkg/common/mount.go Show resolved Hide resolved
pkg/common/mount.go Outdated Show resolved Hide resolved
pkg/common/mount.go Outdated Show resolved Hide resolved
Comment on lines 76 to 80
tmpErofs.Close()
os.Remove(tmpErofs.Name())
defer os.Remove(tmpErofs.Name())
args := []string{tmpErofs.Name(), rootfs}
compression := GzipCompression
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably should be parameter right?

pkg/erofs/fs.go Outdated Show resolved Hide resolved
pkg/erofs/verity.go Outdated Show resolved Hide resolved
pkg/verity/verity.go Show resolved Hide resolved
@rchincha rchincha force-pushed the ero branch 2 times, most recently from fb69535 to 3541b62 Compare November 16, 2024 05:27
Copy link

codecov bot commented Nov 16, 2024

Codecov Report

Attention: Patch coverage is 14.26799% with 691 lines in your changes missing coverage. Please review.

Project coverage is 13.85%. Comparing base (eaa7b43) to head (cdf16db).

Files with missing lines Patch % Lines
pkg/erofs/erofs.go 5.40% 413 Missing and 7 partials ⚠️
pkg/common/mount.go 0.00% 74 Missing ⚠️
pkg/common/exclude.go 0.00% 45 Missing ⚠️
pkg/erofs/fs.go 0.00% 39 Missing ⚠️
pkg/squashfs/fs.go 0.00% 39 Missing ⚠️
pkg/erofs/superblock.go 59.55% 24 Missing and 12 partials ⚠️
pkg/common/fs.go 36.66% 18 Missing and 1 partial ⚠️
pkg/molecule/molecule.go 31.25% 11 Missing ⚠️
pkg/squashfs/squashfs.go 40.00% 6 Missing ⚠️
pkg/erofs/mediatype.go 75.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #25      +/-   ##
==========================================
- Coverage   15.68%   13.85%   -1.84%     
==========================================
  Files           7       16       +9     
  Lines        1358     1718     +360     
==========================================
+ Hits          213      238      +25     
- Misses       1117     1440     +323     
- Partials       28       40      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@rchincha rchincha force-pushed the ero branch 6 times, most recently from 88dfd93 to 9dc7dfd Compare November 17, 2024 06:36
@rchincha rchincha force-pushed the ero branch 10 times, most recently from 97e2b40 to 15cdcb2 Compare November 21, 2024 23:13
@rchincha
Copy link
Author

https://lore.kernel.org/all/[email protected]/T/
^ looks like parallel mkfs.erofs has landed on tip
ubuntu 24.04 doesn't have this part yet

@rchincha
Copy link
Author

Going to deprecate the double +
image+ztsd+verity

and just rely on the root_hash annotation for verity

pkg/erofs/erofs.go Outdated Show resolved Hide resolved
@rchincha rchincha force-pushed the ero branch 8 times, most recently from 8e5f19b to 8ab7e49 Compare December 3, 2024 04:14
@rchincha
Copy link
Author

rchincha commented Dec 3, 2024

Just tested build erofs image, atomfs mount and then lxc-execute. Works nicely!

Before this commit, only squashfs was supported.
However, there are other filesystems such as erofs that fit the same
theme, and additional filesystem support requires refactoring and
exposing a more generic filesystem interface.

pkg/fs/fs.go    - Filesystem interface
pkg/squashfs    - squashfs
pkg/erofs       - erofs
pkg/common      - filesystem-agnostic common routines
pkg/verity      - verity routines

Signed-off-by: Ramkumar Chinchani <[email protected]>
BREAKING-CHANGE: the layer media-type no longer contains "+verity"

For a layer media-type, we add the fstype+compression+verity_present.
Only one '+' is allowed as per following RFC.
https://datatracker.ietf.org/doc/html/rfc6838#section-4.2

Instead just rely on "root_hash" annotation.

Signed-off-by: Ramkumar Chinchani <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants