Add ID-mapping capabilities #1392

sondavidb · 2024-10-09T01:28:51Z

Issue #, if available:

Description of changes:
Working off the POC at Kern--/idmap-v2, added id-mapping capabilities to the snapshotter.

The PR additionally has a commit to add a separate ctr binary to the container. This is because containerd doesn't support id-mapping in ctr in an official release yet, so we need a specific binary to pass the proper labels. This is done by adding a new Makefile target and copying it in the Dockerfile, but I'm open to different approaches here.

Testing performed:
make test and make integration

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

austinvazquez

Thanks for including the modified ctr as a individual commit. Makes it easy to revert if containerd accepts the upstream contribution later.

Overall nice work iterating on this. I would like to see the logging cleaned up a bit. Some considerations:

Quite a bit of info logging occurring. Should some of these be reduced to debug or trace logs? Users running in production operate a large scale, so cost of logs increases quickly. So make each log count.
Prefer structured logging over formatting. Think about the tech that gets paged and has a hard time finding where in code an error originated because it has been formatted.

austinvazquez · 2024-10-09T15:55:08Z

Makefile

+    # Use a custom fork for testing ID-mapping as containerd doesn't fully support this yet.
+	git clone https://github.com/sondavidb/containerd.git tempfolder
+	cd tempfolder && \
+	git checkout multi-uid-gid && \


Instead of taking a dependency on a developer fork, can we carry a patch in SOCI and pull a known good tag from upstream as a git submodule.

austinvazquez · 2024-10-09T15:58:02Z

fs/fs.go

@@ -455,6 +459,79 @@ func (fs *filesystem) getSociContext(ctx context.Context, imageRef, indexDigest,
 	return c, err
 }

+func (fs *filesystem) getIDMappedMountpoint(mountpoint, activeLayerKey string) string {


Function doesn't use any state from filesystem. Could this function be standalone?

austinvazquez · 2024-10-09T16:01:57Z

fs/fs.go

+
+func (fs *filesystem) IDMapMount(ctx context.Context, mountpoint, activeLayerKey string, idmapper idtools.IDMap) (string, error) {
+	newMountpoint := fs.getIDMappedMountpoint(mountpoint, activeLayerKey)
+	log.G(ctx).WithField("mountpoint", newMountpoint).Info("creating remote id-mapped mount")


It looks like most logs in this function add this field. Perhaps something like this would be more concise.

Suggested change

log.G(ctx).WithField("mountpoint", newMountpoint).Info("creating remote id-mapped mount")

logger := log.G(ctx).WithField("mountpoint", newMountPoint)

logger.Info("creating remote id-mapped mount")

austinvazquez · 2024-10-09T16:10:34Z

fs/fs.go

+		log.G(ctx).WithField("mountpoint", newMountpoint).Info("failed to create remote id-mapped mount")
+		return "", errdefs.ErrNotFound
+	}
+	fs.layer[newMountpoint] = l


So this adds the new mountpoint, but does not remove the "old" mountpoint. Is that intentional?

It just reuses the same layer and builds on top of it AFAIK. I don't believe we need to unassign this in any way?

austinvazquez · 2024-10-09T19:27:49Z

fs/fs.go

+	if _, err := exec.LookPath(fusermountBin); err == nil {
+		mountOpts.Options = []string{"suid"} // option for fusermount; allow setuid inside container
+	} else {
+		log.G(ctx).WithError(err).Infof("%s not installed; trying direct mount", fusermountBin)


Prefer structured logging instead of formatting. (handful of other places if you can get those as well)

Suggested change

log.G(ctx).WithError(err).Infof("%s not installed; trying direct mount", fusermountBin)

logger.WithField("binary", fusermountBin).WithError(err).Info("fuse binary not found; trying direct mount")

austinvazquez · 2024-10-09T20:20:01Z

snapshot/snapshot.go

+	for i := range s.ParentIDs {
+		id := s.ParentIDs[i]
+		if _, ok := o.idmapped[s.ID]; ok {
+			id = fmt.Sprintf("%s_%s", s.ParentIDs[i], s.ID)


Can use id here instead of reading again.

austinvazquez · 2024-10-09T20:20:35Z

snapshot/snapshot.go

+}
+
+func (o *snapshotter) createIDMapMounts(ctx context.Context, s storage.Snapshot, idmap idtools.IDMap) error {
+	log.G(ctx).Infof("mapping ids")


Formatting is not required, but also is this logging too verbose? Perhaps consider removing or reducing to trace.

austinvazquez · 2024-10-09T20:24:07Z

snapshot/snapshot.go

+	// s.ID is the shortest unique identifier for each new container,
+	// so append it to the end of the new mountpoint
+	_, err := o.fs.IDMapMount(ctx, path, id, idmap)
+	if err == errdefs.ErrNotFound {


For readability, let's use errdefs.IsNotFound(err) here instead.

austinvazquez · 2024-10-09T20:29:14Z

fs/fs.go

+		return "", err
+	}
+
+	go server.Serve()


Does this goroutine need to be tethered to the passed in context? If not, it looks like the passed in context is only being used for logging.

Is it a bad thing to tether logging to the context?

austinvazquez · 2024-10-09T20:30:58Z

fs/layer/node.go

@@ -808,7 +812,9 @@ func entryToAttr(ino uint64, e metadata.Attr, out *fuse.Attr) fusefs.StableAttr
 	mtime := e.ModTime
 	out.SetTimes(nil, &mtime, nil)
 	out.Mode = fileModeToSystemMode(e.Mode)
-	out.Owner = fuse.Owner{Uid: uint32(e.UID), Gid: uint32(e.GID)}
+	// Potentially dangerous casting int -> uint32? But probably fine.


I'm generally cautious of casting from a signed int to an unsigned int, perhaps overly so 😅 I'd feel a bit more comfortable if we have some sort of bounds check before this?

Kern-- · 2024-10-09T22:55:27Z

fs/fs.go

+		return "", err
+	}
+
+	rawFS := fusefs.NewNodeFS(node, &fusefs.Options{


Is this all the same as in fs.Mount? Can we have a helper function instead?

Signed-off-by: David Son <[email protected]>

Taken from containerd commit 83aaa89, this adds the necessary tools to add idmapping capabilities to SOCI. Signed-off-by: David Son <[email protected]>

sondavidb · 2024-10-13T16:09:05Z

New changes address logging concerns as well (EDIT: missed some) as moved the FUSE server setup to a new function. ~~I think this also made the tests pass properly, possibly because I previously didn't setup the fuse logging properly before?~~ NVM, still not good, seems the FUSE servers aren't being set up properly. Might have to do with the layer bit Austin mentioned...There's also parts w/ overlay fallback which means that the layer isn't being setup properly, but the new code shouldn't even be called unless id-mapping is turned on? So I'm a bit confused...

Also added a new internal patch which will make the diff look even bigger but it's just so we can store the patch locally.

Signed-off-by: David Son <[email protected]>

sondavidb · 2024-10-14T01:41:11Z

FWIW, I tested these changes on the ubuntu runners and they all seemed to pass?? Not sure what it might be indicative of.

sondavidb#20

Kern--

I'm still reviewing, but I wanted to get my comments in so far before they drift too far.

Kern-- · 2024-10-09T23:02:15Z

fs/fs.go

@@ -692,7 +770,8 @@ func (fs *filesystem) Unmount(ctx context.Context, mountpoint string) error {
 	l, ok := fs.layer[mountpoint]
 	if !ok {
 		fs.layerMu.Unlock()
-		return fmt.Errorf("specified path %q isn't a mountpoint", mountpoint)
+		log.G(ctx).Errorf("specified path %q isn't a remote mount", mountpoint)


Can we wrap the error instead? We always try to Unmount before UnmountLocal so this is going to cause a lot of log messages.

Kern-- · 2024-10-09T23:06:20Z

fs/fs.go

@@ -692,7 +770,8 @@ func (fs *filesystem) Unmount(ctx context.Context, mountpoint string) error {
 	l, ok := fs.layer[mountpoint]
 	if !ok {
 		fs.layerMu.Unlock()
-		return fmt.Errorf("specified path %q isn't a mountpoint", mountpoint)
+		log.G(ctx).Errorf("specified path %q isn't a remote mount", mountpoint)
+		return errdefs.ErrNotFound
 	}
 	delete(fs.layer, mountpoint) // unregisters the corresponding layer
 	l.Done()


Since we're reusing layers, I think this invalidates the entire image when any idmapped layer is cleaned up.

Kern-- · 2024-10-09T23:08:24Z

fs/fs.go

@@ -706,6 +785,10 @@ func (fs *filesystem) Unmount(ctx context.Context, mountpoint string) error {
 	return syscall.Unmount(mountpoint, syscall.MNT_FORCE)
 }

+func (fs *filesystem) UnmountLocal(ctx context.Context, mountpoint string) error {


What does it mean to unmount a local mount?

I think both idmapped and non-idmapped layers are handled in Unmount.

sondavidb requested a review from a team as a code owner October 9, 2024 01:28

github-actions bot added dependencies Pull requests that update a dependency file go Pull requests that update Go code testing Unit and/or integration tests labels Oct 9, 2024

sondavidb force-pushed the add-uid-gid-mapping branch 4 times, most recently from 2d0cf6d to 7f2fd08 Compare October 9, 2024 03:32

austinvazquez requested changes Oct 9, 2024

View reviewed changes

Kern-- reviewed Oct 9, 2024

View reviewed changes

sondavidb added 2 commits October 14, 2024 01:05

Add id-mapped ctr to Makefile

d8bfc0d

Signed-off-by: David Son <[email protected]>

Add idtools package

6b0d083

Taken from containerd commit 83aaa89, this adds the necessary tools to add idmapping capabilities to SOCI. Signed-off-by: David Son <[email protected]>

sondavidb force-pushed the add-uid-gid-mapping branch from 7f2fd08 to b87f962 Compare October 13, 2024 16:06

Add support for idmapped layers

79b17e1

Signed-off-by: David Son <[email protected]>

sondavidb force-pushed the add-uid-gid-mapping branch from b87f962 to 79b17e1 Compare October 13, 2024 16:50

Kern-- reviewed Oct 14, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ID-mapping capabilities #1392

Add ID-mapping capabilities #1392

sondavidb commented Oct 9, 2024

austinvazquez left a comment

austinvazquez Oct 9, 2024

austinvazquez Oct 9, 2024

austinvazquez Oct 9, 2024

austinvazquez Oct 9, 2024

sondavidb Oct 13, 2024

austinvazquez Oct 9, 2024

austinvazquez Oct 9, 2024

austinvazquez Oct 9, 2024

austinvazquez Oct 9, 2024

austinvazquez Oct 9, 2024

sondavidb Oct 13, 2024

austinvazquez Oct 9, 2024

sondavidb Oct 13, 2024

Kern-- Oct 9, 2024

sondavidb commented Oct 13, 2024 •

edited

Loading

sondavidb commented Oct 14, 2024

Kern-- left a comment

Kern-- Oct 9, 2024

Kern-- Oct 9, 2024

Kern-- Oct 9, 2024

	log.G(ctx).WithField("mountpoint", newMountpoint).Info("creating remote id-mapped mount")
	logger := log.G(ctx).WithField("mountpoint", newMountPoint)
	logger.Info("creating remote id-mapped mount")

	log.G(ctx).WithError(err).Infof("%s not installed; trying direct mount", fusermountBin)
	logger.WithField("binary", fusermountBin).WithError(err).Info("fuse binary not found; trying direct mount")

Add ID-mapping capabilities #1392

Are you sure you want to change the base?

Add ID-mapping capabilities #1392

Conversation

sondavidb commented Oct 9, 2024

austinvazquez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sondavidb commented Oct 13, 2024 • edited Loading

sondavidb commented Oct 14, 2024

Kern-- left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sondavidb commented Oct 13, 2024 •

edited

Loading