Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for executing OSD class methods #308

Open
phlogistonjohn opened this issue Jun 16, 2020 · 2 comments
Open

Add support for executing OSD class methods #308

phlogistonjohn opened this issue Jun 16, 2020 · 2 comments
Labels
ceph-libs Fix or enhancement needed in ceph libraries enhancement

Comments

@phlogistonjohn
Copy link
Collaborator

In ceph/ceph-csi#1071 we started a side conversation about further improvements to the approach used by csi to work with omaps.

There @dillaman wrote:

As a further (future) optimization, you could eliminate the journal lock and use a RADOS class call to execute the cmpomp.cmp_set_vals method which would allow you to atomically reserve the UUID and then, combined w/ a second omap setvals call, store all the values with a single round-trip to the OSD.

and

You need to expose this API [1] in go-ceph to support calling arbitrary OSD-side cls methods. The particular cls method I was referring to is located here [2] and the blob (bufferlist) for the input parameters is here [3].

[1] https://github.com/ceph/ceph/blob/master/src/include/rados/librados.h#L3004
[2] https://github.com/ceph/ceph/blob/master/src/cls/cmpomap/client.cc#L21
[3] https://github.com/ceph/ceph/blob/master/src/cls/cmpomap/ops.h#L29

Looking at the links I see that wrapping the linked function and those similar to it (see #307 #306 #305 #304 etc) would be generally straightforward†.

The cls and method arguments are strings so that's a matter of choosing the correct inputs.

It's the last item that has induced me to file a specific issue to track the topic.
Looking at the linked code I see serialization code that appears to be C++ outside of the librados C wrapper. It looks like it relies on the encoding functions also in ceph's src/include/encoding.h.
(Is this ceph's wire format?)

If this is useful functionality for our users I am wondering how we ought to make this available to our users as, by default, Go doesn't wrap C++ and this appears outside of librados.

Options I see:

  • Add additional data serialization functions to librados C layer
  • Implement the encoding/decoding in pure-Go
  • Somehow try to start consuming C++ outside of librados

Am I missing anything?
@dillaman I'd like your input on this, especially if I've overlooked or misunderstood how this is meant to be used. Later, we can discuss feasibility & effort of these various approaches.

There doesn't seem to be wrappers for this in pybind so I can't steal^W borrow ideas from there, at the moment. :-)

† - I want to do more proper support for write op and read op, and I started looking into it but I haven't followed up much on that. Fie on me.

@phlogistonjohn
Copy link
Collaborator Author

Poking around a bit more, and it appears this isn't in any released versions of ceph yet, not even octopus, so it would appear we'd need to wait a while for csi to be able to consume it anways.

@dillaman
Copy link

Poking around a bit more, and it appears this isn't in any released versions of ceph yet, not even octopus, so it would appear we'd need to wait a while for csi to be able to consume it anways.

Good catch -- so this is just a backlog item for the time being then.

@phlogistonjohn phlogistonjohn added ceph-libs Fix or enhancement needed in ceph libraries enhancement labels Jul 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ceph-libs Fix or enhancement needed in ceph libraries enhancement
Projects
None yet
Development

No branches or pull requests

2 participants