Baggage Propagation Across Process Boundaries

[mannyluvstacos]

Hi Otel-JS Folks!

I hope y'all have been well.
Thank you ahead of time for taking the time to review and answer my question.

I am trying to understand and make sure my implementation of inject into the HTTP.headers for inter-process boundary propagation is correct.

I created a Fastify based demo application to base my question on:
https://github.com/mannyluvstacos/otel-baggage-example

• config: https://github.com/mannyluvstacos/otel-baggage-example/blob/main/config.ts
• server (baggage manipulation): https://github.com/mannyluvstacos/otel-baggage-example/blob/main/index.ts#L29-L60

Question(s)

• Is using propagation.inject of PropagationAPI on the request.header the appropriate way to propagate baggage across process boundaries to the next process in the trace?

• What is the purpose then of HttpBaggagePropagator? what does it serve?
  ref: https://github.com/mannyluvstacos/otel-baggage-example/blob/main/config.ts#L54

• In my example I am making use of the propagation api entry point to inject the Baggage onto the request.headers (in this case the request.raw.headers), is that all there is to it?

• Additionally, after using propagation.inject(...), it returns a new context and I am curious how to set this context to be the active context?

I look forward to hearing back!
-Manny

---

Context and My Understanding

I read Propagation of the OTel Specification to try and understand baggage propagation across boundaries.

In the sample project I am creating a baggage entry and then using the propagation api, I am injecting that baggage onto the headers in the IncomingHTTP request.

        const baggageEntry: BaggageEntry = {
        value: 'iAmATestValue',
        };
        const newBaggage = propagation.createBaggage({
            'testEntry': baggageEntry
        })
        const newContextWithBaggage = propagation.setBaggage(context.active(), newBaggage)
        propagation.inject(newContextWithBaggage,request.raw.headers)

ref: https://github.com/mannyluvstacos/otel-baggage-example/blob/main/index.ts#L35-L42

This request can be forwarded onto the next process (maybe another node.js service)
where one can then extract the baggage header from the IncomingHTTP request via propagation api, and then call getBaggage followed by .getEntry([key]

 const rawHeadersToExtract = request.raw.headers;

        const extractedPropagationContext = propagation.extract(context.active(), rawHeadersToExtract)
        console.log({ extractedPropagationContext})

        const extractedBaggage = propagation.getBaggage(extractedPropagationContext)
        console.log({extractedBaggage})

        const entryByName = extractedBaggage.getEntry('testEntry')
        console.log({ entryByName })

[mannyluvstacos]

Hi @dyladan!

I had seen the gitter conversation around inject/extract for propagation (I believe that context was not for baggage), and I would love your feedback/guidance when you have a moment :)

cc: @vmarchaud , Valentin, if you are available and able to speak to my questions above, I would love your feedback!

Thank you again for your time.

Respectfully,
Manny

[vmarchaud]

Is using propagation.inject of PropagationAPI on the request.header the appropriate way to propagate baggage across process boundaries to the next process in the trace?

Http instrumentation should automatically propagate baggage for outgoing http request (if HttpBaggagePropagator is registered)

What is the purpose then of HttpBaggagePropagator? what does it serve?

We have a concept of propagator that you can register on the api and then instrumentation will use them to know what to inject on outgoing request. In this case the HttpBaggagePropagator will pickup baggage from the context and inject it into what is given by the http instrumentation (headers for http)

In my example I am making use of the propagation api entry point to inject the Baggage onto the request.headers (in this case the request.raw.headers), is that all there is to it?

Even if you could manually, the http instrumentation should do that for you to avoid writing those lines. Normally in this log you should see the baggage header (official spec)

Additionally, after using propagation.inject(...), it returns a new context and I am curious how to set this context to be the active context?

If you want to set the new context as active, you should use api.context.with and returning at the end of your handler (see the restify instrumentation code)

[mannyluvstacos]

I appreciate the guidance, @vmarchaud !

Http instrumentation should automatically propagate baggage for outgoing http request ... the http instrumentation should do that for you

Understood! The Http Instrumentation will automatically propagate. :)
But I would need to update the headers for http, correct?

• If I wanted to add to baggage, then I should make updates the the baggage on the active context correct?
• What I am unclear on is how to update baggage (let's say in a request handler I want to add some additional K:V pairs before any outgoing request), should I update the baggage header directly? add to context via setValue?

I am confusing myself here, as I thought propagation.inject was the way to go.

If you want to set the new context as active, you should use api.context.with
(thank you for that reference!)

I need to chew on this a bit more, I saw other examples where api.context.with was used to wrap the route.
That looks to be what is going on in the restify instrumentation code yes?

I might be confusing myself again here and this is where the propagators come in as updating baggage should then be picked up by the HttpBaggagePropagator

[vmarchaud]

But I would need to update the headers for http, correct?

Nope, the http instrumentation should/is taking care of that. As soon as you register propagators, it should call them for every outgoing/incoming request.

What I am unclear on is how to update baggage (let's say in a request handler I want to add some additional K:V pairs before any outgoing request), should I update the baggage header directly? add to context via setValue?

You should use the context.with() and setBaggage method from the API:

import { propagation, context } from '@opentelemetry/api'
// retrieve the current baggage
const baggage = propagation.getBaggage(context.active())
// mutate it
const updatedBaggage = baggage.setEntry('key', 'value')
// create a new context with the mutated baggage
const newContext = propagation.setBaggage(context.active(), updatedBaggage)
context.with(newContext, () => {
  // here the value of current baggage is updatedBaggage and not the original baggage
  const newBaggage = propagation.getBaggage(context.active())
  assert.strictEqual(newBaggage, updatedBaggage)
})
// however here the value of current baggage is baggage since we are outside of the context.with
const newBaggage = propagation.getBaggage(context.active())
assert.strictEqual(newBaggage, baggage)

I might be confusing myself again here and this is where the propagators come in as updating baggage should then be picked up by the HttpBaggagePropagator

Propagator is a concept to contain specific propagation logic (which header to use etc) however it doesn't propagate by itself. Its instrumentation that pickup current propagator registered and call them to know which data to put in which header.

[nata7che]

Hi @mannyluvstacos and @vmarchaud! I hope it's ok I'm jumping in on this interesting discussion (I am too in search of how to properly utilize the Baggage capabilities and am a bit lost..)

So from what I understand setting the Baggage is a way to pass information throughout an entire trace (along all spans and over distributed trace context). So if I have serviceA talking to serviceB with Http, serviceB writing to Redis and serviceC reading from Redis, and I add a Baggage entry in serviceA before the first span is created then I'd see the same info in spans exported from serviceB and serviceC. Is my understanding correct?

Two followup questions (in case I got the gist of things) are:

1. Are all instrumentations support collecting-and-passing-on the Baggage or is it something that depends on the instrumentation?
2. You said "As soon as you register propagators". Almost same question: Should this be defined for each instrumentation?

Sorry if my questions are missing some basic understanding of the mechanism and thanks in advance for taking the time to read this..

[nata7che]

Thanks @vmarchaud for your quick answer!

Do you have any thoughts on the best course of action for me to explore if I want to accomplish what I described? i.e. set an entry at the start of the flow/trace and have all spans exported with it?

[vmarchaud]

I don't think this would be feasible with redis transparantly because of how the protocol works (it doesn't contains metadata which could be used to propagate trace id/span id). It would require to add data to each message and force code handling it the message on other side to also implement the same logic

[nata7che]

So you mean that the instrumentation should be responsible for extracting the Baggage from the current context, and set it as part of the message, and the same for reading it and propagating it to the context? is Baggage the right object to use to pass such data or should I try and mount it on the context/some other object?

Again, I apologize if these are very basic questions, I was a bit lost in all the documentation.

[vmarchaud]

the instrumentation should be responsible for extracting the Baggage from the current context, and set it as part of the message, and the same for reading it and propagating it to the context?

It should be responsible for asking propagators what to inject in outgoing requests and extract from incoming request (i would advise to lookup the http instrumentation).

is Baggage the right object to use to pass such data or should I try and mount it on the context/some other object?

It's totally designed for that so yep.

[nata7che]

Thanks @vmarchaud super helpful!