OpenTelemetry Inner Spans/Manual Telemetry

[rjawesome]

biothings/biothings_explorer#741

[tokebe]

@rjawesome This is a great start but something seems off...testing on my local it appears that context isn't properly being carried, so various spans end up having no or incorrect parents when viewed in Jaeger.

[rjawesome]

Seems to be working on my end for sync query... If you could let me know some more details about how you are testing then I can try to reproduce it.

Another note, JaegerExporter is being deprecated soon and it seems like the library wants people to use HTTP instead of UDP?

[tokebe]

It may be possible to switch back from JaegerExporter, will need some testing against CI that I can do later. For now, could you tell me your test setup to see if I can replicate the behavior you're seeing?

[rjawesome]

environment variables: JAEGER_SERVER=localhost
jaeger server: docker run -p 16686:16686 -p 4318:4318 -p 6831:6831/udp -p 6832:6832/udp --restart=no --label='desktop.docker.io/wsl-distro=Ubuntu-16.04' --runtime=runc -d jaegertracing/all-in-one:latest

bte run: pnpm run start
(it should say 5 times that opentelemetry init started before querying)
I'm querying /v1/query

[tokebe]

I wasn't able to get any reports to show up in jaeger with your setup. Trying the following setup did work for me (after changing JaegerExporter back to OTLPTraceExporter), but only received traces from biothings-explorer, not biothings-explorer-thread:

Amend package.json > scripts > watch to the following: "nodemon -e js,mjs,json,ts --watch './packages/**/src/*' './packages/**/data/*' --exec 'pnpm run build && node --require ./packages/bte-server/built/controllers/opentelemetry.js .'"

docker run -d --name jaeger -p 16686:16686 -p 6832:6832/udp -p 4318:4318 -p 6381:6831/udp --runtime=runc jaegertracing/all-in-one:latest

JAEGER_HOST=localhost pnpm start

I ran a basic /v1/query as a test.

[rjawesome]

Using your setup, adding a timeout seemed to make it more reliable for me (sometimes for me it doesn't work on the first query, so try running it a couple times). Also the HTTP exporter seems more reliable than the Proto for OTLP (see the second code thing)

In taskHandler.js (should require opentelemetry.js at the top as well which is in this branch)

...
 try {
    transaction.finish();
    span.end();
    Telemetry.removeOtelSpan();
  } catch (error) {
    debug("Sentry/OpenTelemetry transaction finish error. This does not affect execution.");
    debug(error);
  }

  await new Promise((resolve, reject) => {
    setTimeout(resolve, 5000);
  })

  debug(`Worker thread ${threadId} completed ${workerData.queue} task.`);
  
...

also, my implementation of trace exporter in opentelemetry.js (seems to be more reliable with HTTP -> see the import)

const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http");
  traceExporter: new OTLPTraceExporter({
    url: `http://${process.env.JAEGER_HOST}:4318/v1/traces`,
  }),