JSON output format modification & compatibility with other tools

[pspacek]

TL;DR

This is a vague proposal to work on modified JSON output format, with an ultimate goal - sharing the same output format with dnsperf and also DNS Shotgun.

Motivation

All three tools do essentially the same thing, send queries and statistically evaluate answers, but each tool does that in slightly different manner and fits slightly different use-case. At the same time, much of my data post-processing is in the end the same - I want to plot response rates, RCODEs, and latencies over time. Because each tool has a different format I have to switch parts of post-procesing pipeline to, in the end, get the same plots.

I think it would be common benefit if all three tools used the same JSON format and could reuse the same post-processing and plotting scripts.

What's going on in other projects

• I'm planning to modify DNS Shotgun to use the new JSON format - and it already uses format with separate output per-generator, so the fundamental split is the same as in Flamethrower
• I have a WIP version of dnsperf with "a" JSON output format - and plan to submit it to DNS-OARC/Jerry for review and merge after Christmas

So, I think that with buy-in on your end we can make the common format happen!

Discussion about format modification

This is not a firm proposal, take it as kickoff for discussion.

1. It would seem cleaner/easier for third-party parsers if each output line/JSON object had a separate field like "msgtype" which would clearly indicate that it is a "header" (currently "cmdline"), "period stats" or "total".

2. Somewhat related is number format and resolution. I would add header field with "timer units per second" information. That would define resolution used for time fields produced by particular implementation:

• Flamethrower varies between 1 sec vs. ms in some fields - I would standardize on one of them, preferably ms to avoid dealing with floating point
• DNS Shotgun uses ms = field value 1 000
• dnsperf uses usec - field value = 1 000 000

3. Deduplication
   I'm not sure why some field names and field values are duplicated... E.g.

• Could we keep runid just once in header instead of repeating it everywhere?
• Similarly, combining period_* fields with total_* fields in the very last message is confusing. Can it be split?
• In a similar vein, if we have "message type" in header maybe we can remove the period_ and total_ prefixes?

4. Header nits

• Maybe cmdline could be an array of strings
• I would add a separate version number for software used to generate the JSON and another for the JSON format itself

Wow, that is a long post. Sorry! If you made it here, what do you think about it?

[pspacek]

Oh, I forgot to mention that plotting scripts already exist. Basic example outputs:

• https://dns-shotgun.readthedocs.io/en/stable/response-rate-chart.html
• https://dns-shotgun.readthedocs.io/en/stable/connection-chart.html
• https://dns-shotgun.readthedocs.io/en/stable/latency-histogram.html

These scripts can do much fancier output, like per-rcode charts:

or show aggregate results for "groups" of runs with different parameters etc., but I'm not going to post all the variants here.

Of course not all of chart types are applicable to Flamethrower today as it does not seem to track fine-grained latencies, but I think it does not preclude us from sharing format.

[weyrick]

Thanks @pspacek, with just a quick look I think it's a great idea and I'd be happy to collaborate here. I'll review in more detail.

[pspacek]

It turns out I'm not the first with the idea! I've just discovered Flent which, besides other things, offers interactive GUI for browsing & editing charts. I plan to have a look if there is a way to converge on something compatible across even larger ecosystem.

Edited to add link: Flent data format

[weyrick]

@pspacek in terms of a common format that allows better interaction with other tools - in my observability project we have standardized on openmetrics and opentelemetry ... since this is time series data, I think it could make some sense. There are some existing libraries available, but in the end the OTLP format is protobuf, which I believe could be serialized to JSON. This would enable users to easily get the data into their observability stacks, since it supports many different types of "exporters".

[pspacek]

I'm familiar with openmetrics but it might make sense as well. So many options 🤯 😀

[pspacek]

Proposal, all values are integers or strings. Time precision us only.

prettyjson.json.zip

[nicki-krizek]

Feedback on the proposed format:

• merged means that the data has been added up from all senders/threads. I don't think this should be a property of the header, but rather the "data" entry (i.e. stats_periodic). It might also be a special reserved value for the threadid instead.
• there should be some value representing bad/discarded packets -- whether due to network conditions, invalid input data, premature exit, or other error conditions that prevented the packet from being sent
• latency buckets: consider whether their range needs to be specified in every data entry or if it's sufficient to specify them once in the header entry
• add data for connection tracking, e.g. shotgun uses:
  • conn_active: number of active connections
  • conn_handshakes: number of handshakes performed
  • conn_handshakes_failed: number of failed handshakes
  • conn_resumed: number of connection resumed with TLS resumption
  • conn_quic_0rtt_loaded: number of connection for which 0RTT was accepted on a QUIC protocol level
• shotgun also has a couple of fields for tracking success rate of 0RTT queries send over QUIC:
  • quic_0rtt_sent: number of requests for which 0RTT data was used over QUIC
  • quic_0rtt_answered: number of answers which were received for requests sent as 0RTT data over QUIC

[pspacek]

We have moved the unified format description into a separate repo: https://github.com/DNS-OARC/dns-metrics