feat: More detailed tracing for distributors

[benclive]

What this PR does / why we need it:

Improves the tracing in the distributor. Currently we just get a single span for the whole request.

• Adds subspan for parseRequest, as this can be overridden by adaptive logs, for example.
• Splits the tracker for Kafka & Ingester pushes and creates separate spans for each of them.
• Instruments the kafka write client's ProduceSync.
• Adds a metric to track record sizes in bytes.

[grobinson-grafana]

Something that is unclear to me is what was the behavior before when sharing a common tracker, and what is the behavior now having separate trackers?

[benclive]

The API behaviour is exactly the same. The only benefit of the separate trackers is that we can monitor when all the ingesters complete vs when all the kafka writes complete. Before, the tracker only fired "done" once both sets of writes had completed and there was no way to tell which one was faster.

Initially I did this so we could create a histogram from the timings, but I've since replaced it with Spans. We do get subspans from the ingesters once we hit their Push endpoint, but there are no sub-spans from writing to Kafka which is what I wanted to amend here.

[grobinson-grafana]

In general LGTM, but a couple questions before I ✅

[JoaoBraveCoding]

lgtm, but it's the first time I reviewed tracing instrumentation so take it with pinch of salt