This repo is a fork of ryotarai/prometheus-tsdb-dump which enables reading a Prometheus TSDB block and converting it to json. For our use case, we extended the functionality to read a tsdb block directly from s3 and convert it to json. The json can be further processed by downstream processes to extract and analyze relevant metrics.
$ git clone [email protected]:DISHDevEx/prometheus-tsdb-dump.git
$ cd prometheus-tsdb-dump
$ make build
go run main.go -bucket <bucket-name> -prefix </path/to/s3-tsdb-block> -local-path </local/folder/to/store/tsdb-block> -block </path/to/local/tsdb-block> > </file-name/of/converted/json.json>
{"metric":{"__name__":"up","job":"node-exporter","instance":"a"},"values":[1,1,1],"timestamps":[1578636058619,1578636118619,1578636178619]}
{"metric":{"__name__":"up","job":"node-exporter","instance":"b"},"values":[1,1,1],"timestamps":[1578636058619,1578636118619,1578636178619]}
-format: Output format (default: victoriametrics)-min-timestamp: Minimum timestamp of exported samples (unix time in msec)-max-timestamp: Maximum timestamp of exported samples (unix time in msec)
Output format can be configured via -format option.
victoriametrics format is a JSON streaming format which can be imported via VictoriaMetrics' /api/v1/import API. It looks like:
{"metric":{"__name__":"up","job":"node-exporter"},"values":[1,1,1],"timestamps":[1578636058619,1578636118619,1578636178619]}
First, take a snapshot of Prometheus TSDB:
$ curl -XPOST http://your-prometheus:9090/api/v1/admin/tsdb/snapshot
{"status":"success","data":{"name":"20200110T104512Z-xxxxxxxxxxxx"}}
Then, import data to VictoriaMetrics:
$ cd /path/to/prometheus/data/snapshots/20200110T104512Z-xxxxxxxxxxxx
$ parallelism="$(nproc)"
$ find . -mindepth 1 -maxdepth 1 -type d | xargs -n1 -P "$parallelism" sh -c 'echo $0; prometheus-tsdb-dump-linux -block "$0" -format victoriametrics | curl http://your-victoriametrics:8428/api/v1/import -T -'