Skip to content

observatorium/api

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Observatorium

CircleCI Go Doc Go Report Card

This project is an API server for Observatorium. The API provides an authenticated and authorized, multi-tenant interface for writing and reading observability signals, i.e. metrics and logs.

Getting started

For a quick start, run:

make test-interactive

This command spins up a full setup for Observatorium API with all dependent services. It is intended for short-lived manual testing on your local environment.

It is also possible to run the test with flags '-v' (to see the output) and '-test.timeout=9999m' to ensure the test is not terminated.

Since this starts all necessary services in a Docker environment, make sure you have Docker installed.

Backends

The Observatorium API server fulfills requests by proxying reads and writes to a backend for each type of observability signal.

Metrics

The Observatorium API server can serve read and write requests for Prometheus metrics. In order to handle requests for metrics, a compatible backend must be configured.

--metrics.read.endpoint

The backend from which to read metrics can be specified with the --metrics.read.endpoint flag. Compatible backends must implement the Prometheus HTTP API, e.g. Prometheus, Thanos querier, Cortex, etc.

--metrics.write.endpoint

The backend to which to write metrics can be specified with the --metrics.write.endpoint flag. Compatible backends must implement the Prometheus remote-write API, e.g. Thanos receiver, Cortex, etc.

--metrics.rules.endpoint

The rules backend to where rules can be stored can be specified with the --metrics.rules.endpoint flag.

Logs

The Observatorium API server can serve read and write requests for logs. In order to handle requests for logs, a compatible backend must be configured.

--logs.read.endpoint

The backend from which to read logs can be specified with the --logs.read.endpoint flag. Compatible backends must implement the Loki read API, e.g. Loki.

--logs.tail.endpoint

The backend from which to tail logs can be specified with the --logs.tail.endpoint flag. Compatible backends must implement the Loki tail API, e.g. Loki.

--logs.write.endpoint

The backend to which to write logs can be specified with the --logs.write.endpoint flag. Compatible backends must implement the Loki write API, e.g. Loki.

Usage

Usage of ./observatorium-api:
  -debug.block-profile-rate int
    	The percentage of goroutine blocking events that are reported in the blocking profile. (default 10)
  -debug.mutex-profile-fraction int
    	The percentage of mutex contention events that are reported in the mutex profile. (default 10)
  -debug.name string
    	A name to add as a prefix to log lines. (default "observatorium")
  -experimental.traces.read.endpoint-template string
    	A template replacing --read.traces.endpoint, such as http://jaeger-{tenant}-query:16686
  -grpc.listen string
    	The address on which the public gRPC server listens.
  -internal.tracing.otlp-http-endpoint string
    	The full URL of OTLP/http endpoint e.g. http://otel-collector:4318. The https:// scheme enables TLS. The certificates and other exporter options can be configured via standard OTEL env variables. If it's not set, tracing will be disabled.
  -internal.tracing.sampling-fraction float
    	The fraction of traces to sample. Thus, if you set this to .5, half of traces will be sampled. (default 0.1)
  -internal.tracing.service-name string
    	The service name to report to the tracing backend. (default "observatorium_api")
  -log.format string
    	The log format to use. Options: 'logfmt', 'json'. (default "logfmt")
  -log.level string
    	The log filtering level. Options: 'error', 'warn', 'info', 'debug'. (default "info")
  -logs.auth.extract-selectors string
    	Comma-separated list of stream selectors that should be extracted from queries and sent to OPA during authorization.
  -logs.read.endpoint string
    	The endpoint against which to make read requests for logs.
  -logs.rules.endpoint string
    	The endpoint against which to make rules requests for logs.
  -logs.rules.label-filters string
    	Allow the following filters to be applied to user rules queries per tenant (e.g. tenantA:namespace,severity;tenantB:severity).
  -logs.rules.read-only
    	Allow only read-only rule requests for logs.
  -logs.rules.tenant-label string
    	The name of the rules label that should hold the tenant ID in logs upstreams. (default "tenant_id")
  -logs.tail.endpoint string
    	The endpoint against which to make tail read requests for logs.
  -logs.tenant-header string
    	The name of the HTTP header containing the tenant ID to forward to the logs upstream. (default "X-Scope-OrgID")
  -logs.tls.ca-file string
    	File containing the TLS CA against which to upstream logs servers. Leave blank to disable TLS.
  -logs.tls.cert-file string
    	File containing the TLS client certificates to authenticate against upstream logs servers. Leave blank to disable mTLS.
  -logs.tls.key-file string
    	File containing the TLS client key to authenticate against upstream logs servers. Leave blank to disable mTLS.
  -logs.tls.watch-certs
    	Watch for certificate changes and reload
  -logs.write-timeout duration
    	The HTTP write timeout for proxied requests to the logs endpoint. (default 10m0s)
  -logs.write.endpoint string
    	The endpoint against which to make write requests for logs.
  -metrics.alertmanager.endpoint string
    	The endpoint against which to make requests for alerts and silences
  -metrics.read.endpoint string
    	The endpoint against which to send read requests for metrics. It used as a fallback to 'query.endpoint' and 'query-range.endpoint'.
  -metrics.rules.endpoint string
    	The endpoint against which to make get requests for listing recording/alerting rules and put requests for creating/updating recording/alerting rules.
  -metrics.tenant-header string
    	The name of the HTTP header containing the tenant ID to forward to the metrics upstreams. (default "THANOS-TENANT")
  -metrics.tenant-label string
    	The name of the PromQL label that should hold the tenant ID in metrics upstreams. (default "tenant_id")
  -metrics.tls.ca-file string
    	File containing the TLS CA against which to upstream metrics servers. Leave blank to disable TLS.
  -metrics.tls.cert-file string
    	File containing the TLS client certificates to authenticate against upstream logs servers. Leave blank to disable mTLS.
  -metrics.tls.key-file string
    	File containing the TLS client key to authenticate against upstream metrics servers. Leave blank to disable mTLS.
  -metrics.tls.watch-certs
    	Watch for certificate changes and reload
  -metrics.write-timeout duration
    	The HTTP write timeout for proxied requests to the metrics endpoint. (default 2m0s)
  -metrics.write.endpoint string
    	The endpoint against which to make write requests for metrics.
  -middleware.backlog-duration-concurrent-requests duration
    	The time duration to buffer up concurrent requests. (default 1ms)
  -middleware.backlog-limit-concurrent-requests int
    	The number of concurrent requests that can buffered.
  -middleware.concurrent-request-limit int
    	The limit that controls the number of concurrently processed requests across all tenants. (default 10000)
  -middleware.rate-limiter.address value
    	The address of the rate limiter. Only used when not using the gRPC nor "local" rate limiters. Can be repeated to specify multiple addresses (i.e. Redis Cluster).
  -middleware.rate-limiter.grpc-address string
    	The gRPC Server Address against which to run rate limit checks when the rate limits are specified for a given tenant. If not specified, local, non-shared rate limiting will be used. Has precedence over other rate limiter options.
  -middleware.rate-limiter.type string
    	The type of rate limiter to use when not using a gRPC rate limiter. Options: 'local' (default), 'redis' (leaky bucket algorithm). (default "local")
  -rbac.config string
    	Path to the RBAC configuration file. (default "rbac.yaml")
  -server.read-header-timeout duration
    	Global server read header timeout. (default 1s)
  -server.read-timeout duration
    	Global server read timeout. (default 5s)
  -server.write-timeout duration
    	Global server read timeout. (default 12m0s)
  -tenants.config string
    	Path to the tenants file. (default "tenants.yaml")
  -tls.cipher-suites string
    	Comma-separated list of cipher suites for the server. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). If omitted, the default Go cipher suites will be used. Note that TLS 1.3 ciphersuites are not configurable.
  -tls.client-auth-type string
    	Policy for TLS client-side authentication. Values are from ClientAuthType constants in https://pkg.go.dev/crypto/tls#ClientAuthType (default "RequestClientCert")
  -tls.healthchecks.server-ca-file string
    	File containing the TLS CA against which to verify servers. If no server CA is specified, the client will use the system certificates.
  -tls.healthchecks.server-name string
    	Server name is used to verify the hostname of the certificates returned by the server. If no server name is specified, the server name will be inferred from the healthcheck URL.
  -tls.internal.server.cert-file string
    	File containing the default x509 Certificate for internal HTTPS. Leave blank to disable TLS.
  -tls.internal.server.key-file string
    	File containing the default x509 private key matching --tls.internal.server.cert-file. Leave blank to disable TLS.
  -tls.max-version string
    	Maximum TLS version supported. Value must match version names from https://golang.org/pkg/crypto/tls/#pkg-constants. (default "VersionTLS13")
  -tls.min-version string
    	Minimum TLS version supported. Value must match version names from https://golang.org/pkg/crypto/tls/#pkg-constants. (default "VersionTLS13")
  -tls.reload-interval duration
    	The interval at which to watch for TLS certificate changes. (default 1m0s)
  -tls.server.cert-file string
    	File containing the default x509 Certificate for HTTPS. Leave blank to disable TLS.
  -tls.server.key-file string
    	File containing the default x509 private key matching --tls.server.cert-file. Leave blank to disable TLS.
  -traces.read.endpoint string
    	The endpoint against which to make HTTP read requests for traces.
  -traces.tempo.endpoint string
    	The endpoint against which to make HTTP read requests for traces using traceQL (tempo API).
  -traces.tenant-header string
    	The name of the HTTP header containing the tenant ID to forward to upstream OpenTelemetry collector. (default "X-Tenant")
  -traces.tls.ca-file string
    	File containing the TLS CA against which to upstream traces servers. Leave blank to disable TLS.
  -traces.tls.cert-file string
    	File containing the TLS client certificates to authenticate against upstream logs servers. Leave blank to disable mTLS.
  -traces.tls.key-file string
    	File containing the TLS client key to authenticate against upstream traces servers. Leave blank to disable mTLS.
  -traces.tls.watch-certs
    	Watch for certificate changes and reload
  -traces.write-timeout duration
    	The HTTP write timeout for proxied requests to the traces endpoint. (default 2m0s)
  -traces.write.otlpgrpc.endpoint string
    	The endpoint against which to make OTLP gRPC write requests for traces.
  -traces.write.otlphttp.endpoint string
    	The endpoint against which to make OTLP HTTP write requests for traces.
  -web.healthchecks.url string
    	The URL against which to run healthchecks. (default "http://localhost:8080")
  -web.internal.listen string
    	The address on which the internal server listens. (default ":8081")
  -web.listen string
    	The address on which the public server listens. (default ":8080")