v0.39.0-rc.0

This is release v0.39.0-rc.0 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

Breaking changes

• otelcol.receiver.prometheus will drop all otel_scope_info metrics when converting them to OTLP. (@wildum)

  • If the otel_scope_info metric has labels otel_scope_name and otel_scope_version,
    their values will be used to set OTLP Instrumentation Scope name and version respectively.
  • Labels of otel_scope_info metrics other than otel_scope_name and otel_scope_version
    are added as scope attributes with the matching name and version.

• The target block in prometheus.exporter.blackbox requires a mandatory name
  argument instead of a block label. (@hainenber)

• In the azure exporter, dimension options will no longer be validated by the Azure API. (@kgeckhart)

  • This change will not break any existing configurations and you can opt in to validation via the validate_dimensions configuration option.
  • Before this change, pulling metrics for azure resources with variable dimensions required one configuration per metric + dimension combination to avoid an error.
  • After this change, you can include all metrics and dimensions in a single configuration and the Azure APIs will only return dimensions which are valid for the various metrics.

Features

• A new discovery.ovhcloud component for discovering scrape targets on OVHcloud. (@ptodev)

Enhancements

• Flow Windows service: Support environment variables. (@jkroepke)

• Allow disabling collection of root Cgroup stats in
  prometheus.exporter.cadvisor (flow mode) and the cadvisor integration
  (static mode). (@hainenber)

• Grafana Agent on Windows now automatically restarts on failure. (@hainenber)

• Added metrics, alerts and dashboard visualisations to help diagnose issues
  with unhealthy components and components that take too long to evaluate. (@thampiotr)

• The http config block may now reference exports from any component.
  Previously, only remote.* and local.* components could be referenced
  without a circular dependency. (@rfratto)

• Add support for Basic Auth-secured connection with Elasticsearch cluster using prometheus.exporter.elasticsearch. (@hainenber)

• Add a resource_to_telemetry_conversion argument to otelcol.exporter.prometheus
  for converting resource attributes to Prometheus labels. (@hainenber)

• pyroscope.ebpf support python on arm64 platforms. (@korniltsev)

• otelcol.receiver.prometheus does not drop histograms without buckets anymore. (@wildum)

• Added exemplars support to otelcol.receiver.prometheus. (@wildum)

• mimir.rules.kubernetes may now retry its startup on failure. (@hainenber)

• Added links between compatible components in the documentation to make it
  easier to discover them. (@thampiotr)

• Allow defining HTTPClientConfig for discovery.ec2. (@cmbrad)

• The remote.http component can optionally define a request body. (@tpaschalis)

• Added support for loki.write to flush WAL on agent shutdown. (@thepalbi)

• Add support for integrations-next static to flow config conversion. (@erikbaranowski)

• Add support for passing extra arguments to the static converter such as -config.expand-env. (@erikbaranowski)

• Added 'country' mmdb-type to log pipeline-stage geoip. (@superstes)

• Azure exporter enhancements for flow and static mode, (@kgeckhart)

  • Allows for pulling metrics at the Azure subscription level instead of resource by resource
  • Disable dimension validation by default to reduce the number of exporter instances needed for full dimension coverage

• Add max_cache_size to prometheus.relabel to allow configurability instead of hard coded 100,000. (@mattdurham)

• Add support for http_sd_config within a scrape_config for prometheus to flow config conversion. (@erikbaranowski)

• discovery.lightsail now supports additional parameters for configuring HTTP client settings. (@ptodev)

• Add sample_age_limit to remote_write config to drop samples older than a specified duration. (@marctc)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.38.1

This is release v0.38.1 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

Security fixes

• Fix CVE-2023-47108 by updating otelgrpc from v0.45.0 to v0.46.0. (@hainenber)

Features

• Agent Management: Introduce support for templated configuration. (@jcreixell)

Bugfixes

• Permit X-Faro-Session-ID header in CORS requests for the faro.receiver
  component (flow mode) and the app_agent_receiver integration (static mode).
  (@cedricziel)

• Fix issue with windows_exporter defaults not being set correctly. (@mattdurham)

• Fix agent crash when process null OTel's fan out consumers. (@hainenber)

• Fix issue in prometheus.operator.* where targets would be dropped if two crds share a common prefix in their names. (@Paul424, @captncraig)

• Fix issue where convert command would generate incorrect Flow Mode config
  when provided promtail configuration that uses docker_sd_configs (@thampiotr)

• Fix converter issue with loki.relabel and max_cache_size being set to 0 instead of default (10_000). (@mattdurham)

Other changes

• Add Agent Deploy Mode to usage report. (@captncraig)

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.38.0

This is release v0.38.0 of Grafana Agent.

v0.38.0 is a huge release, with many new features, improvements, and bugfixes. This page only covers a very small portion of those changes. For a full list of changes, refer to our CHANGELOG.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

Breaking changes

• The deprecated otelcol.exporter.jaeger component has been removed as scheduled in v0.36.
• Some metrics for the mysqld exporter integration ahve been removed or renamed.

Features

• New flow mode components:
  • otelcol.processor.filter filters OTLP telemetry data using OpenTelemetry Transformation Language (OTTL).
  • otelcol.receiver.vcenter receives metrics telemetry data from vCenter.
• Changes to River:
  • River now supports "raw strings" to avoid needing to escape special characters.
  • River now permits using [] to access non-existent keys in an object, rather than generating an error.

Enhancements

• Improvements to log collection in flow mode:
  • Improve detection of rolled log files in loki.source.kubernetes and loki.source.podlogs.
  • Clustering is now supported in loki.source.kubernetes and loki.source.podlogs.
  • Added support for replaying unsent data for loki.write when WAL is enabled.
• Improvements to the flow controller:
  • Improved resilience of graph evaluation in presence of slow components.

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.38.0-rc.1

This is release v0.38.0-rc.1 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

The following changes are included on top of v0.38.0-rc.0:

Enhancements

• Allow agent to start with module.git config if cached before. (@hainenber)

Bugfixes

• Fix default configuration file grafana-agent-flow.river used in downstream packages. (@bricewge)

• Fix converter output for prometheus.exporter.windows to not unnecessarily add empty blocks. (@erikbaranowski)

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.38.0-rc.0

This is release v0.38.0-rc.0 of Grafana Agent.

v0.38.0-rc.0 is a huge release, with many new features, improvements, and bugfixes. This page only covers a very small portion of those changes. For a full list of changes, refer to our CHANGELOG.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes

Breaking changes

• The deprecated otelcol.exporter.jaeger component has been removed as scheduled in v0.36.
• Some metrics for the mysqld exporter integration ahve been removed or renamed.

Features

• New flow mode components:
  • otelcol.processor.filter filters OTLP telemetry data using OpenTelemetry Transformation Language (OTTL).
  • otelcol.receiver.vcenter receives metrics telemetry data from vCenter.
• Changes to River:
  • River now supports "raw strings" to avoid needing to escape special characters.
  • River now permits using [] to access non-existent keys in an object, rather than generating an error.

Enhancements

• Improvements to log collection in flow mode:
  • Improve detection of rolled log files in loki.source.kubernetes and loki.source.podlogs.
  • Clustering is now supported in loki.source.kubernetes and loki.source.podlogs.
  • Added support for replaying unsent data for loki.write when WAL is enabled.
• Improvements to the flow controller:
  • Improved resilience of graph evaluation in presence of slow components.

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.37.4

This is release v0.37.4 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

This patch release has two small changes

• Added an add_metric_suffixes option to otelcol.exporter.prometheus in flow mode, which configures whether to add type and unit suffixes to metrics names. (@mar4uk)

• Fixes a bug where reloading the configuration of a loki.write component lead to a panic. (@tpaschalis)

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.37.3

This is release v0.37.3 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

This patch release fixes a couple of bugs with the Agent's handing of Prometheus Native Histograms, as well as improving small issues with the configuration converters.

It also fixes an issue with the remote.vault component ignoring the namespace argument, and performs routine updates to some of our dependencies.

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.37.2

This is release v0.37.2 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

v0.37.2 contains a few bugfixes and enhancements.

Bugfixes

• Fix the handling of the --cluster.join-addresses flag causing an invalid
  comparison with the mutually-exclusive --cluster.discover-peers. (@tpaschalis)

• Fix an issue with the static to flow converter for blackbox exporter modules
  config not being included in the river output. (@erikbaranowski)

Enhancements

• Update Prometheus dependency to v2.47.2. (@tpaschalis)

• Allow Out of Order writing to the WAL for metrics. (@mattdurham)

Other changes

• Use Go 1.21.3 for builds. (@tpaschalis)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.37.1

This is release v0.37.1 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

NOTE: v0.37.1 fixes a bug that caused a panic when using the Operator

The biggest news is that Flow mode has now feature parity with Static mode! That means that you can use Flow components for all static mode use cases.

The v0.37 release contains some breaking changes:

Breaking changes

• Set retry_on_http_429 to true by default in the queue_config block in static mode's remote_write. (@wildum)
• Renamed non_indexed_labels Loki processing stage to structured_metadata. (@vlad-diachenko)
• Include otel_scope_name and otel_scope_version in all metrics for otelcol.exporter.prometheus
  by default using a new argument include_scope_labels. (@erikbaranowski)
• Static mode Windows Certificate Filter no longer restricted to TLS 1.2 and specific cipher suites. (@mattdurham)
• The __meta_agent_integration* and __meta_agent_hostname labels have been
  removed from the targets exposed by prometheus.exporter.* components and
  got replaced by the pair of __meta_component_name and __meta_component_id
  labels. (@tpaschalis)
• Flow: Allow prometheus.exporter.unix to be specified multiple times and used in modules. This now means all
  prometheus.exporter.unix references will need a label prometheus.exporter.unix "example". (@mattdurham)

Also, it includes a number of new features, including 23 (!) new Flow components, improved evaluation of the Flow component graph, new config converters as well as the ability to run multiple River files at once.

Features

• New Grafana Agent Flow components:

  • discovery.consulagent discovers scrape targets from Consul Agent. (@wildum)
  • discovery.kuma discovers scrape targets from the Kuma control plane. (@tpaschalis)
  • discovery.linode discovers scrape targets from the Linode API. (@captncraig)
  • discovery.marathon discovers scrape targets from Marathon servers. (@wildum)
  • discovery.ionos discovers scrape targets from the IONOS Cloud API. (@wildum)
  • discovery.triton discovers scrape targets from Triton Container Monitor. (@erikbaranowski)
  • discovery.nerve discovers scrape targets from AirBnB's Nerve. (@tpaschalis)
  • discovery.serverset discovers Serversets stored in Zookeeper. (@thampiotr)
  • discovery.scaleway discovers scrape targets from Scaleway virtual
    instances and bare-metal machines. (@rfratto)
  • faro.receiver accepts Grafana Faro-formatted telemetry data over the
    network and forwards it to other components. (@megumish, @rfratto)
  • prometheus.exporter.azure collects metrics from Azure. (@wildum)
  • discovery.dockerswarm discovers scrape targets from Docker Swarm. (@wildum)
  • otelcol.connector.servicegraph creates service graph metrics from spans. It is the
    flow mode equivalent to static mode's service_graphs processor. (@ptodev)
  • otelcol.connector.spanlogs creates logs from spans. It is the flow mode equivalent
    to static mode's automatic_logging processor. (@ptodev)
  • otelcol.processor.k8sattributes adds Kubernetes metadata as resource attributes
    to spans, logs, and metrics. (@acr92)
  • otelcol.processor.probabilistic_sampler samples logs and traces based on configuration options. (@mar4uk)
  • otelcol.processor.transform transforms OTLP telemetry data using the
    OpenTelemetry Transformation Language (OTTL). It is most commonly used
    for transformations on attributes.
  • remote.kubernetes.configmap loads a configmap's data for use in other components (@captncraig)
  • remote.kubernetes.secret loads a secret's data for use in other components (@captncraig)
  • prometheus.exporter.agent exposes the agent's internal metrics. (@hainenber)
  • prometheus.exporter.azure collects metrics from Azure. (@wildum)
  • prometheus.exporter.cadvisor exposes cAdvisor metrics. (@tpaschalis)
  • prometheus.exporter.vsphere exposes vmware vsphere metrics. (@marctc)

• Flow: allow the HTTP server to be configured with TLS in the config file
  using the new http config block. (@rfratto)

• Clustering: add new flag --cluster.max-join-peers to limit the number of peers the system joins. (@wildum)

• Clustering: add a new flag --cluster.name to prevent nodes without this identifier from joining the cluster. (@wildum)

• Clustering: add IPv6 support when using advertise interfaces to assign IP addresses. (@wildum)

• Add a file_watch block in loki.source.file to configure how often to poll files from disk for changes via min_poll_frequency and max_poll_frequency.
  In static mode it can be configured in the global file_watch_config via min_poll_frequency and max_poll_frequency. (@wildum)

• Flow: In prometheus.exporter.blackbox, allow setting labels for individual targets. (@spartan0x117)

• Add optional nil_to_zero config flag for YACE which can be set in the static, discovery, or metric config blocks. (@berler)

• The cri stage in loki.process can now be configured to limit line size.

• Flow: Allow grafana-agent run to accept a path to a directory of *.river files.
  This will load all River files in the directory as a single configuration;
  component names must be unique across all loaded files. (@rfratto, @hainenber)

• Added support for static configuration conversion in grafana-agent convert and grafana-agent run commands. (@erikbaranowski)

• Flow: the prometheus.scrape component can now configure the scraping of
  Prometheus native histograms. (@tpaschalis)

• Flow: the prometheus.remote_write component now supports SigV4 and AzureAD authentication. (@ptodev)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.37.0

This is release v0.37.0 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

The biggest news is that Flow mode has now feature parity with Static mode! That means that you can use Flow components for all static mode use cases.

The v0.37.0 release contains some breaking changes:

Breaking changes

• Set retry_on_http_429 to true by default in the queue_config block in static mode's remote_write. (@wildum)
• Renamed non_indexed_labels Loki processing stage to structured_metadata. (@vlad-diachenko)
• Include otel_scope_name and otel_scope_version in all metrics for otelcol.exporter.prometheus
  by default using a new argument include_scope_labels. (@erikbaranowski)
• Static mode Windows Certificate Filter no longer restricted to TLS 1.2 and specific cipher suites. (@mattdurham)
• The __meta_agent_integration* and __meta_agent_hostname labels have been
  removed from the targets exposed by prometheus.exporter.* components and
  got replaced by the pair of __meta_component_name and __meta_component_id
  labels. (@tpaschalis)
• Flow: Allow prometheus.exporter.unix to be specified multiple times and used in modules. This now means all
  prometheus.exporter.unix references will need a label prometheus.exporter.unix "example". (@mattdurham)

Also, it includes a number of new features, including 23 (!) new Flow components, improved evaluation of the Flow component graph, new config converters as well as the ability to run multiple River files at once.

Features

• New Grafana Agent Flow components:

  • discovery.consulagent discovers scrape targets from Consul Agent. (@wildum)
  • discovery.kuma discovers scrape targets from the Kuma control plane. (@tpaschalis)
  • discovery.linode discovers scrape targets from the Linode API. (@captncraig)
  • discovery.marathon discovers scrape targets from Marathon servers. (@wildum)
  • discovery.ionos discovers scrape targets from the IONOS Cloud API. (@wildum)
  • discovery.triton discovers scrape targets from Triton Container Monitor. (@erikbaranowski)
  • discovery.nerve discovers scrape targets from AirBnB's Nerve. (@tpaschalis)
  • discovery.serverset discovers Serversets stored in Zookeeper. (@thampiotr)
  • discovery.scaleway discovers scrape targets from Scaleway virtual
    instances and bare-metal machines. (@rfratto)
  • faro.receiver accepts Grafana Faro-formatted telemetry data over the
    network and forwards it to other components. (@megumish, @rfratto)
  • prometheus.exporter.azure collects metrics from Azure. (@wildum)
  • discovery.dockerswarm discovers scrape targets from Docker Swarm. (@wildum)
  • otelcol.connector.servicegraph creates service graph metrics from spans. It is the
    flow mode equivalent to static mode's service_graphs processor. (@ptodev)
  • otelcol.connector.spanlogs creates logs from spans. It is the flow mode equivalent
    to static mode's automatic_logging processor. (@ptodev)
  • otelcol.processor.k8sattributes adds Kubernetes metadata as resource attributes
    to spans, logs, and metrics. (@acr92)
  • otelcol.processor.probabilistic_sampler samples logs and traces based on configuration options. (@mar4uk)
  • otelcol.processor.transform transforms OTLP telemetry data using the
    OpenTelemetry Transformation Language (OTTL). It is most commonly used
    for transformations on attributes.
  • remote.kubernetes.configmap loads a configmap's data for use in other components (@captncraig)
  • remote.kubernetes.secret loads a secret's data for use in other components (@captncraig)
  • prometheus.exporter.agent exposes the agent's internal metrics. (@hainenber)
  • prometheus.exporter.azure collects metrics from Azure. (@wildum)
  • prometheus.exporter.cadvisor exposes cAdvisor metrics. (@tpaschalis)
  • prometheus.exporter.vsphere exposes vmware vsphere metrics. (@marctc)

• Flow: allow the HTTP server to be configured with TLS in the config file
  using the new http config block. (@rfratto)

• Clustering: add new flag --cluster.max-join-peers to limit the number of peers the system joins. (@wildum)

• Clustering: add a new flag --cluster.name to prevent nodes without this identifier from joining the cluster. (@wildum)

• Clustering: add IPv6 support when using advertise interfaces to assign IP addresses. (@wildum)

• Add a file_watch block in loki.source.file to configure how often to poll files from disk for changes via min_poll_frequency and max_poll_frequency.
  In static mode it can be configured in the global file_watch_config via min_poll_frequency and max_poll_frequency. (@wildum)

• Flow: In prometheus.exporter.blackbox, allow setting labels for individual targets. (@spartan0x117)

• Add optional nil_to_zero config flag for YACE which can be set in the static, discovery, or metric config blocks. (@berler)

• The cri stage in loki.process can now be configured to limit line size.

• Flow: Allow grafana-agent run to accept a path to a directory of *.river files.
  This will load all River files in the directory as a single configuration;
  component names must be unique across all loaded files. (@rfratto, @hainenber)

• Added support for static configuration conversion in grafana-agent convert and grafana-agent run commands. (@erikbaranowski)

• Flow: the prometheus.scrape component can now configure the scraping of
  Prometheus native histograms. (@tpaschalis)

• Flow: the prometheus.remote_write component now supports SigV4 and AzureAD authentication. (@ptodev)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode