Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature: Add option to exclude consumer from Stream metrics #906

Merged
merged 6 commits into from
May 30, 2024
Merged

Feature: Add option to exclude consumer from Stream metrics #906

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a78079c
Feature: Add option to exclude consumer from Stream metrics
nantiferov May 18, 2024
1b2825c
Clarify exclude consumer metrics from streams
nantiferov May 20, 2024
d97b9bb
Merge branch 'oliver006:master' into feature-exclude-consumer-stream
nantiferov May 21, 2024
60ca0d6
Add TestStreamsExtractStreamMetricsExcludeConsumer to cover new option
nantiferov May 21, 2024
7ee9fed
Update Readme for new streams-exclude-consumer-metrics option
nantiferov May 25, 2024
cec04c8
Add check for excluded metrics to TestStreamsExtractStreamMetricsExcl…
nantiferov May 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ Prometheus uses file watches and all changes to the json file are applied immedi
| check-single-keys | REDIS_EXPORTER_CHECK_SINGLE_KEYS | Comma separated list of keys to export value and length/size, eg: `db3=user_count` will export key `user_count` from db `3`. db defaults to `0` if omitted. The keys specified with this flag will be looked up directly without any glob pattern matching. Use this option if you don't need glob pattern matching; it is faster than `check-keys`. |
| check-streams | REDIS_EXPORTER_CHECK_STREAMS | Comma separated list of stream-patterns to export info about streams, groups and consumers. Syntax is the same as `check-keys`. |
| check-single-streams | REDIS_EXPORTER_CHECK_SINGLE_STREAMS | Comma separated list of streams to export info about streams, groups and consumers. The streams specified with this flag will be looked up directly without any glob pattern matching. Use this option if you don't need glob pattern matching; it is faster than `check-streams`. |
| streams-exclude-consumer | REDIS_EXPORTER_STREAMS_EXCLUDE_CONSUMER | Don't collect per consumer metrics for streams (decreases amount of metrics and cardinality). |
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REDIS_EXPORTER_STREAMS_EXCLUDE_CONSUMER -> REDIS_EXPORTER_STREAMS_EXCLUDE_CONSUMER_METRICS ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, sure.

My only concern was that it would be quite long, but not too much I think)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, committed changes.

Copy link
Contributor Author

@nantiferov nantiferov May 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently tests are failing, but I don't see exact reason.

In the end of output only:

FAIL
coverage: 90.0% of statements
FAIL	github.com/oliver006/redis_exporter/exporter	22.273s
FAIL

And before something looks like errors, but nothing says FAIL

=== RUN   TestParseKeyArg/invalid_args_with_args_separator_skipped
time="2024-05-20T17:54:09Z" level=error msg="parseKeyArg(): Empty value parsed in pair '=', skip"
=== RUN   TestParseKeyArg/empty_args_with_comma_separators_skipped
=== RUN   TestParseKeyArg/multiple_invalid_args_skipped
time="2024-05-20T17:54:09Z" level=error msg="parseKeyArg(): Empty value parsed in pair '=', skip"
time="2024-05-20T17:54:09Z" level=error msg="parseKeyArg(): Empty value parsed in pair '=', skip"
=== RUN   TestParseKeyArg/empty_key_with_args_separator_skipped
time="2024-05-20T17:54:09Z" level=error msg="parseKeyArg(): Empty value parsed in pair '0=', skip"
=== RUN   TestParseKeyArg/empty_database_with_args_separator_skipped
time="2024-05-20T17:54:09Z" level=error msg="parseKeyArg(): Empty value parsed in pair '=my-key', skip"
=== RUN   TestParseKeyArg/string_database_index
    keys_test.go:265: Expected failure for test: string_database_index, got err: Invalid database index for db "wrong": strconv.Atoi: parsing "wrong": invalid syntax
=== RUN   TestParseKeyArg/prefixed_string_database_index
    keys_test.go:265: Expected failure for test: prefixed_string_database_index, got err: Invalid database index for db "wrong": strconv.Atoi: parsing "wrong": invalid syntax
=== RUN   TestParseKeyArg/wrong_args_count
    keys_test.go:265: Expected failure for test: wrong_args_count, got err: invalid key list argument: wrong=wrong=wrong
=== RUN   TestParseKeyArg/wrong_args
    keys_test.go:265: Expected failure for test: wrong_args, got err: invalid key list argument: wrong=wrong=1
=== RUN   TestParseKeyArg/negative_database_index
    keys_test.go:265: Expected failure for test: negative_database_index, got err: Invalid database index for db "-1": %!s(<nil>)

| check-keys-batch-size | REDIS_EXPORTER_CHECK_KEYS_BATCH_SIZE | Approximate number of keys to process in each execution. This is basically the COUNT option that will be passed into the SCAN command as part of the execution of the key or key group metrics, see [COUNT option](https://redis.io/commands/scan#the-count-option). Larger value speeds up scanning. Still Redis is a single-threaded app, huge `COUNT` can affect production environment. |
| count-keys | REDIS_EXPORTER_COUNT_KEYS | Comma separated list of patterns to count, eg: `db3=sessions:*` will count all keys with prefix `sessions:` from db `3`. db defaults to `0` if omitted. Warning: The exporter runs SCAN to count the keys. This might not perform well on large databases. |
| script | REDIS_EXPORTER_SCRIPT | Comma separated list of path(s) to Redis Lua script(s) for gathering extra metrics. |
Expand Down
7 changes: 4 additions & 3 deletions exporter/exporter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ type Options struct {
CheckSingleKeys string
CheckStreams string
CheckSingleStreams string
StreamsExcludeConsumer bool
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, add Metrics ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

CheckKeysBatchSize int64
CheckKeyGroups string
MaxDistinctKeyGroups int64
Expand Down Expand Up @@ -189,10 +190,10 @@ func NewRedisExporter(redisURI string, opts Options) (*Exporter, error) {
"rdb_last_bgsave_status": "rdb_last_bgsave_status",
"rdb_last_bgsave_time_sec": "rdb_last_bgsave_duration_sec",
"rdb_current_bgsave_time_sec": "rdb_current_bgsave_duration_sec",
"rdb_saves": "rdb_saves_total",
"rdb_saves": "rdb_saves_total",
"rdb_last_cow_size": "rdb_last_cow_size_bytes",
"rdb_last_load_keys_expired": "rdb_last_load_expired_keys",
"rdb_last_load_keys_loaded": "rdb_last_load_loaded_keys",
"rdb_last_load_keys_expired": "rdb_last_load_expired_keys",
"rdb_last_load_keys_loaded": "rdb_last_load_loaded_keys",
"aof_enabled": "aof_enabled",
"aof_rewrite_in_progress": "aof_rewrite_in_progress",
"aof_rewrite_scheduled": "aof_rewrite_scheduled",
Expand Down
8 changes: 5 additions & 3 deletions exporter/streams.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,9 +205,11 @@ func (e *Exporter) extractStreamMetrics(ch chan<- prometheus.Metric, c redis.Con
e.registerConstMetricGauge(ch, "stream_group_last_delivered_id", parseStreamItemId(g.LastDeliveredId), dbLabel, k.key, g.Name)
e.registerConstMetricGauge(ch, "stream_group_entries_read", float64(g.EntriesRead), dbLabel, k.key, g.Name)
e.registerConstMetricGauge(ch, "stream_group_lag", float64(g.Lag), dbLabel, k.key, g.Name)
for _, c := range g.StreamGroupConsumersInfo {
e.registerConstMetricGauge(ch, "stream_group_consumer_messages_pending", float64(c.Pending), dbLabel, k.key, g.Name, c.Name)
e.registerConstMetricGauge(ch, "stream_group_consumer_idle_seconds", float64(c.Idle)/1e3, dbLabel, k.key, g.Name, c.Name)
if !e.options.StreamsExcludeConsumer {
for _, c := range g.StreamGroupConsumersInfo {
e.registerConstMetricGauge(ch, "stream_group_consumer_messages_pending", float64(c.Pending), dbLabel, k.key, g.Name, c.Name)
e.registerConstMetricGauge(ch, "stream_group_consumer_idle_seconds", float64(c.Idle)/1e3, dbLabel, k.key, g.Name, c.Name)
}
}
}
}
Expand Down
2 changes: 2 additions & 0 deletions main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ func main() {
checkKeyGroups = flag.String("check-key-groups", getEnv("REDIS_EXPORTER_CHECK_KEY_GROUPS", ""), "Comma separated list of lua regex for grouping keys")
checkStreams = flag.String("check-streams", getEnv("REDIS_EXPORTER_CHECK_STREAMS", ""), "Comma separated list of stream-patterns to export info about streams, groups and consumers, searched for with SCAN")
checkSingleStreams = flag.String("check-single-streams", getEnv("REDIS_EXPORTER_CHECK_SINGLE_STREAMS", ""), "Comma separated list of single streams to export info about streams, groups and consumers")
streamsExcludeConsumer = flag.Bool("streams-exclude-consumer", getEnvBool("REDIS_EXPORTER_STREAMS_EXCLUDE_CONSUMER", false), "Don't collect per consumer metrics for streams (decreases cardinality)")
countKeys = flag.String("count-keys", getEnv("REDIS_EXPORTER_COUNT_KEYS", ""), "Comma separated list of patterns to count (eg: 'db0=production_*,db3=sessions:*'), searched for with SCAN")
checkKeysBatchSize = flag.Int64("check-keys-batch-size", getEnvInt64("REDIS_EXPORTER_CHECK_KEYS_BATCH_SIZE", 1000), "Approximate number of keys to process in each execution, larger value speeds up scanning.\nWARNING: Still Redis is a single-threaded app, huge COUNT can affect production environment.")
scriptPath = flag.String("script", getEnv("REDIS_EXPORTER_SCRIPT", ""), "Comma separated list of path(s) to Redis Lua script(s) for gathering extra metrics")
Expand Down Expand Up @@ -169,6 +170,7 @@ func main() {
MaxDistinctKeyGroups: *maxDistinctKeyGroups,
CheckStreams: *checkStreams,
CheckSingleStreams: *checkSingleStreams,
StreamsExcludeConsumer: *streamsExcludeConsumer,
CountKeys: *countKeys,
LuaScript: ls,
InclSystemMetrics: *inclSystemMetrics,
Expand Down
Loading