Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Indices filter flag #574

Closed
wants to merge 50 commits into from
Closed

Add Indices filter flag #574

wants to merge 50 commits into from

Conversation

chapost1
Copy link

As can be seen earlier, certain people asked for the option to filter the indices they are exporting.

Especially if you have a lot of them and you need to monitor just certain, the rest of it, is just a waste of resources.

ATM, fetching indices settings/stats/mappings queries all the existing ones using this kind of GET requests:
GET /_all/_settings, GET /_all/_stats and GET /_all/_mappings

As mentioned in the docs:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-settings.html

It's possible to just retrieve the ones which match 2 types of queries:

  1. Matching using wildcard(*) expression. i.e:
    GET /prefix_*/_stats

  2. Matching a list of indices, using comma:
    GET /first_name,second_name/_stats

It is possible to mix them both. ie:
GET /prefix_1_*,prefix_2_*/_stats

The new added flag:
es.indices_filter (default is '_all')

Note: The flag will be used only if any export indices flag is used.
To use it, pass an expression to match your use-case indices, default '_all' just like before.
By passing one of the options mentioned above as a value (statement with a wildcard, list with commas), it will be used in the http request to retrieve statistics.

This change would be very appreciated,
Thanks.

@shani-e
Copy link

shani-e commented Jul 31, 2023

Any update about this one?

@chapost1
Copy link
Author

chapost1 commented Aug 2, 2023

I've fixed the merge conflicts, but I'm failing to pass the DCO due to invalid signature.
Is it possible to work it out?

@sysadmind
Copy link
Contributor

The details button on the DCO should help you to resolve. My suggestion is to rebase onto the master branch, squashing all of your commits into a single commit. This makes it easy to know that there's only one commit to sign.

bobo333 and others added 26 commits August 17, 2023 19:51
@bobo333 @chapost1
Add support for adding aliases to index and shard stats (#563)
fff73a6 
* Add support for adding aliases to index and shard stats

Optionally will include aliases in a label to index and shard stats

Signed-off-by: Steven Cipriano <[email protected]>

* move aliases to separate informational metrics

Signed-off-by: Steven Cipriano <[email protected]>

* address review

- switch boolean flag
- fix linting errors

Signed-off-by: Steven Cipriano <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@chapost1
added indices filter flag and using it where appropriate to customize…
be1bacd 
… queries

Signed-off-by: Shahar Tal <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@dependabot @chapost1
Bump github.com/go-kit/log from 0.2.0 to 0.2.1 (#576)
cdb04b8 
Bumps [github.com/go-kit/log](https://github.com/go-kit/log) from 0.2.0 to 0.2.1.
- [Release notes](https://github.com/go-kit/log/releases)
- [Commits](go-kit/log@v0.2.0...v0.2.1)

---
updated-dependencies:
- dependency-name: github.com/go-kit/log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <[email protected]>
@prombot @chapost1
Update common Prometheus files (#583)
a4fe57c 
Signed-off-by: prombot <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@prombot @chapost1
Update common Prometheus files (#586)
2d8d30e 
Signed-off-by: prombot <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@thbkrkr @chapost1
Fixes the case of the word Elasticsearch (#588)
d45b68f 
Signed-off-by: Thibault Richard <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@dependabot @chapost1
Bump github.com/prometheus/common from 0.34.0 to 0.35.0 (#598)
4b97dcb 
Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.34.0 to 0.35.0.
- [Release notes](https://github.com/prometheus/common/releases)
- [Commits](prometheus/common@v0.34.0...v0.35.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/common
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <[email protected]>
@dependabot @chapost1
Bump github.com/imdario/mergo from 0.3.12 to 0.3.13 (#577)
b6b874e 
Bumps [github.com/imdario/mergo](https://github.com/imdario/mergo) from 0.3.12 to 0.3.13.
- [Release notes](https://github.com/imdario/mergo/releases)
- [Commits](darccio/mergo@0.3.12...v0.3.13)

---
updated-dependencies:
- dependency-name: github.com/imdario/mergo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <[email protected]>
@dependabot @chapost1
Bump github.com/prometheus/client_golang from 1.12.1 to 1.12.2 (#578)
26a7054 
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.1 to 1.12.2.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.12.1...v1.12.2)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <[email protected]>
@sysadmind @chapost1
Update changelog and version for 1.4.0 (#593)
333dd7c 
Signed-off-by: Joe Adams <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@rosspatil @chapost1
Support for AWS Elasticsearch using AWS SDK v2 (#597)
8eaa8ed 
* Support for AWS Elasticsearch using AWS SDK v2 (#1)

Signed-off-by: Roshan Patil <[email protected]>

* Fixes PR comments

Signed-off-by: Roshan Patil <[email protected]>

* Resolved PR comments

Signed-off-by: Roshan Patil <[email protected]>

* Fix PR comments

Signed-off-by: Roshan Patil <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@Evesy @chapost1
Add metrics collection for data stream statistics (#592)
b891075 
Signed-off-by: Mike Eves <[email protected]>
@akazs @chapost1
Fix cluster settings collection when max_shards_per_node is manually …
9637554 
…set (#603)

Signed-off-by: akazs <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@SuperQ @chapost1
Update semver package (#606)
b97e27b 
Use semver/v4 for better Go module compatibility.
* Update all Go modules.

Signed-off-by: SuperQ <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@SuperQ @chapost1
Release 1.5.0 (#607)
6b723df 
* [FEATURE] Add metrics collection for data stream statistics #592
* [FEATURE] Support for AWS Elasticsearch using AWS SDK v2 #597
* [BUGFIX] Fix cluster settings collection when max_shards_per_node is manually set. #603

Signed-off-by: SuperQ <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@rosspatil @chapost1
Update Readme for AWS region (#613)
572b6c3 
Signed-off-by: Roshan Patil <[email protected]>

Signed-off-by: Roshan Patil <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@Sijoma @chapost1
Update to Go 1.19 (#614)
b93fd23 
Signed-off-by: Simon Zengerling <[email protected]>

Signed-off-by: Simon Zengerling <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@yuriykis @sysadmind @chapost1
Add https to metrics server (#615)
d3439d8 
* Add https to metrics server

Signed-off-by: Yuriy Kis <[email protected]>

* Delete tls-config.yml

Signed-off-by: Yuriy Kis <[email protected]>

* Apply suggestions

Co-authored-by: Joe Adams <[email protected]>
Signed-off-by: Yuriy Kis <[email protected]>

* Add webflag import

Signed-off-by: Yuriy Kis <[email protected]>

Signed-off-by: Yuriy Kis <[email protected]>
Signed-off-by: Yuriy Kis <[email protected]>
Co-authored-by: Joe Adams <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@prombot @chapost1
Update common Prometheus files (#616)
90b12e3 
Signed-off-by: prombot <[email protected]>

Signed-off-by: prombot <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@dependabot @chapost1
Bump github.com/aws/aws-sdk-go-v2/config from 1.15.14 to 1.17.8 (#631)
f744447 
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.15.14 to 1.17.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@config/v1.15.14...config/v1.17.8)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <[email protected]>
@abhilashmandaliya @chapost1
Fixed typo in README.md (#625)
c4af068 
Signed-off-by: Abhilash Mandaliya <[email protected]>

Signed-off-by: Abhilash Mandaliya <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@prombot @chapost1
Update common Prometheus files (#632)
2f1981a 
Signed-off-by: prombot <[email protected]>

Signed-off-by: prombot <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@mac2000 @SuperQ
@sysadmind @chapost1
number_of_replicas (#483)
593c5ac 
* number_of_replicas

Signed-off-by: Alexandr Marchenko <[email protected]>

* simplify metric name by removing redundant number_of_ prefix

Co-authored-by: Ben Kochie <[email protected]>
Signed-off-by: Alexandr Marchenko <[email protected]>

* Update README.md

Thanks indeed

Co-authored-by: Joe Adams <[email protected]>
Signed-off-by: Alexandr Marchenko <[email protected]>

Signed-off-by: Alexandr Marchenko <[email protected]>
Co-authored-by: Ben Kochie <[email protected]>
Co-authored-by: Joe Adams <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@dependabot @chapost1
Bump github.com/aws/aws-sdk-go-v2/config from 1.17.8 to 1.18.7 (#670)
564d79a 
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.17.8 to 1.18.7.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@config/v1.17.8...config/v1.18.7)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <[email protected]>
@iishabakaev @chapost1
feat(*): add ilm metrics (#513)
957bc53 
* feat(*): add ilm metrics

Signed-off-by: iishabakaev <[email protected]>

* Refactor ilm collectors

Signed-off-by: Шабакаев Илья Исмаилович <[email protected]>
Signed-off-by: Шабакаев Илья Исмаилович <[email protected]>

* fix lint checks

Signed-off-by: Шабакаев Илья Исмаилович <[email protected]>

Signed-off-by: iishabakaev <[email protected]>
Signed-off-by: Шабакаев Илья Исмаилович <[email protected]>
Signed-off-by: Шабакаев Илья Исмаилович <[email protected]>
Co-authored-by: Шабакаев Илья Исмаилович <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@hartfordfive @chapost1
Fix index field count (#675)
3500035 
* Fixing field counting logic in countFieldsRecursive function for the indices_mappings feature

Signed-off-by: Alain Lefebvre <[email protected]>

* Added test case to validate counting of fields in index mapping

Signed-off-by: Alain Lefebvre <[email protected]>

* Updated conditional statement to exclude counting field when the type field is set to object as it will be counted in the recursion case

Signed-off-by: Alain Lefebvre <[email protected]>

* Reverting version change

Signed-off-by: Alain Lefebvre <[email protected]>

Signed-off-by: Alain Lefebvre <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
prombot and others added 24 commits August 17, 2023 19:53
@prombot @chapost1
Update common Prometheus files (#668)
917d973 
Signed-off-by: prombot <[email protected]>

Signed-off-by: prombot <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@steveteuber @chburmeister @chapost1
Add support to assume an AWS role and renew expired credentials (#653)
8cb13f4 
Co-authored-by: Christoph Burmeister <[email protected]>
Signed-off-by: Steve Teuber <[email protected]>

Signed-off-by: Steve Teuber <[email protected]>
Co-authored-by: Christoph Burmeister <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@dependabot @chapost1
Bump github.com/aws/aws-sdk-go-v2/credentials from 1.13.7 to 1.13.10 (#…
7048c11 
…679)

Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.13.7 to 1.13.10.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@service/mq/v1.13.7...service/mq/v1.13.10)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <[email protected]>
@dependabot @chapost1
Bump github.com/aws/aws-sdk-go-v2/service/sts from 1.18.2 to 1.18.3 (#…
531450e 
…682)

Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) from 1.18.2 to 1.18.3.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@config/v1.18.2...config/v1.18.3)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <[email protected]>
@dependabot @chapost1
Bump github.com/prometheus/common from 0.37.0 to 0.40.0 (#687)
ebdfda2 
Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.37.0 to 0.40.0.
- [Release notes](https://github.com/prometheus/common/releases)
- [Commits](prometheus/common@v0.37.0...v0.40.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/common
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <[email protected]>
@dependabot @chapost1
Bump golang.org/x/net from 0.4.0 to 0.7.0 (#688)
c201f4b 
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](golang/net@v0.4.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <[email protected]>
@dependabot @chapost1
Bump golang.org/x/crypto from 0.0.0-20210616213533-5ff15b29337e to 0.…
c98c504 
…1.0 (#689)

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.0.0-20210616213533-5ff15b29337e to 0.1.0.
- [Release notes](https://github.com/golang/crypto/releases)
- [Commits](https://github.com/golang/crypto/commits/v0.1.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <[email protected]>
@christophemorio @chapost1
feat: report ElasticCloud data tiers nodes roles (#652)
24c0d7b 
* feat: additional elasticcloud roles

---------

Signed-off-by: Christophe MORIO <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@sysadmind @chapost1
Refactor cluster settings collector (#656)
ce225e7 
* Refactor cluster settings collector

Refactor cluster settings collector into new Collector interface. Update tests to use prometheus/testutil to validate the output instead of checking the parsed elasticsearch response.

Signed-off-by: Joe Adams <[email protected]>

* Fix rebase

Signed-off-by: Joe Adams <[email protected]>

---------

Signed-off-by: Joe Adams <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@SuperQ @chapost1
Update build and cleanup
7812ad0 
* Update Go to 1.20.
* Update Makefile.common
* Update golangci-lint.
* Update exporter-toolkit.
* Cleanup useless `_` assignment for go-kit/log.
* Cleanup deprecated and unused code.

Signed-off-by: SuperQ <[email protected]>
@sysadmind @chapost1
Fix ilm metric labels (#677)
ef20704 
The metric labels in the prometheus description were set to all 3 status options, instead of the name of the status label. The code exports a metric for each of the statuses individually, not all 3 at the same time.

fixes #677

Signed-off-by: Joe Adams <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@prombot @chapost1
Update common Prometheus files
92638f2 
Signed-off-by: prombot <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@innerpeacez @chapost1
add elasticsearch_indices_docs_total metric doc (#705)
b4677f2 
Signed-off-by: innerpeacez <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@innerpeacez @chapost1
add elasticsearch_indices_deleted_docs_primary metric doc (#707)
c32775a 
Signed-off-by: innerpeacez <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@jorgelbg @chapost1
fix: set the value of elasticsearch_clusterinfo_version_info to 1 (#728)
ff5e056 
In the Prometheus ecosystem, info metrics (those with the `_info`
suffix) usually have a value of 1. Although not enforced, this has
become a standard pattern across Prometheus and several exporters (see
`prometheus_build_info`).

Signed-off-by: Jorge Luis Betancourt Gonzalez <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@sysadmind @chapost1
Upgrade direct go dependencies
561c689 
Signed-off-by: Joe Adams <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@bagmeg @chapost1
Fix comment typo error
47ffd91 
Fix comment typo error in Retriever struct of the clusterinfo package

Signed-off-by: bagmeg <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@raz-amir @chapost1
Update README: Add required privileges for es.indices_mappings
c79e9c6 
Signed-off-by: Raz Amir <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@prombot @chapost1
Update common Prometheus files (#741)
bbc572b 
Signed-off-by: prombot <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@sysadmind @chapost1
Updates for v1.6.0 release
fc6f1b6 
Signed-off-by: Joe Adams <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@sysadmind @chapost1
Adjust language for breaking changes
90d9f31 
Signed-off-by: Joe Adams <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@prombot @chapost1
Update common Prometheus files (#747)
01a9c82 
Signed-off-by: prombot <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@tomachine @chapost1
add watermark metrics (#611)
42d5cd6 
Signed-off-by: Tomáš Kadaně <[email protected]>
Co-authored-by: Tomáš Kadaně <[email protected]>
Signed-off-by: Shahar Tal <[email protected]>
@chapost1
fixed indices mapping test
b4db178 
Signed-off-by: Shahar Tal <[email protected]>
@chapost1 chapost1 closed this by deleting the head repository Aug 17, 2023
@chapost1 chapost1 mentioned this pull request Aug 17, 2023
Open
@chapost1
Copy link
Author

chapost1 commented Aug 17, 2023 •
edited
Loading

@sysadmind
@shani-e

The details button on the DCO should help you to resolve. My suggestion is to rebase onto the master branch, squashing all of your commits into a single commit. This makes it easy to know that there's only one commit to sign.

I've tried to use the DCO guidelines, but it went off so I've closed this PR and created a new fork.
New PR can be found in here:
#764

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shani-e shani-e shani-e approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.