Why it's so complicated? I'm not sure I've understand the solution

[XDavidT]

For a few days, I've been looking for a way to get logs from NetApp storage to show up on our Grafana.
This is the closest solution I've found, but also that, the command that generates docker-compose.yaml create a complete solution that includes grafana and Prometheus together, the wizard does not provide you any details to add for the progress and links it to your NetApp server.

Why there is no basic image I can use to export logs from the NetApp dashboard, using 3 parameters only: Host, User, and Password?

[rahulguptajss]

@XDavidT Harvest collects its counters via EMS, ZAPI, REST, and the private CLI that REST exposes from ONTAP.
Are you trying to collect event logs event log show from ONTAP using Harvest?

Could you provide more details on which wizard are you refrerring to? Harvest Host,user,password needs to be configured in harvest.yml as mentioned here before docker-compose steps.

[rahulguptajss]

It takes few minutes for poller to start (2-3 minutes). I have edited your harvest.yml as it was missing exporter mapping.

Exporters:
  prometheus:
    exporter: Prometheus
    local_http_addr: 0.0.0.0
    port: 12990
  prometheus1:
    exporter: Prometheus
    port_range: 13000-14000

Defaults:
  collectors:
    - Zapi
    - ZapiPerf
    - Ems
  use_insecure_tls: false
  exporters:
    - prometheus1

Pollers:
  # add your cDOT / 7Mode pollers below, example:
 cluster-01:
   datacenter: OURCLUSTERNAME
   addr: 192.168.1.2
   auth_style: basic_auth
   username: user-admin
   password: --------   
   use_insecure_tls: true

It may be possible that your container is failing. Could you share logs for the poller containerr?
docker logs poller-cluster-01

[XDavidT]

Getting restart loop:

2023-02-14T13:33:08Z INF ./poller.go:202 > Init Poller=cluster-01 configPath=/opt/harvest.yml logLevel=info version="harvest version 22.11.1-1 (commit 689c1cfb) (build date 2022-12-08T14:29:52+0000) linux/amd64\n"
2023-02-14T13:33:08Z INF ./poller.go:239 > started in foreground Poller=cluster-01 pid=1
2023-02-14T13:33:08Z WRN ./poller.go:678 > abort collector Poller=cluster-01 collector=Zapi object=AggregateEfficiency
2023-02-14T13:33:08Z WRN ./poller.go:320 > no collectors initialized, stopping Poller=cluster-01
2023-02-14T13:33:08Z INF ./poller.go:545 > cleaning up and stopping [pid=1] Poller=cluster-01
2023-02-14T13:33:09Z INF ./poller.go:202 > Init Poller=cluster-01 configPath=/opt/harvest.yml logLevel=info version="harvest version 22.11.1-1 (commit 689c1cfb) (build date 2022-12-08T14:29:52+0000) linux/amd64\n"
2023-02-14T13:33:09Z INF ./poller.go:239 > started in foreground Poller=cluster-01 pid=1
2023-02-14T13:33:09Z WRN ./poller.go:678 > abort collector Poller=cluster-01 collector=ZapiPerf object=CopyManager
2023-02-14T13:33:09Z WRN ./poller.go:320 > no collectors initialized, stopping Poller=cluster-01
2023-02-14T13:33:09Z INF ./poller.go:545 > cleaning up and stopping [pid=1] Poller=cluster-01
2023-02-14T13:33:09Z INF ./poller.go:202 > Init Poller=cluster-01 configPath=/opt/harvest.yml logLevel=info version="harvest version 22.11.1-1 (commit 689c1cfb) (build date 2022-12-08T14:29:52+0000) linux/amd64\n"
2023-02-14T13:33:09Z INF ./poller.go:239 > started in foreground Poller=cluster-01 pid=1
2023-02-14T13:33:09Z WRN ./poller.go:678 > abort collector Poller=cluster-01 collector=Zapi object=SecurityAuditDestination
2023-02-14T13:33:09Z WRN ./poller.go:320 > no collectors initialized, stopping Poller=cluster-01
2023-02-14T13:33:09Z INF ./poller.go:545 > cleaning up and stopping [pid=1] Poller=cluster-01

[rahulguptajss]

@XDavidT It seems like , there is a connection issue with cluster. Could you share the output of below commands from harvest directory.

bin/harvest doctor --print

bin/zapi --poller cluster-01 show system

[XDavidT]

Sure!

$ bin/harvest doctor --print
Admin:
Tools:
Exporters:
    prometheus:
        exporter: Prometheus
        local_http_addr: 0.0.0.0
        port: 12990
    prometheus1:
        exporter: Prometheus
        port_range: 13000-14000
Defaults:
    collectors:
        - Zapi
        - ZapiPerf
        - Ems
    use_insecure_tls: false
    exporters:
        - prometheus1
Pollers:
    cluster-01:
        datacenter: DC-01
        addr: -REDACTED-
        auth_style: basic_auth
        username: -REDACTED-
        password: -REDACTED-

and (censored ip)

$ bin/zapi --poller cluster-01 show system
2023/02/14 18:13:01 connection error => Post "https://192.168.x.x:443/servlets/netapp.servlets.admin.XMLrequest_filer": x509: certificate has expired or is not yet valid: current time 2023-02-14T18:13:01+02:00 is after 2023-02-01T16:48:05Z

Edit: Thx for those commands it's helped me to better understand what the issue is.
I've added use_insecure_tls: true and it seems to be working.

[cgrinds]

Glad you got it working @XDavidT looks like you might have an expired cert, which as you noted, use_insecure_tls: true will ignore.

[cgrinds]

@XDavidT we agree on point 3, it's captured in #1505.

We have changes that use a simple docker compose, as you suggested, but we're still iterating on a solution for a single command that also allows customers to modify the files bundled in the container. Things like templates, dashboards, and your harvest.yml config file.

We discussed mounting the conf and certs directories via a volume mount, but it would still require copying files out of the container or extracting from the tar.gz or git. The current thinking is to provide a simple compose cmd that requires you to pass your harvest.yml via an environment var. That works, until you want to make template changes, but perhaps there is still value in providing a simple getting started and additional instructions when you want to modify templates. Would that better fit what you're looking for?

[XDavidT]

What you are describing is a normal docker-compose.yaml solution.
You can describe a docker-compose YAML to have a metric based on a few options:

1. Configuration file, ini type, or other
2. Env variables
   The example can describe an expanded solution with Grafana & Prometheus for customers who want the whole pack, if most of the customers use it for multi-server metrics - you need to read more than one instance using a config file.
   If most of them read one instance only, then people who want to get more than one cluster will open a few sessions with the same docker-compose.yaml
   For example - users who want a PostgreSQL exporter can run 1 command, and it's easy to go: https://github.com/prometheus-community/postgres_exporter
   Give me as a developer the simple way to get /metrics and if someone needs more than that, you can extend the guide as needed, but giving a full solution on quick start it's providing a ship for a boat ride.

About certs and confs: you can describe what folder is relevant for cert or configs, and the user will bind volume as needed.

[cgrinds]

thanks for the feedback. We'll track improvements in #1505 and regardless of those improvements, we'll get you working