Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

when docker warn: "error sending batch, will retry", container can not stop #13339

Open
CxistHans opened this issue Jun 27, 2024 · 2 comments
Open

Comments

@CxistHans
Copy link

Describe the bug
when docker warn: "error sending batch, will retry", container can not stop

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki (2.9.8)
  2. dockers plugin loki driver 2.9.8
  3. stop loki instance
  4. try to stop container: docker stop

Expected behavior
when loki instance stop or error, other use loki driver service can stop and start normal.

Environment:

  • Infrastructure: [docker: 20.10.8 ]
  • Deployment tool: [e.g., helm, jsonnet]

Screenshots, Promtail config, or terminal output

docker log:

Jun 27 15:45:26 data-transfer-test dockerd[2764940]: time="2024-06-27T15:45:26+08:00" level=info msg="level=warn ts=2024-06-27T07:45:26.617187418Z caller=client.go:419 container_id=df24cb99aaf27d0b3e57952048cfe9bf5810cf8fcfda1228936fa6ed1b2e8344 component=client host=192.168.69.125:10030 msg=\"error sending batch, will retry\" status=-1 tenant= error=\"Post \\\"http://192.168.69.125:10030/loki/api/v1/push\\\": dial tcp 192.168.69.125:10030: connect: connection refused\"" plugin=d29798387bd4f14f18baf6744f2564e3b3b3dd365f7c032fbd9436477c800b10

service congfig:

    logging:
      driver: "loki:latest"
      options:
        mode: "non-blocking"
        max-buffer-size: "10m"
        loki-url: "http://192.168.69.125:10030/loki/api/v1/push"
        loki-timeout: 10s
        loki-retries: "3"
        keep-file: "true"
        loki-batch-size: "400"
        max-file: "100"
        max-size: "10m"
        loki-batch-wait: "10s"
        loki-min-backoff: "10s"
        loki-max-backoff: "10m"
@CxistHans
Copy link
Author

image
and all unhealthy container can not restart

@CxistHans
Copy link
Author

@cyriltovena

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants