Lots of "failed to receive the first 2 bytes" errors #12847

zyiqian · 2024-04-10T21:19:11Z

zyiqian
Apr 10, 2024

Hi everyone..

When kong receives a large number of long-term connection requests. Kong will keep restarting. I see a lot of errors like this in the logs. What causes it?

proxy  [error] 1129#0: *153 [lua] worker.lua:248: communicate(): event worker failed: failed to receive the first 2 bytes: closed, context: ngx.timer
proxy  [error] 1127#0: *10 [lua] worker.lua:248: communicate(): event worker failed: failed to receive the first 2 bytes: closed, context: ngx.timer
proxy  [error] 1127#0: *10 [lua] worker.lua:248: communicate(): event worker failed: failed to receive the first 2 bytes: closed, context: ngx.timer
proxy  [error] 1129#0: *152 [lua] worker.lua:248: communicate(): event worker failed: failed to receive the first 2 bytes: closed, context: ngx.timer
proxy  [error] 1126#0: *22896 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 106.x.x.7, server: kong, request: "GET /cloud/connect HTTP/1.1", upstream: "http://10.x.x.73:8080/cloud/connect", host: "connector.xxx.com:10443"
proxy  [alert] 1126#0: *140545 open socket #14601 left in connection 11271
ingress-controller level=error msg="checking config status failed" error="making HTTP request: Get \"http://localhost:8001/status\": dial tcp 127.0.0.1:8001: connect: connection refused"

My kong config

{
   "admin": {
      "annotations": { },
      "enabled": true,
      "http": {
         "enabled": true
      },
      "ingress": {
         "annotations": { },
         "enabled": false,
         "hostname": null,
         "ingressClassName": "kong-gateway",
         "path": "/"
      },
      "labels": { },
      "tls": {
         "enabled": false
      },
      "type": "ClusterIP"
   },
   "affinity": { },
   "autoscaling": {
      "enabled": true,
      "maxReplicas": 11,
      "metrics": [
         {
            "resource": {
               "name": "cpu",
               "target": {
                  "averageUtilization": 80,
                  "type": "Utilization"
               }
            },
            "type": "Resource"
         }
      ],
      "minReplicas": 11,
      "targetCPUUtilizationPercentage": null
   },
   "cluster": {
      "annotations": { },
      "enabled": false,
      "labels": { },
      "tls": {
         "containerPort": 8005,
         "enabled": false,
         "parameters": [ ],
         "servicePort": 8005
      },
      "type": "ClusterIP"
   },
   "clustertelemetry": {
      "annotations": { },
      "enabled": false,
      "labels": { },
      "tls": {
         "containerPort": 8006,
         "enabled": false,
         "parameters": [ ],
         "servicePort": 8006
      },
      "type": "ClusterIP"
   },
   "containerSecurityContext": { },
   "dblessConfig": {
      "config": "",
      "configMap": ""
   },
   "deploy": true,
   "deployOrder": 100,
   "deployment": {
      "daemonset": false,
      "hostNetwork": false,
      "kong": {
         "enabled": true
      },
      "serviceAccount": {
         "create": true
      },
      "test": {
         "enabled": false
      }
   },
   "deploymentAnnotations": { },
   "doc": { },
   "enterprise": {
      "enabled": false,
      "portal": {
         "enabled": false
      },
      "rbac": {
         "admin_gui_auth": "basic-auth",
         "admin_gui_auth_conf_secret": "CHANGEME-admin-gui-auth-conf-secret",
         "enabled": false,
         "session_conf_secret": "kong-session-config"
      },
      "smtp": {
         "admin_emails_from": "[email protected]",
         "admin_emails_reply_to": "[email protected]",
         "auth": {
            "smtp_password_secret": "CHANGEME-smtp-password",
            "smtp_username": ""
         },
         "enabled": false,
         "portal_emails_from": "[email protected]",
         "portal_emails_reply_to": "[email protected]",
         "smtp_admin_emails": "[email protected]",
         "smtp_auth_type": "",
         "smtp_host": "smtp.example.com",
         "smtp_port": 587,
         "smtp_ssl": "nil",
         "smtp_starttls": true
      },
      "vitals": {
         "enabled": true
      }
   },
   "env": {
      "admin_access_log": "/dev/stdout",
      "admin_error_log": "/dev/stderr",
      "admin_gui_access_log": "/dev/stdout",
      "admin_gui_error_log": "/dev/stderr",
      "database": "off",
      "lua_ssl_protocols": "TLSv1.2",
      "mem_cache_size": "1024m",
      "portal_api_access_log": "/dev/stdout",
      "portal_api_error_log": "/dev/stderr",
      "prefix": "/kong_prefix/",
      "proxy_access_log": "/dev/stdout",
      "proxy_error_log": "/dev/stderr",
      "proxy_listen": "0.0.0.0:8443 http2 ssl",
      "real_ip_header": "X-Forwarded-For",
      "real_ip_recursive": "on",
      "ssl_cipher_suite": "custom",
      "ssl_ciphers": "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:AES128-SHA:AES256-SHA:DES-CBC3-SHA",
      "ssl_protocols": "TLSv1.2",
      "trusted_ips": "0.0.0.0/0,::/0"
   },
   "extraLabels": { },
   "extraSecrets": [ ],
   "image": {
      "pullPolicy": "IfNotPresent",
      "repository": "kong",
      "tag": "3.1"
   },
   "ingressController": {
      "admissionWebhook": {
         "certificate": {
            "provided": false
         },
         "enabled": false,
         "failurePolicy": "Fail",
         "port": 8080
      },
      "args": [ ],
      "enabled": true,
      "env": {
         "kong_admin_tls_skip_verify": true
      },
      "image": {
         "effectiveSemver": null,
         "pullPolicy": "IfNotPresent",
         "repository": "kong/kubernetes-ingress-controller",
         "tag": "2.8"
      },
      "ingressClass": "kong-gateway",
      "ingressClassAnnotations": { },
      "installCRDs": false,
      "livenessProbe": {
         "failureThreshold": 3,
         "httpGet": {
            "path": "/healthz",
            "port": 10254,
            "scheme": "HTTP"
         },
         "initialDelaySeconds": 5,
         "periodSeconds": 10,
         "successThreshold": 1,
         "timeoutSeconds": 5
      },
      "rbac": {
         "create": true
      },
      "readinessProbe": {
         "failureThreshold": 3,
         "httpGet": {
            "path": "/readyz",
            "port": 10254,
            "scheme": "HTTP"
         },
         "initialDelaySeconds": 5,
         "periodSeconds": 10,
         "successThreshold": 1,
         "timeoutSeconds": 5
      },
      "resources": {
         "limits": {
            "cpu": "2000m",
            "memory": "4096Mi"
         },
         "requests": {
            "cpu": "1000m",
            "memory": "1024Mi"
         }
      },
      "watchNamespaces": [ ]
   },
   "lifecycle": {
      "preStop": {
         "exec": {
            "command": [
               "kong",
               "quit",
               "--wait=15"
            ]
         }
      }
   },
   "livenessProbe": {
      "failureThreshold": 3,
      "httpGet": {
         "path": "/status",
         "port": "status",
         "scheme": "HTTP"
      },
      "initialDelaySeconds": 30,
      "periodSeconds": 10,
      "successThreshold": 1,
      "timeoutSeconds": 5
   },
   "manager": {
      "annotations": { },
      "enabled": false,
      "http": {
         "containerPort": 8002,
         "enabled": true,
         "parameters": [ ],
         "servicePort": 8002
      },
      "ingress": {
         "annotations": { },
         "enabled": false,
         "hostname": null,
         "ingressClassName": "kong-gateway",
         "path": "/"
      },
      "labels": { },
      "tls": {
         "containerPort": 8445,
         "enabled": true,
         "parameters": [
            "http2"
         ],
         "servicePort": 8445
      },
      "type": "ClusterIP"
   },
   "migrations": {
      "annotations": {
         "sidecar.istio.io/inject": false
      },
      "backoffLimit": null,
      "jobAnnotations": { },
      "postUpgrade": true,
      "preUpgrade": true,
      "resources": { }
   },
   "name": "kong-gateway",
   "namespace": "prod",
   "nodeSelector": { },
   "plugins": { },
   "podAnnotations": {
      "prometheus.io/path": "/metrics",
      "prometheus.io/port": "8100",
      "prometheus.io/scrape": "true",
      "sidecar.istio.io/inject": "false"
   },
   "podDisruptionBudget": {
      "enabled": false
   },
   "podLabels": { },
   "podSecurityPolicy": {
      "enabled": false,
      "spec": {
         "allowPrivilegeEscalation": false,
         "fsGroup": {
            "rule": "RunAsAny"
         },
         "hostIPC": false,
         "hostNetwork": false,
         "hostPID": false,
         "privileged": false,
         "readOnlyRootFilesystem": true,
         "runAsGroup": {
            "rule": "RunAsAny"
         },
         "runAsUser": {
            "rule": "RunAsAny"
         },
         "seLinux": {
            "rule": "RunAsAny"
         },
         "supplementalGroups": {
            "rule": "RunAsAny"
         },
         "volumes": [
            "configMap",
            "secret",
            "emptyDir"
         ]
      }
   },
   "portal": {
      "annotations": { },
      "enabled": true,
      "http": {
         "containerPort": 8003,
         "enabled": true,
         "parameters": [ ],
         "servicePort": 8003
      },
      "ingress": {
         "annotations": { },
         "enabled": false,
         "hostname": null,
         "ingressClassName": null,
         "path": "/"
      },
      "labels": { },
      "tls": {
         "containerPort": 8446,
         "enabled": true,
         "parameters": [
            "http2"
         ],
         "servicePort": 8446
      },
      "type": "ClusterIP"
   },
   "portalapi": {
      "annotations": { },
      "enabled": true,
      "http": {
         "containerPort": 8004,
         "enabled": true,
         "parameters": [ ],
         "servicePort": 8004
      },
      "ingress": {
         "annotations": { },
         "enabled": false,
         "hostname": null,
         "ingressClassName": null,
         "path": "/"
      },
      "labels": { },
      "tls": {
         "containerPort": 8447,
         "enabled": true,
         "parameters": [
            "http2"
         ],
         "servicePort": 8447
      },
      "type": "ClusterIP"
   },
   "postgresql": {
      "enabled": false
   },
   "priorityClassName": "",
   "proxy": {
      "enabled": true,
      "externalTrafficPolicy": "Local",
      "http": {
         "containerPort": 8000,
         "enabled": false,
         "parameters": [ ],
         "servicePort": 80
      },
      "ingress": {
         "annotations": { },
         "enabled": false,
         "hostname": null,
         "ingressClassName": null,
         "path": "/"
      },
      "labels": {
         "enable-metrics": "true"
      },
      "stream": [ ],
      "tls": {
         "containerPort": 8443,
         "enabled": true,
         "parameters": [
            "http2"
         ],
         "servicePort": 10443
      },
      "type": "LoadBalancer"
   },
   "readinessProbe": {
      "failureThreshold": 3,
      "httpGet": {
         "path": "/status",
         "port": "status",
         "scheme": "HTTP"
      },
      "initialDelaySeconds": 30,
      "periodSeconds": 10,
      "successThreshold": 1,
      "timeoutSeconds": 5
   },
   "replicaCount": 2,
   "resources": {
      "limits": {
         "cpu": "3000m",
         "memory": "8192Mi"
      },
      "requests": {
         "cpu": "1000m",
         "memory": "2048Mi"
      }
   },
   "secretVolumes": [ ],
   "securityContext": { },
   "serviceMonitor": {
      "enabled": false
   },
   "status": {
      "enabled": true,
      "http": {
         "containerPort": 8100,
         "enabled": true,
         "parameters": [ ]
      },
      "tls": {
         "containerPort": 8543,
         "enabled": false,
         "parameters": [ ]
      }
   },
   "statusHttpPort": 8100,
   "terminationGracePeriodSeconds": 30,
   "tolerations": [ ],
   "udpProxy": {
      "annotations": { },
      "enabled": false,
      "labels": { },
      "stream": [ ],
      "type": "LoadBalancer"
   },
   "updateStrategy": {
      "rollingUpdate": {
         "maxSurge": 1,
         "maxUnavailable": 0
      },
      "type": "RollingUpdate"
   },
   "waitImage": {
      "enabled": true,
      "pullPolicy": "IfNotPresent"
   }
}

ingress

{
"apiVersion": "networking.k8s.io/v1",
"kind": "Ingress",
"metadata": {
    "annotations": {
        "konghq.com/plugins": "prometheus-plugin-grpc"
    },
    "labels": { },
    "name": "connector-ingress",
    "namespace": "prod"
},
"spec": {
    "ingressClassName": "kong-gateway",
    "rules": [
        {
            "host": "connector.xxx.com",
            "http": {
            "paths": [
                {
                    "backend": {
                        "service": {
                        "name": "connector-service",
                        "port": {
                            "number": 80
                        }
                        }
                    },
                    "path": "/cloud/connect",
                    "pathType": "Prefix"
                }
            ]
            }
        },
        {
            "host": "connector1.xxx.com",
            "http": {
            "paths": [
                {
                    "backend": {
                        "service": {
                        "name": "connector-service",
                        "port": {
                            "number": 80
                        }
                        }
                    },
                    "path": "/cloud/connect",
                    "pathType": "Prefix"
                }
            ]
            }
        },
        {
            "host": "connector2.xx.com",
            "http": {
            "paths": [
                {
                    "backend": {
                        "service": {
                        "name": "connector-service",
                        "port": {
                            "number": 80
                        }
                        }
                    },
                    "path": "/cloud/connect",
                    "pathType": "Prefix"
                }
            ]
            }
        }
    ],
    "tls": [
        {
            "hosts": [
            "connector.xxx.com",
            "connector1.xxx.com"
            ],
            "secretName": "connector-tls"
        },
        {
            "hosts": [
            "connector2.xx.com"
            ],
            "secretName": "connector2-tls"
        }
    ]
}
}

zyiqian · 2024-04-11T04:57:16Z

zyiqian
Apr 11, 2024
Author

Add some monitoring capture information.

kong worker lua vm usage by node

nginx connection state

Latencies

Request rate

kong proxy container memory usage

Do you need any more information from me?

0 replies

chronolaw · 2024-04-11T05:20:46Z

chronolaw
Apr 11, 2024
Collaborator

Kong 3.x uses lua-resty-events as its event propagation mechanism, this library works with unix domain socket, the error message you saw are lua-resty-events generated, because frequently rebooting interrupt socket communication.

I think that it will not harm kong's working.

2 replies

zyiqian Apr 11, 2024
Author

Hi @chronolaw , Thank you very much for your help many times.

We are based on websocket long connection communication. If Kong interrupts communication, client will reconnect. Is it possible that Kong restarts frequently due to repeated reconnection and interrupted communication?

Because we have run the same configuration in the test environment for a long time and have not found such a problem. So it's a bit confusing.

zyiqian Apr 11, 2024
Author

Looking at the k8s Events,

fieldPath: "spec.containers{ingress-controller}"
reason: "Unhealthy"
message: "Readiness probe failed: Get "http://10.116.233.95:10254/readyz": dial tcp 10.116.233.95:10254: connect: connection refused"

fieldPath: "spec.containers{proxy}"
reason: "Unhealthy"
message: "Readiness probe failed: Get "http://10.116.233.95:8100/status": dial tcp 10.116.233.95:8100: connect: connection refused"

fieldPath: "spec.containers{proxy}"
reason: "FailedPreStopHook"
message: "Exec lifecycle hook ([kong quit --wait=15]) for Container "proxy" in Pod "kong-gateway-858f9b8b79-27cwb_aispeech-prod(e51cdf94-f73b-49ae-bf66-748ae071a57a)" failed - error: command 'kong quit --wait=15' exited with 137: , message: "Timeout, Kong stopped forcefully\n""

fieldPath: "spec.containers{proxy}"
reason: "Killing"
message: "Stopping container proxy"

chronolaw · 2024-04-11T05:22:11Z

chronolaw
Apr 11, 2024
Collaborator

Here is the blog post: https://konghq.com/blog/engineering/nginx-openresty-event-handling-strategy-for-cpu-efficiency

0 replies

chronolaw · 2024-04-11T07:58:53Z

chronolaw
Apr 11, 2024
Collaborator

It seems that the k8s' health probe restarts kong gateway frequently, perhaps we should check this.

4 replies

zyiqian Apr 11, 2024
Author

Very good, looking forward to your reply.

zyiqian Apr 13, 2024
Author

hi @chronolaw , Sorry to bother you again, is there any update? This problem will prevent us from continuing to migrate the prod environment.

chronolaw Apr 13, 2024
Collaborator

Sorry, we can not identify any useful clue from k8s events, could you check the k8s environment to figure out?

zyiqian Apr 13, 2024
Author

My doubt is why we haven't had this problem after testing with the same configuration in the test environment for a long time. When we switched the traffic of the prod environment, this error kept reporting. Or where can I check it?

If the backend service is unavailable, will it cause Kong to restart?

chronolaw · 2024-04-14T04:01:12Z

chronolaw
Apr 14, 2024
Collaborator

I am sure that the unavailable backend service will not cause Kong to restart, you should check why k8s health probe fails.

1 reply

zyiqian Apr 14, 2024
Author

Okay, understand, let me check again. Thanks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lots of "failed to receive the first 2 bytes" errors #12847

{{title}}

Replies: 5 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Lots of "failed to receive the first 2 bytes" errors #12847

zyiqian Apr 10, 2024

Replies: 5 comments · 7 replies

zyiqian Apr 11, 2024 Author

chronolaw Apr 11, 2024 Collaborator

zyiqian Apr 11, 2024 Author

zyiqian Apr 11, 2024 Author

chronolaw Apr 11, 2024 Collaborator

chronolaw Apr 11, 2024 Collaborator

zyiqian Apr 11, 2024 Author

zyiqian Apr 13, 2024 Author

chronolaw Apr 13, 2024 Collaborator

zyiqian Apr 13, 2024 Author

chronolaw Apr 14, 2024 Collaborator

zyiqian Apr 14, 2024 Author

zyiqian
Apr 10, 2024

Replies: 5 comments 7 replies

zyiqian
Apr 11, 2024
Author

chronolaw
Apr 11, 2024
Collaborator

zyiqian Apr 11, 2024
Author

zyiqian Apr 11, 2024
Author

chronolaw
Apr 11, 2024
Collaborator

chronolaw
Apr 11, 2024
Collaborator

zyiqian Apr 11, 2024
Author

zyiqian Apr 13, 2024
Author

chronolaw Apr 13, 2024
Collaborator

zyiqian Apr 13, 2024
Author

chronolaw
Apr 14, 2024
Collaborator

zyiqian Apr 14, 2024
Author