Logstash retries indefinitely

[rssawhney]

• Version: plugin (10.5.1) & Elasticsearch + logstash (7.7.0)
• Operating System: Amazon Linux
• Config File (if you have sensitive info, please remove it):
• Sample Data:
• Steps to Reproduce:

while using filebeat and multiline input I am experiencing issues after logstash updates the mapping for index and after it all the events are not processed due to indefinite retry.

logstash conf

input {
    beats {
        port => 5044
    }
}

output {
     elasticsearch {
          document_id => "%{log_line_uuid}"
          index => "index_%{[@metadata][index_name_day]}"
          hosts => ["http://localhost:9200"]
          template => "path to template.json"
          template_name => "template name"
          template_overwrite => "true"
      }
}

Logstash logs

2021-07-06T05:31:15,302][ERROR][logstash.outputs.elasticsearch][log_monitoring][7338da4e6115dea6be7dbf86d42de2614d74a3a42465274043c908858710a98c] Encountered a retryable error. Will Retry with expone
ntial backoff  {:code=>400, :url=>"http://10.90.10.92:9200/_bulk"}
[2021-07-06T05:31:17,471][ERROR][logstash.outputs.elasticsearch][log_monitoring][7338da4e6115dea6be7dbf86d42de2614d74a3a42465274043c908858710a98c] Encountered a retryable error. Will Retry with expone
ntial backoff  {:code=>400, :url=>"http://10.90.10.92:9200/_bulk"}
[2021-07-06T05:31:21,480][ERROR][logstash.outputs.elasticsearch][log_monitoring][7338da4e6115dea6be7dbf86d42de2614d74a3a42465274043c908858710a98c] Encountered a retryable error. Will Retry with expone
ntial backoff  {:code=>400, :url=>"http://10.90.10.92:9200/_bulk"}
[2021-07-06T05:31:29,514][ERROR][logstash.outputs.elasticsearch][log_monitoring][7338da4e6115dea6be7dbf86d42de2614d74a3a42465274043c908858710a98c] Encountered a retryable error. Will Retry with expone
ntial backoff  {:code=>400, :url=>"http://10.90.10.92:9200/_bulk"}
[2021-07-06T05:31:45,522][ERROR][logstash.outputs.elasticsearch][log_monitoring][7338da4e6115dea6be7dbf86d42de2614d74a3a42465274043c908858710a98c] Encountered a retryable error. Will Retry with expone
ntial backoff  {:code=>400, :url=>"http://10.90.10.92:9200/_bulk"}

Elasticsearch logs

[2021-07-06T05:34:33,671][INFO ][o.e.c.m.MetaDataMappingService] [ip-10-90-10-92] [index_2021_07_06/MHCfijNbSAuMrrRh_W_O5g] update_mapping [_doc]

Mapping change
addition of flags inside logs section of mapping with following data

"flags" : {
    "type" : "text",
    "fields" : {
      "keyword" : {
        "type" : "keyword",
        "ignore_above" : 256
      }
    }
  },

[kares]

LS is retry-ing 400s since if the ES issue got fixed on the cluster than data would eventually get ingested...
we lack information on the ES part here around the _bulk errors thus there isn't a specific advice to give you.

[rssawhney]

Is there any config that I can set to work around with?