-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Getting "A plugin had an unrecoverable error" #41
Comments
I am wondering, why the plugin is not already failing at
At least, if I read it correctly, this is the line where the logstash-input-dead_letter_queue/src/main/java/org/logstash/input/DeadLetterQueueInputPlugin.java Line 80 in f7ef85a
|
Hey Robert, this does not seem like a new issue:
... given this is a known issue, does it persist after a restart or two? You concern seems to me that this is somehow caused by a change in the 8.4.0-SNAPSHOT build you're using? If that is the case we could try a different SNAPSHOT that would contain the patch you're after but would not be based off main but rather a more stable Logstash branch such as |
Hi @rpasche, The PR for the symbolic link is under review and we are giving a priority to make the change happen. And I assume you are building Logstash from the PQ symbolic link change source. I have also tried on my local that it doesn't seem belong to the changes. Let me know if I can help or if my understanding is not correct. Thanks. |
Hi @kares I am back from vacation now. I would need to test this again. But I think I heard, that 8.3.0 should get released on 28th of June. So I think, we will anyhow test this release within our environment. |
Hi @kares @jsvd @richardgilm, yesterday, version This is what we see in our logs
Further information (might be helpful). We forced the deadletter to trigger, because we changed the type from an The version (from API)
|
Hi Robert we’re looking into it, the issue comes from code we had to
introduce after upgrading jackson-datatabind because of known CVEs in
previous versions.
…On Wed, 29 Jun 2022 at 08:07, Robert Paschedag ***@***.***> wrote:
Hi @kares <https://github.com/kares> @jsvd <https://github.com/jsvd>
@richardgilm <https://github.com/richardgilm>,
yesterday, version 8.3.0 got officially released. We have tested this
version within our stack and we get again a A plugin had an unrecoverable
error.....but with another error message.
This is what we see in our logs
{"level":"ERROR","loggerName":"logstash.javapipeline","timeMillis":1656485427851,"thread":"[deadletter]<dead_letter_queue","logEvent":{"message":"A plugin had an unrecoverable error. Will restart this plugin.\n Pipeline_id:deadletter\n Plugin: <LogStash::Inputs::DeadLetterQueue pipeline_id=>\"in-https\", path=>\"/opt/logstash/data/dead_letter_queue\", id=>\"0b6cd156bd02d597addab476308a1e811786c47e231d05a6902564983c580525\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_17bb2b8f-d426-4206-a621-ac596c26aeb7\", enable_metric=>true, charset=>\"UTF-8\">, commit_offsets=>true>\n Error: Could not resolve type id 'java.lang.Long' as a subtype of `java.lang.Object`: Configured `PolymorphicTypeValidator` (of type `com.fasterxml.jackson.databind.jsontype.BasicPolymorphicTypeValidator`) denied resolution\n at [Source: (byte[])[5161 bytes]; byte offset: #146] (through reference chain: java.util.HashMap[\"DATA\"]->org.logstash.ConvertedMap[\"process\"]->org.logstash.ConvertedMap[\"ppid\"])\n Exception: Java::ComFasterxmlJacksonDatabindExc::InvalidTypeIdException\n Stack: com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(com/fasterxml/jackson/databind/exc/InvalidTypeIdException.java:43)\ncom.fasterxml.jackson.databind.DeserializationContext.invalidTypeIdException(com/fasterxml/jackson/databind/DeserializationContext.java:2073)\ncom.fasterxml.jackson.databind.DatabindContext._throwSubtypeClassNotAllowed(com/fasterxml/jackson/databind/DatabindContext.java:287)\ncom.fasterxml.jackson.databind.DatabindContext.resolveAndValidateSubType(com/fasterxml/jackson/databind/DatabindContext.java:244)\ncom.fasterxml.jackson.databind.jsontype.impl.ClassNameIdResolver._typeFromId(com/fasterxml/jackson/databind/jsontype/impl/ClassNameIdResolver.java:72)\ncom.fasterxml.jackson.databind.jsontype.impl.ClassNameIdResolver.typeFromId(com/fasterxml/jackson/databind/jsontype/impl/ClassNameIdResolver.java:66)\ncom.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(com/fasterxml/jackson/databind/jsontype/impl/TypeDeserializerBase.java:159)\ncom.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer._deserialize(com/fasterxml/jackson/databind/jsontype/impl/AsArrayTypeDeserializer.java:97)\ncom.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer.deserializeTypedFromAny(com/fasterxml/jackson/databind/jsontype/impl/AsArrayTypeDeserializer.java:71)\ncom.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserializeWithType(com/fasterxml/jackson/databind/deser/std/UntypedObjectDeserializer.java:781)\ncom.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(com/fasterxml/jackson/databind/deser/std/MapDeserializer.java:611)\ncom.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com/fasterxml/jackson/databind/deser/std/MapDeserializer.java:437)\ncom.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com/fasterxml/jackson/databind/deser/std/MapDeserializer.java:32)\ncom.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer._deserialize(com/fasterxml/jackson/databind/jsontype/impl/AsArrayTypeDeserializer.java:120)\ncom.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer.deserializeTypedFromAny(com/fasterxml/jackson/databind/jsontype/impl/AsArrayTypeDeserializer.java:71)\ncom.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserializeWithType(com/fasterxml/jackson/databind/deser/std/UntypedObjectDeserializer.java:781)\ncom.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(com/fasterxml/jackson/databind/deser/std/MapDeserializer.java:611)\ncom.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com/fasterxml/jackson/databind/deser/std/MapDeserializer.java:437)\ncom.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com/fasterxml/jackson/databind/deser/std/MapDeserializer.java:32)\ncom.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer._deserialize(com/fasterxml/jackson/databind/jsontype/impl/AsArrayTypeDeserializer.java:120)\ncom.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer.deserializeTypedFromAny(com/fasterxml/jackson/databind/jsontype/impl/AsArrayTypeDeserializer.java:71)\ncom.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserializeWithType(com/fasterxml/jackson/databind/deser/std/UntypedObjectDeserializer.java:781)\ncom.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(com/fasterxml/jackson/databind/deser/std/MapDeserializer.java:611)\ncom.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com/fasterxml/jackson/databind/deser/std/MapDeserializer.java:437)\ncom.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com/fasterxml/jackson/databind/deser/std/MapDeserializer.java:32)\ncom.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer._deserialize(com/fasterxml/jackson/databind/jsontype/impl/AsArrayTypeDeserializer.java:120)\ncom.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer.deserializeTypedFromObject(com/fasterxml/jackson/databind/jsontype/impl/AsArrayTypeDeserializer.java:61)\ncom.fasterxml.jackson.databind.deser.std.MapDeserializer.deserializeWithType(com/fasterxml/jackson/databind/deser/std/MapDeserializer.java:482)\ncom.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(com/fasterxml/jackson/databind/deser/impl/TypeWrappedDeserializer.java:74)\ncom.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(com/fasterxml/jackson/databind/deser/DefaultDeserializationContext.java:323)\ncom.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(com/fasterxml/jackson/databind/ObjectMapper.java:4674)\ncom.fasterxml.jackson.databind.ObjectMapper.readValue(com/fasterxml/jackson/databind/ObjectMapper.java:3723)\norg.logstash.Event.fromSerializableMap(org/logstash/Event.java:234)\norg.logstash.Event.deserialize(org/logstash/Event.java:510)\norg.logstash.DLQEntry.deserialize(org/logstash/DLQEntry.java:96)\norg.logstash.common.io.DeadLetterQueueReader.pollEntry(org/logstash/common/io/DeadLetterQueueReader.java:172)\norg.logstash.input.DeadLetterQueueInputPlugin.run(org/logstash/input/DeadLetterQueueInputPlugin.java:104)\njdk.internal.reflect.GeneratedMethodAccessor155.invoke(jdk/internal/reflect/GeneratedMethodAccessor155)\njdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)\norg.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:456)\norg.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:317)\nopt.logstash_minus_8_dot_3_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_dead_letter_queue_minus_1_dot_1_dot_12.lib.logstash.inputs.dead_letter_queue.run(/opt/logstash-8.3.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-dead_letter_queue-1.1.12/lib/logstash/inputs/dead_letter_queue.rb:74)\nopt.logstash_minus_8_dot_3_dot_0.logstash_minus_core.lib.logstash.java_pipeline.inputworker(/opt/logstash-8.3.0/logstash-core/lib/logstash/java_pipeline.rb:410)\nopt.logstash_minus_8_dot_3_dot_0.logstash_minus_core.lib.logstash.java_pipeline.start_input(/opt/logstash-8.3.0/logstash-core/lib/logstash/java_pipeline.rb:401)\norg.jruby.RubyProc.call(org/jruby/RubyProc.java:318)\njava.lang.Thread.run(java/lang/Thread.java:829)"}}
The version (from API)
$ curl localhost:9600?pretty
{
"host" : "container",
"version" : "8.3.0",
"http_address" : "0.0.0.0:9600",
"id" : "24a6d1f6-37c1-4a21-995d-0cafcba724a8",
"name" : "container",
"ephemeral_id" : "6710e4ec-57d8-4680-808b-14f385702ef6",
"status" : "green",
"snapshot" : false,
"pipeline" : {
"workers" : 16,
"batch_size" : 125,
"batch_delay" : 50
},
"build_date" : "2022-06-23T22:50:24Z",
"build_sha" : "e6ba1107a07b0468ca11d2765a4ab99a2c974eef",
"build_snapshot" : false
}
$
—
Reply to this email directly, view it on GitHub
<#41 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAHYQOJRHKLXMU72WUGOSDVRPY3HANCNFSM5YNN53JA>
.
You are receiving this because you were mentioned.Message ID:
<logstash-plugins/logstash-input-dead_letter_queue/issues/41/1169613000@
github.com>
|
Hi. @jsvd , I just wanted to provide a very simple example for you. Strange enough, in this case, the deadletter plugin worked as expected. I need to ask colleague, what exactly he tested. At least, this is what "I" did.
This created the event as expected in the target index. Additionally, I explicitly set the type to Then, forcing a mapping conflict with
And this got correctly processed by our deadletter pipeline (which we are writing to a dedicated index). |
Yes this is what I'm seeing as well, there are code paths where java.lang.Long is deserialized correctly:
|
@rpasche if there's any way y'all can pass us the DLQ files that trigger this issue it'd facilitate debugging. In the meantime we'll change the type validation to be "allow all", while we identify the issue. elastic/logstash#14312 |
@jsvd Where should I paste it? It's ~8k |
https://upload.elastic.co/u/92d7a242-6feb-49e3-8b2e-2204e8d3b736 (expires in 1 day) |
@jsvd Uploaded the file |
The |
Hi @jsvd sounds great. We'll test this soon. Thanks for the input (and fix) |
Logstash information:
Please include the following information:
based on brach referenced at based on the branch referenced at elastic/logstash#14212
JVM (e.g.
java -version
):bundled JDK version
Steps to reproduce:
I have rebuild logstash using referenced branch with the following steps:
rake
andbundler
like mentioned in the logstash READMEProvide logs (if relevant):
Logstash.yml
pipelines.yml
deadletter configuration
Error we get
Then, constantly following message like
While now looking into the "running" container, we see that the path exists
In the past, we sometimes also had "race-condition" looking like issues. Our "main" pipeline is quite big...and for us, it looked like the deadletter pipeline is started, before the "monitored" main pipelines is fully started up (and also the directories within the "dead_letter_queue" folder have been created). We were able to workaround this issue in the past and then have sent a
SIGHUP
to logstash (to reload its pipelines). But this does not work anymore.See logstash
_node/pipelines
APIThe text was updated successfully, but these errors were encountered: