Pipeline terminating immediately after start up

[TSASM]

Hi,

I'm trying to set logstash up in my cluster to get the app protect logs but after using the pipeline config provided it only starts up to then terminate it's self right away

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1/jruby15032451142012517880jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2020-11-12 11:35:50.403 [main] runner - Starting Logstash {"logstash.version"=>"7.9.3", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10-LTS on 11.0.8+10-LTS +jit [linux-x86_64]"}
[INFO ] 2020-11-12 11:35:50.416 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2020-11-12 11:35:50.418 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2020-11-12 11:35:50.608 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-11-12 11:35:50.617 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"e2bc3af3-102b-4d42-9e1f-b8ecf260fe59", :path=>"/usr/share/logstash/data/uuid"}
[WARN ] 2020-11-12 11:35:50.691 [LogStash::Runner] pipelineregisterhook - Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
Please configure Metricbeat to monitor Logstash. Documentation can be found at: 
https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
[INFO ] 2020-11-12 11:35:51.072 [LogStash::Runner] licensereader - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://waf-logstash:xxxxxx@quickstart-es-http:9200/]}}
[WARN ] 2020-11-12 11:35:51.204 [LogStash::Runner] licensereader - Restored connection to ES instance {:url=>"http://waf-logstash:xxxxxx@quickstart-es-http:9200/"}
[INFO ] 2020-11-12 11:35:51.364 [LogStash::Runner] licensereader - ES Output version determined {:es_version=>7}
[WARN ] 2020-11-12 11:35:51.364 [LogStash::Runner] licensereader - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2020-11-12 11:35:51.429 [LogStash::Runner] internalpipelinesource - Monitoring License OK
[INFO ] 2020-11-12 11:35:51.429 [LogStash::Runner] internalpipelinesource - Validated license for monitoring. Enabling monitoring pipeline.
[INFO ] 2020-11-12 11:35:51.442 [Agent thread] configpathloader - No config files found in path {:path=>"/usr/share/logstash/ /usr/share/logstash/pipeline"}
[INFO ] 2020-11-12 11:35:52.070 [Converge PipelineAction::Create<.monitoring-logstash>] Reflections - Reflections took 37 ms to scan 1 urls, producing 22 keys and 45 values 
[INFO ] 2020-11-12 11:35:52.141 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://waf-logstash:xxxxxx@quickstart-es-http:9200/]}}
[WARN ] 2020-11-12 11:35:52.151 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Restored connection to ES instance {:url=>"http://waf-logstash:xxxxxx@quickstart-es-http:9200/"}
[INFO ] 2020-11-12 11:35:52.158 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - ES Output version determined {:es_version=>7}
[WARN ] 2020-11-12 11:35:52.158 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2020-11-12 11:35:52.186 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["//quickstart-es-http:9200"]}
[WARN ] 2020-11-12 11:35:52.187 [[.monitoring-logstash]-pipeline-manager] javapipeline - 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[INFO ] 2020-11-12 11:35:52.210 [[.monitoring-logstash]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x393ee466@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[INFO ] 2020-11-12 11:35:52.961 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.75}
[INFO ] 2020-11-12 11:35:52.969 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[INFO ] 2020-11-12 11:35:52.991 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[]}
[INFO ] 2020-11-12 11:35:53.040 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>5144}
[INFO ] 2020-11-12 11:35:54.188 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}

These are the logs, I've also added the following logstash config file

http.host: "0.0.0.0"
http.port: "5144"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: "quickstart-es-http:9200"
xpack.monitoring.elasticsearch.username: "waf-logstash"
xpack.monitoring.elasticsearch.password: "blahblah"

Any help would be greatly appreciated with this

[464d41]

Hi @TSASM,

How about this message: [INFO ] 2020-11-12 11:35:51.442 [Agent thread] configpathloader - No config files found in path {:path=>"/usr/share/logstash/ /usr/share/logstash/pipeline"}

Looks logstash fails to find config files. Are they mounted properly/have right permissions?