elasticsearch container quits during stack build, but can be started again in docker

[wkhchow]

When building new stack with latest branch opendrr-api (update_psra_apr2022) for some reason es container quits while the db is still building. You can still enable it in docker before it reaches the that stage to continue the build process.

kibana-opendrr_1 | 2022-05-10T20:55:22.837488770Z {"type":"log","@timestamp":"2022-05-10T20:55:22+00:00","tags":["warning","plugins","licensing"],"pid":8,"message":"License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 172.18.0.2:9200 error"}

[anthonyfok]

The following log (saved by @wkhchow using docker compose logs -t) may hopefully offer some clues:

• opendrr_may_10_2002.log

[anthonyfok]

Will, Drew and I brainstormed a little on Slack earlier today, and here are some initial observations/thoughts/ideas:

• Drew thinks it could be resource issue, e.g. the Elasticsearch container ran out of memory.
• Anthony agrees; the size of our dataset is growing after all, and it seems Elasticsearch itself is taking up quite a bit of RAM.
• Will wonders why it is Elasticsearch container and not the Python script (in python-env container) that ran out of memory (and exit with signal 9).
• There is no way to shutdown Elasticsearch (Java server) remotely from another container, not since Elasticsearch 2.x (see https://stackoverflow.com/questions/17191539/how-to-stop-shut-down-an-elasticsearch-node), so it seems unlikely the (random?) shutdown is caused by any errors in our bash, Python or PostgreSQL scripts.
• Perhaps something like the following crude CPU/RAM monitoring script can be extended to record Docker containers status etc. to give us more clues?

  #!/bin/bash
  
  instance_type=$(sudo dmesg -t | grep 'Amazon EC2' | cut -f4 -d' ' | sed -e 's/\/.*//')
  user="$1"
  logfile="${HOME}/logs/log_$(date +"%Y-%m-%d_%H-%M-%S")_${instance_type}_${user}_cpu-ram-process.log"
  
  while true; do
    ( date; uptime; free -h; ps auxwww | grep "^${user}" ; echo) | tee -a "${logfile}"
    sleep 15
  done

• Docker Compose logs do not seem to record when a container stopped abnormally. Maybe those starting/stopping info are shown on the running console? Perhaps script ("make typescript of terminal session", from bsdutils) or similar utilities can be used to capture those starting/stopping messages?