Problem with Postgresql 13 on a K8s Cluster

[alihoda]

Describe the Bug:
We had deployed golart (v0.31.1) and a postgresql 11 on our k8s cluster. Everything was good till we upgraded the postgresql to version 13. After about two days we realized that the postgresql usage memory is going up. After some digging, we saw this in goalert log: goalert-web-9c5cc6f9-llkfd tw-03 web time="2024-02-18T04:18:54Z" level=error AuthSystemComponent=Engine Trigger=INTERVAL error="parse error response: invalid character '<' looking for beginning of value".

Steps to Reproduce:

1. A postgresql 13 on k8s
2. A goalert instance on k8s

Expected Behavior:
Works like when postgresql version is 11.

Observed Behavior:
Postgresql crashed because of ram oom.

Application Version:
Using golart v0.31.1 docker image.

[mastercactapus]

This is a known issue with Postgres on Linux:
https://github.com/target/goalert/blob/master/docs/getting-started.md#database

Note: If you are using default install of Postgres on Debian (maybe others) you may run into an issue where the OOM (out of memory) killer terminates the supervisor process. More information along with steps to resolve can be found here.

https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT

The default virtual memory behavior on Linux is not optimal for PostgreSQL. Because of the way that the kernel implements memory overcommit, the kernel might terminate the PostgreSQL postmaster (the supervisor server process) if the memory demands of either PostgreSQL or another process cause the system to run out of virtual memory.