Latest builds requiring 8589934528 bytes of ram to launch

[Bonejob]

This may be an issue with CKAN/BASE, but I am using OKFN as my base for our project.

when building the latest OKFN, I now receive from CKAN the following error on a 4GB test server.

ckan  | lock engine: pthread robust mutexes
ckan  | thunder lock: disabled (you can enable it with --thunder-lock)
ckan  | uWSGI http bound on 0.0.0.0:5000 fd 3
ckan  | uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 6
ckan  | Python version: 3.8.15 (default, Nov 10 2022, 13:35:13)  [GCC 10.2.1 20201203]
ckan  | Python main interpreter initialized at 0x7f01e8d5ac20
ckan  | python threads support enabled
ckan  | your server socket listen backlog is limited to 100 connections
ckan  | your mercy for graceful operations on workers is 60 seconds
ckan  | mapped 304776 bytes (297 KB) for 2 cores
ckan  | *** Operational MODE: preforking ***
ckan  | *** uWSGI is running in multiple interpreter mode ***
ckan  | spawned uWSGI master process (pid: 26)
ckan  | spawned uWSGI worker 1 (pid: 27, cores: 1)
ckan  | spawned uWSGI worker 2 (pid: 28, cores: 1)
ckan  | spawned uWSGI http 1 (pid: 29)
ckan  | malloc(): Out of memory [core/utils.c line 1801]
ckan  | !!! tried memory allocation of 8589934528 bytes !!!

CKAN from my last build uses

CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
c13eec16c2ca   ckan           0.01%     326.9MiB / 3.644GiB   8.76%     1.21MB / 3.31MB   7.54MB / 0B       7

now it requires 8GB when spooling up and settles back to 5GB

CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
00e113c9a83e   ckan           0.01%     5.282GiB / 7.723GiB   68.40%    521kB / 440kB     56.1MB / 422kB   7

Has anybody run into this before, and can you point me in the correct direction to fix this?

[aivuk]

That is a bit weird. I just download the latest version from this repository and when I ran I got:

CONTAINER ID   NAME                    CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
a99bc4b44d4b   ckanc                   0.03%     295.6MiB / 15.24GiB   1.89%     197kB / 179kB     64.1MB / 1.51MB   7
c1980107a572   datapusherc             0.00%     58.88MiB / 15.24GiB   0.38%     9.08kB / 0B       37.3MB / 111kB    3
ade322c9d91c   redisc                  0.22%     3.023MiB / 15.24GiB   0.02%     15kB / 2.97kB     3.48MB / 0B       5
0a95f602e79f   dbc                     0.01%     59MiB / 15.24GiB      0.38%     169kB / 148kB     30.6MB / 885kB    22
ba3369fade18   solrc                   0.89%     360.9MiB / 15.24GiB   2.31%     21.2kB / 41.5kB   95.5MB / 69.6kB   55

[Bonejob]

@aivuk glad to see that it is working for you. That actually gives me some hope. I still need to rule out that it's not one of the plugins we use.

[Bonejob]

Ok I pulled a fresh copy of the repo modify the .env and did a fresh compile and it is using
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS c94a9d351313 ckan 0.13% 6.874GiB / 15.59GiB 44.10% 370kB / 408kB 45.8MB / 381kB 8 23b4cf0acf0d solr 0.62% 223.2MiB / 15.59GiB 1.40% 28kB / 55.7kB 41.2MB / 119kB 57 7e1ace564ca8 db 0.00% 48.61MiB / 15.59GiB 0.30% 383kB / 302kB 15.4MB / 93.6MB 19 128a6e340ccf redis 0.17% 5.211MiB / 15.59GiB 0.03% 15.7kB / 6.94kB 5.89MB / 0B 5 4239f5492e08 datapusher 0.00% 3.951GiB / 15.59GiB 25.35% 6.13kB / 0B 831kB / 139kB 3
I assumed at first it was one of my plugins causing the issue but the memory footprint has gone up a lot.

Anybody else able to confirm this?

[aivuk]

Since your last post I pulled docker-ckan in two other VPS and these are the stats:

root@ubuntu-4gb-hel1-2:~#  docker stats --no-stream
CONTAINER ID   NAME         CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
71e871adefc6   ckan         84.45%    83.4MiB / 3.738GiB    2.18%     71.7kB / 66.2kB   5.53MB / 0B       3
320a882de539   db           13.36%    28.9MiB / 3.738GiB    0.75%     62.7kB / 59.3kB   6.29MB / 422kB    13
48e61d1bcc41   solr         8.39%     262.5MiB / 3.738GiB   6.86%     5.72kB / 12.2kB   46.2MB / 69.6kB   42
39ed953c813d   redis        0.30%     2.52MiB / 3.738GiB    0.07%     1.98kB / 676B     541kB / 0B        5
c0a90c70faab   datapusher   0.01%     53.13MiB / 3.738GiB   1.39%     962B / 0B         2.44MB / 0B       3
8316947de911   ckan-solr    0.13%     375.4MiB / 3.738GiB   9.81%     786kB / 4.21MB    96.2MB / 279MB    38
root@ubuntu-4gb-hel1-2:~# free
               total        used        free      shared  buff/cache   available
Mem:         3919844     1394132      133156       38288     2392556     2205044
Swap:              0           0           0
root@ubuntu-4gb-hel1-2:~# 

and:

root@ubuntu-4gb-hel1-1:/home/aivuk#  docker stats --no-stream
CONTAINER ID   NAME         CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
58bbc2f82701   ckan         0.04%     431.8MiB / 3.738GiB   11.28%    10.3MB / 30.9MB   67.4MB / 16.4kB   7
5e67d2250860   datapusher   0.01%     59.73MiB / 3.738GiB   1.56%     51.2kB / 24.2kB   36.1MB / 4.1kB    3
c4b4e43d5f03   db           0.00%     58.8MiB / 3.738GiB    1.54%     9.56MB / 6.78MB   28MB / 1.92GB     22
7145d9ff5d45   solr         1.10%     528.9MiB / 3.738GiB   13.82%    762kB / 1.28MB    95.6MB / 1.5MB    42
63f0690fe978   redis        0.28%     5.18MiB / 3.738GiB    0.14%     1.45MB / 1.25MB   6.75MB / 709kB    5
root@ubuntu-4gb-hel1-1:/home/aivuk# free
               total        used        free      shared  buff/cache   available
Mem:         3919836     1397608      255320       35980     2266908     2228640
Swap:              0           0           0
root@ubuntu-4gb-hel1-1:/home/aivuk# 

Both servers are VPS with 4gb of RAM in Hetzner provider. Which docker/docker-compose versions and OS are you using?