You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The http server currently spawns one thread per core, however, only a single database connection is opened with sqlite compiled with SQLITE_THREADSAFE 0
What this basically means is that one core will run hot when querying with the others idle.
We should either develop or document a workflow which utilizes all cores.
A similar situation applies for imports, these are also single-threaded.
One completely valid solution to this issue is to simply say that the application is single-threaded but it can be scaled at the infrastructure level.
Eg. For imports:
it is possible to run multiple imports in parallel and then merge the databases into one, this has the con of taking some minutes to merge the files and the pro of not having to worry about threading, write locks, mutexes etc. which make the application code complex and prone to errors.
Eg. For queries:
it is possible to run multiple containers of the same image in a load-balancer setup, giving each container one CPU. The default read mode for the QueryService is mmap so the database pages are cached in the Kernel FileSystem Cache which is then shared between processes by the kernel; all this means that it does not require any additional RAM to generate additional caches for each container.
The simplest solution to this issue might be to just avoid it altogether and document how to achieve full CPU utilization while keeping the application single-core.
If that's the case, we can simplify the server/http.js code to also be single-threaded.
The http server currently spawns one thread per core, however, only a single database connection is opened with
sqlite
compiled withSQLITE_THREADSAFE 0
What this basically means is that one core will run hot when querying with the others idle.
We should either develop or document a workflow which utilizes all cores.
A similar situation applies for imports, these are also single-threaded.
One completely valid solution to this issue is to simply say that the application is single-threaded but it can be scaled at the infrastructure level.
Eg. For imports:
it is possible to run multiple imports in parallel and then merge the databases into one, this has the con of taking some minutes to merge the files and the pro of not having to worry about threading, write locks, mutexes etc. which make the application code complex and prone to errors.
Eg. For queries:
it is possible to run multiple containers of the same image in a load-balancer setup, giving each container one CPU. The default read mode for the
QueryService
ismmap
so the database pages are cached in the Kernel FileSystem Cache which is then shared between processes by the kernel; all this means that it does not require any additional RAM to generate additional caches for each container.The simplest solution to this issue might be to just avoid it altogether and document how to achieve full CPU utilization while keeping the application single-core.
If that's the case, we can simplify the
server/http.js
code to also be single-threaded.Related: #7
The text was updated successfully, but these errors were encountered: