Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: add POSTGRES_MAX_CONNECTIONS env. #1922

Merged
merged 6 commits into from
Dec 27, 2024
Merged

perf: add POSTGRES_MAX_CONNECTIONS env. #1922

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1d04b0b
build: remove pyc compilation.
liqiang-fit2cloud Dec 26, 2024
b4834b1
Merge branch 'main' of https://github.com/maxkb-dev/maxkb
liqiang-fit2cloud Dec 26, 2024
0ba733e
Merge branch 'main' of https://github.com/maxkb-dev/maxkb
liqiang-fit2cloud Dec 26, 2024
982be95
perf: increase MAX_OVERFLOW of database connection pool.
liqiang-fit2cloud Dec 26, 2024
68897b9
Merge branch 'main' of https://github.com/maxkb-dev/maxkb
liqiang-fit2cloud Dec 26, 2024
a0b03d3
perf: add POSTGRES_MAX_CONNECTIONS env.
liqiang-fit2cloud Dec 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/smartdoc/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ def get_db_setting(self) -> dict:
"ENGINE": self.get('DB_ENGINE'),
"POOL_OPTIONS": {
"POOL_SIZE": 20,
"MAX_OVERFLOW": 80
"MAX_OVERFLOW": self.get('DB_MAX_OVERFLOW')
}
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks mostly correct for retrieving database settings from configuration data. However, there's a few points that could be improved:

Improvements

  1. Use Configurables: Instead of using get('DB_MAX_OVERFLOW') directly, consider defining a configurable variable if you plan to change it frequently. This can make the code more maintainable.

  2. Type Annotations: Add type annotations where applicable to improve readability and help with static analysis tools like mypy.

  3. Ensure Values are Positive Integers: Validate that POOL_SIZE and MAX_OVERFLOW are positive integers before proceeding.

Here's an adjusted version of the function:

from typing import Any, Dict

def get_db_setting(self) -> Dict[str, Any]:
    """
    Retrieve database settings from the configuration.
    
    Returns:
        A dictionary containing database connection settings.
    """
    return {
        "ENGINE": self.get('DB_ENGINE'),
        "POOL_OPTIONS": {
            "POOL_SIZE": int(self.get('DB_POOL_SIZE', 5)),  # Default size is 5
            "MAX_OVERFLOW": int(self.get('DB_MAX_OVERFLOW', 20))  # Default overflow is 20
        }
    }

# Example usage:
# db_settings = get_db_setting()

Optimization Suggestions

  • Early Return for Non-Configurable Values: If certain values (like POOL_SIZE) have default configurations, early returning these can reduce unnecessary processing.

This revised code includes simple types and adds some basic validation to ensure the necessary keys exist and have valid numeric values.

Expand Down
2 changes: 2 additions & 0 deletions installer/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,15 @@ ENV MAXKB_VERSION="${DOCKER_IMAGE_TAG} (build at ${BUILD_AT}, commit: ${GITHUB_C
MAXKB_DB_PORT=5432 \
MAXKB_DB_USER=root \
MAXKB_DB_PASSWORD=Password123@postgres \
MAXKB_DB_MAX_OVERFLOW=80 \
MAXKB_EMBEDDING_MODEL_NAME=/opt/maxkb/model/embedding/shibing624_text2vec-base-chinese \
MAXKB_EMBEDDING_MODEL_PATH=/opt/maxkb/model/embedding \
MAXKB_SANDBOX=1 \
LANG=en_US.UTF-8 \
PATH=/opt/py3/bin:$PATH \
POSTGRES_USER=root \
POSTGRES_PASSWORD=Password123@postgres \
POSTGRES_MAX_CONNECTIONS=1000 \
PIP_TARGET=/opt/maxkb/app/sandbox/python-packages \
PYTHONPATH=/opt/maxkb/app/sandbox/python-packages \
PYTHONUNBUFFERED=1
Expand Down
2 changes: 1 addition & 1 deletion installer/run-maxkb.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash
rm -f /opt/maxkb/app/tmp/*.pid
# Start postgresql
docker-entrypoint.sh postgres &
docker-entrypoint.sh postgres -c max_connections=${POSTGRES_MAX_CONNECTIONS} &
sleep 10
# Wait postgresql
until pg_isready --host=127.0.0.1; do sleep 1 && echo "waiting for postgres"; done
Expand Down
Loading