From e65c0a27ddd0cdb4b2362de5a0c2cb06adfd928b Mon Sep 17 00:00:00 2001 From: Cesar <275373127+sanzakicesarr@users.noreply.github.com> Date: Mon, 15 Jun 2026 23:43:18 +0200 Subject: [PATCH] feat: rotate the internal PostgreSQL logs The internal vector-db PostgreSQL redirected all output to a single "$DATA_DIR/logfile" (pg_ctl -l) that grew without bound and was never rotated, eventually filling the persistent storage on long-running instances. Drop the -l redirect and enable PostgreSQL's built-in logging collector with the documented weekday scheme: one file per day named postgresql-Mon.log through postgresql-Sun.log under "$DATA_DIR/log". Size-based rotation stays off (log_rotation_size=0): with weekday filenames PostgreSQL reopens the same file on a size rotation without truncating, so a size cap cannot bound a day's file anyway. The collector overwrites a weekday file only when it rotates while running across a day boundary, not on startup, so a container that restarts frequently would keep appending to the same-named weekday file week after week. To keep the logs bounded to about a week regardless of restart frequency, remove the legacy single logfile and any weekday log at least six days old before starting the server, so last week's file is always gone before its name is reused. This also reclaims the old unbounded logfile on upgraded installs. The options are passed on the pg_ctl command line, so they also take effect for already-initialised data directories on the next restart and need no edits to persisted configuration. Document the new log location and retention in the README. Fixes #303 Signed-off-by: Cesar <275373127+sanzakicesarr@users.noreply.github.com> --- README.md | 1 + changelog.md | 1 + dockerfile_scripts/pgsql/setup.sh | 15 +++++++++++++-- 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0cc3b450..a0303183 100644 --- a/README.md +++ b/README.md @@ -104,6 +104,7 @@ volumes: Logs are stored in the `logs/` directory in the persistent directory. In a docker container, it should be at `/nc_app_context_chat_backend/logs/`. The log file is named `ccb.log` and is set to otate at 20 MB with 10 backups. These logs are in JSONL format, i.e. each line is a valid JSON object. Now only warning and above logs are printed to the console. All the debug logs are written to the log file if `debug` is set to `true` in the config file. The logs of the embedding server are written to `logs/embedding_server_[date].log` in the persistent directory, it rotates with date change and is not in JSONL format, just raw stdout and stderr from the embedding server's process. +The internal vector database (PostgreSQL) writes its logs to `vector_db_data/pgsql/log/` in the persistent directory, one file per weekday (e.g. `postgresql-Mon.log`); each is overwritten when that weekday comes round again, so the logs are kept for about a week. ## Configuration Configuration resides inside the persistent storage as `config.yaml`. The location is `$APP_PERSISTENT_STORAGE`. By default it would be at `/nc_app_context_chat_backend_data/config.yaml` inside the container. diff --git a/changelog.md b/changelog.md index e5e5b063..b0f5e18c 100644 --- a/changelog.md +++ b/changelog.md @@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/). - add gh workflows for docker builds and do separate cpu, cuda and rocm (vulkan) images (#295) @kyteinsky ### Changed +- rotate the internal PostgreSQL logs into one file per weekday instead of a single ever-growing logfile (#312) @sanzakicesarr - update readme according to the latest changes (#300) @kyteinsky - bump llama_cpp_python to 0.3.23 (#301) @kyteinsky diff --git a/dockerfile_scripts/pgsql/setup.sh b/dockerfile_scripts/pgsql/setup.sh index 7578ed83..ccae73ce 100755 --- a/dockerfile_scripts/pgsql/setup.sh +++ b/dockerfile_scripts/pgsql/setup.sh @@ -11,7 +11,7 @@ source "$(dirname $(realpath $0))/env" if [ x"$1" == "xstop" ]; then echo -n Stopping PostgreSQL... - sudo -u postgres ${PG_BIN}/pg_ctl -D "$DATA_DIR" -l "${DATA_DIR}/logfile" -o "-p ${PG_PORT}" stop + sudo -u postgres ${PG_BIN}/pg_ctl -D "$DATA_DIR" -o "-p ${PG_PORT}" stop exit 0 fi @@ -47,8 +47,19 @@ if [ ! -d "$DATA_DIR/base" ]; then sudo -u postgres ${PG_BIN}/initdb -D "$DATA_DIR" -E UTF8 fi +# Rotate the internal PostgreSQL logs with the built-in logging collector: one +# file per weekday (postgresql-Mon.log ... postgresql-Sun.log) under +# "$DATA_DIR/log". The collector overwrites a weekday file only on time-based +# rotation, not on startup, so a frequently-restarted container would otherwise +# append to last week's same-named file. Before starting, drop the pre-rotation +# single logfile and any weekday log at least six days old (find -mtime +5), so +# last week's file is always gone before its name is reused. See #303. +rm -f "${DATA_DIR}/logfile" +find "${DATA_DIR}/log" -maxdepth 1 -type f -name 'postgresql-*.log' -mtime +5 -delete 2>/dev/null || true +PG_LOG_OPTS="-c logging_collector=on -c log_directory=log -c log_filename=postgresql-%a.log -c log_rotation_age=1d -c log_rotation_size=0 -c log_truncate_on_rotation=on" + echo "Starting PostgreSQL..." -sudo -u postgres ${PG_BIN}/pg_ctl -D "$DATA_DIR" -l "${DATA_DIR}/logfile" -o "-p ${PG_PORT}" start +sudo -u postgres ${PG_BIN}/pg_ctl -D "$DATA_DIR" -o "-p ${PG_PORT} ${PG_LOG_OPTS}" start echo "Waiting for PostgreSQL to start..." until sudo -u postgres ${PG_SQL} -c "SELECT 1" > /dev/null 2>&1; do