Skip to content

fix: guarantee DB connections are closed on any exit path#44

Merged
wanqqq31 merged 2 commits into
mainfrom
fix/db-connection-cleanup
Jun 9, 2026
Merged

fix: guarantee DB connections are closed on any exit path#44
wanqqq31 merged 2 commits into
mainfrom
fix/db-connection-cleanup

Conversation

@monodera

Copy link
Copy Markdown
Contributor

Summary

  • All DB-querying functions in dbutils.py opened a connection at the start and called close() at the end, but without try/finally. An exception during query or data processing would skip the close() call, leaving idle connections on the PostgreSQL server.
  • Wrapped all four generate_*_from_targetdb() functions in try/finally so that db.close() is guaranteed on any exit (normal or exception).
  • For psycopg2 connections (gaiadb, qadb): try/finally: conn.close() + with conn.cursor() as cur: for automatic cursor cleanup.

Affected functions

Function Change
generate_targets_from_targetdb() Wrapped in try/finally
generate_fluxstds_from_targetdb() Wrapped in try/finally (including fallback Gaia query)
generate_skyobjects_from_targetdb() Wrapped in try/finally
generate_fillers_from_targetdb() Wrapped in try/finally
generate_targets_from_gaiadb() try/finally: conn.close() + cursor context manager
fixcols_filler_targetdb() qadb branch try/finally: conn.close() + cursor context manager

Related

Companion fix in ets_target_database: TargetDB.close() now also calls engine.dispose() to release pooled connections (see Subaru-PFS/ets_target_database#146).

Test plan

  • Verify that after a function call (or after an exception inside one), the connection no longer appears in pg_stat_activity
  • Run normal SFA / PPP workflow and confirm results are unchanged

🤖 Generated with Claude Code

All DB-querying functions in dbutils.py opened a connection at the
start and called close() at the end, but without try/finally. An
exception raised during query execution or data processing would
skip the close() call, leaving connections open on the server.

- Wrap all four generate_*_from_targetdb() functions in try/finally
  so that db.close() is guaranteed even on exception.
- For psycopg2 connections (gaiadb, qadb), use try/finally for
  conn.close() and "with conn.cursor() as cur:" for cursor cleanup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates database access helpers in dbutils.py to ensure database connections are reliably closed even when exceptions occur during queries or downstream DataFrame processing, reducing the risk of leaking idle PostgreSQL connections during normal workflows.

Changes:

  • Wrapped TargetDB-based query functions in try/finally to guarantee db.close() on all exit paths.
  • Updated psycopg2 usage to close connections via try/finally and manage cursors with with conn.cursor() as cur:.
  • Applied the same connection-lifetime pattern to the qaDB query path used when a cached CSV is missing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/pfs_design_tool/pointing_utils/dbutils.py
Comment thread src/pfs_design_tool/pointing_utils/dbutils.py
Comment thread src/pfs_design_tool/pointing_utils/dbutils.py
Comment thread src/pfs_design_tool/pointing_utils/dbutils.py
Comment thread src/pfs_design_tool/pointing_utils/dbutils.py
Comment thread src/pfs_design_tool/pointing_utils/dbutils.py
@wanqqq31 wanqqq31 merged commit 11fff32 into main Jun 9, 2026
@wtgee

wtgee commented Jun 9, 2026

Copy link
Copy Markdown
Member

pfs_utils has database tools that already handle all of these db connections cleanly and consistently. We should be using those in all circumstances rather than writing (yet another) databases access library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants