fix: guarantee DB connections are closed on any exit path#44
Merged
Conversation
All DB-querying functions in dbutils.py opened a connection at the start and called close() at the end, but without try/finally. An exception raised during query execution or data processing would skip the close() call, leaving connections open on the server. - Wrap all four generate_*_from_targetdb() functions in try/finally so that db.close() is guaranteed even on exception. - For psycopg2 connections (gaiadb, qadb), use try/finally for conn.close() and "with conn.cursor() as cur:" for cursor cleanup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merged
2 tasks
There was a problem hiding this comment.
Pull request overview
This PR updates database access helpers in dbutils.py to ensure database connections are reliably closed even when exceptions occur during queries or downstream DataFrame processing, reducing the risk of leaking idle PostgreSQL connections during normal workflows.
Changes:
- Wrapped TargetDB-based query functions in
try/finallyto guaranteedb.close()on all exit paths. - Updated psycopg2 usage to close connections via
try/finallyand manage cursors withwith conn.cursor() as cur:. - Applied the same connection-lifetime pattern to the qaDB query path used when a cached CSV is missing.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Member
|
pfs_utils has database tools that already handle all of these db connections cleanly and consistently. We should be using those in all circumstances rather than writing (yet another) databases access library. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dbutils.pyopened a connection at the start and calledclose()at the end, but withouttry/finally. An exception during query or data processing would skip theclose()call, leaving idle connections on the PostgreSQL server.generate_*_from_targetdb()functions intry/finallyso thatdb.close()is guaranteed on any exit (normal or exception).gaiadb,qadb):try/finally: conn.close()+with conn.cursor() as cur:for automatic cursor cleanup.Affected functions
generate_targets_from_targetdb()try/finallygenerate_fluxstds_from_targetdb()try/finally(including fallback Gaia query)generate_skyobjects_from_targetdb()try/finallygenerate_fillers_from_targetdb()try/finallygenerate_targets_from_gaiadb()try/finally: conn.close()+ cursor context managerfixcols_filler_targetdb()qadb branchtry/finally: conn.close()+ cursor context managerRelated
Companion fix in
ets_target_database:TargetDB.close()now also callsengine.dispose()to release pooled connections (see Subaru-PFS/ets_target_database#146).Test plan
pg_stat_activity🤖 Generated with Claude Code