Skip to content

fix: resolve logging.py issues from code audit - #255

Closed
fwh888 wants to merge 1 commit into
ONSdigital:mainfrom
fwh888:fix-logging-issues
Closed

fix: resolve logging.py issues from code audit#255
fwh888 wants to merge 1 commit into
ONSdigital:mainfrom
fwh888:fix-logging-issues

Conversation

@fwh888

@fwh888 fwh888 commented Aug 30, 2026

Copy link
Copy Markdown

Summary

This PR fixes issues found during a code audit of the core rdsa-utils modules (see issue #254).

Changes

1. Bitwise AND → Logical AND (logging.py:237)

# Before
if show_records & stop_pipeline:
# After
if show_records and stop_pipeline:

Bitwise & on boolean values works accidentally but is not Pythonic and will behave unexpectedly with non-standard boolean types (e.g., numpy.bool_).

2. try/finally for Spark cache cleanup (log_rows_in_spark_df)

# Before: kwargs['df'].unpersist() was outside the try block
# After: moved into finally block
try:
    df_return = func(*args, **kwargs)
finally:
    if kwargs.get('df') and isinstance(kwargs['df'], SparkDF):
        kwargs['df'].unpersist()

If the decorated function raises an exception, the input DataFrame's cache was never released, causing a persistent Spark memory leak. Added a type guard to avoid AttributeError on non-DataFrame inputs.

Remaining issues from audit

See issue #254 for the full audit report. Other findings (handler validation order, logger init idempotency, MEMORY_ONLY → MEMORY_AND_DISK) are higher-risk changes that require more careful review.

Three fixes in rdsa_utils/logging.py:

1. & -> and operator (line 237): Changed bitwise AND to logical AND
   in show_records & stop_pipeline condition.

2. try/finally for cache cleanup (log_rows_in_spark_df): Moved
   kwargs['df'].unpersist() into a finally block so the input
   DataFrame cache is always released, even if the decorated
   function raises an exception. Prevents Spark cache leak / OOM.

3. Added isinstance check before unpersist to avoid AttributeError
   when kwargs['df'] is not a Spark DataFrame.

See issue ONSdigital#254 for full audit report.
@coatet

coatet commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Closed, as we do not accept submissions from users outside the ONS and I suspect this may be an LLM.

@coatet coatet closed this Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants