Skip to content

Mask noise-only channels in cleaning - #399

Merged
e-koch merged 3 commits into
masterfrom
skip-noise-channels
Nov 18, 2026
Merged

e-koch merged 3 commits into
masterfrom
skip-noise-channels

Conversation

@thomaswilliamsastro

@thomaswilliamsastro thomaswilliamsastro commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator

This PR adds a function to test if channels in a cube are noise-only, and if so to mask them in the clean mask and skip through multiscale/singlescale cleaning.

There are two checks that go on here: the first is that the sample-size-independent departure score (calculated from skew and kurtosis of the data) is above a sigma-threshold to a null test. The second is that the Jarque-Bera statistic is above a sigma-threshold. There is a little complication here that statistics need to be corrected for the effective number of independent measurements (number of beams), but this is all handled within the code.

By default, this check is off. There's one exposed parameter, channel_noise_sigma_threshold, for which a value of 3 seems reasonable. Lower values are more stringent (i.e. will say there's signal in the channel at lower significance).

Speed up is good! For an example 7m single field, processing time goes from 367s to 211s. For a 7m mosaic, it's 4.2h down to 2.4h. So about twice as fast.

The cubes are also basically identical. Plots attached for single field and mosaic, the blue show the median of the data and 1/99th percentiles, the black the same for the difference between the two cubes. Any deviations are well within the noise of the data.

ngc5236_1_7m_co21 ngc5236_1_flux_ref_7m_co21

Across these two cases, if we take the 99th percentile of the absolute differences of the cubes, the differences are (as a sigma value):

  • Mosaic field: 0.39sigma
  • Single field: 0.55sigma

this is well within the typical noise. Taking the maximum of the residuals, this rises to 3.21/1.64sigma. Given the number of effective samples in the data, a purely Gaussian field should have maximum outliers of 4.47/3.85sigma, so we're within this tolerance.

Note that this test becomes more stringent with number of effective samples (effectively number of pixels in the image). So these tests on 7m data should really be the lower limits on how strict the tests are.

Changes:

  • Added functions to casaMaskingRoutines to calculate and mask noise-only channels
  • Add task_mask_noise_channels function to handlerImagingChunked to mask noise channels, post-dirty beam but pre-multiscale
  • Add skip_multiscale_if_mask_empty to handlerImagingChunked, which mimics the existing skip_singlescale_if_mask_empty. True by default
  • Add task_mask_noise_channels function to handlerImaging to mask noise channels, post-dirty beam but pre-multiscale
  • Update CHANGELOG.md

@codecov

codecov Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 8.25688% with 200 lines in your changes missing coverage. Please review.
✅ Project coverage is 12.40%. Comparing base (b96bbc6) to head (39e00f6).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #399      +/-   ##
==========================================
- Coverage   12.45%   12.40%   -0.06%     
==========================================
  Files          43       43              
  Lines       16980    17190     +210     
  Branches     4028     4068      +40     
==========================================
+ Hits         2115     2132      +17     
- Misses      14631    14824     +193     
  Partials      234      234              
Components Coverage Δ
casa 4.93% <9.39%> (+0.12%) ⬆️
handler 14.38% <5.79%> (-0.08%) ⬇️
spectral-cube 10.16% <ø> (ø)
utils 20.28% <ø> (ø)

@thomaswilliamsastro
thomaswilliamsastro force-pushed the skip-noise-channels branch 5 times, most recently from 263984d to a202312 Compare October 1, 2026 16:02
@thomaswilliamsastro
thomaswilliamsastro force-pushed the skip-noise-channels branch 2 times, most recently from 1d90814 to 8b5c6da Compare October 7, 2026 07:08
@thomaswilliamsastro

Copy link
Copy Markdown
Collaborator Author

@low-sky I've added in some numbers here to try and quantify some worst-case outliers we see, and how they compare to what we'd expect just given a random noise field. The tl;dr here is that within the noise of the cube, these two methods are identical

@thomaswilliamsastro
thomaswilliamsastro force-pushed the skip-noise-channels branch 3 times, most recently from d8d2240 to 62edab1 Compare October 21, 2026 08:06
@thomaswilliamsastro
thomaswilliamsastro force-pushed the skip-noise-channels branch 2 times, most recently from 54fd3b0 to 4e5c4d5 Compare November 1, 2026 07:42
@thomaswilliamsastro
thomaswilliamsastro force-pushed the skip-noise-channels branch 3 times, most recently from 778aece to 7438994 Compare November 11, 2026 15:13
This PR adds a function to test if channels in a cube are noise-only, and if so to mask them in the clean mask and skip through multiscale/singlescale cleaning.

There are two checks that go on here: the first is that the sample-size-independent departure score (calculated from skew and kurtosis of the data) is above a sigma-threshold to a null test. The second is that the Jarque-Bera statistic is above a sigma-threshold. There is a little complication here that statistics need to be corrected for the effective number of independent measurements (number of beams), but this is all handled within the code.

By default, this check is off. There's one exposed parameter, channel_noise_sigma_threshold, for which a value of 3 seems reasonable. Lower values are more stringent (i.e. will say there's signal in the channel at lower significance).

Speed up is good! For an example 7m single field, processing time goes from 367s to 211s. For a 7m mosaic, it's 4.2h down to 2.4h. So about twice as fast.

The cubes are also basically identical. Plots attached for single field and mosaic, the blue show the median of the data and 1/99th percentiles, the black the same for the difference between the two cubes. Any deviations are well within the noise of the data.

- Added functions to casaMaskingRoutines to calculate and mask noise-only channels
- Add `task_mask_noise_channels` function to handlerImagingChunked to mask noise channels, post-dirty beam but pre-multiscale
- Add `skip_multiscale_if_mask_empty` to handlerImagingChunked, which mimics the existing `skip_singlescale_if_mask_empty`. True by default
- Add `task_mask_noise_channels` function to handlerImaging to mask noise channels, post-dirty beam but pre-multiscale
- Update CHANGELOG.md

@e-koch e-koch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few questions, mostly on places where I think we can avoid reading in additional copies of the data.

Comment thread phangsPipeline/casaMaskingRoutines.py Resolved
Comment thread phangsPipeline/casaMaskingRoutines.py Resolved
Comment thread phangsPipeline/handlerImagingChunked.py Resolved
@e-koch

e-koch commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

@thomaswilliamsastro -- how well is the default checks for this masking working for line wings? I'm wondering whether we could overly mask the blue/red shifted sides of the target's velocity range

@thomaswilliamsastro

Copy link
Copy Markdown
Collaborator Author

@e-koch as far as I can tell it's fine. The test is pretty conservative, and the fluxes are identical to the noise level. There might be a case for testing with something like broad+very diffuse structure, but I suspect then our regular clean masking would fail anyway

I think the comments are addressed. I realised the rebase removed actually passing the parameters along, so that's now fixed as well

@thomaswilliamsastro

Copy link
Copy Markdown
Collaborator Author

And just for a bit of added safety, we now do a couple of rounds of binary dilation to make sure we capture potential wings.

@e-koch
e-koch merged commit 34d000b into master Nov 18, 2026
7 of 8 checks passed
@thomaswilliamsastro
thomaswilliamsastro deleted the skip-noise-channels branch November 18, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants