feat: Add BAQ mode decoder, chunk info extraction & fix FDBAQ channel desync by liuzl621 · Pull Request #41 · Rich-Hall/sentinel1decoder

liuzl621 · 2026-05-28T09:17:41Z

Hi! I've made some improvements, including implementing the BAQ mode decoder and fixing a critical bug in the FDBAQ decoder.

New Features

BAQ Mode Decoder: Integrated necessary lookup table data and implemented BAQ decoding logic, fully supporting 3-bit, 4-bit, and 5-bit modes.
Chunks Info Module: Added chunks_info.py for clearer chunk information output. It displays chunk IDs in a list format, grouped by signal category, which is incredibly helpful for IW mode data analysis.

Bug Fixes

FDBAQ overread issue: Fixed an overread issue in the FDBAQ mode decoder that caused incorrect byte consumption and channel desynchronization.
Example of problematic scenes resolved:
- S1A_S3_RAW__0SDV_20160522T094430_20160522T094451_011370_011459_7607 (Previously crashed immediately during decoding)
- S1A_S3_RAW__0SDV_20151218T094429_20151218T094449_009095_00D110_0BB2 (Previously generated abnormal white spots during imaging without reporting an error)

Additional changes

Added comprehensive unit tests for test_baq_decoder_rust.py and test_chunks_info.py.
Cleaned up minor formatting details.

I have run the test suite locally and all tests passed.

I realize this PR contains both a bug fix and new feature additions. If you feel the new features (like the Chunks Info module) don't currently align with the project's roadmap, or if you prefer I split the bug fix and the new features into separate smaller PRs, please just let me know! I am completely open to modifying, splitting, or dropping any part of this to match your vision for the project.

Looking forward to your thoughts!

Rich-Hall · 2026-05-28T14:46:09Z

👀

This looks great, thank you! It'll take me a bit of time to properly review, but I wanted to send a holding response.

liuzl621 · 2026-05-28T15:00:39Z

Take your time with the review, there's absolutely no rush. Feel free to let me know whenever you're ready or if you need any clarification on the changes.

Rich-Hall

Looks good, that bug in particular must have been quite hard to find and fix! I think we need to fix a couple of things before merging:

Please remove termcolor imports and calls - coloring log text is not worth adding a new project dependency, IMO.
We should discuss the chunks_info.py functions - I think these duplicate some logic that already exists in Level0File.iter_chunks_matching(...), so I'd like to understand better the justification / use-case for these.

Otherwise, it's just a few nits around docstrings.

Rich-Hall · 2026-06-19T20:39:55Z

-        if let Some(&byte) = data.get(*byte_idx) {
-            boundary_state_bits = (state.state_bits as u32) << 8 | byte as u32;
-            boundary_state_len = state.state_len + 8;
-            *byte_idx += 1;
+
+        // Determine how many bits we strictly need to read the header for this block.
+        // - BRC needs 3 bits
+        // - THIDX needs 8 bits
+        // - Other channels (IO, QO) need 0 bits (no header)
+        let bits_needed_for_header = if read_brc { 3 } else if read_thidx { 8 } else { 0 };
+
+        // CRITICAL FIX: Only fetch a new byte from `data` if the leftover bits in `state` are  
+        // strictly insufficient for the header. If we unconditionally fetch a byte here (especially  
+        // on the last block of a channel), we might consume a byte that actually belongs to the NEXT  
+        // channel. Since `state`is discarded at the end of `decode_channel`, that over-read byte 
+        // would be lost forever, causing the next channel to lose its synchronization.
+        if state.state_len < bits_needed_for_header {
+            if let Some(&byte) = data.get(*byte_idx) {
+                boundary_state_bits = (state.state_bits as u32) << 8 | byte as u32;
+                boundary_state_len = state.state_len + 8;
+                *byte_idx += 1;
+            } else {
+                boundary_state_bits = state.state_bits as u32;
+                boundary_state_len = state.state_len;
+            }


Great find! Thank you for fixing this.

Rich-Hall · 2026-06-19T20:40:16Z

+
+pub fn decode_single_baq_packet_inner(data: &[u8], num_quads: usize, baq_bits: u8) -> Result<Vec<Complex32>, String> {


Could you add a docstring for this function?

Sure, I'll add a docstring for this function.

Rich-Hall · 2026-06-19T20:42:48Z

+#[pyfunction]
+fn decode_single_baq_packet(
+    data: &[u8],
+    num_quads: usize,
+    baq_bits: usize,
+    py: Python,
+) -> PyResult<Py<PyAny>> {
+    let complex_samples = decode_single_baq_packet_inner(data, num_quads, baq_bits as u8)
+        .map_err(|e| PyValueError::new_err(e))?;
+
+    Ok(complex_samples.into_pyarray(py).to_owned().into())
+}
+
+#[pyfunction]
+fn decode_batched_baq_packets(
+    packets: &Bound<'_, PyList>,
+    num_quads: usize,
+    baq_bits: usize,
+    py: Python,
+) -> PyResult<Py<PyAny>> {
+    decode_batched_packets_helper(packets, num_quads, py, |packet_data, nq| {
+        decode_batched_baq_packets_inner(packet_data, nq, baq_bits as u8)
+    })
+}


Let's add some docstrings to these as well.

Rich-Hall · 2026-06-19T20:44:00Z

+def decode_single_baq_packet(
+    data: bytes,
+    num_quads: int,
+    baq_bits: int,
+) -> np.ndarray:
+    ...
+
+def decode_batched_baq_packets(
+    packets: List[bytes],
+    num_quads: int,
+    baq_bits: int,
+) -> np.ndarray:
+    ...
+


Docstrings for these as well please!

Rich-Hall · 2026-06-19T20:55:21Z

@@ -0,0 +1,171 @@
+from termcolor import colored


This introduces an additional project dependency - termcolor is not currently a package requirement. I'm not convinced we need to introduce a new dependency for these purposes - is there a strong reason you'd like this to be included?

Got it, I’ll remove this termcolor import —it was just for temporary log highlighting.👌

Rich-Hall · 2026-06-19T21:10:33Z

+def extract_SM_chunk_info(
+        inputfile: str,
+        visualize: bool = True
+    ) -> tuple["Level0File", dict[str, list]]:
+    """
+    Extract the data block information of the Sentinel-1 SM mode
+    Important Note:
+    When visualize=True, the function will output logs using logger.info.
+    Please configure logging before calling the function
+    """


I think this function duplicates a lot of logic that already exists using the iter_chunk_matching function on Level0File. For example, you can get each of this function's results using the following:

list(l0file.iter_chunks_matching(signal_type=SignalType.ECHO)) list(l0file.iter_chunks_matching(signal_type=SignalType.NOISE)) list(l0file.iter_chunks_matching(signal_type=SignalType.RX_CAL)) list(l0file.iter_chunks_matching(signal_type=SignalType.TX_CAL)) list(l0file.iter_chunks_matching(signal_type=SignalType.EPDN_CAL)) list(l0file.iter_chunks_matching(signal_type=SignalType.APDN_CAL_S1AB_ONLY)) list(l0file.iter_chunks_matching(signal_type=SignalType.TA_CAL_OR_TX_CAL_ISO)) list(l0file.iter_chunks_matching(signal_type=SignalType.TXH_CAL_ISO_S1AB_ONLY))

I wasn't familiar with this existing API before. I'll refactor this to reuse iter_chunks_matching instead.🤣

Rich-Hall · 2026-06-19T21:19:24Z

+def extract_IW_chunk_info(
+        inputfile: str,
+        visualize: bool = True
+    ) -> tuple["Level0File", dict[str, list]]:
+    """
+    Extract the data block information of the Sentinel-1 IW mode
+    Important Note:
+    When visualize=True, the function will output logs using logger.info.
+    Please configure logging before calling the function
+    """


Similarly, there's a lot of overlap with logic that already exists here too. Most of the buckets map directly:

list(l0file.iter_chunks_matching(signal_type=SignalType.NOISE, swath_num=10)) list(l0file.iter_chunks_matching(signal_type=SignalType.NOISE, swath_num=11)) list(l0file.iter_chunks_matching(signal_type=SignalType.NOISE, swath_num=12)) list(l0file.iter_chunks_matching(signal_type=SignalType.RX_CAL)) list(l0file.iter_chunks_matching(signal_type=SignalType.TX_CAL)) list(l0file.iter_chunks_matching(signal_type=SignalType.EPDN_CAL)) list(l0file.iter_chunks_matching(signal_type=SignalType.APDN_CAL_S1AB_ONLY)) list(l0file.iter_chunks_matching(signal_type=SignalType.TA_CAL_OR_TX_CAL_ISO)) list(l0file.iter_chunks_matching(signal_type=SignalType.TXH_CAL_ISO_S1AB_ONLY)) list(l0file.iter_chunks_matching(signal_type=SignalType.ECHO, swath_num=10)) list(l0file.iter_chunks_matching(signal_type=SignalType.ECHO, swath_num=11)) list(l0file.iter_chunks_matching(signal_type=SignalType.ECHO, swath_num=12))

We don't currently have a way of listing skipped packets, because iter_chunks_matching doesn't have a filter for number of packets. I think it might be better to add the logic there, though, rather than introducing a new API surface here. Do you have a strong reason for having these utility functions exist separately here?

You're right, these two new functions are redundant now that I know about this existing API. 😢
It makes sense to add the skipped packets listing feature directly to iter_chunks_matching.

Instead of two standalone functions, I suggest adding a single get_chunks_summary() method on Level0File.

It does a single O(N) pass to group all chunks by type & swath, avoiding the O(n×N) cost of calling iter_chunks_matching 10+ times for a full overview. It complements the existing API, needs no new files.

Would you be happy with this approach?😁

Rich-Hall · 2026-06-19T21:20:33Z

 )
 from sentinel1decoder.enums import BaqMode

+from termcolor import colored


Similarly to above, I don't think we should add a new package dependency just to color log text.

liuzl621 · 2026-06-20T14:49:04Z

Thanks for the review and feedback!
Good points. I'll take some time to go through the comments properly and update the PR.

liuzl621 · 2026-06-24T06:27:43Z

I noticed the CI / build-and-verify-installability (3.8) check keeps failing across my pushes. I haven't touched any configuration files like pyproject.toml. This is likely because Python 3.8 has been dropped from the manylinux images: History for README.rst - pypa/manylinux
🤔

Rich-Hall · 2026-08-20T03:35:11Z

Hi again - I've updated the project dependencies and took the liberty of rebasing your branch here onto that change to fix CI. Hope that's okay. I'm going to get this merged in shortly.

Rich-Hall

Looks like all feedback has been addressed - approving! Thanks. Going to make a couple of additional tweaks after merge here before releasing new version.

Rich-Hall · 2026-08-20T15:50:03Z

Just a quick follow-up to say that I've released v2.1.0 now, which features your changes. I've credited you for these in the changelog.md. Thanks again! 😁

liuzl621 · 2026-08-21T01:35:24Z

Thanks for merging and crediting me! Glad my changes could help.😊

Rich-Hall requested changes Jun 19, 2026

View reviewed changes

Rich-Hall mentioned this pull request Aug 20, 2026

Issue Reading Stripmap Chunks Using Sentine1Decoder v2.0 #42

Closed

liuzl621 added 20 commits August 20, 2026 03:20

first submission

ff0774c

test: ensure the iw_chunk_info() be used normally

6760e95

feat: add BAQ decoding logic

ea5fd11

style: optimize the visual effect

3c39150

fix(fdbaq): avoid over-reading boundary bytes in channel decoding

382b3d8

refactor: chunk info extraction logic

4615b2a

style: show the currently used decoder version

1b794e8

fix: fill in the correct data into the LUT of the BAQ mode

1ef46c0

test: add the test_xx.py to test the baq decoder

db6a1f2

a minor adjustment

17d17d2

chore(deps): remove termcolor dependency for color output

c2ef9b3

refactor(chunks_info): move the code logic to l0file

cff18ed

chore:remove chunks_info.py after migrating logic to Level0File

d7dc540

docs(baq): add docstrings for all BAQ decoding related logic

70cf8df

refactor(baq): extract similar code blocks into a shared subfunction

113a459

test(baq): optimize test code and add new test cases

8f486d1

chore: adjust code style for pr check

5ef6241

chore: adjust code style for pr check 1

1b162c4

chore: adjust code style for pr check 2

aab66cb

chore: adjust code style for pr check 3

915a592

Rich-Hall force-pushed the lzl_functions branch from 91ba75a to 915a592 Compare August 20, 2026 03:26

Rich-Hall approved these changes Aug 20, 2026

View reviewed changes

Rich-Hall merged commit cf8b917 into Rich-Hall:main Aug 20, 2026
9 checks passed


		pub fn decode_single_baq_packet_inner(data: &[u8], num_quads: usize, baq_bits: u8) -> Result<Vec<Complex32>, String> {

Conversation

liuzl621 commented May 28, 2026

New Features

Bug Fixes

Additional changes

Uh oh!

Rich-Hall commented May 28, 2026

Uh oh!

liuzl621 commented May 28, 2026

Uh oh!

Rich-Hall left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liuzl621 commented Jun 20, 2026

Uh oh!

liuzl621 commented Jun 24, 2026

Uh oh!

Rich-Hall commented Aug 20, 2026

Uh oh!

Rich-Hall left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Rich-Hall commented Aug 20, 2026

Uh oh!

liuzl621 commented Aug 21, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants