Skip to content

Reject a tuple whose column count disagrees with the relation - #103

Open
lukashes wants to merge 1 commit into
mainfrom
bugfix/converter-column-count-guard
Open

Reject a tuple whose column count disagrees with the relation#103
lukashes wants to merge 1 commit into
mainfrom
bugfix/converter-column-count-guard

Conversation

@lukashes

Copy link
Copy Markdown
Owner

Closes #98.

Problem

tupleToRowData indexes rel_info.columns[i] for every tuple column without checking the two agree in length. A decoded tuple that is wider than the registered relation reads past the columns slice: a panic under ReleaseSafe (release Docker image), undefined behavior under ReleaseFast (load stand), where it can silently corrupt the event sent to Kafka.

Solution

  • Guard tupleToRowData with tuple.columns.len == rel_info.columns.len and return error.ColumnCountMismatch otherwise. pgoutput always sends one TupleData column per live relation column (non-identity columns in a key tuple come through as null, not dropped), so an inequality is a real desync, and the error propagates to ConversionFailed, which fails the process for the supervisor to restart.
  • Fix a latent leak the new error path exposed: convert built the event metadata before the row, so a failing row build leaked the three metadata strings (the event's data is undefined, so it can't be deinit'd). Metadata now has an errdefer, and UPDATE frees the already-built new row if the old row fails.

Test

Unit test: a registered 2-column relation with a 1-column INSERT tuple returns error.ColumnCountMismatch and leaks nothing. Integration and e2e (real pgoutput INSERT/UPDATE/DELETE) stay green, confirming the strict check does not reject valid streams.

@github-actions

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

Current run is the minimum over 3 passes, compared against the base branch (main @ a93ff4b), built on the same runner.

Benchmark Baseline Current Δ Time Allocs Status
Converter DELETE 126.68μs 126.57μs -0.1% 12 → 12 ➡️
Converter INSERT 121.52μs 121.31μs -0.2% 12 → 12 ➡️
Converter UPDATE 172.91μs 175.67μs +1.6% 21 → 21 ➡️
JsonSerializer 41.53μs 41.59μs +0.1% 3 → 3 ➡️
KafkaProducer produce 277.04μs 281.92μs +1.8% 1 → 1 ➡️
KafkaProducer sendMessage 385.02μs 346.43μs -10.0% 0 → 0 🟢 faster
PgOutputDecoder 94.00μs 93.62μs -0.4% 6 → 6 ➡️
getPartitionKeyValue boolean 16.14μs 16.34μs +1.2% 1 → 1 ➡️
getPartitionKeyValue integer 50.46μs 50.62μs +0.3% 3 → 3 ➡️
getPartitionKeyValue not found 0.03μs 0.03μs +0.0% 0 → 0 ⚪ noise
getPartitionKeyValue string 18.19μs 18.30μs +0.6% 1 → 1 ➡️
matchStreams found 17.33μs 16.40μs -5.4% 1 → 1 ➡️
matchStreams not found 0.05μs 0.05μs +0.0% 0 → 0 ⚪ noise

Summary: 🟢 1 faster · ➡️ 10 neutral · ⚪ 2 ignored (sub-μs)

Thresholds: <1μs ignore · 1–20μs 15% · 20–50μs 10% · ≥50μs 5%. Measured on a shared CI runner — treat small deltas as noise. Informational only; this check never fails the build.

@lukashes

lukashes commented Jul 12, 2026

Copy link
Copy Markdown
Owner Author

Parking this: the value is questionable.

The guard catches our own decoder bug (a cursor desync), but internal fragility already fails fast — under ReleaseSafe (the release Docker image) an out-of-bounds read panics and the process restarts. In production the guard adds nothing.

It also hurts diagnostics: a ReleaseSafe panic gives a stack trace next to the bug, while the guard catches the symptom far away in the converter and turns it into a crash loop with an unhelpful message.

Data is only at risk under ReleaseFast on the load stand, which is a test tool. If that matters, building the load stand with ReleaseSafe is cleaner than a runtime guard.

@lukashes lukashes added the marinating Deliberately paused to settle before deciding label Jul 12, 2026
@lukashes lukashes added this to the v0.3.0: GA release milestone Jul 18, 2026
@lukashes lukashes removed the marinating Deliberately paused to settle before deciding label Jul 18, 2026
@lukashes

Copy link
Copy Markdown
Owner Author

Moved to v0.4.0. The release Docker image builds with ReleaseSafe, so on the GA artifact the unguarded path panics and restarts instead of corrupting data: the guard is a diagnosability upgrade there, not a safety fix. The UB exposure is limited to ReleaseFast builds (load stand), which check-gaps monitors.

Needs a rebase before merging: #114 rewrote the same converter functions (convert() takes lsn and lost the io parameter, buildMetadata moved into the struct and now allocates meta.lsn). Note for the rebase: freeMetadata must free the new lsn string too, or the error path leaks it.

@lukashes lukashes added the marinating Deliberately paused to settle before deciding label Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

marinating Deliberately paused to settle before deciding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Converter: tuple column count is not checked against relation metadata

1 participant