Skip to content

Lock in the duplicate (id, ts) guard (#304) - #574

Merged
GermanBluefox merged 1 commit into
masterfrom
fix/duplicate-key-guard-tests
Oct 3, 2026
Merged

GermanBluefox merged 1 commit into
masterfrom
fix/duplicate-key-guard-tests

Conversation

@GermanBluefox

Copy link
Copy Markdown
Contributor

Follow-up to #573. #304 is already fixed too — this adds the tests that were missing, and the executable proof on the engine it was reported against.

#304 is not #294

The comment on #294 says they are the same report. They are not:

#294 #304
error SQL syntax error primary key violation
cause two INSERT in one query two rows with the same (id, ts) in one INSERT

In #304 the same datapoint is logged from two sources that land on the same millisecond — ts 1681102802872 appears twice for id 4, once from source 2 and once from source 3.

Current state

dialect guard
MySQL ON DUPLICATE KEY UPDATE id=id
PostgreSQL ON CONFLICT DO NOTHING
SQLite ON CONFLICT DO NOTHING
MS SQL none — and that is correct

The MS SQL exception is not an oversight: init() creates CREATE INDEX i_id on ...(id, ts), a plain non-unique index rather than a primary key, so no uniqueness conflict can arise. Nothing tested any of this.

What the tests pin

test/testInsertStatements.js:

  • the guard per dialect, on all three keyed tables — including the deliberate absence for MS SQL, so that someone levelling the dialects does not add one there;
  • DO NOTHING against DO UPDATE. The duplicate sits inside one statement. DO NOTHING copes; DO UPDATE answers with "cannot affect row a second time". Making the last value win is the obvious improvement here, and it would bring the bug straight back;
  • the reported batch inserting against a live SQLite schema with the real PRIMARY KEY(id, ts).

Executable proof on MariaDB

Since #304 was reported on MariaDB, that dialect gets a real check rather than a string match. test/testMySQLInsertDuplicates.js talks to the server directly — no js-controller, no adapter instance, about a second — in its own throwaway database:

  1. the reported batch inserts cleanly, 3 rows from 4 input rows, the first value for a timestamp winning;
  2. the same batch without the guard still fails with ER_DUP_ENTRY: Duplicate entry '4-1681102802872' for key 'PRIMARY' — the exact error from the issue;
  3. a genuinely broken statement still raises ER_BAD_FIELD_ERROR.

Case 3 is the answer to #407, which proposes INSERT IGNORE: that would swallow it. The code already says so in a comment; now a test holds it.

Verified against MariaDB 10.11.14 — 3 passing with a server, 3 pending without, so npx mocha 'test/testMySQL*.js' stays usable on a machine with none.

Wiring

No workflow change needed: the MySQL job's existing test/testMySQL*.js glob picks it up, and it sorts after testMySQLExistingNoNulls.js, so the documented ordering of the existing-data tests is untouched. It requires only build/lib/mysql, never build/main.js — the constraint from #567.

184 unit tests pass, prettier --check clean.

#294, #304 and #407 can all be closed.

🤖 Generated with Claude Code

#304 is a different bug from #294 despite the comment there: not two statements
in one query, but two rows with the same (id, ts) in one INSERT - the same
datapoint logged from two sources that landed on the same millisecond - which
aborted the whole batch with a primary key violation.

It is already fixed. MySQL carries `ON DUPLICATE KEY UPDATE id=id`, PostgreSQL
and SQLite `ON CONFLICT DO NOTHING`. MS SQL has nothing, and that is correct:
init() creates a plain `CREATE INDEX i_id on ...(id, ts)` rather than a primary
key, so no uniqueness conflict can arise there. Nothing tested any of it.

test/testInsertStatements.js now asserts the guard per dialect, including the
deliberate absence for MS SQL, so that someone levelling the dialects does not
add one there. It also pins DO NOTHING against DO UPDATE: the duplicate sits
*inside* one statement, which DO NOTHING handles but DO UPDATE answers with
"cannot affect row a second time" - the obvious way to make the last value win
would bring the bug straight back.

Since the issue was reported on MariaDB, that dialect gets an executable check
rather than a string match. test/testMySQLInsertDuplicates.js talks to the
server directly - no js-controller, no adapter instance - creates its own
throwaway database, and covers three things: the reported batch inserts
cleanly, the same batch without the guard still fails with ER_DUP_ENTRY
'4-1681102802872', and a genuinely broken statement still raises
ER_BAD_FIELD_ERROR. That last one is why the guard is not INSERT IGNORE, as
proposed in #407: IGNORE would swallow it.

It is picked up by the MySQL job's existing `test/testMySQL*.js` glob and sorts
after testMySQLExistingNoNulls.js, so the documented order of the
existing-data tests is unaffected. Without a reachable server the cases skip
with the reason, so the glob stays usable locally.

Verified against MariaDB 10.11.14: 3 passing with a server, 3 pending without.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@GermanBluefox
GermanBluefox merged commit 3021c2a into master Oct 3, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant