Skip to content

feat: relax name validation and harden recovery - #756

Merged
JalinWang merged 12 commits into
alibaba:mainfrom
zhourrr:feat/relax_validation
Sep 17, 2026
Merged

JalinWang merged 12 commits into
alibaba:mainfrom
zhourrr:feat/relax_validation

Conversation

@zhourrr

@zhourrr zhourrr commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Relax document ID and collection name rules with centralized validation and clearer errors.

Fix related WAL recovery, schema, decoding, and Python/C binding issues while preserving storage formats.

Fixes #395

Validation Rules

The new rules are written as below, please review this first.

Identifier Previous rules New rules
Document ID 1–64 bytes; ASCII letters, digits, and _!@#$%+=.- 1–1024 bytes; valid UTF-8, excluding C0/C1 control characters, DEL, and Unicode line/paragraph separators (U+2028/U+2029)
Collection name 3–64 bytes; ASCII letters, digits, _, and - 1–256 bytes; same UTF-8 and character restrictions as document IDs
Field name 1–32 bytes; ASCII letters, digits, _, and - 1–64 bytes; same ASCII character set, excluding five reserved names: _zvec_row_id_, _zvec_g_doc_id_, _zvec_uid_, _zvec_score, and _zvec_group_id

Identifier Validation Performance

Summary

For valid ASCII inputs accepted by both implementations, the new validators were approximately 30–63× faster than the previous std::regex checks in this microbenchmark.

Valid Inputs Accepted by Both Implementations

Input Previous (ns/call) New (ns/call) Speedup
Document ID, 8 ASCII bytes 674.91 14.70 45.91×
Document ID, 32 ASCII bytes 2,041.66 59.98 34.04×
Document ID, 64 ASCII bytes 3,563.33 110.87 32.14×
Collection name, 8 ASCII bytes 551.01 14.50 38.00×
Collection name, 32 ASCII bytes 1,796.46 60.26 29.81×
Collection name, 64 ASCII bytes 3,335.49 111.10 30.02×
Field name, 8 ASCII bytes 653.86 11.68 55.98×
Field name, 16 ASCII bytes 1,101.34 17.40 63.30×
Field name, 32 ASCII bytes 1,824.54 28.75 63.46×

Newly Supported Inputs

These inputs were rejected by the previous implementation. Only the new validation cost is reported, since the two implementations perform different work.

Input New (ns/call)
Document ID, 256 ASCII bytes 420.46
Document ID, 1,024 ASCII bytes 1,793.87
Collection name, 256 ASCII bytes 436.82
Field name, 64 ASCII bytes 50.99
Document ID, 28-byte mixed Unicode input 42.13
Collection name, 28-byte mixed Unicode input 41.04

Comment thread src/db/sqlengine/common/util.h Outdated
@zhourrr
zhourrr force-pushed the feat/relax_validation branch from cf55d83 to ccf6250 Compare September 16, 2026 14:41
JalinWang
JalinWang previously approved these changes Sep 17, 2026
@JalinWang
JalinWang merged commit 20da9ae into alibaba:main Sep 17, 2026
3 checks passed
@zhourrr
zhourrr deleted the feat/relax_validation branch September 17, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Enhance]: Relax collection, field, and document ID name validation

2 participants