Commit 076b7e5
authored
Speed up native AST materialization (#388)
## Summary
- reuse one `WP_MySQL_Parser` instance inside the SQLite driver and
reset its token stream per query
- add `reset_tokens()` to the PHP parser polyfill and the Rust native
parser
- restore native parser-node accessor fast paths in
`WP_MySQL_Native_Parser_Node`, while keeping PHP child materialization
for mutation
- fix the local native extension build helper for Nix/libclang bindgen
by undefining `__SSE2__` during binding generation
## Stack
This is the top PR in the native MySQL lexer/parser stack. The stack is
split so each GitHub diff shows one reviewable concern:
1. [#384 Extract MySQL lexer and parser
polyfills](#384)
- `trunk` -> `codex/native-parser-php-facade`
- extraction-only PHP refactor
- moves the existing PHP lexer/parser implementations into polyfill
classes
- keeps public `WP_MySQL_Lexer` and `WP_MySQL_Parser` as thin PHP
subclasses
2. [#385 Add optional native parser
routing](#385)
- `codex/native-parser-php-facade` ->
`codex/native-parser-class-routing`
- adds fallback `WP_MySQL_Native_*` PHP classes
- routes the public lexer/parser classes through native classes when the
Rust extension provides them
- adds the minimal PHP grammar-export bridge for the native parser
3. [#386 Add lazy native parser node
facade](#386)
- `codex/native-parser-class-routing` ->
`codex/native-parser-node-facade`
- keeps `WP_Parser_Node` as the plain PHP tree node
- adds `WP_MySQL_Native_Parser_Node extends WP_Parser_Node` for
native-backed lazy AST nodes
- keeps native AST handles and native accessor delegation out of the
base node class
4. [#381 Add lazy native AST
facade](#381)
- `codex/native-parser-node-facade` -> `codex/native-lazy-ast-facade`
- implements the Rust lexer/parser extension and lazy native AST facade
- makes the Rust extension instantiate `WP_MySQL_Native_Parser_Node`
- adds native-extension CI coverage for the SQLite driver and WordPress
PHPUnit tests
- includes the local SQLite facade smoke benchmark
5. [#387 Cache native grammar on parser grammar
object](#387)
- `codex/native-lazy-ast-facade` ->
`codex/native-parser-object-grammar-cache`
- restores the object-attached native grammar cache
- adds only `WP_Parser_Grammar::$native_grammar` on the PHP side
- removes the Rust content-hash cache that walked the whole exported
grammar on every parser construction
6. This PR, [#388 Speed up native AST
materialization](#388)
- `codex/native-parser-object-grammar-cache` ->
`codex/native-parser-bulk-materialization`
- optimizes native-to-PHP AST access after the grammar-cache performance
restoration
- reuses the SQLite driver's parser instance instead of constructing it
per query
## Why
The native lexer/parser itself is fast, but the PHP-facing path can lose
that benefit if each query repeatedly rebuilds native parser state or
forces full PHP AST materialization. On the current stack, #387 already
removes the large grammar export/hash cost. This PR removes the
remaining per-query parser construction churn and restores the native
AST accessor path for descendant-heavy SQLite driver workloads.
## Measurements
Environment: local PHP 8.2 via the native build helper, release Rust
extension, current top of this PR.
Focused constructor/reset benchmark over 5000 unique SELECT queries:
| Phase | Time |
| --- | ---: |
| native tokenize | 22.62 us/query |
| fresh native parser constructor only | 2.31 us/query |
| reusable parser `reset_tokens()` only | 0.32 us/query |
| reusable parser reset + parse + `get_descendants()` | 157.06 us/query
|
| constructor/reset ratio | 7.3x |
The previously reported ~622 us/query constructor cost does not
reproduce on this stack because #387 already caches the native grammar
on the PHP grammar object. Parser reuse still removes most of the
remaining constructor overhead.
SQLite facade smoke workload:
Command:
```bash
TMP_TEST_NATIVE_QUERY_COUNT=250 ./tmp-test-native/run.sh
```
| Workload | PHP fallback | Native extension | Speedup |
| --- | ---: | ---: | ---: |
| 250 generated queries, including 1 x 2000-row insert | 4.060s | 0.525s
| 7.73x |
## Testing
- `cargo fmt --check`
- `git diff --check`
- `composer run check-cs`
- `composer run test` from `packages/mysql-on-sqlite`
- `php -d
extension=packages/mysql-on-sqlite/ext/wp-mysql-parser/target/release/libwp_mysql_parser.so
packages/mysql-on-sqlite/vendor/bin/phpunit -c
packages/mysql-on-sqlite/phpunit.xml.dist`
- `TMP_TEST_NATIVE_QUERY_COUNT=250 ./tmp-test-native/run.sh`1 parent 09b9c1a commit 076b7e5
5 files changed
Lines changed: 164 additions & 29 deletions
File tree
- .github/workflows
- packages
- mysql-on-sqlite/src
- mysql
- sqlite
- php-ext-wp-mysql-parser/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
81 | 81 | | |
82 | 82 | | |
83 | 83 | | |
84 | | - | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
85 | 88 | | |
86 | 89 | | |
87 | 90 | | |
| |||
149 | 152 | | |
150 | 153 | | |
151 | 154 | | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
57 | | - | |
58 | | - | |
| 57 | + | |
| 58 | + | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| |||
Lines changed: 11 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
11 | 22 | | |
12 | 23 | | |
13 | 24 | | |
| |||
Lines changed: 25 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
410 | 410 | | |
411 | 411 | | |
412 | 412 | | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
413 | 420 | | |
414 | 421 | | |
415 | 422 | | |
| |||
1160 | 1167 | | |
1161 | 1168 | | |
1162 | 1169 | | |
1163 | | - | |
| 1170 | + | |
1164 | 1171 | | |
1165 | 1172 | | |
1166 | 1173 | | |
1167 | | - | |
| 1174 | + | |
| 1175 | + | |
| 1176 | + | |
| 1177 | + | |
| 1178 | + | |
| 1179 | + | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
1168 | 1191 | | |
1169 | 1192 | | |
1170 | 1193 | | |
| |||
0 commit comments