Skip to content

Commit c43113d

Browse files
adamzieladamziel
authored
Add native Rust-based MySQL parser extension (#381)
## What it does Adds an optional Rust PHP extension for the MySQL lexer/parser. When the extension is loaded, the existing public PHP API stays the same, but `WP_MySQL_Lexer` and `WP_MySQL_Parser` delegate lexing/parsing to native Rust code. ```bash php -d extension=/path/to/libwp_mysql_parser.so your-script.php ``` ```php require 'packages/mysql-on-sqlite/src/load.php'; $driver = new WP_PDO_MySQL_On_SQLite( 'mysql-on-sqlite:path=:memory:;dbname=wp;' ); $parser = $driver->create_parser( 'SELECT ID, post_title FROM wp_posts WHERE ID IN (1, 2, 3)' ); $parser->next_query(); $ast = $parser->get_query_ast(); echo $ast->rule_name; // query ``` Without the extension, the same code uses the existing PHP parser. ## Rationale The PHP parser is correct but expensive on large query sets. On the MySQL corpus, the native path measured against trunk's PHP implementation was: | Scenario | Trunk PHP | Native extension | | --- | ---: | ---: | | Parse only | 14.3114s / 4,862 QPS / 68.0MB | 1.1574s / 60,105 QPS / 30.0MB | | Parse + walk | 20.0804s / 3,465 QPS / 70.0MB | 2.9751s / 23,383 QPS / 48.0MB | That is about **12.37x faster** for parse-only and **6.75x faster** for parse+walk. Raw numbers are in #381 (comment). ## Implementation The extension lives in `packages/php-ext-wp-mysql-parser/` and exports `WP_MySQL_Native_Lexer`, `WP_MySQL_Native_Token_Stream`, `WP_MySQL_Native_Parser`, and `WP_MySQL_Native_Parser_Node`. The SQLite driver selects the native path only when the native lexer class is active: ```php $lexer = new WP_MySQL_Lexer( $sql ); if ( $lexer instanceof WP_MySQL_Native_Lexer ) { $tokens = $lexer->native_token_stream(); } else { $tokens = $lexer->remaining_tokens(); } ``` Native AST nodes are lazy PHP wrappers over a Rust-owned AST. Wrapper identity is stable through a per-AST cache, and Rust state is stored in a Rust-side registry keyed by the PHP wrapper object pointer. That avoids the previous PHP/Rust reference cycle while keeping repeated child reads referentially stable. The pure-PHP parser remains the fallback and `WP_MySQL_Parser` remains an `instanceof WP_Parser`. ## Testing instructions Run the PHP suite normally: ```bash cd packages/mysql-on-sqlite php ./vendor/bin/phpunit -c ./phpunit.xml.dist ``` Run it against the extension: ```bash cd packages/php-ext-wp-mysql-parser cargo build cd ../mysql-on-sqlite WP_SQLITE_REQUIRE_NATIVE_PARSER_EXTENSION=1 \ php -d extension=../php-ext-wp-mysql-parser/target/debug/libwp_mysql_parser.so \ ./vendor/bin/phpunit -c ./phpunit.xml.dist ``` CI currently passes on `bde34d5` for PHP 7.2-8.5, including extension-loaded SQLite integration tests on PHP 8.0-8.5 and WordPress PHPUnit with the extension loaded.
1 parent fa5a7ba commit c43113d

26 files changed

Lines changed: 10567 additions & 32 deletions

‎.github/workflows/mysql-parser-extension-tests.yml‎

Lines changed: 263 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,263 @@
1+
name: MySQL Parser Extension Tests
2+
3+
on:
4+
push:
5+
branches:
6+
- trunk
7+
paths:
8+
- '.github/workflows/mysql-parser-extension-tests.yml'
9+
- 'packages/mysql-on-sqlite/**'
10+
- 'packages/php-ext-wp-mysql-parser/**'
11+
pull_request:
12+
paths:
13+
- '.github/workflows/mysql-parser-extension-tests.yml'
14+
- 'packages/mysql-on-sqlite/**'
15+
- 'packages/php-ext-wp-mysql-parser/**'
16+
workflow_dispatch:
17+
18+
concurrency:
19+
group: ${{ github.workflow }}-${{ github.ref }}
20+
cancel-in-progress: true
21+
22+
jobs:
23+
extension-tests:
24+
name: PHP ${{ matrix.php }} / ${{ matrix.coverage }} / ubuntu-latest
25+
runs-on: ubuntu-latest
26+
timeout-minutes: 30
27+
strategy:
28+
fail-fast: false
29+
matrix:
30+
include:
31+
- php: '7.2'
32+
sqlite: '3.27.0'
33+
native: false
34+
coverage: SQLite integration
35+
- php: '7.3'
36+
sqlite: '3.31.1'
37+
native: false
38+
coverage: SQLite integration
39+
- php: '7.4'
40+
sqlite: '3.34.1'
41+
native: false
42+
coverage: SQLite integration
43+
- php: '8.0'
44+
sqlite: '3.37.0'
45+
native: true
46+
coverage: SQLite integration + Rust extension
47+
- php: '8.1'
48+
sqlite: '3.40.1'
49+
native: true
50+
coverage: SQLite integration + Rust extension
51+
- php: '8.2'
52+
sqlite: '3.45.1'
53+
native: true
54+
coverage: SQLite integration + Rust extension
55+
- php: '8.3'
56+
sqlite: '3.46.1'
57+
native: true
58+
coverage: SQLite integration + Rust extension
59+
- php: '8.4'
60+
sqlite: '3.51.2'
61+
native: true
62+
coverage: SQLite integration + Rust extension
63+
- php: '8.5'
64+
sqlite: latest
65+
native: true
66+
coverage: SQLite integration + Rust extension
67+
68+
steps:
69+
- name: Checkout repository
70+
uses: actions/checkout@v4
71+
72+
- name: Set up SQLite
73+
run: |
74+
VERSION='${{ matrix.sqlite }}'
75+
if [ "$VERSION" = 'latest' ]; then
76+
TAG='release'
77+
else
78+
TAG="version-${VERSION}"
79+
fi
80+
SQLITE_SOURCE="https://sqlite.org/src/tarball/sqlite.tar.gz?r=${TAG}"
81+
SQLITE_MIRROR="https://github.com/sqlite/sqlite/archive/refs/tags/${TAG}.tar.gz"
82+
DOWNLOADED=0
83+
for url in "$SQLITE_SOURCE" "$SQLITE_MIRROR"; do
84+
for attempt in 1 2 3 4 5; do
85+
if wget -O sqlite.tar.gz "$url"; then
86+
DOWNLOADED=1
87+
break 2
88+
fi
89+
if [ "$attempt" -lt 5 ]; then
90+
sleep $(( attempt * 10 ))
91+
fi
92+
done
93+
done
94+
if [ "$DOWNLOADED" -ne 1 ]; then
95+
exit 1
96+
fi
97+
tar xzf sqlite.tar.gz
98+
if [ ! -d sqlite ]; then
99+
SQLITE_DIR=$(find . -maxdepth 1 -type d -name 'sqlite-*' | head -n 1)
100+
if [ -z "$SQLITE_DIR" ]; then
101+
exit 1
102+
fi
103+
mv "$SQLITE_DIR" sqlite
104+
fi
105+
cd sqlite
106+
./configure --prefix=/usr/local CFLAGS="-DSQLITE_ENABLE_COLUMN_METADATA -DSQLITE_ENABLE_FTS5 -DSQLITE_USE_URI -DSQLITE_ENABLE_JSON1" LDFLAGS="-lm"
107+
make -j$(nproc)
108+
sudo make install
109+
sudo ldconfig
110+
111+
- name: Set up PHP
112+
uses: shivammathur/setup-php@v2
113+
with:
114+
php-version: ${{ matrix.php }}
115+
coverage: none
116+
tools: phpunit-polyfills
117+
118+
- name: Verify SQLite version in PHP
119+
run: |
120+
EXPECTED='${{ matrix.sqlite }}'
121+
if [ "$EXPECTED" = 'latest' ]; then
122+
EXPECTED=$(cat sqlite/VERSION)
123+
fi
124+
PDO=$(php -r "echo (new PDO('sqlite::memory'))->query('SELECT SQLITE_VERSION();')->fetch()[0];")
125+
echo "Expected SQLite version: $EXPECTED"
126+
echo "PHP PDO SQLite version: $PDO"
127+
if [ "$EXPECTED" != "$PDO" ]; then
128+
echo "Error: Expected SQLite version $EXPECTED, but PHP PDO uses $PDO"
129+
exit 1
130+
fi
131+
132+
- name: Set up Rust
133+
if: matrix.native
134+
uses: dtolnay/rust-toolchain@stable
135+
136+
- name: Install native build dependencies
137+
if: matrix.native
138+
run: |
139+
sudo apt-get update
140+
sudo apt-get install -y libclang-dev
141+
echo "PHP_CONFIG=$(command -v php-config)" >> "$GITHUB_ENV"
142+
LIBCLANG_SO="$(find /usr/lib -name 'libclang.so*' | head -n 1)"
143+
echo "LIBCLANG_PATH=$(dirname "$LIBCLANG_SO")" >> "$GITHUB_ENV"
144+
145+
- name: Install Composer dependencies (root)
146+
uses: ramsey/composer-install@v3
147+
with:
148+
ignore-cache: "yes"
149+
composer-options: "--optimize-autoloader"
150+
151+
- name: Install Composer dependencies (mysql-on-sqlite)
152+
uses: ramsey/composer-install@v3
153+
with:
154+
working-directory: packages/mysql-on-sqlite
155+
ignore-cache: "yes"
156+
composer-options: "--optimize-autoloader"
157+
158+
- name: Check Rust formatting
159+
if: matrix.php == '8.2' && matrix.native
160+
run: cargo fmt --check
161+
working-directory: packages/php-ext-wp-mysql-parser
162+
163+
- name: Build parser extension
164+
if: matrix.native
165+
run: cargo build
166+
working-directory: packages/php-ext-wp-mysql-parser
167+
168+
- name: Run native parser smoke tests
169+
if: matrix.native
170+
run: |
171+
php -d extension="$GITHUB_WORKSPACE/packages/php-ext-wp-mysql-parser/target/debug/libwp_mysql_parser.so" -r '
172+
require "src/load.php";
173+
$lexer = new WP_MySQL_Lexer( "SELECT ID, post_title FROM wp_posts WHERE ID IN (1, 2, 3)" );
174+
if ( ! ( $lexer instanceof WP_MySQL_Native_Lexer ) ) {
175+
fwrite( STDERR, "Native lexer is not available.\n" );
176+
exit( 1 );
177+
}
178+
$tokens = $lexer->native_token_stream();
179+
$rules = include "src/mysql/mysql-grammar.php";
180+
$grammar = new WP_Parser_Grammar( $rules );
181+
$parser = new WP_MySQL_Parser( $grammar, $tokens );
182+
$parser_reflection = new ReflectionObject( $parser );
183+
if ( ! $parser_reflection->hasProperty( "native" ) ) {
184+
fwrite( STDERR, "WP_MySQL_Parser did not select the native parser delegate.\n" );
185+
exit( 1 );
186+
}
187+
$native_property = $parser_reflection->getProperty( "native" );
188+
$native_property->setAccessible( true );
189+
if ( ! ( $native_property->getValue( $parser ) instanceof WP_MySQL_Native_Parser ) ) {
190+
fwrite( STDERR, "WP_MySQL_Parser did not select the native parser delegate.\n" );
191+
exit( 1 );
192+
}
193+
$ast = $parser->parse();
194+
if ( ! $ast instanceof WP_MySQL_Native_Parser_Node || "query" !== $ast->rule_name ) {
195+
fwrite( STDERR, "Native parser did not produce the expected query AST.\n" );
196+
exit( 1 );
197+
}
198+
'
199+
working-directory: packages/mysql-on-sqlite
200+
201+
- name: Verify SQLite driver selects the native parser path
202+
if: matrix.native
203+
run: |
204+
php -d extension="$GITHUB_WORKSPACE/packages/php-ext-wp-mysql-parser/target/debug/libwp_mysql_parser.so" -r '
205+
require "packages/mysql-on-sqlite/src/load.php";
206+
$lexer = new WP_MySQL_Lexer( "SELECT 1" );
207+
if ( ! ( $lexer instanceof WP_MySQL_Native_Lexer ) ) {
208+
fwrite( STDERR, "Native lexer is not available.\n" );
209+
exit( 1 );
210+
}
211+
$driver = new WP_PDO_MySQL_On_SQLite( "mysql-on-sqlite:path=:memory:;dbname=wp;" );
212+
$parser = $driver->create_parser( "SELECT 1" );
213+
$parser_reflection = new ReflectionObject( $parser );
214+
if ( ! $parser_reflection->hasProperty( "native" ) ) {
215+
fwrite( STDERR, "SQLite driver did not create a native parser delegate.\n" );
216+
exit( 1 );
217+
}
218+
$native_property = $parser_reflection->getProperty( "native" );
219+
$native_property->setAccessible( true );
220+
if ( ! ( $native_property->getValue( $parser ) instanceof WP_MySQL_Native_Parser ) ) {
221+
fwrite( STDERR, "SQLite driver did not create a native parser delegate.\n" );
222+
exit( 1 );
223+
}
224+
$parser->next_query();
225+
$ast = $parser->get_query_ast();
226+
if ( ! ( $ast instanceof WP_MySQL_Native_Parser_Node ) ) {
227+
fwrite( STDERR, "SQLite driver did not return a native-backed AST.\n" );
228+
exit( 1 );
229+
}
230+
$reflection = new ReflectionObject( $ast );
231+
if ( $reflection->hasProperty( "native_ast" ) || $reflection->hasProperty( "native_node_index" ) ) {
232+
fwrite( STDERR, "Native wrapper still stores Rust AST handle properties.\n" );
233+
exit( 1 );
234+
}
235+
$first = $ast->get_first_child_node();
236+
if ( ! ( $first instanceof WP_MySQL_Native_Parser_Node ) ) {
237+
fwrite( STDERR, "Native wrapper did not return a native-backed child node.\n" );
238+
exit( 1 );
239+
}
240+
if ( $first !== $ast->get_first_child_node() ) {
241+
fwrite( STDERR, "Native wrapper identity is not stable across reads.\n" );
242+
exit( 1 );
243+
}
244+
$synthetic = new WP_Parser_Node( 0, "synthetic" );
245+
$first->append_child( $synthetic );
246+
$same_first = $ast->get_first_child_node();
247+
if ( $same_first !== $first || ! in_array( $synthetic, $same_first->get_children(), true ) ) {
248+
fwrite( STDERR, "Materialized native wrapper was lost from the parent cache.\n" );
249+
exit( 1 );
250+
}
251+
'
252+
253+
- name: Run full PHPUnit suite with parser extension
254+
if: matrix.native
255+
env:
256+
WP_SQLITE_REQUIRE_NATIVE_PARSER_EXTENSION: '1'
257+
run: php -d extension="$GITHUB_WORKSPACE/packages/php-ext-wp-mysql-parser/target/debug/libwp_mysql_parser.so" ./vendor/bin/phpunit -c ./phpunit.xml.dist
258+
working-directory: packages/mysql-on-sqlite
259+
260+
- name: Run full PHPUnit suite
261+
if: ${{ ! matrix.native }}
262+
run: php ./vendor/bin/phpunit -c ./phpunit.xml.dist
263+
working-directory: packages/mysql-on-sqlite

‎.github/workflows/phpunit-tests-run.yml‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,31 @@ jobs:
3838
else
3939
TAG="version-${VERSION}"
4040
fi
41-
wget -O sqlite.tar.gz "https://sqlite.org/src/tarball/sqlite.tar.gz?r=${TAG}"
41+
SQLITE_SOURCE="https://sqlite.org/src/tarball/sqlite.tar.gz?r=${TAG}"
42+
SQLITE_MIRROR="https://github.com/sqlite/sqlite/archive/refs/tags/${TAG}.tar.gz"
43+
DOWNLOADED=0
44+
for url in "$SQLITE_SOURCE" "$SQLITE_MIRROR"; do
45+
for attempt in 1 2 3 4 5; do
46+
if wget -O sqlite.tar.gz "$url"; then
47+
DOWNLOADED=1
48+
break 2
49+
fi
50+
if [ "$attempt" -lt 5 ]; then
51+
sleep $(( attempt * 10 ))
52+
fi
53+
done
54+
done
55+
if [ "$DOWNLOADED" -ne 1 ]; then
56+
exit 1
57+
fi
4258
tar xzf sqlite.tar.gz
59+
if [ ! -d sqlite ]; then
60+
SQLITE_DIR=$(find . -maxdepth 1 -type d -name 'sqlite-*' | head -n 1)
61+
if [ -z "$SQLITE_DIR" ]; then
62+
exit 1
63+
fi
64+
mv "$SQLITE_DIR" sqlite
65+
fi
4366
cd sqlite
4467
./configure --prefix=/usr/local CFLAGS="-DSQLITE_ENABLE_COLUMN_METADATA -DSQLITE_ENABLE_FTS5 -DSQLITE_USE_URI -DSQLITE_ENABLE_JSON1" LDFLAGS="-lm"
4568
make -j$(nproc)

0 commit comments

Comments
 (0)