Bump Rust to 1.98.1 and the CI nightly to 2026-09-02 - #38620
Conversation
QA LLM Review1. HIGH -- 1.98.0 is the one stable release carrying an open, unfixed UB miscompilation
Rust 1.98.0 miscompiles some Detailsrustc 1.98.0 can decide that an impl's predicates are impossible when they involve associated-type projections plus an opaque type, emit I reproduced it on the platform we build for, The trigger needs dyn dispatch into an impl whose where-clauses involve associated-type projections or associated-type bounds, with an Suggested fix: hold |
antiguru
changed the title
|
Good catch, and it holds up on every point I could check independently:
The point about a green CI run proving nothing is the one that decides it. This PR is now a draft, held until 1.98.1 lands. I will then repin both On splitting: the nightly half is genuinely unaffected, since master has carried the fix since 2026-07-15 and the new pin is 2026-09-02. I am keeping it here rather than landing it separately, so the toolchain moves as one reviewable change. Happy to split it out instead if you would rather unblock the doc and nightly jobs now. Two things already surfaced by the nightly half, for the record. Advancing past 2026-08-02 required removing the stale blocker note in 🤖 Posted by Claude Code |
f113d67 to
7be4db7
Compare
|
Addressed. The pin now reads 1.98.1 in both 1.98.1 published on 2026-09-03, carrying the rust-lang/rust#161555 backport of the rust-lang/rust#158993 fix. I verified the difference with the self-contained reproducer from rust-lang/rust#161441 on The branch is rebased onto The commit message and the PR description now state why the pin skips 1.98.0 rather than leaving it implicit. The remaining item before merge is unchanged: 🤖 Posted by Claude Code |
7be4db7 to
75db7b9
Compare
| # lint-versions - Check rust version | ||
|
|
||
| grep "rust-version = " Cargo.toml | grep -q "1\.97\.1" || \ | ||
| grep "rust-version = " Cargo.toml | grep -q "1\.98\.1" || \ |
There was a problem hiding this comment.
This line is the record that a Rust version has been validated for compilation time regressions, and the description says that validation has not been performed for 1.98.1.
doc/developer/upgrade-rust.md step 7 is the mechanism, and it has not run yet: the only Buildkite build on antiguru:rust-1-98-bump is test #134971, there is no nightly build on the branch. A toolchain bump changes codegen for every crate, so the compile-time and performance data this line asserts only exists once Nightly is green here. Worth doing while the PR is open rather than after it merges, per that same step. I am happy to trigger and read the run if that is easier.
There was a problem hiding this comment.
Agreed, and triggered: Nightly build #18327 is running on bc29533, the current head. You were right that nothing had run: test #134971 was the only build on the branch.
Two related notes while I was in doc/developer/upgrade-rust.md.
Step 6 had also not been checked, and it comes out clean: UNICODE_VERSION is (17, 0, 0) at both 1.97.1 and 1.98.1, so there is nothing for the release-notes classifier to pick up and the "No user-visible changes" line stands.
The line this thread is on stays as-is until #18327 reports. I will post the result here either way, and the description keeps the Team Testing item open until then.
🤖 Posted by Claude Code
There was a problem hiding this comment.
Nightly #18327 has run on bc29533: 251 jobs passed, 8 failed, 2 still running, 38 broken. No failure is attributable to the toolchain, and every performance-sensitive step passed.
The steps doc/developer/upgrade-rust.md step 7 exists to protect are all green: feature-benchmark-against-merge-base-or-latest 1 through 9, rust-cargo-bench-against-ancestor 1 through 9, parallel-benchmark 1 through 5, the three scalability-benchmark variants, and all five maelstrom jobs. The two jobs still running are console-scalability-workload-replay 1 and 2, which replay a Console workload and do not bear on the toolchain.
The 8 failures group into four root causes, and two of them are already fixed on main by commits this branch predates.
- Platform-checks upgrade scenarios, 3 jobs.
ALTER CLUSTER ... WITH (WAIT UNTIL READY ...)fails with "the requested feature (enable_zero_downtime_cluster_reconfiguration) is in private preview". Fixed by 03783c3 (mzcompose: default the zero-downtime reconfiguration flag on for binaries before v26.42 #38830), which is onmainbut after this branch's base 6e55add. - SLT (2 replicas) shard 3, 1 job. Every failure is in
distinct_arrangements.slt, the curated metric sink golden breakage. Fixed by 8a0c315 (sqllogictest: disable curated metric sinks in distinct_arrangements.slt #38835), also after the base. This branch's copy of the file still lacks theenable_metric_sink = falsestanza. - Orchestratord + documentation defaults, 2 jobs.
kubectl wait -n materialize --for=condition=Available --timeout=300s deployment/minioexits non-zero. Both parallel jobs failed onmainnightlies 18324 and 18320 as well. - Testdrive with SIZE 8 shard 1, plain and azurite, 2 jobs.
kafka-compression.td:186reads"world"where"hello"is expected fromlz4_sink_implicit. Main history on these jobs is one pass in the last five nightlies.
Next step is a rebase onto current upstream/main to absorb #38830 and #38835, which should clear four of the eight. The other four are chronic main flakes that a rebase will not touch.
One caveat to record: that rebase moves the branch off bc29533, so the green Nightly will no longer sit on the head commit. The conclusion still holds, because a rebase only adds main commits and changes neither the rust-version pin nor the nightly date, but the evidence above is tied to bc29533 rather than to whatever head ends up being.
🤖 Posted by Claude Code
There was a problem hiding this comment.
Nightly #18327 finished: the last three console-scalability shards passed, so every benchmark step in the run is green and the failure triage above is final.
Rebased onto ad1150f222, head is now 87973eb755. That absorbs #38830 and #38835, which should clear four of the eight failures if Nightly is run again. The pin, the nightly date, and the seven-file diff are unchanged by the rebase.
🤖 Posted by Claude Code
| /// Any structure implementing the [ToAvro] trait will be usable | ||
| /// from a [Writer](../writer/struct.Writer.html). |
There was a problem hiding this comment.
[ToAvro] is the only link touched in this PR without backticks; the rest of the diff uses the [`Name`] form, and so does the line immediately below.
The Writer link is the same legacy shape as the ones being fixed, but rustdoc cannot flag it here: writer is a private module (mod writer; at src/avro/src/lib.rs:338), so a plain bin/doc run emits no writer/ directory and ../writer/struct.Writer.html resolves to nothing in the published docs. Writer is re-exported at the crate root (src/avro/src/lib.rs:356), so routing through the re-export both renders and gets checked by rustdoc. Bare Writer is not in scope in this module, so the explicit target should not trip redundant_explicit_links, but it is worth a local bin/doc to confirm.
| /// Any structure implementing the [ToAvro] trait will be usable | |
| /// from a [Writer](../writer/struct.Writer.html). | |
| /// Any structure implementing the [`ToAvro`] trait will be usable | |
| /// from a [`Writer`](crate::Writer). |
There was a problem hiding this comment.
Applied, with one correction to the mechanism and one extra fix.
The correction: rustdoc does emit something at mz_avro/writer/struct.Writer.html. It is a 400-byte redirect stub to ../../mz_avro/struct.Writer.html, generated because the item is re-exported at the crate root, so the old link resolved for a reader rather than 404ing. The reason to change it is the other half of your argument, which holds: an explicit HTML path is never checked, so it rots silently on a rename, whereas crate::Writer is checked by broken_intra_doc_links.
The extra fix: types.rs:354 carries the same legacy shape, [Schema](../schema/enum.Schema.html), and that one is genuinely dead. Schema is a struct (src/avro/src/schema.rs:355), so rustdoc emits schema/struct.Schema.html and nothing at enum.Schema.html, not even a redirect. Neither Writer nor Schema is in scope in types.rs, which is why redundant_explicit_links never flagged either one. Both now route through the crate-root re-export.
Verified with a full bin/doc at the pinned nightly (5db7f4be8, the 2026-09-02 dist), RUSTDOCFLAGS="-D warnings --cfg nightly_doc_features", exit 0. The rendered hrefs are ../struct.Writer.html and ../schema/struct.Schema.html, and both files exist in target/doc.
🤖 Posted by Claude Code
75db7b9 to
bc29533
Compare
CI derives its stable toolchain from the `rust-version` field in the root `Cargo.toml`, so that field is what decides which warnings CI can see. Holding it at 1.97.1 meant the lints Rust 1.98 introduced only showed up when someone built locally. Raising it to 1.98.1 closes that gap. Cargo.lock needs no change, which matters because the doc test job resolves with `--locked`. The pin skips 1.98.0, which miscompiles some `dyn Trait` calls into a null vtable slot. rustc 1.98.0 can decide that an impl's predicates are impossible when they involve associated-type projections plus an opaque type, emit `VtblEntry::Vacant` for that impl's method, and leave a zero in the method slot of a compiler-generated vtable, after which safe code dispatches through a null pointer. The bug is rust-lang/rust#161441, and the fix reached stable in the 1.98.1 point release. Running the self-contained reproducer from that issue on `x86_64-unknown-linux-gnu` under edition 2024 confirms the difference: 1.98.0 aborts with SIGILL at `-O` and SIGSEGV without it, while 1.98.1 runs to completion in both profiles. A green CI run on 1.98.0 would not have ruled the tree out, because rustc emits the bad vtable with no diagnostic. Rust 1.98.1 uses LLVM 22.1.8, matching the `clang-22`, `lld-22`, and `llvm-22` packages the CI builder image already installs, so the Dockerfile needs no accompanying change. The comment on that apt stanza asks for the two to move together, and they still agree. Bumping `rust-version` does change the builder image tag, because the tag hashes the build arguments and `RUST_VERSION` is one of them. `ci/mkpipeline.sh` detects the missing tag and inserts bootstrap steps that build and push the stable, min, and console flavors for both architectures, so the first build on this branch will be slow but needs no manual intervention. The nightly pin moves to 2026-09-02. The note that pinned it to 2026-08-02 pointed at rust-lang/rust#160439, a rustdoc hang that broke the Doctests job, and that issue was closed as completed on 2026-08-06. The note is removed rather than reworded, because the constraint it described no longer exists. Advancing the nightly does make rustdoc's `redundant_explicit_links` lint fire, and `bin/doc` runs with `RUSTDOCFLAGS=-D warnings`, so those become errors. Eight doc comments in `mz-avro` and `mz-pgtest` spell an intra-doc link as a label plus an explicit legacy HTML path that resolves to the same destination. Dropping the explicit target is the rewrite rustdoc itself suggests, and every referenced item is in scope at the link site. The `flush` links in the Avro writer keep their explicit targets, because a fragment path is not redundant with its label and rustdoc does not flag them. Two links in `mz-avro` keep an explicit target but get a different one. Neither `Writer` nor `Schema` is in scope in `src/avro/src/types.rs`, so rustdoc cannot flag their legacy paths as redundant and has never checked them. One of the two was wrong: `Schema` is a struct, so `../schema/enum.Schema.html` names a page that rustdoc does not emit. Routing both through the crate-root re-export, `crate::Writer` and `crate::Schema`, puts them under `broken_intra_doc_links` so the next rename breaks the build instead of the docs. `bin/lint-versions` records the Rust version that has been checked for compilation time regressions, and it is updated here so `bin/lint` passes. That validation has not been performed. Team Testing should confirm 1.98.1 before this merges. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bc29533 to
87973eb
Compare
3 participants
CI derives its stable toolchain from the
rust-versionfield in the rootCargo.toml, so that field is what decides which warnings CI can see. Holding it at 1.97.1 meant the lints Rust 1.98 introduced only showed up when someone built locally, which is how the warnings fixed in #38619 went unnoticed. Raising it closes that gap. Cargo.lock needs no change, which matters because the doc test job resolves with--locked.The pin is 1.98.1 rather than 1.98.0. 1.98.0 shipped an open
P-criticalmiscompilation, rust-lang/rust#161441: rustc could wrongly decide an impl's predicates were impossible when they involved associated-type projections plus an opaque type, emit a vacant vtable entry, and leave a zero in the method slot, so safe code dispatched through a null pointer. That was silent at compile time, andrust-versionselects the toolchain in thestableci-builder flavor that builds the shipped images, so it would have reached release artifacts. rust-lang/rust#158993 fixed it after the 1.98 beta cutoff and rust-lang/rust#161555 backported it for 1.98.1, which is now the current stable release.Rust 1.98.1 uses LLVM 22.1.8, matching the
clang-22,lld-22, andllvm-22packages the CI builder image already installs, so the Dockerfile needs no accompanying change. The comment on that apt stanza asks for the two to move together, and they still agree. Bumpingrust-versiondoes change the builder image tag, because the tag hashes the build arguments andRUST_VERSIONis one of them.ci/mkpipeline.shdetects the missing tag and inserts bootstrap steps that build and push the stable, min, and console flavors for both architectures, so the first build on this branch will be slow but needs no manual intervention.The nightly pin moves to 2026-09-02. The note that pinned it to 2026-08-02 pointed at rust-lang/rust#160439, a rustdoc hang that broke the Doctests job, and that issue was closed as completed on 2026-08-06. The note is removed rather than reworded, because the constraint it described no longer exists.
Advancing the nightly does make rustdoc's
redundant_explicit_linkslint fire, andbin/docruns withRUSTDOCFLAGS=-D warnings, so those become errors. Eight doc comments inmz-avroandmz-pgtestspell an intra-doc link as a label plus an explicit legacy HTML path that resolves to the same destination. Dropping the explicit target is the rewrite rustdoc itself suggests, and every referenced item is in scope at the link site. Theflushlinks in the Avro writer keep their explicit targets, because a fragment path is not redundant with its label and rustdoc does not flag them.Outstanding before merge
bin/lint-versionsrecords the Rust version that has been checked for compilation time regressions, and it is updated here sobin/lintpasses. That validation has not been performed. Team Testing should confirm 1.98.1 before this merges.Two further caveats for reviewers. The
cargo test --docjob could not be exercised locally because that machine has noprotoc, so it is covered only by CI. Building the nightly builder image also runscargo miri setupand installscargo-fuzz, neither of which can be checked outside an image build; both fail loudly in the bootstrap step rather than silently. Finally, a toolchain bump surfaces latent problems anywhere in the tree, not only in the diff, so a failure on this branch may point at code it does not touch.Release notes
No user-visible changes.
🤖 Generated with Claude Code