Repository navigation
Cap kMaxArcs by Unsigned in CompactArcStore and ConstFstImpl. - #351
Merged
Merged
Conversation
copybara-service
Bot
force-pushed
the
copybara/995274956
branch
from
October 8, 2026 00:39
87d7c1a to
6ea3b79
Compare
`CompactArcStore<Element, Unsigned>` and `ConstFstImpl<Arc, Unsigned>` can be instantiated with smaller unsigned index types such as `uint8_t` and `uint16_t` (`compact8_*`, `compact16_*`, `const8`, `const16`), where state offsets, arc counts, and epsilon counts are stored in `Unsigned`. Previously: 1. `kMaxArcs` was hardcoded to `1ULL << 52` regardless of `Unsigned`, so `Read()` did not reject headers or state-offset tables whose arc/compact counts exceeded `std::numeric_limits<Unsigned>::max()`. 2. The constructors did not check `ncompacts_ > kMaxArcs` or `narcs_ > kMaxArcs` before truncating offsets/counts into `Unsigned`, silently wrapping around for inputs exceeding the index type's capacity. 3. In `CompactFstImpl(const Fst<Arc>&, ...)`, `if (compactor_->Error())` called `SetProperties(kError, kError)` without returning, allowing the subsequent 1-argument `SetProperties(...)` call at the end of the constructor to overwrite `kError`. Fix by: - Capping `kMaxArcs` on `CompactArcStore` and `ConstFstImpl` at `std::min<uint64_t>(0x10000000000000ull, std::numeric_limits<Unsigned>::max())`. - Checking `ncompacts_ > kMaxArcs` in `CompactArcStore`'s FST and iterator constructors, and checking `data->nstates_ > kMaxArcs / arc_compactor.Size()` in the fixed out-degree branch of `CompactArcStore::Read`. - Returning early in `CompactFstImpl(const Fst<Arc>&, ...)` when `compactor_->Error()` is true. - Checking `narcs_ > kMaxArcs` in `ConstFstImpl(const Fst<Arc>&)`. - Adding unit tests in `fst_limits_test` for `uint8_t` index overflow and 64-bit `kMaxArcs` bounds. PiperOrigin-RevId: 995469267
copybara-service
Bot
force-pushed
the
copybara/995274956
branch
from
October 8, 2026 00:50
6ea3b79 to
7f3847b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cap
kMaxArcsbyUnsignedinCompactArcStoreandConstFstImpl.CompactArcStore<Element, Unsigned>andConstFstImpl<Arc, Unsigned>can beinstantiated with smaller unsigned index types such as
uint8_tanduint16_t(
compact8_*,compact16_*,const8,const16), where state offsets, arccounts, and epsilon counts are stored in
Unsigned. Previously:kMaxArcswas hardcoded to1ULL << 52regardless ofUnsigned, soRead()did not reject headers or state-offset tables whose arc/compactcounts exceeded
std::numeric_limits<Unsigned>::max().ncompacts_ > kMaxArcsornarcs_ > kMaxArcsbefore truncating offsets/counts into
Unsigned, silently wrapping aroundfor inputs exceeding the index type's capacity.
CompactFstImpl(const Fst<Arc>&, ...),if (compactor_->Error())calledSetProperties(kError, kError)without returning, allowing the subsequent1-argument
SetProperties(...)call at the end of the constructor tooverwrite
kError.Fix by:
kMaxArcsonCompactArcStoreandConstFstImplatstd::min<uint64_t>(0x10000000000000ull, std::numeric_limits<Unsigned>::max()).ncompacts_ > kMaxArcsinCompactArcStore's FST and iteratorconstructors, and checking
data->nstates_ > kMaxArcs / arc_compactor.Size()in the fixed out-degree branch of
CompactArcStore::Read.CompactFstImpl(const Fst<Arc>&, ...)whencompactor_->Error()is true.narcs_ > kMaxArcsinConstFstImpl(const Fst<Arc>&).fst_limits_testforuint8_tindex overflow and64-bit
kMaxArcsbounds.