Fix token_to_sequence returning Some(0) for out-of-bounds index

[Osamaali313]

Problem

Encoding::token_to_sequence guards the index with > instead of >=:

pub fn token_to_sequence(&self, token: usize) -> Option<usize> {
    if token > self.len() {
        None
    } else if self.sequence_ranges.is_empty() {
        Some(0)
    } else {
        self.sequence_ranges.iter().find_map(|(seq_id, range)| {
            if range.contains(&token) { Some(*seq_id) } else { None }
        })
    }
}

Valid token indices are 0..len, so token == len is out of bounds and should return None. But token > self.len() lets it through, and for an unprocessed encoding (empty sequence_ranges, e.g. from Encoding::from_tokens or a raw model output) it falls to Some(0).

Evidence (sibling accessors prove intent)

Every other index-based accessor in the same file treats index >= len as out of bounds:

• token_to_chars → self.offsets.get(token) → None at token == len
• token_to_word → self.words.get(token) → None at token == len

Only token_to_sequence admits token == len. token_to_sequence is public API (also exposed through the Python and Node bindings), so a caller doing encoding.token_to_sequence(len) on a single-sequence/unprocessed encoding gets 0 rather than None.

Fix

if token >= self.len() {

Processed encodings are unaffected — their sequence_ranges are populated and never contain len, so the boundary already resolved to None via find_map; existing tests (processors::bert, processors::roberta, Python test_encoding.py) still pass. A regression test is added for the unprocessed-encoding boundary.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.