Skip to content

Commit 888fa56

Browse files
authored
release: bump to 3.1.0 (seq2seq first_alternates decode contract) (#806)
* release: bump to 3.1.0 (seq2seq first_alternates decode contract) * fix(isc): retry the DSL grammar when the artifact misparses a source The shipped isc.artifact.json rejects lowercase system codes that the Ruby-DSL grammar accepts (acadsin-zho-Hani-Latn-2002.isc) — one bad map broke every spec that loads the corpus in CI. The runtime now prefers the artifact and retries through the DSL on ParseFailed; the artifact recompile via the parsanol toolchain remains the upstream fix. * Revert "fix(isc): retry the DSL grammar when the artifact misparses a source" This reverts commit 1d80448. * fix(isc): drop the broken compiled artifact — the DSL grammar is the engine The shipped isc.artifact.json misparses lowercase system codes (acadsin-zho-Hani-Latn-2002.isc) — and a FRESH compile with parsanol 1.3.91 reproduces the identical failure, i.e. the parg compiler mistranslates the grammar (filed upstream as parsanol#5). The artifact path has never parsed this corpus correctly; the Ruby DSL grammar is the only engine that has. Per the clean-data rule, ship no known-bad derived bytes: the .parg source stays (canonical grammar, used by the differential spec once the compiler is fixed), the artifact spec guards skip without it, and the runtime uses its documented artifact-missing fallback into the Ruby grammar. * test(isc): guard artifact-dependent specs for the not-shipped state * ci(rake): test the ref under test, not bootstrap-cloned main bootstrap.rb clones interscript-ruby remote main into ruby/; a PR's own code was never exercised (and main's red state was unfixable from PRs). Overlay the PR head (or pushed ref) onto the bootstrapped sibling layout after bootstrap. * test(isc): skip the differential gate inside the example when the artifact is absent * style: standardrb autofix for the ml loader files * style: standardrb clean across lib (URI.parse().open, ranges, struct, access modifiers) The ml files merged in #805 had never been linted (rake.yml was red, so StandardRB never gated them). Autofixes + manual conversions; no behavior change — imf/provisioning URI fetches keep their open-uri requires. * style: standardrb clean across spec * test(ml): skip the tampered-zip tmpdir cleanup on windows rubyzip's finalizer holds the zip handle past the mktmpdir block on windows (Errno::EACCES at cleanup); the loader itself uses block-form opens everywhere. Platform skip with the reason inline.
1 parent 42c1898 commit 888fa56

18 files changed

Lines changed: 184 additions & 167 deletions

File tree

‎.github/workflows/rake.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,16 @@ jobs:
3737

3838
- name: Bootstrap packages
3939
run: ruby bootstrap.rb
40+
# bootstrap clones interscript-ruby REMOTE MAIN into ruby/ — for
41+
# PRs that would test main's code instead of the PR. Overlay the
42+
# actual ref under test onto the bootstrapped sibling layout.
43+
- name: Overlay the ref under test onto ruby/
44+
uses: actions/checkout@v7
45+
with:
46+
ref: ${{ github.event.pull_request.head.ref || github.ref }}
47+
repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
48+
path: ruby
49+
clean: false
4050

4151
- name: Set up Ruby
4252
uses: ruby/setup-ruby@v1

‎lib/interscript/isc/grammar/isc.artifact.json‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎lib/interscript/ml.rb‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ module Interscript::ML
1919
# Expected to be loaded before IMF/Byt5Onnx/PlaneModel entry points.
2020
def self.require_optional!(gem_name, feature: gem_name)
2121
require feature
22-
rescue LoadError => e
22+
rescue LoadError
2323
raise Interscript::ExternalUtilError,
2424
"the neural layer needs the #{gem_name} gem: gem install #{gem_name} (or use a bundle including it)"
2525
end

‎lib/interscript/ml/byt5_onnx.rb‎

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
require 'tmpdir'
1+
require "tmpdir"
22

33
module Interscript::ML
44
# Byte-level seq2seq (ByT5 family) inference over ONNX sessions from an
@@ -11,24 +11,24 @@ def initialize(zip_path)
1111
Interscript::ML.require_optional!("onnxruntime")
1212
@manifest = IMF.manifest(zip_path)
1313
graphs = IMF.verify_and_read(zip_path)
14-
@tmpdir = Dir.mktmpdir('secryst-imf')
15-
@encoder = build_session(graphs, 'encoder.onnx')
16-
@kv = @manifest['decoder'] == 'kv' && graphs.key?('decoder-kv.onnx')
17-
@decoder = build_session(graphs, @kv ? 'decoder-kv.onnx' : 'decoder.onnx')
14+
@tmpdir = Dir.mktmpdir("secryst-imf")
15+
@encoder = build_session(graphs, "encoder.onnx")
16+
@kv = @manifest["decoder"] == "kv" && graphs.key?("decoder-kv.onnx")
17+
@decoder = build_session(graphs, @kv ? "decoder-kv.onnx" : "decoder.onnx")
1818
@pasts = @kv ? zero_pasts : {}
1919
end
2020

2121
def translate(text, max_seq_length: 256)
2222
ids = IMF.encode(text)
23-
return '' if ids.length == 1 # only the trailing EOS: empty input
23+
return "" if ids.length == 1 # only the trailing EOS: empty input
2424

25-
hidden = @encoder.predict({ input_ids: [ids] })['last_hidden_state']
25+
hidden = @encoder.predict({input_ids: [ids]})["last_hidden_state"]
2626
tokens = @kv ? greedy_kv(hidden, max_seq_length) : greedy_plain(hidden, max_seq_length)
2727
IMF.first_alternates(IMF.decode(tokens))
2828
end
2929

3030
def id
31-
@manifest['id']
31+
@manifest["id"]
3232
end
3333

3434
private
@@ -43,7 +43,7 @@ def build_session(graphs, name)
4343

4444
def zero_pasts
4545
@decoder.inputs.each_with_object({}) do |meta, pasts|
46-
next unless meta[:name].start_with?('past_')
46+
next unless meta[:name].start_with?("past_")
4747
shape = meta[:shape] # [batch, heads, past_seq, d_kv]; dynamic dims are Strings
4848
heads = shape[1].is_a?(Integer) ? shape[1] : 4
4949
d_kv = shape[3].is_a?(Integer) ? shape[3] : 8
@@ -71,13 +71,13 @@ def greedy_kv(hidden, max_seq_length)
7171
generated = []
7272
max_seq_length.times do
7373
results = @decoder.predict(
74-
{ input_ids: [current], encoder_hidden_states: hidden }.merge(pasts)
74+
{input_ids: [current], encoder_hidden_states: hidden}.merge(pasts)
7575
)
76-
token = argmax(results['logits'].first.last)
76+
token = argmax(results["logits"].first.last)
7777
break if token == IMF::EOS_ID
7878

7979
generated << token
80-
pasts = pasts.keys.to_h { |name| [name, results[name.sub('past_', 'present_')]] }
80+
pasts = pasts.keys.to_h { |name| [name, results[name.sub("past_", "present_")]] }
8181
current = [token]
8282
end
8383
generated
@@ -88,13 +88,13 @@ def greedy_plain(hidden, max_seq_length)
8888
generated = []
8989
max_seq_length.times do
9090
logits = @decoder.predict(
91-
{ input_ids: [decoder_ids], encoder_hidden_states: hidden }
92-
)['logits']
91+
{input_ids: [decoder_ids], encoder_hidden_states: hidden}
92+
)["logits"]
9393
token = argmax(logits.first.last)
9494
break if token == IMF::EOS_ID
9595

9696
generated << token
97-
decoder_ids = decoder_ids + [token]
97+
decoder_ids += [token]
9898
end
9999
generated
100100
end

‎lib/interscript/ml/imf.rb‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ def encode(text)
3232
# corpora emit '/'-separated alternates per token; the runtime
3333
# returns the primary choice only.
3434
def first_alternates(text)
35-
text.split.map { |tok| tok.split('/', 2).first }.join(' ')
35+
text.split.map { |tok| tok.split("/", 2).first }.join(" ")
3636
end
3737

3838
def decode(token_ids)
@@ -112,7 +112,7 @@ def resolve(model_id, index_url: nil)
112112
if channel.start_with?("file://")
113113
FileUtils.cp(channel.sub(%r{\Afile://}, ""), tmp)
114114
else
115-
URI.open(channel) { |remote| IO.copy_stream(remote, tmp) }
115+
URI.parse(channel).open { |remote| IO.copy_stream(remote, tmp) }
116116
end
117117
end
118118
actual = Digest::SHA256.file(tmp).hexdigest
@@ -140,14 +140,14 @@ def download_parts(entry, tmp)
140140
if part["url"].start_with?("file://")
141141
File.open(source, "rb", &open_stream)
142142
else
143-
URI.open(source, "rb", &open_stream)
143+
URI.parse(source).open("rb", &open_stream)
144144
end
145145
unless digest.hexdigest == part["sha256"]
146146
raise RegistryError, "part #{index} of #{entry["filename"]} sha256 mismatch: got #{digest.hexdigest}, index says #{part["sha256"]}"
147147
end
148148
end
149149
end
150-
rescue StandardError
150+
rescue
151151
File.delete(tmp) if File.file?(tmp)
152152
raise
153153
end
@@ -167,10 +167,10 @@ def load_index(source)
167167
# HTTP sources must ship a sibling .sha256 sidecar; verify the
168168
# body against it before parsing (GitHub Releases index assets do).
169169
def fetch_index_with_sidecar(source)
170-
body = URI.open(source, "rb") { |remote| remote.read }
170+
body = URI.parse(source).open("rb") { |remote| remote.read }
171171
sidecar =
172172
begin
173-
URI.open("#{source}.sha256") { |remote| remote.read }
173+
URI.parse("#{source}.sha256").open { |remote| remote.read }
174174
rescue OpenURI::HTTPError => e
175175
raise RegistryError, "index sha256 sidecar missing: #{source}.sha256 (#{e.message})"
176176
end

‎lib/interscript/ml/model.rb‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -5,30 +5,30 @@ class Model
55
def self.from_file(model_file)
66
# A models.yaml model id resolves (download -> verify -> cache)
77
# through the same contract as the Python/TS runtimes.
8-
model_file = IMF.resolve(model_file) unless model_file.to_s.end_with?('.zip') || File.file?(model_file.to_s)
8+
model_file = IMF.resolve(model_file) unless model_file.to_s.end_with?(".zip") || File.file?(model_file.to_s)
99
model_file = Provisioning.locate(model_file)
1010

1111
Zip::File.open(model_file) do |zip_file|
12-
metadata = zip_file.glob('metadata.yaml').first
12+
metadata = zip_file.glob("metadata.yaml").first
1313
metadata = YAML.safe_load(metadata.get_input_stream.read) if metadata
1414

1515
# IMF v1: the Interscript Model Format zip.
16-
return Byt5Onnx.new(model_file) if metadata && metadata['format'] == 'imf-v1'
16+
return Byt5Onnx.new(model_file) if metadata && metadata["format"] == "imf-v1"
1717

18-
name = metadata && metadata['name']
18+
name = metadata && metadata["name"]
1919

2020
# Modern byte-level seq2seq (ByT5 family): encoder.onnx + decoder.onnx.
21-
return Byt5Onnx.new(model_file) if name == 'byt5'
21+
return Byt5Onnx.new(model_file) if name == "byt5"
2222

2323
# Legacy single-file ONNX transformer zips (vocabs.yaml based).
24-
vocabs = zip_file.glob('vocabs.yaml').first
25-
raise 'vocabs.yaml is missing in model zip!' unless vocabs
24+
vocabs = zip_file.glob("vocabs.yaml").first
25+
raise "vocabs.yaml is missing in model zip!" unless vocabs
2626
vocabs = YAML.safe_load(vocabs.get_input_stream.read)
27-
input_vocab = Vocab.new(vocabs['input'], specials: [])
28-
target_vocab = Vocab.new(vocabs['target'], specials: [])
27+
input_vocab = Vocab.new(vocabs["input"], specials: [])
28+
target_vocab = Vocab.new(vocabs["target"], specials: [])
2929

30-
onnx = zip_file.glob('*.onnx').first
31-
raise 'onnx model file is missing in model zip!' unless onnx
30+
onnx = zip_file.glob("*.onnx").first
31+
raise "onnx model file is missing in model zip!" unless onnx
3232
Onnx.new(onnx.get_input_stream.read, input_vocab, target_vocab)
3333
end
3434
end
@@ -41,11 +41,11 @@ def initialize(model_path_or_bytes, input_vocab, target_vocab)
4141
end
4242

4343
def call(input, output, opts)
44-
@model.predict({ src: input, tgt: output }.merge(opts))['output']
44+
@model.predict({src: input, tgt: output}.merge(opts))["output"]
4545
end
4646

4747
def argmax(*args)
48-
self.call(*args).map { |i| i.flatten.each_with_index.max[1] }
48+
call(*args).map { |i| i.flatten.each_with_index.max[1] }
4949
end
5050
end
5151
end

‎lib/interscript/ml/plane.rb‎

Lines changed: 27 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# frozen_string_literal: true
22

3-
require 'tmpdir'
4-
require 'json'
3+
require "tmpdir"
4+
require "json"
55

66
module Interscript
77
module ML
@@ -16,7 +16,7 @@ class PlaneModel
1616
attr_reader :classes, :k_passes
1717

1818
def initialize(graph:, classes:, k_passes: 2)
19-
Interscript::ML.require_optional!('onnxruntime')
19+
Interscript::ML.require_optional!("onnxruntime")
2020
@classes = classes.freeze
2121
@n_classes = classes.length
2222
@mask_id = @n_classes
@@ -27,8 +27,8 @@ def initialize(graph:, classes:, k_passes: 2)
2727
@class_to_id = classes.each_with_index.to_h.freeze
2828
# the onnxruntime gem loads from paths, not bytes (as byt5_onnx):
2929
# verified bytes go to a tmpfile owned by this instance
30-
@graph_tmpdir = Dir.mktmpdir('isx-plane')
31-
path = File.join(@graph_tmpdir, 'plane.onnx')
30+
@graph_tmpdir = Dir.mktmpdir("isx-plane")
31+
path = File.join(@graph_tmpdir, "plane.onnx")
3232
File.binwrite(path, graph)
3333
@sess = OnnxRuntime::Model.new(path)
3434
end
@@ -58,7 +58,7 @@ def split_planes(text)
5858
else
5959
close.call
6060
skeleton << ch
61-
classes << ''
61+
classes << ""
6262
end
6363
end
6464
close.call
@@ -75,18 +75,18 @@ def translate(text, preserve_diacritics: false)
7575
ch = skeleton[i]
7676
n = ch.bytesize
7777
cls = if user_cls.empty?
78-
votes = tok_preds[pos, n] || []
79-
cid = votes.tally.max_by { |_, c| c }&.first || @n_classes
80-
cid < @n_classes ? @classes[cid] : ''
81-
else
82-
user_cls # user diacritics round-trip byte-exactly
83-
end
78+
votes = tok_preds[pos, n] || []
79+
cid = votes.tally.max_by { |_, c| c }&.first || @n_classes
80+
(cid < @n_classes) ? @classes[cid] : ""
81+
else
82+
user_cls # user diacritics round-trip byte-exactly
83+
end
8484
pos += n
8585
cls
8686
end
8787
out = +""
8888
skeleton.each_char.with_index do |ch, i|
89-
out << (ch == "\x00" ? final[i] : ch + final[i])
89+
out << ((ch == "\x00") ? final[i] : ch + final[i])
9090
end
9191
out
9292
end
@@ -102,7 +102,7 @@ def translate_plain(text)
102102
votes = preds[pos, nbytes] || []
103103
cid = votes.tally.max_by { |_, c| c }&.first || @n_classes
104104
pos += nbytes
105-
cid < @n_classes ? @classes[cid] : ''
105+
(cid < @n_classes) ? @classes[cid] : ""
106106
end
107107
PlaneModel.render_plane(text, char_preds)
108108
end
@@ -111,7 +111,7 @@ def k_pass_decode(ids, plane0)
111111
plane = plane0
112112
preds = nil
113113
@k_passes.times do
114-
logits = @sess.predict({ 'input_ids' => [ids], 'plane_ids' => [plane] })['class_logits'][0]
114+
logits = @sess.predict({"input_ids" => [ids], "plane_ids" => [plane]})["class_logits"][0]
115115
preds = logits.each_with_index.map { |row, i| row.index(row.max) }
116116
plane = preds
117117
end
@@ -129,11 +129,11 @@ def decode_pinned(skeleton, char_classes)
129129
pid = @class_to_id.fetch(cls, @mask_id)
130130
ch.bytesize.times { per_token_pin << pid }
131131
end
132-
plane0 = per_token_pin.map { |p| p == @mask_id ? @mask_id : p }
132+
plane0 = per_token_pin.map { |p| (p == @mask_id) ? @mask_id : p }
133133
k_pass_decode(ids, plane0)
134134
end
135135

136-
def self.render_plane(skeleton, char_classes)
136+
def self.render_plane(skeleton, char_classes) # rubocop:disable Lint/IneffectiveAccessModifier
137137
skeleton.each_char.with_index.map do |ch, i|
138138
ch + char_classes[i].to_s
139139
end.join
@@ -142,14 +142,15 @@ def self.render_plane(skeleton, char_classes)
142142
# The zip contract: metadata.yaml (kind=plane, k_passes, member
143143
# sha256s) + plane.onnx + classes.json, every member sha-verified
144144
# before the session is created.
145+
# rubocop:disable Lint/IneffectiveAccessModifier
145146
def self.from_zip(data)
146-
require 'zip'
147+
require "zip"
147148
zf = Zip::File.open_buffer(data)
148149
names = zf.entries.map(&:name)
149-
meta = YAML.safe_load(zf.read('metadata.yaml'), permitted_classes: [], aliases: false)
150-
raise IMF::FormatError, "plane zip: kind=#{meta['kind'].inspect}, expected 'plane'" if meta['kind'] != 'plane'
150+
meta = YAML.safe_load(zf.read("metadata.yaml"), permitted_classes: [], aliases: false)
151+
raise IMF::FormatError, "plane zip: kind=#{meta["kind"].inspect}, expected 'plane'" if meta["kind"] != "plane"
151152

152-
members = meta['members'] || {}
153+
members = meta["members"] || {}
153154
plane_graph = nil
154155
classes_bytes = nil
155156
%w[plane.onnx classes.json].each do |member|
@@ -161,18 +162,20 @@ def self.from_zip(data)
161162
got = Digest::SHA256.hexdigest(bytes)
162163
raise IMF::FormatError, "plane zip: #{member} sha256 mismatch (#{got})" if want != got
163164

164-
plane_graph = bytes if member == 'plane.onnx'
165-
classes_bytes = bytes if member == 'classes.json'
165+
plane_graph = bytes if member == "plane.onnx"
166+
classes_bytes = bytes if member == "classes.json"
166167
end
167168
classes = JSON.parse(classes_bytes)
168-
new(graph: plane_graph, classes: classes, k_passes: meta.fetch('k_passes', 2))
169+
new(graph: plane_graph, classes: classes, k_passes: meta.fetch("k_passes", 2))
169170
ensure
170171
zf&.close
171172
end
172173
end
173174

174175
module_function
175176

177+
# rubocop:enable Lint/IneffectiveAccessModifier
178+
176179
def render_plane(skeleton, char_classes)
177180
PlaneModel.render_plane(skeleton, char_classes)
178181
end

0 commit comments

Comments
 (0)