From 46e10d7b9d509e05fde05f581ff605f2ef2de06d Mon Sep 17 00:00:00 2001 From: Brian Yahn Date: Fri, 31 Jul 2026 14:59:09 +0000 Subject: [PATCH 01/11] Compiler fixes, tooling, and test coverage from the self-host run Everything from self-host-i except the generated self-hosted tree (compiler/src, compiler/.src-transpiled), the ruby-to-clear transpiler (gems/ruby-to-clear), and the fact-mine/lineage gem work (carried separately on fact-mine-aliasing-exp), so this can land on master while self-host-i stays unmerged. Compiler fixes (each with a regression test): - Type#needs_cleanup? made non-owning strings owning under a wrapper. Symbols are interned and raw strings borrowed, so ?String@symbol / !String@symbol must not need cleanup; the classifier then asked for a DROP the lifecycle plan refused and lowering aborted. Both wrapper branches now use owning_string?. - visit_GetIndex crashed with an internal RuntimeError when indexing a struct with no element type instead of reporting UNSUPPORTED_INDEX. - resolve_inherent_static_call! left the resolved return type unstamped. - A lazy require of WithMatchCheck used a wrong path, masked by load order. - Removed the dead HasExpression module: all 22 includers are Structs whose generated reader shadows it, and it broke AOT compilation. Tooling: - tools/sorbet_strip.rb mirrors compiler/ruby to compiler/.ruby-rbs with Sorbet removed plus RBS sig export; 2.2x faster builds (203s -> 93s on the self-host parser), verified by byte-identical Zig on 5452 programs (tools/zig_equivalence.rb). - tools/selfhost_build.sh builds against that mirror and regenerates it when compiler/ruby is newer -- a stale mirror silently rebuilds pre-fix sources. - Incremental compilation now fingerprints tokens outside function bodies rather than raw source, so a comment edit no longer forces a full rebuild (15.8s -> 2.7s on vm.clear). - parser_compat.rb reads both REQUIRE spellings, so cyclic module clusters are found and merged into multi-file packages instead of reporting a circular dependency. Also: 4 fuzz cells covering non-owning strings under a wrapper (an axis the corpus never composed), and a .gitignore rule that was silently ignoring the self-hosted parser sources. CI follow-ups on this branch: - Sorbet EnforceSignatures: four attr_readers in type.rb schema classes had no sig; typed from their T.let declarations rather than autocorrected to T.untyped. - sorbet/config now ignores compiler/.ruby-rbs, the generated stripped mirror. It is gitignored so CI never saw it, but locally it collided with compiler/ruby and produced 9488 phantom errors. - Three benchmarks still used the retired `~?T[]` stream syntax, which this branch's parser rejects; converted to `[~]T` / `[~]@split T`. - IF-EXISTS captures: 119dc9506f excluded node_reference? from the mutable-slot-payload rule, so `IF nodes[i] EXISTS AS n THEN n.f = ...` failed as an immutable-field assignment. A @node handle is a pointer into the NodeStore, so assigning through the capture lands in the stored node; the exclusion is now Rc-only. Regression test added. - Two corpus sources needed migrating to the new carrier rules (rule 6, added deliberately by 9f39b5628e): mal's envSet fans `val` out to a map store and a recursive call, so it takes MONOMORPHIC and KEEPs at the fan-out; the kvstore benchmark assigned a plain local across three exclusive arms with no ELSE, so it uses COPY. Sorbet type-check: master is clean, this branch had 237 errors. Fixed: - 153 redundant T.cast/T.must the branch's sharper types made useless, plus the multi-line cases Sorbet's own autocorrect removes. - T.bind(self, ...) added to the module methods that call into their includer, matching each file's existing idiom (the annotator domains bind without `rescue nil`, which otherwise makes Sorbet ignore the bind). - escape_analysis#matches? takes a BasicObject, so its is_a? checks need T.unsafe, as master does. - protocol_projection_resolver#issue takes **values; a kwargs splat is typed by its value type, not as a Hash. - dotted_type_value_zig_name called module_eval(namespace), which is Ruby's Module#module_eval and does not exist on the instance -- it would raise at runtime. Restored master's "#{namespace}.#{node.field}". - Loop-reassigned locals declared with T.let; DiagnosticRegistry.format_from_hash can return nil; LSP diagnostics templates are only Strings; mod_alias normalized to String-or-nil; make_rc_retain guarded on Identifier. - sorbet/config ignores compiler/.ruby-rbs and scratch/, both generated or local and gitignored. scratch/struct_flow.rb defines a conflicting `Token = Struct.new(:type, :value)`, which made Sorbet believe Token#initialize took 0..2 args -- that is what led `srb tc -a` to silently delete arguments from Token.new/ParserError.new call sites. - sorbet/rbi/clear-attr-accessors.rbi regenerated; CI diffs it against the generator and the committed copy was 35 lines stale. Architecture SARIF: the job hung until its 45-minute deadline on every branch, holding a runner and blocking its concurrency group, so queued jobs on other PRs starved behind it. fact-mine's call-resolution pass was being handed gems/lineage/src/ui/assets/diff/assets/index--_D05yx7.js -- a 222 KB Vite bundle, 61 lines of minified JavaScript. corpus_common already skips *.min.js and dist/build/node_modules, but these bundles sit under .../src/ui/assets/, which nothing matched. Excluded by path prefix rather than adding `assets` to EXCLUDE_DIRS, so hand-written assets elsewhere stay in the corpus; the three other assets/ directories are already covered by `docs`. The gigasail path is the same tree after the lineage -> gigasail rename, which is why gigasail-master hangs too -- that branch moved the bundles, it did not exclude them. Corpus drops from 723 to 721 files (exactly the two bundles). The job's three scripts now finish in ~4.5 min instead of timing out: cycle_report 136s, reach_through 133s, change_coupling 1s, each emitting valid SARIF. Gates run locally: 7877 specs, 664 transpile-tests (0 leaks), buildable corpus, benchmark transpile coverage shards 1 and 2, Rubocop EnforceSignatures, RBI freshness, and srb tc clean on a pristine checkout. Co-authored-by: Codex Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01AkBJZMTAuVZCVrghaLWXEh --- .gitignore | 8 +- .../cheat-lang/syntaxes/clear.tmLanguage.json | 5 - CLAUDE.md | 2 +- .../concurrent/05_backpressure/bench.clear | 2 +- benchmarks/concurrent/08_pubsub/README.md | 2 +- benchmarks/concurrent/08_pubsub/bench.clear | 4 +- .../concurrent/12_false_sharing/bench.clear | 2 +- .../concurrent/14_nested_lock/bench.clear | 2 +- .../concurrent/16_observables/bench.clear | 2 +- benchmarks/server/01_tcp_kvstore/server.clear | 6 +- clear | 63 +- .../ruby/annotator/domains/control_flow.rb | 129 +- compiler/ruby/annotator/domains/errors.rb | 68 +- .../annotator/domains/execution_boundaries.rb | 11 +- .../ruby/annotator/domains/expressions.rb | 39 +- compiler/ruby/annotator/domains/lifetimes.rb | 185 +- .../ruby/annotator/domains/member_access.rb | 22 +- compiler/ruby/annotator/domains/variables.rb | 175 +- .../ruby/annotator/helpers/auto_inference.rb | 2 - .../ruby/annotator/helpers/capabilities.rb | 2 +- .../ruby/annotator/helpers/fixable_helpers.rb | 56 +- .../annotator/helpers/function_analysis.rb | 238 ++- .../annotator/helpers/function_context.rb | 16 +- .../ruby/annotator/helpers/function_return.rb | 36 +- .../annotator/helpers/function_signature.rb | 57 +- .../helpers/function_signature_returns.rb | 2 +- .../annotator/helpers/generic_analysis.rb | 150 +- .../ruby/annotator/helpers/method_analysis.rb | 8 +- .../ruby/annotator/helpers/pipe_analysis.rb | 129 +- compiler/ruby/annotator/helpers/union.rb | 17 +- .../annotator/phases/annotation_products.rb | 6 +- .../phases/capability_audit_session.rb | 6 + .../annotator/phases/capability_evidence.rb | 27 + .../phases/conformance_registration.rb | 13 +- .../annotator/phases/declaration_index.rb | 14 +- .../annotator/phases/deferred_validation.rb | 46 + .../annotator/phases/expression_domains.rb | 15 +- .../annotator/phases/import_resolution.rb | 2 +- .../phases/signature_registration.rb | 2 +- .../annotator/phases/signature_registry.rb | 31 +- .../annotator/phases/type_analysis_session.rb | 173 +- .../annotator/phases/type_registration.rb | 91 +- .../phases/whole_program_semantics.rb | 4 +- .../annotator/protocol_projection_resolver.rb | 38 +- compiler/ruby/ast/ast.rb | 497 ++++- compiler/ruby/ast/async_result_shape.rb | 1 + compiler/ruby/ast/diagnostic_buckets.rb | 26 +- compiler/ruby/ast/diagnostic_examples.rb | 17 +- compiler/ruby/ast/diagnostic_registry.rb | 207 +- compiler/ruby/ast/error_registry.rb | 54 +- compiler/ruby/ast/fixable_error.rb | 77 +- compiler/ruby/ast/frontend_resource_budget.rb | 15 +- compiler/ruby/ast/lexer.rb | 78 +- compiler/ruby/ast/param.rb | 50 + compiler/ruby/ast/parser.rb | 43 +- .../parser/declarations_and_definitions.rb | 61 +- .../ast/parser/expressions_and_postfix.rb | 52 +- .../ast/parser/predicates_and_refinements.rb | 6 +- compiler/ruby/ast/parser/state.rb | 81 +- .../ast/parser/statements_and_control_flow.rb | 53 +- compiler/ruby/ast/parser/types.rb | 53 +- compiler/ruby/ast/schemas.rb | 530 ------ compiler/ruby/ast/scope.rb | 131 +- compiler/ruby/ast/source_error.rb | 78 +- compiler/ruby/ast/std_lib.rb | 30 +- compiler/ruby/ast/struct_field.rb | 8 +- compiler/ruby/ast/symbol_entry.rb | 295 ++- compiler/ruby/ast/syntax_typo_scanner.rb | 16 +- compiler/ruby/ast/type.rb | 1683 +++++++++++------ compiler/ruby/ast/type_capabilities.rb | 260 +++ compiler/ruby/ast/type_expression.rb | 640 ++++--- compiler/ruby/backends/mir_emitter.rb | 144 +- compiler/ruby/backends/transpiler.rb | 33 +- compiler/ruby/backends/type_zig_renderer.rb | 2 + compiler/ruby/backends/zig_type.rb | 1 + compiler/ruby/backends/zig_type_mapper.rb | 4 +- compiler/ruby/compiler/compiler_frontend.rb | 15 +- compiler/ruby/compiler/module_importer.rb | 82 +- compiler/ruby/compiler/package_source.rb | 195 ++ compiler/ruby/ffi/c_header_importer.rb | 119 +- .../ruby/incremental/dependency_snapshot.rb | 21 +- compiler/ruby/incremental/portable_cache.rb | 8 +- compiler/ruby/incremental/source_catalog.rb | 86 +- compiler/ruby/lsp/diagnostics.rb | 2 +- compiler/ruby/lsp/logger.rb | 2 +- compiler/ruby/lsp/server.rb | 2 +- compiler/ruby/mir/alloc.rb | 5 +- compiler/ruby/mir/cleanup_classifier.rb | 58 +- compiler/ruby/mir/cleanup_entry.rb | 2 +- compiler/ruby/mir/control_flow.rb | 6 +- compiler/ruby/mir/fsm_lowering.rb | 35 +- compiler/ruby/mir/fsm_ops.rb | 26 +- compiler/ruby/mir/fsm_transform/emit.rb | 6 +- compiler/ruby/mir/fsm_transform/liveness.rb | 15 +- .../mir/fsm_transform/recursive_splitter.rb | 34 +- compiler/ruby/mir/fsm_transform/segments.rb | 180 +- compiler/ruby/mir/hoist.rb | 127 +- .../pipeline/pipeline_batch_window_lowerer.rb | 35 +- .../pipeline_binding_chain_lowerer.rb | 37 +- .../pipeline/pipeline_concurrent_lowerer.rb | 4 +- .../mir/lower/pipeline/pipeline_context.rb | 199 +- .../lower/pipeline/pipeline_each_lowerer.rb | 24 +- .../ruby/mir/lower/pipeline/pipeline_host.rb | 234 ++- .../lower/pipeline/pipeline_list_lowerer.rb | 173 +- .../pipeline/pipeline_lowering_bridge.rb | 14 + .../lower/pipeline/pipeline_materializer.rb | 137 +- .../pipeline/pipeline_placeholder_usage.rb | 45 +- .../lower/pipeline/pipeline_range_lowerer.rb | 346 ++-- .../mir/lower/pipeline/pipeline_records.rb | 13 + .../pipeline/pipeline_set_index_lowerer.rb | 140 +- compiler/ruby/mir/lowering/capabilities.rb | 33 +- compiler/ruby/mir/lowering/concurrency.rb | 55 +- compiler/ruby/mir/lowering/control_flow.rb | 102 +- compiler/ruby/mir/lowering/counters.rb | 30 +- compiler/ruby/mir/lowering/expressions.rb | 355 +++- compiler/ruby/mir/lowering/functions.rb | 315 ++- compiler/ruby/mir/lowering/literals.rb | 8 + compiler/ruby/mir/lowering/schema_registry.rb | 26 +- compiler/ruby/mir/lowering/state.rb | 16 + compiler/ruby/mir/lowering/variables.rb | 161 +- compiler/ruby/mir/mir.rb | 172 +- compiler/ruby/mir/mir_checker.rb | 72 +- compiler/ruby/mir/mir_lowering.rb | 575 ++++-- compiler/ruby/mir/mir_pass.rb | 39 +- compiler/ruby/mir/pre_mir_type_check.rb | 7 +- compiler/ruby/mir/program_mir_facts.rb | 18 +- .../ruby/mir/rewriters/pipeline_rewriter.rb | 154 +- .../mir/rewriters/string_concat_rewriter.rb | 90 +- compiler/ruby/mir/test_lowering.rb | 35 +- .../mir/thunk_transform/recursive_splitter.rb | 53 +- compiler/ruby/semantic/capability_plan.rb | 22 +- compiler/ruby/semantic/capture_strategy.rb | 1 + compiler/ruby/semantic/effect_inference.rb | 21 +- compiler/ruby/semantic/effect_set.rb | 32 +- compiler/ruby/semantic/escape_analysis.rb | 422 ++++- compiler/ruby/semantic/keep_analysis.rb | 77 + compiler/ruby/semantic/lifecycle_plan.rb | 298 ++- .../ruby/semantic/ownership_edge_planner.rb | 115 ++ compiler/ruby/semantic/ownership_graph.rb | 79 +- compiler/ruby/semantic/ownership_identity.rb | 15 +- compiler/ruby/semantic/ownership_transport.rb | 173 +- compiler/ruby/semantic/pass_work_profiler.rb | 38 +- .../ruby/semantic/tense_operation_plan.rb | 196 +- compiler/ruby/tools/clear_build_support.rb | 83 +- compiler/ruby/tools/clear_fix_support.rb | 7 +- compiler/ruby/tools/lint_fix_rewriter.rb | 5 +- compiler/ruby/tools/method_rewriter.rb | 23 +- compiler/ruby/tools/predicate_rewriter.rb | 23 +- compiler/spec/allocation_strategy_spec.rb | 56 + compiler/spec/annotator_gap_burndown_spec.rb | 15 +- compiler/spec/annotator_spec.rb | 21 +- compiler/spec/architecture_invariants_spec.rb | 4 +- compiler/spec/ast_coverage_burndown_spec.rb | 12 +- compiler/spec/auto_box_tdd_spec.rb | 2 +- .../spec/auto_ownership_transport_spec.rb | 4 +- compiler/spec/capabilities_spec.rb | 54 +- compiler/spec/carrier_contract_symbol_spec.rb | 30 + .../spec/carrier_fanout_diagnostic_spec.rb | 39 + compiler/spec/carrier_static_rules_spec.rb | 54 + compiler/spec/cleanup_plan_spec.rb | 3 +- compiler/spec/clear_build_support_spec.rb | 84 + .../spec/clear_cli_module_scope_owned_spec.rb | 95 + .../spec/clear_cli_registered_pkg_spec.rb | 121 ++ compiler/spec/copy_retained_unique_spec.rb | 71 + compiler/spec/error_emission_coverage_spec.rb | 93 +- compiler/spec/error_registry_spec.rb | 32 +- .../extern_resource_raii_integration_spec.rb | 2 +- .../spec/fixable_use_after_move_clone_spec.rb | 18 +- compiler/spec/fsm_classifier_spec.rb | 6 +- compiler/spec/fsm_lowering_spec.rb | 4 +- compiler/spec/fsm_ops_spec.rb | 2 +- compiler/spec/gen_attr_rbi_spec.rb | 14 +- compiler/spec/generic_map_protocol_spec.rb | 4 +- .../spec/generic_protocol_conformance_spec.rb | 10 +- compiler/spec/generics_spec.rb | 26 +- compiler/spec/higher_order_spec.rb | 45 +- .../incremental_compilation_spec.rb | 9 +- .../fixtures/frontend_oracles/smoke.ast.json | 3 +- .../seed_20260715_case_3.clear.bin | 0 .../seed_20260715_case_8.clear.bin | Bin 0 -> 15 bytes .../pipeline_type_preservation_spec.rb | 4 +- compiler/spec/keep_carrier_lowering_spec.rb | 92 + compiler/spec/keep_carrier_op_stamp_spec.rb | 36 + compiler/spec/keep_optional_strict_spec.rb | 47 + compiler/spec/kept_identity_spec.rb | 621 ++++++ compiler/spec/lexer_compat_harness_spec.rb | 60 + compiler/spec/lexer_spec.rb | 20 + compiler/spec/lifecycle_plan_spec.rb | 99 + compiler/spec/mir_checker_spec.rb | 30 +- compiler/spec/mir_emitter_spec.rb | 21 +- compiler/spec/mir_gap_burn_spec.rb | 16 +- compiler/spec/mir_lowering_spec.rb | 169 +- compiler/spec/monomorphic_lowering_spec.rb | 125 ++ compiler/spec/multi_file_package_spec.rb | 129 ++ compiler/spec/observable_array_no_set_spec.rb | 4 +- .../spec/observable_cleanup_codegen_spec.rb | 4 +- ...observable_collection_next_consume_spec.rb | 4 +- compiler/spec/observable_i1_fix_spec.rb | 2 +- compiler/spec/observable_lockdown_spec.rb | 8 +- compiler/spec/observable_nested_spec.rb | 10 +- compiler/spec/observable_pipe_dest_spec.rb | 6 +- .../observable_terminal_inference_spec.rb | 8 +- .../observable_terminal_validation_spec.rb | 12 +- ...composite_argument_materialization_spec.rb | 86 +- compiler/spec/ownership_edge_planner_spec.rb | 53 + compiler/spec/ownership_graph_spec.rb | 11 + .../spec/ownership_surface_matrix_spec.rb | 312 +++ compiler/spec/package_source_spec.rb | 132 ++ compiler/spec/parser_carrier_contract_spec.rb | 110 ++ .../spec/parser_collection_capability_spec.rb | 16 +- .../spec/parser_mutation_contract_spec.rb | 20 + compiler/spec/parser_typed_routes_spec.rb | 2 +- .../spec/pipeline_backend_coverage_spec.rb | 91 +- compiler/spec/pipeline_differential_spec.rb | 119 ++ .../spec/pipeline_downstream_register_spec.rb | 95 + compiler/spec/pipeline_legacy_matrix_spec.rb | 2 +- .../spec/pipeline_position_matrix_spec.rb | 435 +++++ ...polymorphic_transaction_acceptance_spec.rb | 4 +- compiler/spec/rc_bind_ownership_spec.rb | 4 +- compiler/spec/retained_plain_slot_spec.rb | 36 + compiler/spec/schemas_spec.rb | 2 +- compiler/spec/scope_composition_spec.rb | 9 + compiler/spec/share_spec.rb | 8 +- compiler/spec/shared_contract_spec.rb | 50 + compiler/spec/stream_spec.rb | 143 +- compiler/spec/symbol_spec.rb | 17 + compiler/spec/tense_operation_plan_spec.rb | 42 +- compiler/spec/type_expression_spec.rb | 190 +- compiler/spec/type_ownership_contract_spec.rb | 26 + compiler/spec/type_system_change_spec.rb | 83 +- compiler/spec/type_zig_type_gap_spec.rb | 2 +- compiler/spec/unique_exclusive_spec.rb | 37 + compiler/spec/use_after_move_dataflow_spec.rb | 4 +- compiler/spec/with_alias_escape_spec.rb | 22 +- compiler/spec/with_view_codegen_spec.rb | 4 +- compiler/spec/with_view_spec.rb | 2 +- docs/agents/carrier-first-pass-tracker.md | 123 ++ docs/agents/error-notes.md | 1373 ++++++++++++++ docs/agents/keep_or_clone.md | 572 ++++++ docs/agents/parser-type-slop.md | 417 ++-- docs/agents/retained-identity-design.md | 648 +++++++ docs/agents/retained-identity-tracker.md | 480 +++++ docs/agents/retained-identity-v5-tracker.md | 206 ++ ...to-clear-semantic-architecture-recovery.md | 196 ++ docs/agents/self-host-plan.md | 1451 +++++++++----- docs/agents/self-host.md | 284 ++- docs/collections.md | 15 +- docs/pipeline-placement-extent.md | 227 +++ docs/tense-composition.md | 14 +- examples/mal/interpreter.clear | 4 +- examples/minivm/types.clear | 2 +- gems/espalier/tools/corpus_common.rb | 14 + package-lock.json | 2 +- sorbet/config | 2 + sorbet/rbi/clear-attr-accessors.rbi | 124 +- stdlib/fs/src/lib.clear | 8 + stdlib/regex/src/lib.clear | 63 + syntaxes/cheat.tmLanguage.json | 5 - syntaxes/cheat.vim | 1 - tools/fuzz/README.md | 23 +- tools/fuzz/coverage_model.rb | 17 + .../patches/carrier_copy_polymorphic.patch | 13 + .../patches/kept_identity_mutable_model.patch | 13 + .../pipeline_element_hoist_escape.patch | 14 + ...educe_owned_accumulator_unclassified.patch | 22 + .../pipeline_skip_borrowed_source_freed.patch | 17 + .../patches/retained_needs_own_copy.patch | 13 + tools/fuzz/mutants/registry.rb | 48 + tools/fuzz/semantic_capability_expansion.rb | 22 +- tools/fuzz/semantic_equivalence.rb | 72 +- tools/fuzz/semantic_full.rb | 2 +- tools/fuzz/templates/bind_capture_cleanup.rb | 4 +- .../call_ownership_contract_matrix.rb | 231 ++- .../fuzz/templates/capability_wrap_matrix.rb | 4 +- .../templates/carrier_ownership_matrix.rb | 261 +++ .../templates/cleanup_classifier_shapes.rb | 66 +- .../fuzz/templates/cleanup_control_matrix.rb | 24 +- tools/fuzz/templates/cross_fiber_consumer.rb | 6 +- .../templates/diagnostic_policy_matrix.rb | 2 +- .../fuzz/templates/heap_ownership_transfer.rb | 6 +- tools/fuzz/templates/kept_identity_matrix.rb | 361 ++++ .../templates/lowering_boundary_matrix.rb | 4 +- tools/fuzz/templates/module_const_matrix.rb | 152 ++ tools/fuzz/templates/next_tense_matrix.rb | 4 +- .../templates/or_heap_destination_matrix.rb | 28 +- .../pipeline_composite_element_matrix.rb | 185 ++ tools/fuzz/templates/pipeline_gap_matrix.rb | 50 + .../templates/pipeline_source_shape_matrix.rb | 34 +- .../templates/rc_generic_collection_matrix.rb | 5 +- tools/fuzz/templates/stream_into_boundary.rb | 14 +- .../fuzz/templates/stream_selector_matrix.rb | 98 + tools/fuzz/templates/takes_move_modality.rb | 2 +- tools/gen_attr_rbi.rb | 11 - tools/lexer_compat.rb | 302 ++- tools/lexer_harness_support.rb | 4 + tools/parser_build.sh | 6 + tools/parser_compat.rb | 148 +- tools/rtoc_golden_facts.json.gz | Bin 0 -> 450452 bytes tools/rtoc_regression_scan.rb | 251 +++ tools/selfhost_build.sh | 38 + tools/sorbet_strip.rb | 735 +++++++ tools/sorbet_strip_test.rb | 100 + tools/spinel_shims/logger.rb | 33 + tools/spinel_shims/open3.rb | 16 + tools/spinel_shims/tmpdir.rb | 4 + tools/zig_equivalence.rb | 123 ++ transpile-tests/225_split_stream.clear | 70 +- .../226_split_stream_clone_edges.clear | 16 +- .../241_open_stream_pipelines.clear | 38 +- transpile-tests/242_concurrent_capacity.clear | 6 +- transpile-tests/243_batch_window.clear | 4 +- .../258_bg_body_copy_capture.clear | 2 +- transpile-tests/303_observable_pipe_sum.clear | 2 +- .../304_observable_with_view.clear | 2 +- transpile-tests/305_observable_collect.clear | 2 +- transpile-tests/306_observable_default.clear | 2 +- .../307_observable_pipe_count.clear | 2 +- transpile-tests/308_observable_pipe_max.clear | 2 +- transpile-tests/309_observable_pipe_min.clear | 2 +- transpile-tests/310_observable_pipe_any.clear | 2 +- transpile-tests/311_observable_pipe_all.clear | 4 +- transpile-tests/312_observable_pipe_avg.clear | 2 +- .../313_observable_pipe_find.clear | 2 +- .../314_observable_pipe_reduce.clear | 2 +- .../315_observable_pipe_distinct.clear | 2 +- .../316_observable_multi_pipe.clear | 4 +- .../317_observable_empty_stream.clear | 20 +- ...18_observable_mixed_shape_multi_pipe.clear | 4 +- ...9_observable_with_view_all_terminals.clear | 16 +- .../320_observable_materialized_view.clear | 4 +- .../321_observable_producer_error.clear | 2 +- .../322_observable_distinct_string.clear | 2 +- ...4_observable_distinct_producer_error.clear | 2 +- .../325_observable_mat_view_error_path.clear | 2 +- .../326_observable_pipe_find_string.clear | 2 +- ...327_observable_reduce_producer_error.clear | 2 +- transpile-tests/612_for_underscore_var.clear | 17 + .../612_stream_pipeline_string_cleanup.clear | 4 +- .../613_fixed_array_contains.clear | 11 + .../614_narrowed_optional_union.clear | 33 + .../615_stream_select_observe_release.clear | 50 + .../616_stream_select_identity_transfer.clear | 29 + .../617_stream_select_move_copy.clear | 87 + .../618_stream_select_kinds_and_limits.clear | 61 + .../619_kept_identity_shared.clear | 22 + .../620_kept_identity_transitive.clear | 24 + .../621_kept_identity_last_use.clear | 17 + .../622_kept_identity_born_as_rc.clear | 21 + .../623_kept_identity_two_params.clear | 26 + .../624_kept_identity_error_path.clear | 27 + .../625_kept_identity_optional_default.clear | 22 + ...6_kept_identity_fallible_sibling_arg.clear | 36 + .../627_kept_identity_expression_args.clear | 32 + .../628_kept_identity_field_assign.clear | 24 + transpile-tests/630_v5_keep_multiowned.clear | 14 + transpile-tests/631_v5_keep_shared.clear | 12 + .../632_v5_shared_to_unique_copy.clear | 17 + transpile-tests/633_v5_last_use_move.clear | 14 + .../634_v5_shared_registration.clear | 19 + transpile-tests/635_v5_unique_exclusive.clear | 16 + transpile-tests/636_v5_own_copy_detach.clear | 16 + .../637_v5_monomorphic_thread.clear | 20 + transpile-tests/638_v5_monomorphic_keep.clear | 27 + .../639_v5_monomorphic_keep_heapfield.clear | 24 + .../640_v5_monomorphic_method.clear | 17 + .../641_v5_monomorphic_own_copy_return.clear | 18 + transpile-tests/642_module_const.clear | 30 + transpile-tests/643_runtime_init_const.clear | 37 + .../644_select_composite_owned_element.clear | 58 + .../645_reduce_owned_accumulator.clear | 40 + ...6_reduce_composite_accumulator_field.clear | 32 + .../647_pipeline_skip_limit_select.clear | 40 + ...648_monomorphic_ctrl_field_collision.clear | 29 + .../649_stream_select_owned_selector.clear | 58 + transpile-tests/650_continue_owned_temp.clear | 19 + transpile-tests/651_break_owned_temp.clear | 19 + .../652_raise_mid_list_build.clear | 28 + .../653_orelse_owned_fallback.clear | 29 + .../654_map_string_value_overwrite.clear | 19 + transpile-tests/655_field_self_update.clear | 17 + transpile-tests/656_nested_string_lists.clear | 30 + .../657_pool_remove_owned_payload.clear | 29 + transpile-tests/658_set_duplicate_freed.clear | 15 + .../659_copy_bag_independence.clear | 22 + .../660_window_owned_fn_result.clear | 13 + .../661_pipeline_in_lambda_body.clear | 6 + .../662_bg_stream_in_test_that.clear | 30 + transpile-tests/663_bg_in_test_that.clear | 15 + transpile-tests/664_defer_statement.clear | 46 + .../665_map_struct_exists_mutation.clear | 32 + .../666_own_copy_binding_detach.clear | 56 + .../667_map_keys_values_return_type.clear | 27 + .../668_symbol_set_and_collection_slots.clear | 46 + .../669_catch_question_mark_name.clear | 26 + .../670_value_block_bare_if_result.clear | 89 + ..._interpolated_assignment_value_block.clear | 54 + transpile-tests/75_open_stream.clear | 75 +- transpile-tests/gen.rb | 3 + .../node_handle_ifexists_field_assign.clear | 32 + .../optional_symbol_struct_field.clear | 42 + zig/build.zig | 1 + zig/lib/data-structures-test.zig | 85 + zig/lib/data-structures.zig | 152 +- zig/rc-keep-edge-test.zig | 86 + zig/runtime/runtime-header.zig | 56 +- 405 files changed, 25311 insertions(+), 5148 deletions(-) delete mode 100644 compiler/ruby/ast/schemas.rb create mode 100644 compiler/ruby/ast/type_capabilities.rb create mode 100644 compiler/ruby/compiler/package_source.rb create mode 100644 compiler/ruby/semantic/keep_analysis.rb create mode 100644 compiler/ruby/semantic/ownership_edge_planner.rb create mode 100644 compiler/spec/carrier_contract_symbol_spec.rb create mode 100644 compiler/spec/carrier_fanout_diagnostic_spec.rb create mode 100644 compiler/spec/carrier_static_rules_spec.rb create mode 100644 compiler/spec/clear_cli_module_scope_owned_spec.rb create mode 100644 compiler/spec/clear_cli_registered_pkg_spec.rb create mode 100644 compiler/spec/copy_retained_unique_spec.rb create mode 100644 compiler/spec/integration/fixtures/hostile_frontend/seed_20260715_case_3.clear.bin create mode 100644 compiler/spec/integration/fixtures/hostile_frontend/seed_20260715_case_8.clear.bin create mode 100644 compiler/spec/keep_carrier_lowering_spec.rb create mode 100644 compiler/spec/keep_carrier_op_stamp_spec.rb create mode 100644 compiler/spec/keep_optional_strict_spec.rb create mode 100644 compiler/spec/kept_identity_spec.rb create mode 100644 compiler/spec/lexer_compat_harness_spec.rb create mode 100644 compiler/spec/monomorphic_lowering_spec.rb create mode 100644 compiler/spec/multi_file_package_spec.rb create mode 100644 compiler/spec/ownership_edge_planner_spec.rb create mode 100644 compiler/spec/ownership_surface_matrix_spec.rb create mode 100644 compiler/spec/package_source_spec.rb create mode 100644 compiler/spec/parser_carrier_contract_spec.rb create mode 100644 compiler/spec/pipeline_differential_spec.rb create mode 100644 compiler/spec/pipeline_downstream_register_spec.rb create mode 100644 compiler/spec/pipeline_position_matrix_spec.rb create mode 100644 compiler/spec/retained_plain_slot_spec.rb create mode 100644 compiler/spec/shared_contract_spec.rb create mode 100644 compiler/spec/unique_exclusive_spec.rb create mode 100644 docs/agents/carrier-first-pass-tracker.md create mode 100644 docs/agents/error-notes.md create mode 100644 docs/agents/keep_or_clone.md create mode 100644 docs/agents/retained-identity-design.md create mode 100644 docs/agents/retained-identity-tracker.md create mode 100644 docs/agents/retained-identity-v5-tracker.md create mode 100644 docs/agents/ruby-to-clear-semantic-architecture-recovery.md create mode 100644 docs/pipeline-placement-extent.md create mode 100644 stdlib/regex/src/lib.clear create mode 100644 tools/fuzz/mutants/patches/carrier_copy_polymorphic.patch create mode 100644 tools/fuzz/mutants/patches/kept_identity_mutable_model.patch create mode 100644 tools/fuzz/mutants/patches/pipeline_element_hoist_escape.patch create mode 100644 tools/fuzz/mutants/patches/pipeline_reduce_owned_accumulator_unclassified.patch create mode 100644 tools/fuzz/mutants/patches/pipeline_skip_borrowed_source_freed.patch create mode 100644 tools/fuzz/mutants/patches/retained_needs_own_copy.patch create mode 100644 tools/fuzz/templates/carrier_ownership_matrix.rb create mode 100644 tools/fuzz/templates/kept_identity_matrix.rb create mode 100644 tools/fuzz/templates/module_const_matrix.rb create mode 100644 tools/fuzz/templates/pipeline_composite_element_matrix.rb create mode 100644 tools/fuzz/templates/stream_selector_matrix.rb create mode 100755 tools/parser_build.sh create mode 100644 tools/rtoc_golden_facts.json.gz create mode 100644 tools/rtoc_regression_scan.rb create mode 100755 tools/selfhost_build.sh create mode 100644 tools/sorbet_strip.rb create mode 100644 tools/sorbet_strip_test.rb create mode 100644 tools/spinel_shims/logger.rb create mode 100644 tools/spinel_shims/open3.rb create mode 100644 tools/spinel_shims/tmpdir.rb create mode 100644 tools/zig_equivalence.rb create mode 100644 transpile-tests/612_for_underscore_var.clear create mode 100644 transpile-tests/613_fixed_array_contains.clear create mode 100644 transpile-tests/614_narrowed_optional_union.clear create mode 100644 transpile-tests/615_stream_select_observe_release.clear create mode 100644 transpile-tests/616_stream_select_identity_transfer.clear create mode 100644 transpile-tests/617_stream_select_move_copy.clear create mode 100644 transpile-tests/618_stream_select_kinds_and_limits.clear create mode 100644 transpile-tests/619_kept_identity_shared.clear create mode 100644 transpile-tests/620_kept_identity_transitive.clear create mode 100644 transpile-tests/621_kept_identity_last_use.clear create mode 100644 transpile-tests/622_kept_identity_born_as_rc.clear create mode 100644 transpile-tests/623_kept_identity_two_params.clear create mode 100644 transpile-tests/624_kept_identity_error_path.clear create mode 100644 transpile-tests/625_kept_identity_optional_default.clear create mode 100644 transpile-tests/626_kept_identity_fallible_sibling_arg.clear create mode 100644 transpile-tests/627_kept_identity_expression_args.clear create mode 100644 transpile-tests/628_kept_identity_field_assign.clear create mode 100644 transpile-tests/630_v5_keep_multiowned.clear create mode 100644 transpile-tests/631_v5_keep_shared.clear create mode 100644 transpile-tests/632_v5_shared_to_unique_copy.clear create mode 100644 transpile-tests/633_v5_last_use_move.clear create mode 100644 transpile-tests/634_v5_shared_registration.clear create mode 100644 transpile-tests/635_v5_unique_exclusive.clear create mode 100644 transpile-tests/636_v5_own_copy_detach.clear create mode 100644 transpile-tests/637_v5_monomorphic_thread.clear create mode 100644 transpile-tests/638_v5_monomorphic_keep.clear create mode 100644 transpile-tests/639_v5_monomorphic_keep_heapfield.clear create mode 100644 transpile-tests/640_v5_monomorphic_method.clear create mode 100644 transpile-tests/641_v5_monomorphic_own_copy_return.clear create mode 100644 transpile-tests/642_module_const.clear create mode 100644 transpile-tests/643_runtime_init_const.clear create mode 100644 transpile-tests/644_select_composite_owned_element.clear create mode 100644 transpile-tests/645_reduce_owned_accumulator.clear create mode 100644 transpile-tests/646_reduce_composite_accumulator_field.clear create mode 100644 transpile-tests/647_pipeline_skip_limit_select.clear create mode 100644 transpile-tests/648_monomorphic_ctrl_field_collision.clear create mode 100644 transpile-tests/649_stream_select_owned_selector.clear create mode 100644 transpile-tests/650_continue_owned_temp.clear create mode 100644 transpile-tests/651_break_owned_temp.clear create mode 100644 transpile-tests/652_raise_mid_list_build.clear create mode 100644 transpile-tests/653_orelse_owned_fallback.clear create mode 100644 transpile-tests/654_map_string_value_overwrite.clear create mode 100644 transpile-tests/655_field_self_update.clear create mode 100644 transpile-tests/656_nested_string_lists.clear create mode 100644 transpile-tests/657_pool_remove_owned_payload.clear create mode 100644 transpile-tests/658_set_duplicate_freed.clear create mode 100644 transpile-tests/659_copy_bag_independence.clear create mode 100644 transpile-tests/660_window_owned_fn_result.clear create mode 100644 transpile-tests/661_pipeline_in_lambda_body.clear create mode 100644 transpile-tests/662_bg_stream_in_test_that.clear create mode 100644 transpile-tests/663_bg_in_test_that.clear create mode 100644 transpile-tests/664_defer_statement.clear create mode 100644 transpile-tests/665_map_struct_exists_mutation.clear create mode 100644 transpile-tests/666_own_copy_binding_detach.clear create mode 100644 transpile-tests/667_map_keys_values_return_type.clear create mode 100644 transpile-tests/668_symbol_set_and_collection_slots.clear create mode 100644 transpile-tests/669_catch_question_mark_name.clear create mode 100644 transpile-tests/670_value_block_bare_if_result.clear create mode 100644 transpile-tests/671_interpolated_assignment_value_block.clear create mode 100644 transpile-tests/node_handle_ifexists_field_assign.clear create mode 100644 transpile-tests/optional_symbol_struct_field.clear create mode 100644 zig/rc-keep-edge-test.zig diff --git a/.gitignore b/.gitignore index 94e60e7a5..f8b7bc13a 100644 --- a/.gitignore +++ b/.gitignore @@ -186,7 +186,6 @@ gems/**/target/** # Self-host bootstrap build outputs (sources use explicit extensions). compiler/src/ast/lexer -compiler/src/ast/parser # Generated gem diagnostics and local analysis output. gems/**/*.log @@ -213,3 +212,10 @@ gems/fact-mine/target/ gems/hazard-contract/target/ gems/nil-kill/target/ kcov-bin/ + +# Generated Sorbet-stripped mirror of compiler/ruby (tools/sorbet_strip.rb) +compiler/.ruby-rbs/ +# Transient stash used by tools/sorbet_strip_test.rb while it swaps trees +compiler/.ruby-original/ +# Local Spinel fork used for the AOT experiment +tmp/spinel/ diff --git a/.vscode/extensions/cheat-lang/syntaxes/clear.tmLanguage.json b/.vscode/extensions/cheat-lang/syntaxes/clear.tmLanguage.json index b8f82cf9e..0c1fed5fc 100644 --- a/.vscode/extensions/cheat-lang/syntaxes/clear.tmLanguage.json +++ b/.vscode/extensions/cheat-lang/syntaxes/clear.tmLanguage.json @@ -176,11 +176,6 @@ }, "sigils": { "patterns": [ - { - "comment": "Explicit panic operator", - "name": "keyword.operator.panic.clear", - "match": "!!" - }, { "comment": "Mutation suffix on identifiers (foo!, increment!) — must follow a word", "name": "keyword.operator.mutation.clear", diff --git a/CLAUDE.md b/CLAUDE.md index 2f1fc7f81..0da97a20b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -111,7 +111,7 @@ Reference docs: `mir-bugs.md` (known MIR violations), `alloc-bugs.md` (frame-the ## Language Semantics -**Sigils:** `$` pipeline/interp, `!` mutation, `|>` SMOOTH (safe pipeline w/ error prop), `_` placeholder, `!!` explicit panic. +**Sigils:** `$` pipeline/interp, `&` explicit mutable call-site path, `|>` SMOOTH (safe pipeline w/ error prop), `_` placeholder, `TRY` explicit propagation. **Ownership / capabilities — bindings, not types.** Two sigil groups: - **Group 1 (sync / ownership wrappers):** `@locked`, `@writeLocked`, `@shared` (Arc), `@multiowned` (Rc), `@local`. Stored on `SymbolEntry#sync` and `#storage`. Composed via `MIR::CapWrap`. diff --git a/benchmarks/concurrent/05_backpressure/bench.clear b/benchmarks/concurrent/05_backpressure/bench.clear index 1ba9533af..289524188 100644 --- a/benchmarks/concurrent/05_backpressure/bench.clear +++ b/benchmarks/concurrent/05_backpressure/bench.clear @@ -22,7 +22,7 @@ FN main() RETURNS Void -> acc = Acc{ value: 0 } @shared:locked; t0 = timestampMs(); - gen: ~?Int64[] = BG STREAM { + gen: [~]Int64 = BG STREAM { MUTABLE i = 0; WHILE i < 100_000 DO YIELD i; diff --git a/benchmarks/concurrent/08_pubsub/README.md b/benchmarks/concurrent/08_pubsub/README.md index 4ca1c2a62..56ed85237 100644 --- a/benchmarks/concurrent/08_pubsub/README.md +++ b/benchmarks/concurrent/08_pubsub/README.md @@ -7,7 +7,7 @@ Total work: 10K x 64 x 2000 = 1.28 billion LCG iterations. ## Implementation - **CLEAR**: publisher writes once to a `SplitStream` ring buffer. Each - subscriber `CLONE`s an independent cursor — zero message copying. Publisher + subscriber `KEEP`s an independent cursor — zero message copying. Publisher blocks only when the slowest subscriber's cursor is a full buffer behind. - **Go**: publisher loops over 64 buffered channels (cap 64), sending one message per subscriber per iteration. Publisher blocks if any channel is diff --git a/benchmarks/concurrent/08_pubsub/bench.clear b/benchmarks/concurrent/08_pubsub/bench.clear index f2e94940b..d01b64df6 100644 --- a/benchmarks/concurrent/08_pubsub/bench.clear +++ b/benchmarks/concurrent/08_pubsub/bench.clear @@ -22,13 +22,13 @@ END FN main() RETURNS Void -> t0 = timestampMs(); - msgs: ~?Msg[]@split = BG STREAM { + msgs: [~]@split Msg = BG STREAM { FOR i IN (0 ..< 10_000) -> YIELD Msg{ seed: i }; }; MUTABLE futures: ~Int64[]@list = []; FOR i IN (0 ..< 64) DO - subscriber_msgs: ~?Msg[]@split = CLONE msgs; + subscriber_msgs: [~]@split Msg = KEEP msgs; &futures .append( BG { diff --git a/benchmarks/concurrent/12_false_sharing/bench.clear b/benchmarks/concurrent/12_false_sharing/bench.clear index a898a7f2f..24641f1d7 100644 --- a/benchmarks/concurrent/12_false_sharing/bench.clear +++ b/benchmarks/concurrent/12_false_sharing/bench.clear @@ -33,7 +33,7 @@ FN main() RETURNS !Void -> MUTABLE wi = 0; WHILE wi < workers DO IF counters[wi] EXISTS AS ref0 THEN - ref = CLONE ref0; + ref = KEEP ref0; &futures .append( BG { diff --git a/benchmarks/concurrent/14_nested_lock/bench.clear b/benchmarks/concurrent/14_nested_lock/bench.clear index 85ad4d563..6ec26ced6 100644 --- a/benchmarks/concurrent/14_nested_lock/bench.clear +++ b/benchmarks/concurrent/14_nested_lock/bench.clear @@ -91,7 +91,7 @@ FN main() RETURNS Void -> MUTABLE acct_refs: []Account@shared:locked = []; WITH bank AS bk { FOR i IN (0 ..< numAccounts) DO - IF bk.accounts[i] EXISTS AS account THEN &acct_refs.append(CLONE account); END + IF bk.accounts[i] EXISTS AS account THEN &acct_refs.append(KEEP account); END END } MUTABLE total = 0; diff --git a/benchmarks/concurrent/16_observables/bench.clear b/benchmarks/concurrent/16_observables/bench.clear index 482e21f87..bd26b4c13 100644 --- a/benchmarks/concurrent/16_observables/bench.clear +++ b/benchmarks/concurrent/16_observables/bench.clear @@ -16,7 +16,7 @@ FN main() RETURNS Void -> n_writes = 2_000_000; expected = (n_writes * (n_writes - 1)) / 2; - gen: ~?Int64[] = BG STREAM { + gen: [~]Int64 = BG STREAM { MUTABLE i = 0; WHILE i < n_writes DO YIELD i; diff --git a/benchmarks/server/01_tcp_kvstore/server.clear b/benchmarks/server/01_tcp_kvstore/server.clear index 7fbe7ab82..8b9a4ba83 100644 --- a/benchmarks/server/01_tcp_kvstore/server.clear +++ b/benchmarks/server/01_tcp_kvstore/server.clear @@ -49,11 +49,11 @@ FN handleClient(client: TCPClient, MUTABLE store: {String}String, MUTABLE counte pos += len; pos += 2; IF ai == 0 THEN - arg0 = val; + arg0 = COPY val; ELSE_IF ai == 1 THEN - arg1 = val; + arg1 = COPY val; ELSE_IF ai == 2 THEN - arg2 = val; + arg2 = COPY val; END END ai += 1; diff --git a/clear b/clear index b71b4cd14..fa7f7eef3 100755 --- a/clear +++ b/clear @@ -357,12 +357,22 @@ def do_build(source, output: nil, opt_level: 'Debug', extra_flags: [], module_mo # require another package, and the Ruby frontend resolves that import before # the later per-package Zig build pass. source_text = File.read(source) - package_imports = source_text.scan(/REQUIRE\s+"pkg:([^"]+)"(?:\s+AS\s+([A-Za-z_]\w*))?/) pkg_paths = begin ClearBuildSupport.collect_package_dependencies(source) rescue ClearBuildSupport::BuildError => e error e.message end + # A package import can live in a locally REQUIREd module, not just the root + # file. The emitted root Zig inlines local modules, so their package aliases + # must be rewritten to package-file imports as well. + closure_paths = begin + ClearBuildSupport.collect_clear_dependencies(source).to_a + rescue ClearBuildSupport::BuildError + [source] + end + package_imports = closure_paths.flat_map do |dep_path| + File.read(dep_path).scan(/REQUIRE\s+"pkg:([^"]+)"(?:\s+AS\s+([A-Za-z_]\w*))?/) + end.uniq pkg_requires = pkg_paths.keys pkg_flags = pkg_paths.map do |pkg_name, pkg_path| "--pkg #{pkg_name}=#{pkg_path}" @@ -513,6 +523,11 @@ def do_build(source, output: nil, opt_level: 'Debug', extra_flags: [], module_mo pkg_zig = pkg_zig.gsub("@import(\"#{zig_name}\")", "@import(\"#{np}.zig\")") pkg_zig = pkg_zig.gsub("@import(\"#{zig_name}.zig\")", "@import(\"#{np}.zig\")") end + # A package can own EXTERN ... FROM "module" declarations; its emitted + # named imports must resolve to the FFI files copied into the build dir. + ffi_modules.each do |m, _src_mod| + pkg_zig = pkg_zig.gsub("@import(\"#{m}\")", "@import(\"#{m}.zig\")") + end pkg_zig_file = File.join(build_dir, "#{pkg_name}.zig") preexisting_pkg = File.exist?(pkg_zig_file) if coverage_module_mode && preexisting_pkg && File.read(pkg_zig_file) != pkg_zig @@ -922,6 +937,16 @@ when 'build', 'watch' when '--mir' # No-op: MIR is the only pipeline. Flag kept for backward compat. i += 1 + when '--pkg' + spec = args[i + 1] + error "--pkg requires name=/path/to/lib.clear" unless spec&.include?('=') + pkg_name, pkg_path = spec.split('=', 2) + begin + ClearBuildSupport.register_packages({ pkg_name => pkg_path }) + rescue ClearBuildSupport::BuildError => e + error e.message + end + i += 2 when '--use-c-allocator' use_c_allocator = true i += 1 @@ -1273,6 +1298,24 @@ when 'test' ENV['ZIG_COVERAGE'] ||= '1' if coverage_mode test_args.delete('--mir') # No-op: MIR is the only pipeline + pkg_specs = [] + while (pkg_index = test_args.index('--pkg')) + spec = test_args[pkg_index + 1] + error "--pkg requires name=/path/to/lib.clear" unless spec&.include?('=') + pkg_specs << spec + test_args.slice!(pkg_index, 2) + end + # Register the packages locally too: FFI module detection walks the REQUIRE + # closure through registered package paths. + pkg_specs.each do |spec| + pkg_name, pkg_path = spec.split('=', 2) + begin + ClearBuildSupport.register_packages({ pkg_name => pkg_path }) + rescue ClearBuildSupport::BuildError => e + error e.message + end + end + # `--tag ` (repeatable). Translates to Zig's `--test-filter "#"` # — each WHEN-block's TAGS are encoded as ` #` suffixes on the # emitted Zig test name, so substring filtering selects tagged tests. @@ -1287,7 +1330,14 @@ when 'test' end source = test_args.first - error "Usage: clear test [--profile] [--strict] [--debug-frame] [--coverage] [--tag ]" unless source + error "Usage: clear test [--profile] [--strict] [--debug-frame] [--coverage] [--tag ]" unless source + if source.start_with?("pkg:") + begin + source = ClearBuildSupport.materialize_package_root(source.delete_prefix("pkg:")) + rescue ClearBuildSupport::BuildError => e + error e.message + end + end frame_env = frame_debug ? "CLEAR_FRAME_DEBUG=1 " : "" if File.directory?(source) @@ -1411,10 +1461,13 @@ when 'test' gen_script = File.join(CLEAR_ROOT, 'transpile-tests', 'gen.rb') # Transpile via gen.rb --single to get a proper zig test block - zig_code = `#{frame_env}ruby #{gen_script} --single #{source} 2>/dev/null` - unless $?.success? + generator_command = [RbConfig.ruby, gen_script, '--single', source] + pkg_specs.each { |spec| generator_command.concat(['--pkg', spec]) } + generator_env = frame_debug ? { 'CLEAR_FRAME_DEBUG' => '1' } : {} + zig_code, generator_error, generator_status = Open3.capture3(generator_env, *generator_command) + unless generator_status.success? # Re-run with stderr visible for error messages - system("ruby #{gen_script} --single #{source} > /dev/null") + $stderr.print(generator_error) error "Transpilation failed" end zig_code = zig_code.gsub('@import("runtime-header.zig")', '@import("runtime/runtime-header.zig")') diff --git a/compiler/ruby/annotator/domains/control_flow.rb b/compiler/ruby/annotator/domains/control_flow.rb index cbe2b1cee..46925c17f 100644 --- a/compiler/ruby/annotator/domains/control_flow.rb +++ b/compiler/ruby/annotator/domains/control_flow.rb @@ -119,6 +119,7 @@ def visit_BlockExpr(node) with_new_scope(current_scope) do node.body.each { |stmt| visit(stmt) } visit(node.result) + promote_to_expr_if!(node, node.result) if node.result.is_a?(AST::IfStatement) stamp_type!(node, node.result.full_type!(context: "catch branch result")) node.storage = node.result.storage end @@ -516,30 +517,27 @@ def visit_IfBind(node) else Type.new(b.expr.full_type!(context: "IF predicate binding expression")) end - unwrapped = if b.predicate == :is_ok - unless ti.error_union? - error!(b.expr, :IS_OK_REQUIRES_FALLIBLE, got: b.expr.resolved_type) - end - T.must(ti.success_type) - else - if ti.stream_step? - b[:predicate] = :stream_item - T.must(ti.stream_step_item_type) - else - unless ti.optional? - error!(b.expr, :IF_AS_NEEDS_OPTIONAL, got: b.expr.resolved_type) - end - T.must(ti.wrapped_type) - end - end + unwrapped = if_bind_unwrapped_type!(b, ti) if b.expr.is_a?(AST::ResolveNode) && (ti.multiowned? || ti.shared?) unwrapped.apply_reference_ownership!(ti.ownership, link_source: ti.link_source) end b.unwrapped_type = unwrapped sym = unwrapped.resolved root = AST.root_identifier(b.expr) + # Plain struct AND plain collection payloads bind mutable + # pointer aliases into the container slot (lowering uses + # getPtr/getAtPtrOpt), so mutation through the capture is + # legal and lands in the container. Rc/node-handle payloads + # are value captures and stay immutable borrows. + # A @node handle is itself a pointer into the NodeStore, so + # assigning through the capture lands in the stored node -- + # excluding it here made `IF nodes[i] EXISTS AS n THEN n.f = ...` + # fail as an immutable-field assignment. Rc payloads stay + # immutable value captures. + mutable_slot_payload = (unwrapped.struct? || unwrapped.collection?) && + !unwrapped.any_rc? mutable_list_alias = b.expr.is_a?(AST::GetIndex) && root && - !current_scope.is_immutable?(root.name) && unwrapped.struct? + !current_scope.is_immutable?(root.name) && mutable_slot_payload current_scope.declare(b.name, nil, unwrapped, mutable_list_alias, false, nil, :stack) entry = current_scope.local_entry!(b.name) b.symbol = entry @@ -549,14 +547,16 @@ def visit_IfBind(node) # of one). IF-AS on `p[i]` / `p.field` where `p` is the alias # makes the new binding a borrow into locked data; it must not # escape the enclosing WITH scope either. - container_source = find_container_source(b.expr) # Compact @node handles and ordinary Copy payloads are returned # by value. Binding them does not borrow the collection storage, # so a later handle assignment cannot invalidate the binding. - if unwrapped.node_reference? || unwrapped.implicitly_copyable? - container_source = nil + container_source = if_bind_container_source(unwrapped, b.expr) + src_sym = T.let(nil, T.nilable(SymbolEntry)) + source_root = AST.root_identifier(b.expr) + if source_root + src_sym = source_root.symbol end - if (src_sym = AST.root_identifier(b.expr)&.symbol) + if src_sym entry.mark_non_escaping! if src_sym.non_escaping if container_source entry.lifetime = SymbolEntry.tied_lifetime([src_sym]) @@ -566,7 +566,8 @@ def visit_IfBind(node) classify_ownership!(entry) og_declare(b.name.to_s, nil, unwrapped) if container_source - ownership_graph[b.name.to_s]&.kind = :borrowed + graph_node = ownership_graph[b.name.to_s] + graph_node.kind = :borrowed if graph_node ownership_graph.borrow(b.name.to_s, container_source, mutable: mutable_list_alias == true) end end @@ -583,6 +584,36 @@ def visit_IfBind(node) stamp_type!(node, :Void) end + sig { params(binding: AST::Binding, type: Type).returns(Type) } + def if_bind_unwrapped_type!(binding, type) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + + if binding.predicate == :is_ok + unless type.error_union? + error!(binding.expr, :IS_OK_REQUIRES_FALLIBLE, got: binding.expr.resolved_type) + end + return T.must(type.success_type) + end + + if type.stream_step? + binding[:predicate] = :stream_item + return T.must(type.stream_step_item_type) + end + + unless type.optional? + error!(binding.expr, :IF_AS_NEEDS_OPTIONAL, got: binding.expr.resolved_type) + end + T.must(type.wrapped_type) + end + + sig { params(type: Type, expression: AST::Node).returns(T.nilable(String)) } + def if_bind_container_source(type, expression) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + return if type.node_reference? || type.implicitly_copyable? + + find_container_source(expression) + end + # Type-checks a struct destructuring pattern against the match subject type. # Verifies field names exist and value types match the struct schema. @@ -739,10 +770,10 @@ def literal_instance_type(node) T.bind(self, Annotator::Phases::TypeAnalysisSession) if node.type_args&.any? - Type.new(NamedTypeExpression.new( + Type.new(TypeExpression.of(NamedTypeExpression.new( name: node.name.to_sym, arguments: node.type_args.map { |argument| Type.new(argument).shape.expression }, - )) + ))) else Type.new(node.name.to_sym) end @@ -810,26 +841,6 @@ def consume_match_subject_if_takes!(node, plan) og_set_moved(source_name, at_token: expr.token, action: :takes) end - sig { params(node: AST::MatchStatement, plan: MatchSubjectPlan).returns(T::Array[T.proc.returns(BasicObject)]) } - def match_branch_logic(node, plan) - T.bind(self, Annotator::Phases::TypeAnalysisSession) - branches = T.let([], T::Array[T.proc.returns(BasicObject)]) - node.cases.each do |match_case| - branches << Kernel.proc { - analyze_match_case!(node, match_case, plan) - with_conditional_context { visit_stmts(match_case.body) } - collect_scope_drops(node: node) - } - end - if node.default_case - branches << Kernel.proc { - with_conditional_context { visit_stmts(node.default_case) } - collect_scope_drops(node: node) - } - end - branches - end - sig { params(node: AST::MatchStatement, match_case: AST::MatchCase, plan: MatchSubjectPlan).void } def analyze_match_case!(node, match_case, plan) T.bind(self, Annotator::Phases::TypeAnalysisSession) @@ -1076,6 +1087,14 @@ def visit_PassStmt(node) stamp_type!(node, :Void) end + sig { params(node: AST::DeferStmt).returns(T.nilable(Symbol)) } + def visit_DeferStmt(node) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + + node.body.each { |stmt| visit(stmt) } + stamp_type!(node, :Void) + end + sig { params(node: AST::MatchStatement).returns(T.nilable(Symbol)) } def visit_MatchStatement(node) T.bind(self, Annotator::Phases::TypeAnalysisSession) @@ -1084,7 +1103,21 @@ def visit_MatchStatement(node) plan = match_subject_plan(node) consume_match_subject_if_takes!(node, plan) - all_drops = analyze_control_flow_branches(match_branch_logic(node, plan)) + branches = T.let([], T::Array[T.proc.returns(BasicObject)]) + node.cases.each do |match_case| + branches << Kernel.proc { + analyze_match_case!(node, match_case, plan) + with_conditional_context { visit_stmts(match_case.body) } + collect_scope_drops(node: node) + } + end + if node.default_case + branches << Kernel.proc { + with_conditional_context { visit_stmts(node.default_case) } + collect_scope_drops(node: node) + } + end + all_drops = analyze_control_flow_branches(branches) if node.default_case node.default_drops = T.must(all_drops).pop @@ -1170,7 +1203,8 @@ def visit_ForEach(node) T.must(node.symbol).mark_borrowed_alias! classify_ownership!(T.must(node.symbol)) og_declare(node.var_name.to_s, nil, elem_ti) - ownership_graph[node.var_name.to_s]&.kind = :borrowed + graph_node = ownership_graph[node.var_name.to_s] + graph_node.kind = :borrowed if graph_node visit_stmts(node.body) finalize_scope(node) node.deferred_drops @@ -1379,7 +1413,8 @@ def visit_ContinueNode(node) private :consume_match_subject_if_takes! private :emit_unknown_destructure_field! private :loop_value_copyable? - private :match_branch_logic + private :if_bind_unwrapped_type! + private :if_bind_container_source private :match_enum_schema private :match_pattern_type_matches_subject? private :match_payload_binding_type diff --git a/compiler/ruby/annotator/domains/errors.rb b/compiler/ruby/annotator/domains/errors.rb index a59d0e0a9..d13c2d934 100644 --- a/compiler/ruby/annotator/domains/errors.rb +++ b/compiler/ruby/annotator/domains/errors.rb @@ -2,6 +2,7 @@ # frozen_string_literal: true require_relative "../../compiler/entrypoint" +require_relative "../helpers/with_match_check" module Annotator module Domains @@ -148,7 +149,6 @@ def validate_sync_policy_body!(decl) def collapse_errors_for_call(sig, args) T.bind(self, Annotator::Phases::TypeAnalysisSession) - require_relative 'helpers/with_match_check' unless defined?(WithMatchCheck) collapsed = Set.new param_indices = sig.params.each_with_index.to_h { |param, index| [param.name.to_s, index] } sig.requires.each do |param_name, _families| @@ -233,7 +233,7 @@ def resolve_catch_clause!(clause) type_sym = item.name.to_sym unless AST.error_type?(type_sym) emit_registry_mismatch!( - item.token, item.name, AST::ERROR_TYPES.keys, + item.token, item.name, AST.error_type_names, "CATCH #{item.name}: error type '#{item.name}' is not registered. A type " \ "must be registered via RAISE/OR_ELSE EXIT before it can be CATCHed.", "closest registered type" @@ -565,18 +565,22 @@ def same_return_capabilities?(expected_t, actual_t) # heap-boxed into a `*T` cell at the RETURN site (escape analysis), # so the returned expression need not already carry :indirect. layout_ok = expected_t.layout == actual_t.layout || expected_t.indirect? - expected_t.ownership == actual_t.ownership && + # nil element ownership IS the default: a declared `[]String@symbol` + # return leaves it nil while registry-built keys()/values() results + # stamp :affine explicitly — the same type, printed identically. + elem_own = ->(t) { t.elem_ownership == :affine ? nil : t.elem_ownership } + (expected_t.ownership == actual_t.ownership && expected_t.sync == actual_t.sync && layout_ok && - expected_t.elem_ownership == actual_t.elem_ownership && - expected_t.elem_sync == actual_t.elem_sync + elem_own.call(expected_t) == elem_own.call(actual_t) && + expected_t.elem_sync == actual_t.elem_sync) == true end sig { params(type: Type).returns(String) } def type_display(type) T.bind(self, Annotator::Phases::TypeAnalysisSession) - parts = [type.resolved.to_s] + parts = [Type.surface_name_type(type)] ownership = type.ownership_surface_name sync = type.sync_surface_name @@ -660,32 +664,44 @@ def visit_OrElse(node) end operation, recovery = or_else_operation(node.right) - begin - plan = TenseOperationPlanner.or_else( - t_left_type, - t_right_type, - operation: operation, - recovery: recovery, - ) - rescue ArgumentError - expected = t_left_type.value_payload_type - error!(node, :TYPE_MISMATCH_IN_OR, expected: expected.resolved, got: t_right_type.resolved) - # Fix-collection mode records the diagnostic and continues. Publish - # the plan the corrected fallback would use so later phases never - # need a nullable compatibility path. - plan = TenseOperationPlanner.or_else( - t_left_type, - expected, - operation: operation, - recovery: recovery, - ) - end + plan = plan_or_else_with_diagnostic(node, t_left_type, t_right_type, operation, recovery) plan = T.must(plan) node.tense_plan = plan coerce_empty_collection_fallback!(node.right, plan.result_type) if recovery == TenseRecovery::Fallback stamp_type!(node, plan.result_type) end + sig do + params( + node: AST::BinaryOp, + left_type: Type, + right_type: Type, + operation: TenseOperationKind, + recovery: TenseRecovery, + ).returns(TenseOperationPlan) + end + def plan_or_else_with_diagnostic(node, left_type, right_type, operation, recovery) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + TenseOperationPlanner.or_else( + left_type, + right_type, + operation: operation, + recovery: recovery, + ) + rescue ArgumentError + expected = left_type.value_payload_type + error!(node, :TYPE_MISMATCH_IN_OR, expected: expected.resolved, got: right_type.resolved) + # Fix-collection mode records the diagnostic and continues. Publish + # the plan the corrected fallback would use so later phases never + # need a nullable compatibility path. + TenseOperationPlanner.or_else( + left_type, + expected, + operation: operation, + recovery: recovery, + ) + end + sig { params(node: AST::Node).returns([TenseOperationKind, TenseRecovery]) } def or_else_operation(node) case node diff --git a/compiler/ruby/annotator/domains/execution_boundaries.rb b/compiler/ruby/annotator/domains/execution_boundaries.rb index 193b41ead..0644a945e 100644 --- a/compiler/ruby/annotator/domains/execution_boundaries.rb +++ b/compiler/ruby/annotator/domains/execution_boundaries.rb @@ -600,7 +600,7 @@ def resolve_error_selectors!(node, clause, is_snapshot_txn = false) when :type unless AST.error_type?(name) emit_registry_mismatch!( - diagnostic_token, name, AST::ERROR_TYPES.keys, + diagnostic_token, name, AST.error_type_names, "Unknown error type '#{name}'. Register it in src/ast/error_registry.rb.", "closest registered type" ) @@ -695,10 +695,10 @@ def visit_BgStreamBlock(node) end element_type = node.declared_yield_type || inferred_join.result_type || Type.new(:Any) - stamp_type!(node, Type.new(StreamTypeExpression.new( + stamp_type!(node, Type.new(TypeExpression.of(StreamTypeExpression.new( cardinality: :FINITE, item: element_type.shape.expression, - ))) + )))) node.capture_analysis = stream_analysis_result @@ -967,6 +967,11 @@ def visit_NextExpr(node) elem_sym = T.must(promise_type.tense_type.element_type).to_sym stamp_type!(node, Type.new(:"#{elem_sym}[]")) node.storage = :heap + elsif promise_type.split_open_stream? + # NEXT on split streams returns ?T — null signals stream exhaustion. + # Split handles advance independently through shared memoized state. + elem_sym = T.must(promise_type.open_stream_element_type).to_sym + stamp_type!(node, Type.new(:"?#{elem_sym}")) elsif promise_type.dynamic_stream? elem_sym = T.must(promise_type.tense_type.element_type).to_sym if promise_type.canonical_stream? diff --git a/compiler/ruby/annotator/domains/expressions.rb b/compiler/ruby/annotator/domains/expressions.rb index 9fdd48d4d..27e42cec4 100644 --- a/compiler/ruby/annotator/domains/expressions.rb +++ b/compiler/ruby/annotator/domains/expressions.rb @@ -117,11 +117,7 @@ def visit_UnaryOp(node) declared = recoverable_result_type(node.right, context: "TRY operand") raw_type = Type.new(node.right.full_type!(context: "TRY operand")) plan_input = declared || raw_type - begin - plan = TenseOperationPlanner.try_value(plan_input) - rescue ArgumentError - error!(node, :UNWRAP_NON_OPTIONAL, got: raw_type) - end + plan = try_value_plan_with_diagnostic(node, plan_input, raw_type) plan = T.must(plan) node.tense_plan = plan stamp_type!(node, plan.result_type) @@ -136,6 +132,15 @@ def visit_UnaryOp(node) node.full_type!(context: "unary expression") end + sig { params(node: AST::UnaryOp, plan_input: Type, raw_type: Type).returns(T.nilable(TenseOperationPlan)) } + def try_value_plan_with_diagnostic(node, plan_input, raw_type) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + TenseOperationPlanner.try_value(plan_input) + rescue ArgumentError + error!(node, :UNWRAP_NON_OPTIONAL, got: raw_type) + nil + end + # ========================================== # LITERALS & BINARY OPS # ========================================== @@ -577,19 +582,17 @@ def publish_tense_navigation_plan!(member, navigation, mapped_type) receiver_type = recoverable_result_type(navigation.target, context: "tense navigation receiver") || navigation.target.full_type!(context: "tense navigation receiver") - plan = begin - TenseOperationPlanner.navigate( - receiver_type, - mapped_type, - markers: navigation.markers, - shared: receiver_type.shared?, - ) - rescue ArgumentError => error - if error.message.include?("nested future") - error!(member, :TENSE_NAVIGATION_NESTED_FUTURE, type: Type.surface_name(mapped_type)) - end - raise - end + receiver_envelope = TenseEnvelope.from_type(receiver_type) + mapped_envelope = TenseEnvelope.from_type(mapped_type) + if receiver_envelope.asynchronous? && mapped_envelope.asynchronous? + error!(member, :TENSE_NAVIGATION_NESTED_FUTURE, type: Type.surface_name(mapped_type)) + end + plan = TenseOperationPlanner.navigate( + receiver_type, + mapped_type, + markers: navigation.markers, + shared: receiver_type.shared?, + ) member.tense_plan = plan if plan.consumes_handle? root = AST.root_identifier(navigation.target) diff --git a/compiler/ruby/annotator/domains/lifetimes.rb b/compiler/ruby/annotator/domains/lifetimes.rb index 66f01b2cf..1748ab605 100644 --- a/compiler/ruby/annotator/domains/lifetimes.rb +++ b/compiler/ruby/annotator/domains/lifetimes.rb @@ -139,14 +139,41 @@ def visit_CopyNode(node) def finish_previsited_copy!(node) T.bind(self, Annotator::Phases::TypeAnalysisSession) + # Static rule 6: bare COPY (a memcpy) on a carrier-polymorphic parameter + # cannot guarantee independent identity (the caller may have passed a + # retained value). Require KEEP or a UNIQUE contract. OWN COPY is exempt: + # it explicitly projects the payload out at comptime and deep-copies it, + # so it yields an independent plain value for every carrier. This applies + # only to a DIRECT reference to the parameter -- COPY of a field or + # element (`COPY items[0]`) copies that projection's own value, whose + # carrier is its element/field type, not the parameter's. + copy_root = node.value + if copy_root.is_a?(AST::Identifier) && !node.own + copy_entry = copy_root.symbol + if copy_entry.is_a?(SymbolEntry) && copy_entry.carrier_polymorphic + error!(node, :COPY_ON_POLYMORPHIC_PARAM, name: copy_root.name) + end + end + # COPY produces an owned deep-copy. The source is NOT consumed. # Clone the Type so mutating provenance doesn't affect the inner node. inner_type = node.value.full_type!(context: "COPY value") - resolver = ->(name) { lookup_type_schema(name) } - if inner_type.is_a?(Type) && inner_type.contains_linear_resource?(resolver) - error!(node, :COPY_NON_COPYABLE, type: inner_type.to_s) + # Retained-identity v5 (V5-3b): a plain-source COPY is a payload_copy. + # A retained source gets :shared_to_unique_copy stamped at the UNIQUE + # call boundary (verify_copy_retained_boundary!), or is rejected. + if inner_type.is_a?(Type) && !inner_type.multiowned? && !inner_type.shared? + node.carrier_op = :payload_copy + end + # OWN COPY detaches: the result is an independent PLAIN value for + # every carrier, so a bound `x = OWN COPY handle` must not inherit + # the source's @shared/@multiowned wrapper (the inline-argument form + # already bypasses the carrier gate via the CopyNode exemption). + result_type = if inner_type.is_a?(Type) + node.own && inner_type.any_rc? ? inner_type.bare_data_type : Type.new(inner_type) + else + inner_type end - stamp_type!(node, inner_type.is_a?(Type) ? Type.new(inner_type) : inner_type) + stamp_type!(node, result_type) ti = node.full_type!(context: "COPY result") # COPY of a primitive, interned symbol, or Id is a semantic no-op @@ -271,37 +298,111 @@ def visit_FreezeNode(node) node.storage = :frozen end - sig { params(node: AST::CloneNode).returns(T.nilable(T::Boolean)) } - def visit_CloneNode(node) + # Retained-identity v5 unified carrier-preserving fan-out KEEP (formerly + # CLONE, the narrow Rc/Arc-retain case, and COPY_OR_CLONE). The value's + # TYPE is unchanged; the ownership OPERATION (Rc retain / Arc retain / + # plain payload copy) is chosen by placement (V5-3a) from the source + # carrier. KEEP is valid on a retained carrier (@multiowned/@shared, + # stream) and on a carrier-polymorphic parameter; static rules 4/5 + # reject it on a statically plain local or a UNIQUE parameter (use COPY). + sig { params(node: AST::KeepNode).returns(T.nilable(T::Boolean)) } + def visit_KeepNode(node) T.bind(self, Annotator::Phases::TypeAnalysisSession) record_capture_site!(node, copied: true) without_capture_moves { visit(node.value) } - finish_previsited_clone!(node) + reject_keep_on_known_carrier!(node) + finish_previsited_keep!(node) end - # See finish_previsited_copy!: this consumes resolved semantic facts and - # deliberately does not revisit the source expression. - sig { params(node: AST::CloneNode).returns(T.nilable(T::Boolean)) } - def finish_previsited_clone!(node) + # Static rules 4 & 5: KEEP is only meaningful on a carrier-polymorphic + # value. A plain local (rule 4) or a UNIQUE parameter (rule 5) has a + # statically known carrier -- use COPY for an independent copy. + sig { params(node: AST::KeepNode).returns(NilClass) } + def reject_keep_on_known_carrier!(node) T.bind(self, Annotator::Phases::TypeAnalysisSession) - type = node.value.full_type!(context: "CLONE value") root = get_root_object(node.value) - if root.is_a?(AST::Identifier) && root.symbol&.non_escaping - error!(node, :CLONE_WITH_SCOPED, name: root.name) + return nil unless root.is_a?(AST::Identifier) + entry = root.symbol + return nil unless entry.is_a?(SymbolEntry) + # A WITH-scoped (non-escaping) borrow is not a "plain local": the + # scoped-escape guard in finish_previsited_keep! handles it. + return nil if entry.non_escaping + + ty = node.value.full_type!(context: "KEEP carrier") + # A retained carrier is any Rc/Arc, split stream, or shared promise -- + # the same set KEEP retains rather than copies. Only a statically plain + # local falls through to rule 4. + retained = ty.is_a?(Type) && + (ty.any_rc? || ty.split_open_stream? || ty.shared_promise? || ty.split?) + carrier = if entry.is_param && entry.carrier_contract == :unique + "UNIQUE parameter" + elsif !entry.is_param && !entry.carrier_polymorphic && !retained + "plain local" end + return nil unless carrier - unless type&.split_open_stream? || type&.shared_promise? || type&.any_rc? - error!(node, :CLONE_BAD_TARGET, got: node.value.resolved_type) + error!(node, :KEEP_ON_KNOWN_CARRIER, name: root.name, carrier: carrier) + end + + # Consumes resolved semantic facts; deliberately does not revisit the + # source. KEEP allows a plain carrier (a payload copy at lowering), so + # there is no bad-target rejection -- only the scoped-escape guard. + sig { params(node: AST::KeepNode).returns(T.nilable(T::Boolean)) } + def finish_previsited_keep!(node) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + + type = node.value.full_type!(context: "KEEP value") + root = get_root_object(node.value) + # A WITH alias itself is only a protected borrow and still cannot + # escape. A retained field reached through that alias is different: + # KEEP duplicates the field's Rc handle while the protected owner is + # live, so the new strong reference is independent of the WITH scope. + retained_projection = type.is_a?(Type) && type.any_rc? && node.value != root + if root.is_a?(AST::Identifier) && root.symbol&.non_escaping && !retained_projection + error!(node, :KEEP_WITH_SCOPED, name: root.name) end - stamp_type!(node, node.value.full_type!(context: "CLONE result")) - node.storage = node.value.storage + stamp_type!(node, node.value.full_type!(context: "KEEP result")) + node.storage = node.value.storage if node.value.respond_to?(:storage) current_fn_ctx&.mark_runtime_used! if type&.any_rc? + stamp_keep_carrier_op!(node, type, root) nil end + # Retained-identity v5 (V5-3a wiring): record the physical KEEP op the + # OwnershipEdgePlanner selects from the source carrier -- the ONE writer. + # A carrier-polymorphic parameter source has no statically known carrier; + # its op is resolved per specialization (Phase 4), marked deferred here. + # Non-Rc/Arc retained carriers (@split streams, promises) stay on the + # existing lowering path until Phase 4 unifies them. + sig { params(node: AST::KeepNode, type: T.nilable(Type), root: AST::Node).void } + def stamp_keep_carrier_op!(node, type, root) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + + if root.is_a?(AST::Identifier) && root.symbol.is_a?(SymbolEntry) && + T.must(root.symbol).carrier_polymorphic + # A MONOMORPHIC param resolves KEEP per concrete carrier at Zig + # comptime (retain a handle, or copy a plain value). An unconstrained + # carrier-polymorphic param has no monomorphization, so its carrier + # stays statically unknown -> the deferred tag path. + node.carrier_op = T.must(root.symbol).carrier_contract == :monomorphic ? + :monomorphic_keep : :deferred_specialization + return + end + return unless type.is_a?(Type) + + carrier = if type.multiowned? then :multiowned + elsif type.shared? then :shared + elsif !type.any_rc? && !type.split_open_stream? && !type.shared_promise? && !type.split? + :plain + end + return unless carrier + + node.carrier_op = OwnershipEdgePlanner.select(source_carrier: carrier, fan_out: :keep).op + end + sig { params(node: AST::ShareNode).void } def visit_ShareNode(node) T.bind(self, Annotator::Phases::TypeAnalysisSession) @@ -347,23 +448,9 @@ def collect_body_identifier_names(nodes) T.bind(self, Annotator::Phases::TypeAnalysisSession) names = Set.new - traverse = T.let(nil, T.untyped) - traverse = lambda do |n| - case n - when nil, Symbol, String, Integer, Float, TrueClass, FalseClass, Type - when Array - n.each { |item| traverse.call(item) } - when Hash - n.each_value { |v| traverse.call(v) } - when AST::FunctionDef - # Don't descend into nested function definitions. - when AST::Identifier - names.add(n.name) - else - n.each_pair { |_, v| traverse.call(v) } if n.respond_to?(:each_pair) - end + AST.each_locatable(nodes) do |node| + names.add(node.name) if node.is_a?(AST::Identifier) end - traverse.call(nodes) names end @@ -955,7 +1042,7 @@ def init_value_contents_heap?(init) init.is_a?(AST::Locatable) && init.heap_storage? when AST::Identifier !!init.symbol&.init_contents_heap - when AST::CopyNode, AST::CloneNode + when AST::CopyNode, AST::KeepNode true when AST::Cast init_value_contents_heap?(init.value) @@ -1316,6 +1403,34 @@ def consume_generic_map_value!(node, value_type) move_if_takes_ownership!(node, action: :takes, consumer_param_type: value_type) end + # Keep-analysis (retained identity v4): a plain param flowing into an + # @multiowned identity field is kept, not borrow-rejected. Stamps + # kept_identity on the param's SymbolEntry; the call edge's cost + # (retain / move / create / copy) is derived later from the caller's + # declared model, never spelled at this use site. + sig { params(val_node: AST::Node, expected_type: Type, container_desc: String).returns(T::Boolean) } + def keep_param_identity!(val_node, expected_type, container_desc) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + + return false unless expected_type.multiowned? + # A generic identity field (T @multiowned) resolves its capability + # after substitution; pre-wrapping the generic param's ABI would + # double-wrap the substituted handle. Generic keeps stay on the + # existing anytype path. + return false if current_function_type_param?(expected_type.resolved) + # Zero-config row: `param OR_ELSE fresh-default` keeps the provided + # identity and constructs a private one when omitted. The param is + # kept; the default branch stays an ordinary owned payload. + if val_node.is_a?(AST::BinaryOp) && val_node.op == :OR_ELSE + return keep_param_identity!(val_node.left, expected_type, container_desc) + end + return false unless val_node.is_a?(AST::Identifier) + entry = current_scope.resolve_entry(val_node.name) + return false unless entry&.is_param + entry.kept_identity ||= KeptIdentityContract.new(family: :multiowned, sink: container_desc) + true + end + # Reject storing a borrowed value into an owned container (struct, union, TAKES param). # Borrows can't outlive the scope they reference. Use COPY for owned data. sig { params(val_node: AST::Node, container_desc: String).returns(NilClass) } diff --git a/compiler/ruby/annotator/domains/member_access.rb b/compiler/ruby/annotator/domains/member_access.rb index 1315808eb..aff56d8cf 100644 --- a/compiler/ruby/annotator/domains/member_access.rb +++ b/compiler/ruby/annotator/domains/member_access.rb @@ -84,13 +84,15 @@ def visit_GetIndex(node) stamp_type!(node, result_type) node.container_borrow = true if op[:container_borrow] - # Validate key types for maps + # Validate the index against the map's declared key type. Maps are + # `{K}V`; K is not restricted to String or numeric primitives. if target_type_info.map? index_type_info = node.index.full_type!(context: "index key") - if target_type_info.numeric_map? - error!(node, :NUMERIC_MAP_KEY_BAD, got: node.index.resolved_type) unless index_type_info&.numeric? - else - error!(node, :STRING_MAP_KEY_BAD, got: node.index.resolved_type) unless index_type_info&.string? + expected_key_type = target_type_info.key_type + unless index_type_info && target_type_info.accepts_map_key?(index_type_info) + error!(node, :GENERIC_MAP_KEY_MISMATCH, + expected: Type.surface_name(expected_key_type), + actual: index_type_info ? Type.surface_name(index_type_info) : node.index.resolved_type) end end @@ -103,7 +105,7 @@ def visit_GetIndex(node) stamp_type!(node, Type.optional_of(Type.new(:"~#{elem_t.resolved}"))) elsif target_type_info.string? && !target_type_info.raw? error!(node, :STRING_INDEX_BY_INT) - elsif node.target.metatype == :struct + elsif node.target.metatype == :struct && target_type_info.element_type # Struct field access via index (rare legacy path) stamp_type!(node, target_type_info.element_type) node.container_borrow = true @@ -531,15 +533,17 @@ def visit_StructLit(node) expected_type = T.let(apply_type_subst(raw_expected, type_subst), Type) # BORROWED fields accept borrowed values — skip ownership checks. - # Non-borrowed fields require owned data. - unless field_is_borrowed + # Non-borrowed fields require owned data, except a kept param + # flowing into an @multiowned identity field (retained identity v4). + kept_param_store = keep_param_identity!(val_node, expected_type, "#{node.name}.#{field_name}") + unless field_is_borrowed || kept_param_store reject_borrowed_value!(val_node, "#{node.name}.#{field_name}") end # Skip CopyNode wrapping for rodata strings in call argument structs. # The struct is a temporary - rodata strings are valid for the call's # lifetime. The callee dupes strings it needs to escape. is_call_arg = struct_literal_call_argument_context? - owned = T.let(unless field_is_borrowed || is_call_arg + owned = T.let(unless field_is_borrowed || is_call_arg || kept_param_store ensure_owned_value!(val_node, expected_type, "#{node.name}.#{field_name}") end, T.nilable(AST::Node)) if owned diff --git a/compiler/ruby/annotator/domains/variables.rb b/compiler/ruby/annotator/domains/variables.rb index cae2ac081..bbbd1c2fb 100644 --- a/compiler/ruby/annotator/domains/variables.rb +++ b/compiler/ruby/annotator/domains/variables.rb @@ -256,33 +256,54 @@ def materialize_inferred_wrapper_annotation!(node) value_type end - node.type = if wants_future + selected_type = declared + if wants_future # `name:~` is a promise/stream-retention assertion, not a way to # re-label a synchronous pipeline result as asynchronous. - value_type.future? ? value_type : declared + if value_type.future? + selected_type = value_type + else + selected_type = declared + end elsif wants_error_optional if value_type.error_union? && value_type.success_type.optional? - value_type - elsif error_type&.error_union? && payload.optional? - error_type + selected_type = value_type + elsif !error_type.nil? && error_type.error_union? && payload.optional? + selected_type = error_type else - Type.error_union_of(Type.optional_of(payload)) + selected_type = Type.error_union_of(Type.optional_of(payload)) end elsif wants_error - value_type.error_union? ? value_type : (error_type&.error_union? ? error_type : Type.error_union_of(payload)) + if value_type.error_union? + selected_type = value_type + elsif !error_type.nil? && error_type.error_union? + selected_type = error_type + else + selected_type = Type.error_union_of(payload) + end else - payload.optional? ? payload : Type.optional_of(payload) + if payload.optional? + selected_type = payload + else + selected_type = Type.optional_of(payload) + end end + node.type = selected_type if wants_error || wants_error_optional - node.value.retain_error_channel = true if node.value.respond_to?(:retain_error_channel=) + value_node = node.value + value_node.retain_error_channel = true if value_node.respond_to?(:retain_error_channel=) end end - sig { params(node: DeclarationNode).void } - def reject_implicit_wrapper_binding!(node) + sig { params(node: DeclarationNode, process_deferred_copy: T::Boolean).void } + def reject_implicit_wrapper_binding!(node, process_deferred_copy: false) return if node.type value = node.value + if value.is_a?(AST::CopyNode) && !process_deferred_copy + @traversal_state.deferred_copy_wrapper_bindings << node + return + end error_type = recoverable_result_type(value, context: "inferred wrapper binding") error_success_type = error_type&.error_union? ? error_type.success_type : nil value_type = Type.new(value.full_type!(context: "inferred wrapper binding")) @@ -473,24 +494,25 @@ def visit_declaration_value!(node) T.bind(self, Annotator::Phases::TypeAnalysisSession) declared = node.type - if node.value.is_a?(AST::BgBlock) && declared&.single_future? + value_node = node.value + if value_node.is_a?(AST::BgBlock) && declared&.single_future? payload = declared.tense_type if payload && !payload.dynamic? && !payload.auto? && !%i[Auto Any].include?(payload.resolved) - T.unsafe(node.value).declared_async_payload = payload + T.unsafe(value_node).declared_async_payload = payload end end # Fixed-array list literals must be storage-stamped before visiting so # downstream list analysis sees the intended stack placement. - if node.value.is_a?(AST::ListLit) && node.type&.fixed? - node.value.storage = :stack + if value_node.is_a?(AST::ListLit) && node.type&.fixed? + value_node.storage = :stack end - if node.value.is_a?(AST::ListLit) && node.type&.tuple? - node.value.coerced_type = node.type + if value_node.is_a?(AST::ListLit) && node.type&.tuple? + value_node.coerced_type = node.type end - if node.value.is_a?(AST::HashLit) && node.type&.map? - node.value.coerced_type = node.type + if value_node.is_a?(AST::HashLit) && node.type&.map? + value_node.coerced_type = node.type end - visit(node.value) + visit(value_node) end sig { params(node: DeclarationNode).void } @@ -530,7 +552,7 @@ def prepare_implicit_ownership_transport!(node) "but mutation occurs on line #{mutation.token&.line || node.token.line} while `#{plan.destination}` " \ "remains live through line #{last_line}. CLEAR will not infer snapshot-versus-shared mutation " \ "semantics in EASY, DEFAULT, or STRICT. Write `#{plan.destination} = COPY #{plan.source}` for " \ - "an independent value, or explicitly use @multiowned/@shared and `CLONE #{plan.source}`." + "an independent value, or explicitly use @multiowned/@shared and `KEEP #{plan.source}`." source_token = node.value.token fixes = T.let([ Fix.new( @@ -544,11 +566,11 @@ def prepare_implicit_ownership_transport!(node) ], T::Array[Fix]) if source_type.any_rc? || source_type.split? fixes << Fix.new( - description: fix_description(:PREFIX_EXPLICIT_OWNERSHIP_COST, keyword: "CLONE"), + description: fix_description(:PREFIX_EXPLICIT_OWNERSHIP_COST, keyword: "KEEP"), confidence: :interactive, edits: [Edit.new( span: Span.new(file: nil, line: source_token.line, col: source_token.column, length: 0), - replacement: "CLONE ", + replacement: "KEEP ", )], ) end @@ -564,10 +586,14 @@ def prepare_implicit_ownership_transport!(node) source = node.value source_type = source.full_type!(context: "implicit ownership materialization source") - keyword = source_type.any_rc? || source_type.split? ? "CLONE" : "COPY" - if language_mode == :strict + keyword = source_type.any_rc? || source_type.split? ? "KEEP" : "COPY" + # Retained-identity v5: KEEP (a refcount retain of a retained carrier) + # is optional in EVERY mode -- the declaration already chose the cost. + # A COPY (payload deep-copy of a plain value) stays explicit in STRICT + # so no hidden allocation occurs (design acceptance #5). + if language_mode == :strict && keyword == "COPY" detail = "STRICT ownership cost: `#{plan.destination} = #{plan.source}` requires an implicit " \ - "#{keyword == 'CLONE' ? 'reference-count retain' : 'deep copy'}. Write " \ + "#{keyword == 'KEEP' ? 'reference-count retain' : 'deep copy'}. Write " \ "`#{plan.destination} = #{keyword} #{plan.source}` explicitly, or shorten the lifetime so this is a move/borrow." fixable!(node, code: :STRICT_IMPLICIT_OWNERSHIP_COST, detail: detail, category: :ownership, level: :error, @@ -582,8 +608,8 @@ def prepare_implicit_ownership_transport!(node) raise_in_collector: true) end - wrapper = if keyword == "CLONE" - AST::CloneNode.new(source.token, source) + wrapper = if keyword == "KEEP" + AST::KeepNode.new(source.token, source) else AST::CopyNode.new(source.token, source) end @@ -606,8 +632,8 @@ def finalize_ownership_transport_facts!(facts) if !node.value.equal?(previous_value) wrapper = node.value record_capture_site!(wrapper, copied: true) - if wrapper.is_a?(AST::CloneNode) - finish_previsited_clone!(wrapper) + if wrapper.is_a?(AST::KeepNode) + finish_previsited_keep!(wrapper) else finish_previsited_copy!(T.cast(wrapper, AST::CopyNode)) end @@ -630,13 +656,13 @@ def finalize_pending_transfer!(decision) end source_type = source.full_type!(context: "finalized ownership transfer") wrapper = if source_type.any_rc? || source_type.split? - AST::CloneNode.new(source.token, source) + AST::KeepNode.new(source.token, source) else AST::CopyNode.new(source.token, source) end record_capture_site!(wrapper, copied: true) - if wrapper.is_a?(AST::CloneNode) - finish_previsited_clone!(wrapper) + if wrapper.is_a?(AST::KeepNode) + finish_previsited_keep!(wrapper) else finish_previsited_copy!(T.cast(wrapper, AST::CopyNode)) end @@ -690,6 +716,24 @@ def finalize_var_declaration!(node) finalize_decl_node!(node, node.mutable) stamp_init_contents_heap!(node) stamp_bg_handle_lifetime!(node) + propagate_carrier_polymorphic!(node) + end + + # Retained-identity v5 ("Why provenance is limited"): a local that + # DIRECTLY aliases a carrier-polymorphic binding inherits the unknown + # carrier, so COPY stays rejected on it. Only a bare identifier (or a + # MOVE of one) propagates; a projection or transformation does not. + sig { params(node: DeclarationNode).void } + def propagate_carrier_polymorphic!(node) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + + rhs = node.value + rhs = rhs.value if rhs.is_a?(AST::MoveNode) + return unless rhs.is_a?(AST::Identifier) + src = rhs.symbol + return unless src.is_a?(SymbolEntry) && src.carrier_polymorphic + dest = current_scope.resolve_entry(node.name.to_s) + dest.carrier_polymorphic = true if dest.is_a?(SymbolEntry) end sig { params(node: DeclarationNode).void } @@ -715,6 +759,7 @@ def finalize_bind_declaration!(node) mark_borrowed_field_bind_alias!(node) stamp_init_contents_heap!(node) stamp_bg_handle_lifetime!(node) + propagate_carrier_polymorphic!(node) end sig { params(node: AST::BindExpr).void } @@ -868,21 +913,22 @@ def classify_ownership!(entry) return unless entry type_obj = entry.type return if type_obj.fn_type? # function signature, not a variable - entry.ownership_kind = if entry.resource - :resource + ownership_kind = T.let(:affine, Symbol) + if entry.resource + ownership_kind = :resource elsif type_obj.multiowned? || type_obj.shared? || entry.rc_stored? - :rc + ownership_kind = :rc elsif entry.sync - :sync + ownership_kind = :sync elsif type_obj.collection? - :collection + ownership_kind = :collection elsif !entry.takes && type_obj.implicitly_copyable? { |t| lookup_type_schema(t) } - :value - else - # TAKES parameters own the data — always affine so cleanup is emitted. - :affine + ownership_kind = :value end + # TAKES parameters own the data — always affine so cleanup is emitted. + entry.ownership_kind = ownership_kind + ownership_kind end # Accumulate stack-local variable bytes for the current function context. @@ -980,11 +1026,11 @@ def mark_var_mutated_via_call(name) AccessPathNode = T.type_alias { T.any(AST::GetField, AST::GetIndex, AST::OptionalUnwrap, AST::Identifier) } - sig { params(node: AccessPathNode).returns(T.nilable(String)) } + sig { params(node: AST::Node).returns(T.nilable(String)) } def chain_root_name(node) T.bind(self, Annotator::Phases::TypeAnalysisSession) - curr = T.let(node, T.any(AST::GetField, AST::GetIndex, AST::OptionalUnwrap, AST::Identifier)) + curr = T.let(node, AST::Node) while curr.is_a?(AST::GetField) || curr.is_a?(AST::GetIndex) || curr.is_a?(AST::OptionalUnwrap) curr = curr.target end @@ -1101,7 +1147,8 @@ def visit_assignment_variable(identifier, node) end validate_assignment_type(node, scope.resolve_type(var_name), node.value.resolved_type) stamp_type!(node, scope.resolve_type(var_name)) - scope.resolve_entry(var_name)&.reassigned = true + entry = scope.resolve_entry(var_name) + entry.reassigned = true if entry mark_var_mutated(var_name) true end @@ -1131,20 +1178,21 @@ def visit_assignment_index(index_node, assignment_node) target_type = index_node.target.full_type!(context: "index assignment collection") protocol_map = map_requires_protocol_lowering?(target_type) - assign_type = if protocol_map + assign_type = Type.new(:Any) + if protocol_map require_generic_map_access_scope!(index_node.target, target_type) index_node.protocol_operation = :map_put - protocol_map_associated_type(target_type, :Value) - elsif target_type&.map? + assign_type = protocol_map_associated_type(target_type, :Value) + elsif !target_type.nil? && target_type.map? # Map reads return ?V because the key may be absent, but map writes # store the declared value type V. If V itself is optional, preserve it. - target_type.value_type + assign_type = target_type.value_type else index_type = index_node.full_type!(context: "index assignment target") - if index_type&.optional? - T.must(index_type.wrapped_type) + if index_type.optional? + assign_type = T.must(index_type.wrapped_type) else - index_type + assign_type = index_type end end @@ -1155,9 +1203,9 @@ def visit_assignment_index(index_node, assignment_node) validate_assignment_type(assignment_node, assign_type, assignment_node.value.resolved_type) - consume_generic_map_value!(assignment_node.value, T.must(assign_type)) if protocol_map + consume_generic_map_value!(assignment_node.value, assign_type) if protocol_map - stamp_type!(assignment_node, T.must(assign_type)) + stamp_type!(assignment_node, assign_type) # HashMap put may allocate, so needs_rt must propagate. if target_type&.map? || protocol_map @@ -1226,6 +1274,20 @@ def visit_assignment_field(field_node, assignment_node) !assignment_node.value.full_type!(context: "pending field assignment").implicitly_copyable? { |name| lookup_type_schema(name) } T.unsafe(assignment_node.value).ownership_pending_transfer = true end + # Assignment after construction is a keep edge too (retained + # identity v4): a param flowing into an @multiowned field via + # `x.f = param [OR_ELSE default]` is kept exactly like a + # struct-literal store. + if assignment_field_type.multiowned? + owner = begin + field_node.target.full_type!(context: "assignment receiver").resolved + rescue StandardError + nil + end + keep_param_identity!(assignment_node.value, assignment_field_type, + "#{owner || 'field'}.#{field_node.field}") + end + validate_assignment_type( assignment_node, assignment_field_type, @@ -1247,12 +1309,13 @@ def validate_assignment_type(node, target_type, value_type) return if target.any? || value.any? || value.untyped? return if target.resolved == :NIL # Allow narrowing from initial NIL union_schema = lookup_type_schema(target.value_payload_type.resolved) + value_node = node.value if UnionPayloadCompatibility.unique_variant(target, value, union_schema) - node.value.coerced_type = target + value_node.coerced_type = target return end if target.accepts?(value) - node.value.coerced_type = target if target != value || + value_node.coerced_type = target if target != value || (target.node_reference? && !value.node_reference?) return end diff --git a/compiler/ruby/annotator/helpers/auto_inference.rb b/compiler/ruby/annotator/helpers/auto_inference.rb index afdcf2a56..9437f2c9b 100644 --- a/compiler/ruby/annotator/helpers/auto_inference.rb +++ b/compiler/ruby/annotator/helpers/auto_inference.rb @@ -249,8 +249,6 @@ def walk(node, current_fn:) # leaf when Array, Set node.each { |c| walk(c, current_fn: current_fn) } - when Set - node.each { |c| walk(c, current_fn: current_fn) } when Hash node.each_value { |v| walk(v, current_fn: current_fn) } else diff --git a/compiler/ruby/annotator/helpers/capabilities.rb b/compiler/ruby/annotator/helpers/capabilities.rb index 72f90b261..76fecaa98 100644 --- a/compiler/ruby/annotator/helpers/capabilities.rb +++ b/compiler/ruby/annotator/helpers/capabilities.rb @@ -17,7 +17,7 @@ module Capabilities extend T::Sig - CaptureSiteNode = T.type_alias { T.any(AST::MoveNode, AST::CopyNode, AST::Copy, AST::CloneNode) } + CaptureSiteNode = T.type_alias { T.any(AST::MoveNode, AST::CopyNode, AST::Copy, AST::KeepNode) } # Capabilities that are mutually exclusive with each other. Conflict = Struct.new(:set_a, :set_b, :message) diff --git a/compiler/ruby/annotator/helpers/fixable_helpers.rb b/compiler/ruby/annotator/helpers/fixable_helpers.rb index 53fc0c4b8..2b8a35938 100644 --- a/compiler/ruby/annotator/helpers/fixable_helpers.rb +++ b/compiler/ruby/annotator/helpers/fixable_helpers.rb @@ -304,7 +304,7 @@ def emit_variant_typo!(anchor, name, candidates, message, fix_label, # When the move site isn't tracked (e.g., branch-merge paths, BG # captures), fall through to the plain `error!` so the legacy # diagnostic still surfaces. - sig { params(use_node: AST::Identifier, og_node: OwnershipGraph::Node).returns(NilClass) } + sig { params(use_node: AST::Identifier, og_node: OwnershipGraph::OwnershipNode).returns(NilClass) } def emit_use_of_moved_error!(use_node, og_node) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil src = source_code @@ -316,6 +316,17 @@ def emit_use_of_moved_error!(use_node, og_node) return error!(use_node, :USE_OF_MOVED_VALUE, detail: msg) end + # Retained-identity v5: a carrier-polymorphic TAKES parameter consumed + # then used again is a fan-out. The caller chose the carrier, so neither + # a per-use COPY/CLONE nor a declaration capability-upgrade is correct + # here (rule 6). Guide to COPY_OR_CLONE at the first fan-out (rule 3), + # or UNIQUE+COPY if independent identity is the intended contract. + fanout_entry = lookup_scope_for(name)&.resolve_entry(name) + if fanout_entry.is_a?(SymbolEntry) && fanout_entry.is_param && + fanout_entry.carrier_contract == :polymorphic + return emit_carrier_fanout_error!(use_node, og_node, name, move_line, move_col) + end + # Pick COPY vs CLONE for the consumer-site fix. CLEAR uses CLONE # for shared / refcounted handles (`@shared`, `@multiowned`, # `@split` streams) and COPY for plain affine values. Picking the @@ -328,8 +339,8 @@ def emit_use_of_moved_error!(use_node, og_node) is_split = type.respond_to?(:split?) ? type.split? : false use_clone = is_shared || is_multi || is_split - consumer_keyword = use_clone ? "CLONE" : "COPY" - consumer_description_code = use_clone ? :WRAP_CONSUMER_WITH_CLONE : :WRAP_CONSUMER_WITH_COPY + consumer_keyword = use_clone ? "KEEP" : "COPY" + consumer_description_code = use_clone ? :WRAP_CONSUMER_WITH_KEEP : :WRAP_CONSUMER_WITH_COPY replacement_col = move_col replacement_length = name.length @@ -429,10 +440,45 @@ def emit_use_of_moved_error!(use_node, og_node) raise_in_collector: true) end + # Retained-identity v5 carrier-polymorphic fan-out (design rule 3 + + # Diagnostics). Offers COPY_OR_CLONE at the consuming site, which preserves + # the caller's carrier, and names UNIQUE+COPY as the independent-identity + # alternative. + sig do + params(use_node: AST::Identifier, og_node: OwnershipGraph::OwnershipNode, + name: String, move_line: Integer, move_col: Integer).returns(NilClass) + end + def emit_carrier_fanout_error!(use_node, og_node, name, move_line, move_col) + T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil + consumer = consumer_source_text(move_line) + consumer_clause = consumer ? "`#{consumer}` consumes it (line #{move_line})" : "it is consumed (line #{move_line})" + msg = "`#{name}` is used again after #{consumer_clause}. `#{name}` is a " \ + "carrier-polymorphic parameter; choose how the additional owner is " \ + "created: wrap the first consuming use as KEEP to preserve " \ + "the caller's ownership model. If both consumers require independent " \ + "identity, constrain the parameter as UNIQUE and use COPY." + + fixes = [Fix.new( + description: fix_description(:WRAP_CONSUMER_WITH_KEEP, line: move_line), + confidence: :interactive, + edits: [Edit.new( + span: Span.new(file: nil, line: move_line, col: move_col, length: name.length), + replacement: "(KEEP #{name})" + )] + )] + fixable!(use_node, + code: :CARRIER_POLYMORPHIC_FANOUT, + detail: msg, + category: :ownership, + level: :error, + fixes: fixes, + raise_in_collector: true) + end + # Loop-body use of a value that was moved on a prior iteration. The # coda "Values can only be TAKEN once; subsequent iterations have # nothing left to GIVE" is the canonical phrasing per WALKTHROUGH.md. - sig { params(node: T.any(AST::WhileBindLoop, AST::WhileLoop), name: String, og_node: T.nilable(OwnershipGraph::Node), code: Symbol).returns(NilClass) } + sig { params(node: T.any(AST::WhileBindLoop, AST::WhileLoop), name: String, og_node: T.nilable(OwnershipGraph::OwnershipNode), code: Symbol).returns(NilClass) } def emit_use_of_moved_in_loop_error!(node, name, og_node = nil, code: :USE_OF_MOVED_IN_LOOP) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil loop_move_line = og_node&.move_line @@ -446,7 +492,7 @@ def emit_use_of_moved_in_loop_error!(node, name, og_node = nil, code: :USE_OF_MO # Sub-path use after the path's owner was consumed elsewhere. Uses # passive voice ("was already TAKEN / GIVEN") because the subject of # the sentence is the owner — what HAPPENED to it — not the consumer. - sig { params(node: AST::GetField, path: T::Array[NameCandidate], og_node: T.nilable(OwnershipGraph::Node)).returns(NilClass) } + sig { params(node: AST::GetField, path: T::Array[NameCandidate], og_node: T.nilable(OwnershipGraph::OwnershipNode)).returns(NilClass) } def emit_use_of_moved_path_error!(node, path, og_node = nil) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil path_str = path.map(&:to_s).join('.') diff --git a/compiler/ruby/annotator/helpers/function_analysis.rb b/compiler/ruby/annotator/helpers/function_analysis.rb index d755ff904..3b60140c7 100644 --- a/compiler/ruby/annotator/helpers/function_analysis.rb +++ b/compiler/ruby/annotator/helpers/function_analysis.rb @@ -10,7 +10,7 @@ module FunctionAnalysis RoutineNode = T.type_alias { T.any(AST::FunctionDef, AST::LambdaLit) } RoutineBody = T.type_alias { T.any(AST::RawBody, AST::Node) } - CallNode = T.type_alias { T.any(AST::FuncCall, AST::MethodCall) } + FunctionCallNode = T.type_alias { T.any(AST::FuncCall, AST::MethodCall) } CallArgList = T.type_alias { T::Array[AST::Locatable] } DeclaredReturn = T.type_alias { T.nilable(Type::TypeInput) } LifetimeSourceList = T.type_alias { T::Array[FunctionSignature::LifetimeSource] } @@ -18,13 +18,14 @@ module FunctionAnalysis class CallSignatureSite < T::Struct extend T::Sig - const :node, CallNode + prop :node, FunctionCallNode const :name, String prop :args, T::Array[AST::Locatable] sig { params(signature: FunctionSignature).void } def assign_signature!(signature) T.unsafe(node).matched_signature = signature if node.respond_to?(:matched_signature=) + self.node = node end sig { params(arg: AST::Locatable).void } @@ -40,21 +41,33 @@ def replace_arg!(index, arg) sig { params(index: Integer).returns(T::Boolean) } def explicit_mutable_argument?(index) method_node = T.cast(node, T.nilable(AST::MethodCall)) if node.is_a?(AST::MethodCall) - if method_node && !args.empty? && args.first.equal?(method_node.object) - return method_node.explicit_mutable_receiver? if index == 0 - return method_node.explicit_mutable_argument?(index - 1) + if method_node && args.length == method_node.args.length + 1 + concrete_method = method_node + return concrete_method.explicit_mutable_receiver? if index == 0 + return concrete_method.explicit_mutable_argument?(index - 1) end - node.explicit_mutable_argument?(index) + if node.is_a?(AST::FuncCall) + func_node = T.cast(node, AST::FuncCall) + return func_node.explicit_mutable_argument?(index) + end + method_call = T.cast(node, AST::MethodCall) + method_call.explicit_mutable_argument?(index) end sig { params(index: Integer).returns(T.nilable(Lexer::Token)) } def explicit_mutable_argument_token(index) method_node = T.cast(node, T.nilable(AST::MethodCall)) if node.is_a?(AST::MethodCall) - if method_node && !args.empty? && args.first.equal?(method_node.object) - return method_node.explicit_mutable_receiver_token if index == 0 - return method_node.explicit_mutable_argument_token(index - 1) + if method_node && args.length == method_node.args.length + 1 + concrete_method = method_node + return concrete_method.explicit_mutable_receiver_token_value if index == 0 + return concrete_method.explicit_mutable_argument_token(index - 1) + end + if node.is_a?(AST::FuncCall) + func_node = T.cast(node, AST::FuncCall) + return func_node.explicit_mutable_argument_token(index) end - node.explicit_mutable_argument_token(index) + method_call = T.cast(node, AST::MethodCall) + method_call.explicit_mutable_argument_token(index) end end @@ -81,10 +94,14 @@ def exact? def injectable_defaults return [] unless given_args < max_args - slice = params[given_args...max_args] - return [] unless slice - - slice.reject(&:required) + defaults = T.let([], T::Array[AST::Param]) + index = given_args + while index < max_args + param = params.fetch(index) + defaults << param unless param.required + index += 1 + end + defaults end end @@ -141,7 +158,10 @@ def analyze_routine(node, body, declared_return, is_implicit) else visit(body) end - finalize_ownership_transport_facts!(transport_facts) unless language_mode == :strict + unless language_mode == :strict + finalize_ownership_transport_facts!(transport_facts) + end + nil ensure unless language_mode == :strict popped = phase_audit_inputs.ownership_transport_frames.pop @@ -297,7 +317,8 @@ def visit_FunctionDef(node) params: node.params.map { |p| AST::Param.new( name: p.name, type: p.type, required: p.default.nil?, default: p.default, mutable: p.mutable, takes: p.takes, - sync: p.type.any_sync? ? p.type.sync : nil + sync: p.type.any_sync? ? p.type.sync : nil, + carrier_contract: p.carrier_contract )}, return_type: node.annotation_return_type, return_lifetime: lifetime_paths, visibility: node.visibility, @@ -311,6 +332,7 @@ def visit_FunctionDef(node) current_scope.declare(node.name, nil, signature, false, false, nil, :static) register_function_node!(node) + warn_monomorphic_variant_explosion!(node) body_identity = body_identity_for_function(node.name) node.semantic_with_blocks = [] @@ -419,7 +441,7 @@ def visit_FunctionDef(node) # (intrinsic, user-defined, fn-type variable, generic), validate args, # and set the call node's full_type. Also tags cross-module, extern, # call result placement is decided later by escape analysis. - sig { params(node: CallNode, args: CallArgList).returns(T.nilable(Symbol)) } + sig { params(node: FunctionCallNode, args: CallArgList).returns(T.nilable(Symbol)) } def resolve_call(node, args) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil func_name = node.name @@ -544,7 +566,7 @@ def resolve_call(node, args) # retaining its declared !T for OR_ELSE and lowering. All resolved call # families use this boundary so generic and protocol dispatch cannot expose # a different expression type from ordinary calls. - sig { params(node: CallNode, return_type: Type, recoverable: T::Boolean).void } + sig { params(node: FunctionCallNode, return_type: Type, recoverable: T::Boolean).void } def stamp_resolved_call_result!(node, return_type, recoverable: false) T.bind(self, Annotator::Phases::TypeAnalysisSession) resolved = Type.new(return_type) @@ -585,7 +607,7 @@ def mark_owned_c_out_parameters!(signature, args) # values in this allocator (per "one collection = one allocator"). # Returns nil when the call has no container context (plain function call, # or receiver storage not yet determined). - sig { params(node: CallNode).returns(T.nilable(Symbol)) } + sig { params(node: FunctionCallNode).returns(T.nilable(Symbol)) } def receiver_container_alloc(node) return nil unless node.is_a?(AST::MethodCall) obj = node.object @@ -597,7 +619,7 @@ def receiver_container_alloc(node) nil end - sig { params(node: CallNode, signature: FunctionSignature, args: T.nilable(CallArgList)).returns(NilClass) } + sig { params(node: FunctionCallNode, signature: FunctionSignature, args: T.nilable(CallArgList)).returns(NilClass) } def verify_function_signature!(node, signature, args = nil) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil args ||= node.args @@ -621,13 +643,16 @@ def verify_function_signature!(node, signature, args = nil) verify_link_argument!(facts) verify_argument_type!(facts, signature, atomic_bare_value_args) verify_argument_aliases!(facts, encountered_args) + verify_copy_retained_boundary!(facts) + verify_unique_argument!(facts) + verify_retained_into_plain_slot!(facts) end warn_multi_atomic_bare_value_call!(site.node, atomic_bare_value_args) nil end - sig { params(node: CallNode, args: T.nilable(CallArgList)).returns(CallSignatureSite) } + sig { params(node: FunctionCallNode, args: T.nilable(CallArgList)).returns(CallSignatureSite) } def call_signature_site(node, args = nil) args ||= node.args source_name = if node.is_a?(AST::MethodCall) && node.source_method_name @@ -719,6 +744,133 @@ def call_argument_facts(site, param, arg_node, index) ) end + # Retained-identity v5: bare COPY is a memcpy and cannot copy a live handle. + # OWN COPY is the sole retained-carrier detach: it derefs the payload and + # deep-copies it (stamped shared_to_unique_copy). A plain COPY of a retained + # carrier is illegal at every boundary; the check is DEFERRED (keep-analysis + # has not run yet) so the v4 kept-edge exception can consult kept_identity + # after placement, before erroring COPY_RETAINED_NEEDS_UNIQUE. + sig { params(facts: CallArgumentFacts).void } + def verify_copy_retained_boundary!(facts) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + arg = facts.arg_node + return unless arg.is_a?(AST::CopyNode) + src_type = arg.value.full_type!(context: "COPY source carrier") rescue nil + return unless src_type.is_a?(Type) + carrier = if src_type.multiowned? then "@multiowned" + elsif src_type.shared? then "@shared" + end + return unless carrier + + # OWN COPY is the explicit handle->owned-RawT downgrade: deref the retained + # payload and deep-copy it into a fresh uniquely-owned value. Legal for any + # retained source, at any parameter (a plain TAKES slot is already a unique + # owner). Bare COPY of a handle is a memcpy and stays illegal (below). + if arg.own + arg.carrier_op = :shared_to_unique_copy + return + end + + # Bare COPY is a memcpy; it cannot copy a live @multiowned/@shared handle + # (that would duplicate the handle bits and skip the refcount). Illegal at + # every boundary, UNIQUE included -- the detach is always spelled OWN COPY. + name = arg.value.is_a?(AST::Identifier) ? T.cast(arg.value, AST::Identifier).name : "the value" + deferred_copy_retained_validations << Annotator::Phases::DeferredCopyRetainedValidation.new( + arg_node: arg, name: name, carrier: carrier, + callee_name: facts.site.node.name.to_s, param_index: facts.index, + ) + end + + # C-1 (carrier first pass): a UNIQUE parameter requires exactly one owner. + # A bare retained (@multiowned/@shared) argument is a live, potentially + # multi-owned handle -- reject it. COPY (which detaches an independent + # payload, handled by verify_copy_retained_boundary!) and plain owned values + # are the valid ways to satisfy UNIQUE. + # A retained @multiowned/@shared handle cannot silently fill a plain (RawT) + # TAKES slot -- crossing the carrier boundary must be explicit. The caller + # writes OWN COPY (detach an independent payload) or the parameter opts into + # keeping the handle via SHARED/MONOMORPHIC. A bare handle here used to + # silently deep-copy the payload out (identity-destroying, no diagnostic). + # Collections carry their own transport rules and are unaffected; UNIQUE is + # handled by verify_unique_argument!. + # C-3d: a MONOMORPHIC parameter is stenciled per concrete carrier + # (plain / @multiowned / @shared) by Zig. Each independent MONOMORPHIC param + # multiplies the variant count by up to 3, so N of them is up to 3^N compiled + # bodies -- a combinatoric-explosion hazard. Surface it once past a threshold + # so the cost is visible where it is chosen. + sig { params(node: AST::FunctionDef).void } + def warn_monomorphic_variant_explosion!(node) + count = node.params.count { |p| p.carrier_contract == :monomorphic } + return if count < 3 + + variants = 3**count + $stderr.puts "\e[36m[Note]\e[0m '#{node.name}' has #{count} MONOMORPHIC parameters, " \ + "so Zig may stencil up to #{variants} (3^#{count}) carrier variants of it. " \ + "This is a combinatoric-explosion hazard for code size and instruction cache; " \ + "if the extra carriers are not needed, constrain some parameters to a concrete " \ + "carrier (plain / SHARED)." + end + + sig { params(facts: CallArgumentFacts).void } + def verify_retained_into_plain_slot!(facts) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + arg = facts.arg_node + return if arg.is_a?(AST::CopyNode) # COPY / OWN COPY: their own checks + return if arg.is_a?(AST::KeepNode) # KEEP: retains, carrier-preserving + return unless facts.param.takes + # Only user functions declare a plain RawT slot whose carrier contract is + # meaningful here. Container builtins (append/insert/...) carry their own + # element-handle transport and must not be gated by this rule. + callee_fn = function_node_for(facts.site.node.name.to_s) + return unless callee_fn + callee_param = callee_fn.params&.fetch(facts.index, nil) + contract = callee_param&.carrier_contract || facts.param.carrier_contract + # SHARED/MONOMORPHIC accept a handle; UNIQUE is checked separately. + return if contract == :shared || contract == :monomorphic || contract == :unique + param_type = facts.param.type + return if param_type.is_a?(Type) && + (param_type.generic_type_parameter? || param_type.any_rc?) + # A bare MONOMORPHIC source may itself be a handle at some monomorphization, + # and a concrete plain slot cannot hold one. Forward it via OWN COPY (detach) + # or make the destination carrier-preserving. (An unconstrained + # carrier-polymorphic param is always plain post-gate, so forwarding it bare + # is a valid move.) + if arg.is_a?(AST::Identifier) && arg.symbol&.carrier_contract == :monomorphic + return error!(arg, :RETAINED_NEEDS_OWN_COPY, name: arg.name, carrier: "MONOMORPHIC", param: facts.param.name.to_s) + end + src_type = arg.full_type!(context: "retained plain-slot argument") rescue nil + return unless src_type.is_a?(Type) + return if src_type.collection? + carrier = if src_type.multiowned? then "@multiowned" + elsif src_type.shared? then "@shared" + end + return unless carrier + + name = arg.is_a?(AST::Identifier) ? arg.name : "the value" + error!(arg, :RETAINED_NEEDS_OWN_COPY, name: name, carrier: carrier, param: facts.param.name.to_s) + end + + sig { params(facts: CallArgumentFacts).void } + def verify_unique_argument!(facts) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + arg = facts.arg_node + # Any COPY is COPY's concern: verify_copy_retained_boundary! accepts OWN COPY + # (detach) and rejects a bare COPY of a handle (COPY_RETAINED_NEEDS_UNIQUE). + return if arg.is_a?(AST::CopyNode) + callee_param = function_node_for(facts.site.node.name.to_s)&.params&.fetch(facts.index, nil) + contract = callee_param&.carrier_contract || facts.param.carrier_contract + return unless contract == :unique + src_type = arg.full_type!(context: "UNIQUE argument carrier") rescue nil + return unless src_type.is_a?(Type) + carrier = if src_type.multiowned? then "@multiowned" + elsif src_type.shared? then "@shared" + end + return unless carrier + + name = arg.is_a?(AST::Identifier) ? arg.name : "the value" + error!(arg, :UNIQUE_NEEDS_EXCLUSIVE, name: name, carrier: carrier, param: facts.param.name.to_s) + end + sig { params(facts: CallArgumentFacts).void } def verify_mutable_argument!(facts) T.bind(self, Annotator::Phases::TypeAnalysisSession) @@ -785,7 +937,10 @@ def promote_mutable_call_argument!(arg_node) def verify_takes_argument!(facts) T.bind(self, Annotator::Phases::TypeAnalysisSession) if facts.is_give && !facts.param.takes - error!(facts.arg_node, :GIVE_TO_BORROW_PARAM, param: facts.param.name) + # Legal when keep-analysis proves the param kept (retained identity + # v4); keep-ness is whole-program, so the check replays after the + # keep fixpoint. + record_deferred_give_validation!(facts) end return unless facts.param.takes || facts.is_give @@ -811,7 +966,7 @@ def verify_takes_argument!(facts) if language_mode != :strict && !facts.is_give && inner_identifier && !inner_identifier.full_type!(context: "pending TAKES transport").implicitly_copyable? { |name| lookup_type_schema(name) } && - !facts.arg_node.is_a?(AST::CopyNode) && !facts.arg_node.is_a?(AST::CloneNode) + !facts.arg_node.is_a?(AST::CopyNode) && !facts.arg_node.is_a?(AST::KeepNode) T.unsafe(inner_identifier).ownership_pending_transfer = true return end @@ -1094,7 +1249,7 @@ def explicit_primitive_atomic_param?(type) type.atomic? && type.primitive? end - sig { params(node: CallNode, atomic_args: CallArgList).void } + sig { params(node: FunctionCallNode, atomic_args: CallArgList).void } def warn_multi_atomic_bare_value_call!(node, atomic_args) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil unique_args = atomic_args.compact @@ -1304,6 +1459,26 @@ def declare_and_verify_params(node) param.symbol = current_scope.local_entry!(param.name) param.symbol.is_param = true param.symbol.param_decl_token = param.name_token + # Retained-identity v5: propagate the parsed carrier contract onto the + # binding (single writer). Downstream consuming-use analysis and + # placement READ it; they never re-derive from syntax. An unconstrained + # TAKES parameter has a statically unknown carrier (the caller chose): + # mark it so COPY is rejected on it and on its direct aliases. + if param.carrier_contract + param.symbol.carrier_contract = param.carrier_contract + # Only a type that can actually carry a retained identity (@multiowned/ + # @shared) is carrier-polymorphic. Collections, strings, and primitives + # have deterministic copy semantics -- COPY of them is unambiguous. + pt = param.type + retainable = pt.is_a?(Type) && !pt.collection? && !pt.string? && !pt.primitive? + # A MONOMORPHIC param is carrier-polymorphic too (its carrier varies per + # call), so the same fan-out rules apply: COPY is rejected on it and a + # non-final reuse requires KEEP. It differs only in that KEEP resolves + # per concrete carrier at monomorphization (stamp_keep_carrier_op!), + # rather than the deferred tag path. + param.symbol.carrier_polymorphic = true if param.takes && retainable && + (param.carrier_contract == :polymorphic || param.carrier_contract == :monomorphic) + end # Preserve REQUIRES disjunctions for call-site effect resolution. if requires_map fams = requires_map[param.name.to_s] @@ -1316,7 +1491,8 @@ def declare_and_verify_params(node) # Non-TAKES parameters are implicit borrows. Mark in OG so the # annotator prevents storing borrowed data into owned containers. unless param.takes - ownership_graph[param.name]&.kind = :borrowed + graph_node = ownership_graph[param.name] + graph_node.kind = :borrowed if graph_node end param.type end @@ -1573,7 +1749,17 @@ def intrinsic_arg_matches?(spec, arg) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil return true if spec.unconstrained_any? return true if spec.type == :"Any[]" && any_array_intrinsic_arg?(spec.type, arg) - return false unless spec.type == :Any || is_safe_autocast?(arg.resolved_type, spec.type) + # The legacy resolved-symbol spelling of a fixed-size array (String@symbol[3]) + # does not round-trip through Type.new, which misreads it as a plain string + # family member. Bracketed spellings therefore match against their real + # parsed Type; anything else keeps the legacy spelling and never touches + # full_type!, so pre-resolution dispatcher recovery stays reachable. + cast_source = T.let(arg.resolved_type, T.untyped) + if cast_source.to_s.include?("[") + full = arg.full_type!(context: "intrinsic argument") + cast_source = full if full.array? + end + return false unless spec.type == :Any || is_safe_autocast?(cast_source, spec.type) return true unless spec.capability_constrained? arg_type = arg.full_type!(context: "intrinsic capability argument") diff --git a/compiler/ruby/annotator/helpers/function_context.rb b/compiler/ruby/annotator/helpers/function_context.rb index 6bda5df5d..e0cb47bff 100644 --- a/compiler/ruby/annotator/helpers/function_context.rb +++ b/compiler/ruby/annotator/helpers/function_context.rb @@ -44,22 +44,22 @@ def return_type=(val) sig { void } def record_frame_use! - self.frame_count += 1 + @frame_count += 1 end sig { void } def record_heap_use! - self.heap_count += 1 + @heap_count += 1 end sig { void } def record_alloc_use! - self.alloc_count += 1 + @alloc_count += 1 end sig { params(bytes: Integer).void } def record_stack_bytes!(bytes) - self.stack_vars_bytes += bytes + @stack_vars_bytes += bytes end sig { void } @@ -69,22 +69,22 @@ def mark_runtime_used! sig { void } def enter_loop! - self.loop_depth += 1 + @loop_depth += 1 end sig { void } def exit_loop! - self.loop_depth -= 1 + @loop_depth -= 1 end sig { void } def enter_conditional! - self.conditional_depth += 1 + @conditional_depth += 1 end sig { void } def exit_conditional! - self.conditional_depth -= 1 + @conditional_depth -= 1 end sig { params(name: String, return_type: T.nilable(Type::TypeInput), lifetime: T::Array[LifetimeSource], type_params: T::Array[Symbol], generic_params: T::Array[AST::GenericParamDecl]).void } diff --git a/compiler/ruby/annotator/helpers/function_return.rb b/compiler/ruby/annotator/helpers/function_return.rb index cd0db3ec4..d3eaa028f 100644 --- a/compiler/ruby/annotator/helpers/function_return.rb +++ b/compiler/ruby/annotator/helpers/function_return.rb @@ -127,16 +127,10 @@ def resolve(receiver, args = []) Type.optional_of(T.must(receiver).value_type) when Kind::ValueList value = T.must(receiver).value_type - list = Type.new(:"#{value.resolved}[]", collection: :list) - list.elem_ownership = value.ownership - list.elem_sync = value.sync - list + element_list(value) when Kind::KeyList key = T.must(receiver).key_type - list = Type.new(:"#{key.resolved}[]", collection: :list) - list.elem_ownership = key.ownership - list.elem_sync = key.sync - list + element_list(key) when Kind::Infer resolve_infer(args) else @@ -144,6 +138,19 @@ def resolve(receiver, args = []) end end + # List-of-element type for keys()/values(). Element capabilities are + # stamped only when NON-default: a declared `[]String@symbol` return + # leaves elem_ownership nil, and explicitly stamping the default + # (:affine) both fails the capability equality in return checking and + # flips the "has element capabilities" predicates. + sig { params(element: Type).returns(Type) } + def element_list(element) + list = Type.new(:"#{element.resolved}[]", collection: :list) + list.elem_ownership = element.ownership unless element.ownership.nil? || element.ownership == :affine + list.elem_sync = element.sync unless element.sync.nil? + list + end + sig { params(args: T::Array[AST::Node]).returns(Type) } def resolve_infer(args) r = case T.must(infer) @@ -163,8 +170,12 @@ def resolve_infer(args) sig { params(args: T::Array[AST::Node]).returns(Type) } def infer_element_type(args) receiver = args.first - type = receiver.is_a?(AST::Locatable) ? receiver.full_type!(context: "element receiver") : nil - type&.element_type || Type.new(:Any) + return Type.new(:Any) unless receiver + + type = receiver.type_object + return Type.new(:Any) unless type + + type.element_type || Type.new(:Any) end sig { params(args: T::Array[AST::Node]).returns(Type) } @@ -175,7 +186,10 @@ def infer_optional_element_type(args) sig { params(args: T::Array[AST::Node]).returns(Type) } def infer_to_list(args) receiver = T.must(args.first) - receiver_type = receiver.full_type!(context: "toList receiver") + receiver_type = receiver.type_object + raise "toList receiver: unresolved type info" unless receiver_type + receiver_type = receiver_type + raise "toList receiver: unresolved type info" if receiver_type.untyped? element_type = if receiver_type.dynamic_stream? || receiver_type.promise_list? receiver_type.tense_type.element_type elsif receiver_type.bounded_stream? diff --git a/compiler/ruby/annotator/helpers/function_signature.rb b/compiler/ruby/annotator/helpers/function_signature.rb index b635def00..0029d26eb 100644 --- a/compiler/ruby/annotator/helpers/function_signature.rb +++ b/compiler/ruby/annotator/helpers/function_signature.rb @@ -20,8 +20,8 @@ class FunctionSignature LifetimeInput = T.type_alias { T.nilable(T.any(LifetimeSource, T::Array[LifetimeSource])) } RequiresMap = T.type_alias { T::Hash[String, T::Set[Symbol]] } GenericBounds = T.type_alias { T::Hash[Symbol, T::Array[Type]] } - ExternEffectValue = T.type_alias { T.any(Symbol, TrueClass) } - ExternEffects = T.type_alias { T::Hash[Symbol, ExternEffectValue] } + ExternEffectValue = T.type_alias { AST::ExternEffectValue } + ExternEffects = T.type_alias { AST::ExternEffects } SignatureEffectSet = T.type_alias { T::Set[Symbol] } SyncSource = T.type_alias { T.any(AST::FunctionDef, Struct) } @@ -288,6 +288,34 @@ def self.unwrap(x) nil end + # Signatures MIRLowering needs to classify every call site: the program's + # own function definitions plus module-imported signatures, so needs_rt and + # can_fail resolve for cross-module calls instead of defaulting to worst case. + # ruby-to-clear: skip + sig { params(ast: AST::Program, scope: Scope).returns(T::Hash[String, FunctionSignature]) } + # ruby-to-clear: skip + def self.lowering_signatures(ast, scope) + sigs = T.let({}, T::Hash[String, FunctionSignature]) + ast.statements.each do |stmt| + next unless stmt.is_a?(AST::FunctionDef) + sigs[stmt.name] = from_function_def(stmt) + end + merge_imported_signatures!(sigs, scope) + sigs + end + + # ruby-to-clear: skip + sig { params(sigs: T::Hash[T.untyped, FunctionSignature], scope: Scope).void } + # ruby-to-clear: skip + def self.merge_imported_signatures!(sigs, scope) + scope.visible_entries.each do |name, entry| + next if sigs.key?(name) + imported = entry.fn_signature + next unless imported && imported.module_alias + sigs[name] = imported + end + end + # ruby-to-clear: skip sig { params(fn: AST::FunctionDef).returns(FunctionSignature) } # ruby-to-clear: skip @@ -519,10 +547,17 @@ def emits_allocating? # Signature can_fail is explicit source-visible behavior. Allocation-only # effects are stamped on call nodes separately and must not turn every - # allocating expression into !T. + # allocating expression into !T. A resolved signature carries the ERROR + # channel in error_fallible; only when that is unset (extern/legacy sigs that + # predate the error_fallible split) do we fall back to can_fail. Reading + # can_fail unconditionally would mark a callee error-recoverable purely + # because it allocates - the exact conflation this contract forbids, and the + # same rule the cross-module import seed applies (see effects.rb). sig { returns(T::Boolean) } def recoverable_result? - @facts.error_fallible == true || can_fail == true + return can_fail == true if @facts.error_fallible.nil? + + @facts.error_fallible == true end sig { returns(T::Boolean) } @@ -744,6 +779,20 @@ def sync_from_function_def!(fn) end public :sync_from_function_def! + # Signature params are fresh AST::Param copies; call-edge lowering reads + # keep-analysis stamps through the shared SymbolEntry. Adopting the + # definition's symbols is the signature's own mutation - callers must + # not reach through the params accessor to write elements (the accessor + # returns copies under CLEAR's ownership model). + sig { params(fn: AST::FunctionDef).void } + def adopt_param_symbols!(fn) + @contract.params.each_with_index do |sig_param, idx| + src = fn.params[idx] + sig_param[:symbol] ||= src&.symbol + end + end + public :adopt_param_symbols! + sig { params(requires: T.nilable(RequiresMap)).void } def replace_requires_storage!(requires) copied_requires = FunctionSignature.copy_requires_for_import(requires || {}) diff --git a/compiler/ruby/annotator/helpers/function_signature_returns.rb b/compiler/ruby/annotator/helpers/function_signature_returns.rb index 5fd0118bc..df71ab8a1 100644 --- a/compiler/ruby/annotator/helpers/function_signature_returns.rb +++ b/compiler/ruby/annotator/helpers/function_signature_returns.rb @@ -36,7 +36,7 @@ def intrinsic_call_validation_signature contract_param = params[index] validation_params << AST::Param.new( name: arg_spec.name || "arg#{index}", - type: arg_spec.type, + type: Type.new(arg_spec.type), required: true, # Receiver mutation and registry-level TAKES metadata are normalized # onto the signature params. Rebuilding solely from arg specs silently diff --git a/compiler/ruby/annotator/helpers/generic_analysis.rb b/compiler/ruby/annotator/helpers/generic_analysis.rb index 3ea8cab18..6e8326885 100644 --- a/compiler/ruby/annotator/helpers/generic_analysis.rb +++ b/compiler/ruby/annotator/helpers/generic_analysis.rb @@ -25,7 +25,7 @@ module GenericAnalysis TargetLongLong TargetULongLong String Any Void Range Map ].freeze DeclarationNode = T.type_alias { T.any(AST::VarDecl, AST::BindExpr) } - TypeShape = T.type_alias { T.any(Type, Symbol, String) } + AnnotationTypeShape = T.type_alias { T.any(Type, Symbol, String) } GenericSchema = T.type_alias { T.any(Schemas::EnumSchema, Schemas::StructSchema, Schemas::UnionSchema, Schemas::ResourceSchema) } GenericCallNode = T.type_alias { T.any(AST::FuncCall, AST::MethodCall) } GenericCallArgs = T.type_alias { T::Array[AST::Locatable] } @@ -566,7 +566,7 @@ def map_requires_protocol_lowering?(type) def protocol_map_associated_type(type, member) return member == :Key ? type.key_type : type.value_type if type.map? - Type.new(TypeProjectionExpression.new(owner: type.resolved, member: member)) + Type.new(TypeExpression.of(TypeProjectionExpression.new(owner: type.resolved, member: member))) end private :protocol_map_associated_type @@ -691,105 +691,103 @@ def apply_type_subst(type_obj, subst) sig { params(expression: TypeExpression, subst: GenericSubstitution).returns(TypeExpression) } def apply_expression_subst(expression, subst) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil - case expression + kind = expression.kind + cap = expression.capabilities + case kind when NamedTypeExpression - if expression.arguments.empty? && subst.key?(expression.name) - replacement = Type.new(T.unsafe(subst[expression.name])) - if replacement.generic_payload_type_arg? && !expression.capabilities.polymorphic_shared + if kind.arguments.empty? && subst.key?(kind.name) + replacement = Type.new(T.unsafe(subst[kind.name])) + if replacement.generic_payload_type_arg? && !cap.polymorphic_shared replacement.strip_runtime_capabilities! end parameter = Type.new(expression) replacement.merge_capabilities_from!(parameter) if generic_type_has_capabilities?(parameter) return replacement.shape.expression end - NamedTypeExpression.new( - name: expression.name, - arguments: expression.arguments.map { |argument| apply_expression_subst(argument, subst) }, - capabilities: expression.capabilities - ) + TypeExpression.new(kind: NamedTypeExpression.new( + name: kind.name, + arguments: kind.arguments.map { |argument| apply_expression_subst(argument, subst) }, + ), capabilities: cap) when TypeProjectionExpression - binding = subst[expression.owner] + binding = subst[kind.owner] return expression unless binding concrete = Type.new(binding) - projected = case expression.member - when :Key then concrete.key_type if (expression.protocol.nil? || expression.protocol == :Map) && concrete.map? - when :Value then concrete.value_type if (expression.protocol.nil? || expression.protocol == :Map) && concrete.map? + projected = T.let(nil, T.nilable(Type)) + if kind.member == :Key + if (kind.protocol.nil? || kind.protocol == :Map) && concrete.map? + projected = concrete.key_type + end + elsif kind.member == :Value + if (kind.protocol.nil? || kind.protocol == :Map) && concrete.map? + projected = concrete.value_type + end end - if projected.nil? && expression.protocol && expression.protocol != :Map - match = conformance_match(expression.protocol.to_s, concrete) + if projected.nil? && kind.protocol && kind.protocol != :Map + match = conformance_match(kind.protocol.to_s, concrete) if match - associated = match.resolution.associated_types[expression.member] + associated = match.resolution.associated_types[kind.member] projected = apply_type_subst(associated, match.substitutions) if associated end end unless projected - return TypeProjectionExpression.new( + return TypeExpression.new(kind: TypeProjectionExpression.new( owner: concrete.resolved, - member: expression.member, - protocol: expression.protocol, - capabilities: expression.capabilities, - ) + member: kind.member, + protocol: kind.protocol, + ), capabilities: cap) end TypeExpressionTree.with_root_capabilities( projected.shape.expression, - expression.capabilities, + cap, ) when FunctionTypeExpression - signature = expression.signature - FunctionTypeExpression.new( - signature: Type::FunctionType.new( - params: signature.params.map do |param| - Type::FunctionTypeParam.new(type: apply_type_subst(param.type, subst)) - end, - return_type: apply_type_subst(signature.return_type, subst), - reentrant: signature.reentrant, - source_signature: signature.source_signature, - abi: signature.abi, - ), - capabilities: expression.capabilities - ) + signature = Type.function_type_for_expression(kind) + TypeExpression.new(kind: Type.function_type_expression_for(Type::FunctionType.new( + params: signature.params.map do |param| + Type::FunctionTypeParam.new(type: apply_type_subst(param.type, subst)) + end, + return_type: apply_type_subst(signature.return_type, subst), + reentrant: signature.reentrant, + source_signature: signature.source_signature, + abi: signature.abi, + )), capabilities: cap) when TupleTypeExpression - TupleTypeExpression.new( - items: expression.items.map { |item| apply_expression_subst(item, subst) }, - capabilities: expression.capabilities - ) + TypeExpression.new(kind: TupleTypeExpression.new( + items: kind.items.map { |item| apply_expression_subst(item, subst) }, + ), capabilities: cap) when OptionalTypeExpression - OptionalTypeExpression.new(inner: apply_expression_subst(expression.inner, subst), - capabilities: expression.capabilities) + TypeExpression.new(kind: OptionalTypeExpression.new(inner: apply_expression_subst(kind.inner, subst)), + capabilities: cap) when FallibleTypeExpression - error_set = expression.error_set - FallibleTypeExpression.new( - inner: apply_expression_subst(expression.inner, subst), + error_set = kind.error_set + TypeExpression.new(kind: FallibleTypeExpression.new( + inner: apply_expression_subst(kind.inner, subst), error_set: error_set.nil? ? nil : apply_expression_subst(error_set, subst), - capabilities: expression.capabilities - ) + ), capabilities: cap) when FutureTypeExpression - FutureTypeExpression.new(inner: apply_expression_subst(expression.inner, subst), - capabilities: expression.capabilities) + TypeExpression.new(kind: FutureTypeExpression.new(inner: apply_expression_subst(kind.inner, subst)), + capabilities: cap) when LinearTypeExpression - LinearTypeExpression.new( - kind: expression.kind, - dimensions: expression.dimensions, - item: apply_expression_subst(expression.item, subst), - allocation_hint: expression.allocation_hint, - capabilities: expression.capabilities - ) + TypeExpression.new(kind: LinearTypeExpression.new( + kind: kind.kind, + dimensions: kind.dimensions, + item: apply_expression_subst(kind.item, subst), + allocation_hint: kind.allocation_hint, + ), capabilities: cap) when MapTypeExpression - MapTypeExpression.new( - key: apply_expression_subst(expression.key, subst), - value: apply_expression_subst(expression.value, subst), - key_implicit: expression.key_implicit, - legacy_separator: expression.legacy_separator, - capabilities: expression.capabilities - ) + TypeExpression.new(kind: MapTypeExpression.new( + key: apply_expression_subst(kind.key, subst), + value: apply_expression_subst(kind.value, subst), + key_implicit: kind.key_implicit, + legacy_separator: kind.legacy_separator, + ), capabilities: cap) when StreamTypeExpression - StreamTypeExpression.new( - cardinality: expression.cardinality, - item: apply_expression_subst(expression.item, subst), - capabilities: expression.capabilities - ) + TypeExpression.new(kind: StreamTypeExpression.new( + cardinality: kind.cardinality, + item: apply_expression_subst(kind.item, subst), + ), capabilities: cap) else expression end @@ -900,7 +898,7 @@ def validate_stream_type!(node) if node.type.multiowned? error!(node, :RC_PROMISE_NEEDS_SHARED) end - if node.type.split? && !node.type.open_stream? + if node.type.split? && !node.type.split_open_stream? error!(node, :ATSPLIT_NEEDS_OPEN_STREAM) end end @@ -908,7 +906,7 @@ def validate_stream_type!(node) # After coerce! validates type compatibility, propagate declared-type metadata # into the value node so the transpiler sees the correct runtime type. # Handles: BgStreamBlock ~T[INF] retyping, shard_count, @shared promise ownership. - sig { params(node: DeclarationNode, final_type: TypeShape).void } + sig { params(node: DeclarationNode, final_type: AnnotationTypeShape).void } def propagate_declared_type_to_value!(node, final_type) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil return unless node.type @@ -996,7 +994,7 @@ def stream_body_contains_break?(body) # Propagate collection, shard_count, soa, and sync metadata from the declared # type annotation (or inferred value type) into node.full_type and node.full_type. # These fields are lost during finalize_storage! and coerce!. - sig { params(node: DeclarationNode, final_type: TypeShape).void } + sig { params(node: DeclarationNode, final_type: AnnotationTypeShape).void } def propagate_collection_metadata!(node, final_type) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil _ = final_type @@ -1042,10 +1040,12 @@ def register_container_borrow!(node) container = find_container_source(node.value) return unless container var_name = node.name.is_a?(String) ? node.name : node.name.to_s - ownership_graph[var_name]&.kind = :borrowed + graph_node = ownership_graph[var_name] + graph_node.kind = :borrowed if graph_node T.must(node.symbol).mark_borrowed_alias! if node.respond_to?(:symbol) && node.symbol node.container_borrow = true - node.value.container_borrow = true if node.value.respond_to?(:container_borrow=) + value_node = node.value + value_node.container_borrow = true if value_node.respond_to?(:container_borrow=) node.storage = :borrow if node.respond_to?(:storage=) true end @@ -1059,7 +1059,7 @@ def find_container_source(expr) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil return nil unless expr # COPY/CLONE produce owned/retained values; no borrow relationship. - return nil if expr.is_a?(AST::CopyNode) || expr.is_a?(AST::CloneNode) + return nil if expr.is_a?(AST::CopyNode) || expr.is_a?(AST::KeepNode) if expr.respond_to?(:container_borrow) && expr.container_borrow receiver = if expr.respond_to?(:object) T.unsafe(expr).object diff --git a/compiler/ruby/annotator/helpers/method_analysis.rb b/compiler/ruby/annotator/helpers/method_analysis.rb index bfa3f7cdd..2308a332a 100644 --- a/compiler/ruby/annotator/helpers/method_analysis.rb +++ b/compiler/ruby/annotator/helpers/method_analysis.rb @@ -111,7 +111,13 @@ def resolve_typed_method(node, obj_type, registry, tag_field, type_label) end # Set tag and return type - node.public_send(:"#{tag_field}=", node.name.to_sym) + case tag_field + when :pool_method then node.pool_method = node.name.to_sym + when :set_method then node.set_method = node.name.to_sym + when :map_method then node.map_method = node.name.to_sym + else + raise "resolve_typed_method: unknown collection tag field #{tag_field}" + end stamp_type!(node, defn.return_def.resolve(obj_type, [])) node.container_borrow = defn.intrinsic_container_borrow? diff --git a/compiler/ruby/annotator/helpers/pipe_analysis.rb b/compiler/ruby/annotator/helpers/pipe_analysis.rb index 1a9073fd3..f685c1b20 100644 --- a/compiler/ruby/annotator/helpers/pipe_analysis.rb +++ b/compiler/ruby/annotator/helpers/pipe_analysis.rb @@ -3,6 +3,7 @@ require_relative "../../ast/ast" require_relative "../../ast/type" require_relative "../../semantic/tense_operation_plan" +require_relative "../../mir/lower/pipeline/pipeline_placeholder_usage" require 'set' module PipeAnalysis @@ -89,6 +90,7 @@ def required_order end sig { returns(Type) } + # ruby-to-clear: fallible def leaf_type plan.leaf_type end @@ -124,12 +126,54 @@ def visit_Smooth(node) error!(node, :PIPE_BAD_DESTINATION) stamp_type!(node, :Any) end + + # Every materializing (finite, list-or-scalar-producing) pipeline + # allocates through the runtime (result list, scratch, sort buffers). + # The per-shape branches above each remember — or forget — to record + # that; a forgotten shape (UNNEST terminal in an IF condition) left + # needs_rt false and the rt parameter was DROPPED from a signature + # whose lowered body references it (undeclared 'rt' Zig). Record once + # here, at the dispatch level: over-recording only threads rt where it + # might be unused; under-recording emits uncompilable code. + result_ti = Type.from_node!(node, context: "pipeline effect recording") + unless result_ti.inf_stream? + current_fn_ctx&.record_frame_use! + if node.storage == :heap || result_ti.heap? + current_fn_ctx&.record_heap_use! + current_fn_ctx&.record_alloc_use! + end + end end smooth_depth end private + # True when a SELECT element expression will lower with per-iteration frame + # transients: a nested materializing pipeline (its whole loop apparatus is + # rebuilt every outer iteration) or a composite literal with constructed + # (owned) field values (each hoists an owned temp per iteration). Both + # force per-iteration rewind in the lowered loop, which is only sound with + # a heap result — see the storage stamp in visit_Smooth. + sig { params(expr: AST::Node).returns(T::Boolean) } + def select_element_forces_heap_result?(expr) + found = T.let(false, T::Boolean) + AST.each_locatable(expr) do |node| + case node + when AST::BinaryOp + found = true if node.smooth? + when AST::StructLit + constructed = node.fields.values.any? do |v| + v.is_a?(AST::CopyNode) || v.is_a?(AST::MoveNode) || + v.is_a?(AST::FuncCall) || v.is_a?(AST::MethodCall) || + v.is_a?(AST::StringConcat) + end + found = true if constructed + end + end + found + end + sig { params(node: T.nilable(AST::Node)).returns(T::Boolean) } def pipe_complex_op?(node) AST.pipeline_complex_op?(node) || node.is_a?(AST::RecoverOp) || node.is_a?(AST::CollectOp) @@ -183,14 +227,14 @@ def stamp_observable_terminal!(node) # raw: :"~Int64") # lift_to_observable_if_terminal!(node, terminal: :distinct, # raw: :"~Int64[]", collection: :set) - sig { params(node: AST::BinaryOp, terminal: Symbol, raw: Symbol, type_kwargs: ObservableTypeKwValue).returns(T.nilable(Type)) } - def lift_to_observable_if_terminal!(node, terminal:, raw:, **type_kwargs) + sig { params(node: AST::BinaryOp, terminal: Symbol, raw: Symbol, collection: T.nilable(Symbol)).returns(T.nilable(Type)) } + def lift_to_observable_if_terminal!(node, terminal:, raw:, collection: nil) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil return unless node.observable_terminal stamp_type!(node, Type.new(raw, observable: true, observable_terminal: terminal, - **type_kwargs)) + collection: collection)) end # M5: collapse the stamp + lift pair that every analyze_*_op call site @@ -202,11 +246,11 @@ def lift_to_observable_if_terminal!(node, terminal:, raw:, **type_kwargs) # stamp_observable_terminal!, the only argumentation a call site # carries is terminal kind + raw type + extra type kwargs, so a # single helper is enough. - sig { params(node: AST::BinaryOp, terminal: Symbol, raw: Symbol, type_kwargs: ObservableTypeKwValue).returns(T.nilable(Type)) } - def mark_observable_terminal!(node, terminal:, raw:, **type_kwargs) + sig { params(node: AST::BinaryOp, terminal: Symbol, raw: Symbol, collection: T.nilable(Symbol)).returns(T.nilable(Type)) } + def mark_observable_terminal!(node, terminal:, raw:, collection: nil) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil stamp_observable_terminal!(node) - lift_to_observable_if_terminal!(node, **T.unsafe({terminal: terminal, raw: raw, **type_kwargs})) + lift_to_observable_if_terminal!(node, terminal: terminal, raw: raw, collection: collection) end sig { params(node: AST::Locatable).returns(T::Boolean) } @@ -219,21 +263,25 @@ def bounded_stream_source?(node) def pipeline_source_fact(source, source_type, include_inf_stream: false) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil if include_inf_stream && source_type.inf_stream? - return PipelineSourceFact.new(kind: :inf_stream, item_type: T.must(source_type.inf_stream_element_type)) + item = source_type.canonical_stream_item_type || source_type.inf_stream_element_type + return PipelineSourceFact.new(kind: :inf_stream, item_type: T.must(item)) end return PipelineSourceFact.new(kind: :range, item_type: range_element_type(source)) if source.is_a?(AST::RangeLit) if source_type.open_stream? - return PipelineSourceFact.new(kind: :open_stream, item_type: T.must(source_type.open_stream_element_type)) + item = source_type.canonical_stream_item_type || source_type.open_stream_element_type + return PipelineSourceFact.new(kind: :open_stream, item_type: T.must(item)) end if source_type.dynamic_stream? - return PipelineSourceFact.new(kind: :dynamic_stream, item_type: T.must(source_type.tense_type.element_type)) + item = source_type.canonical_stream_item_type || source_type.tense_type.element_type + return PipelineSourceFact.new(kind: :dynamic_stream, item_type: T.must(item)) end if source_type.bounded_stream? - return PipelineSourceFact.new(kind: :bounded_stream, item_type: T.must(source_type.tense_type.element_type)) + item = source_type.canonical_stream_item_type || source_type.tense_type.element_type + return PipelineSourceFact.new(kind: :bounded_stream, item_type: T.must(item)) end element_type = source_type.element_type @@ -395,6 +443,12 @@ def analyze_select_family_op(node) validate_where_effect_contract!(node.right, selector_effect) elsif node.right.is_a?(AST::SelectOp) validate_select_effect_contract!(node.right, selector_effect) + # An infinite rendezvous stream never drains: a selector that MOVES + # the item (GIVE / TAKES) can leave moved payloads in flight at + # teardown, which the runtime cannot reclaim safely. Fail closed. + if source.inf_stream? && PipelinePlaceholderUsage.moved_placeholder?(node.right.expression) + error!(node.right, :INF_STREAM_SELECT_MOVES_ITEM) + end end end @@ -430,10 +484,31 @@ def analyze_select_family_op(node) # container have one coherent cleanup allocator. node.storage = node.full_type!(context: "pipeline result storage").promise_list? ? :heap : :frame + # An element expression that allocates frame transients EVERY iteration + # (a nested materializing pipeline, an owned composite construction) + # forces a per-iteration arena rewind in the lowered loop. Rewind frees + # everything frame-allocated during the iteration, so the escaping result + # must live on the heap — the one storage decision that makes rewind + # sound. Stamped HERE (the storage fact owner) so hoist cleanup, binding + # classification, and lowering all read one coherent placement. + if node.storage == :frame && node.right.is_a?(AST::SelectOp) && + select_element_forces_heap_result?(node.right.expression) + node.storage = :heap + end + # WHERE/SELECT/ORDER_BY allocate intermediate ArrayListUnmanaged at the - # transpiler level via rt.frameAlloc(). InfStream results are not materialized; - # only count frame allocation for finite (list-producing) results. - current_fn_ctx&.record_frame_use! unless source.inf_stream? || node.storage == :heap + # transpiler level via rt.frameAlloc() / heapAlloc(). InfStream results are + # not materialized. A heap-storage result still ALLOCATES — recording + # nothing for it left needs_rt false, dropping the rt parameter from the + # signature while the lowered body references it (undeclared 'rt' Zig). + unless source.inf_stream? + if node.storage == :heap + current_fn_ctx&.record_heap_use! + current_fn_ctx&.record_alloc_use! + else + current_fn_ctx&.record_frame_use! + end + end nil end @@ -442,8 +517,8 @@ def analyze_select_family_op(node) .returns(Type) end def select_stream_result_type(source_type, source, effect) - cardinality = if source.stream? && source_type.shape.expression.is_a?(StreamTypeExpression) - T.cast(source_type.shape.expression, StreamTypeExpression).cardinality + cardinality = if source.stream? && source_type.shape.expression.kind.is_a?(StreamTypeExpression) + T.cast(source_type.shape.expression.kind, StreamTypeExpression).cardinality else :FINITE end @@ -1355,8 +1430,11 @@ def emit_multi_map_warning(conc, sharded_names) T.bind(self, Annotator::Phases::TypeAnalysisSession) rescue nil shard_counts = sharded_names.map do |name| sc = lookup_scope_for(name)&.resolve_entry(name)&.type - t = sc.is_a?(Type) ? sc : Type.new(T.unsafe(sc)) - t.shard_count + if sc.is_a?(Type) + sc.shard_count + else + Type.new(T.unsafe(sc)).shard_count + end end.compact.uniq names_str = sharded_names.to_a.join(', ') if shard_counts.length == 1 @@ -1480,7 +1558,7 @@ def each_shard_scan_node(node, &blk) return end if node.is_a?(AST::Capability) - node.each_pair do |_, val| + [node[:var_node], node[:guard_expr], node[:view_length]].each do |val| if val.is_a?(Array) || val.is_a?(AST::Capability) || val.is_a?(AST::Locatable) each_shard_scan_node(val, &blk) end @@ -1564,16 +1642,11 @@ def analyze_shard_op(node) error!(shard_op.target_map, :SHARD_TARGET_BAD, remediation: "SHARD routes items to owning schedulers — :locked maps don't have ownership.") end - map_key_type = target_info&.key_type&.resolved - if map_key_type == :String || map_key_type.nil? - unless key_type == :String - error!(shard_op.key_expr, :SHARD_KEY_NEEDS_STRING, got: key_type) - end - else - # Numeric-keyed map: key expression must match the map's key type - unless Type.new(key_type).numeric? - error!(shard_op.key_expr, :SHARD_KEY_NEEDS_NUMERIC, map_key_type: map_key_type, got: key_type) - end + map_key = target_info&.key_type + actual_key = shard_op.key_expr.full_type!(context: "SHARD key expression") + if map_key && target_info && !target_info.accepts_map_key?(actual_key) + error!(shard_op.key_expr, :GENERIC_MAP_KEY_MISMATCH, + expected: Type.surface_name(map_key), actual: Type.surface_name(actual_key)) end # SHARD is consumed by the subsequent CONCURRENT EACH — not standalone. diff --git a/compiler/ruby/annotator/helpers/union.rb b/compiler/ruby/annotator/helpers/union.rb index afab807b8..cb154bbbf 100644 --- a/compiler/ruby/annotator/helpers/union.rb +++ b/compiler/ruby/annotator/helpers/union.rb @@ -23,17 +23,24 @@ def self.unique_variant(expected_type, actual_type, schema) actual_type end - matches = schema.variants.filter_map do |variant_name, payload| - next unless payload.is_a?(Type) - payload_matches?(payload, compared_actual) ? variant_name : nil + matches = T.let([], T::Array[T.any(String, Symbol)]) + schema.variants.keys.each do |variant_name| + payload = schema.variants[variant_name] + next unless payload + + concrete_payload = payload + case concrete_payload + when Type + matches << variant_name if payload_matches?(concrete_payload, compared_actual) + end end matches.one? ? matches.first : nil end sig { params(payload_type: Type, actual_type: Type).returns(T::Boolean) } def self.payload_matches?(payload_type, actual_type) - payload_surface = Type.coercion_surface_name(payload_type) - actual_surface = Type.coercion_surface_name(actual_type) + payload_surface = Type.coercion_surface_name_type(payload_type) + actual_surface = Type.coercion_surface_name_type(actual_type) return true if payload_surface == actual_surface return false if payload_type.string? || actual_type.string? diff --git a/compiler/ruby/annotator/phases/annotation_products.rb b/compiler/ruby/annotator/phases/annotation_products.rb index a860baa94..60477e332 100644 --- a/compiler/ruby/annotator/phases/annotation_products.rb +++ b/compiler/ruby/annotator/phases/annotation_products.rb @@ -106,6 +106,8 @@ class TypedProgramFacts attr_reader :ownership_graph sig { returns(Semantic::LifecycleRegistry) } attr_reader :lifecycle_registry + sig { returns(Semantic::LinearResourceFacts) } + attr_reader :linear_resource_facts sig do params( @@ -115,10 +117,11 @@ class TypedProgramFacts unresolved_node_count: Integer, ownership_graph: OwnershipGraph, lifecycle_registry: Semantic::LifecycleRegistry, + linear_resource_facts: Semantic::LinearResourceFacts, local_function_facts: T.nilable(LocalFacts) ).void end - def initialize(resolution:, body_summaries:, typed_node_count:, unresolved_node_count:, ownership_graph:, lifecycle_registry: Semantic::LifecycleRegistry.empty, local_function_facts: nil) + def initialize(resolution:, body_summaries:, typed_node_count:, unresolved_node_count:, ownership_graph:, lifecycle_registry: Semantic::LifecycleRegistry.empty, linear_resource_facts: Semantic::LinearResourceFacts.empty, local_function_facts: nil) raise "typed program cannot publish unresolved nodes" unless unresolved_node_count.zero? @resolution = T.let(resolution, ResolutionFacts) @@ -129,6 +132,7 @@ def initialize(resolution:, body_summaries:, typed_node_count:, unresolved_node_ @unresolved_node_count = T.let(unresolved_node_count, Integer) @ownership_graph = T.let(ownership_graph, OwnershipGraph) @lifecycle_registry = T.let(lifecycle_registry, Semantic::LifecycleRegistry) + @linear_resource_facts = T.let(linear_resource_facts, Semantic::LinearResourceFacts) freeze end diff --git a/compiler/ruby/annotator/phases/capability_audit_session.rb b/compiler/ruby/annotator/phases/capability_audit_session.rb index 2f2e3d3e7..f9f5d7a8e 100644 --- a/compiler/ruby/annotator/phases/capability_audit_session.rb +++ b/compiler/ruby/annotator/phases/capability_audit_session.rb @@ -140,6 +140,12 @@ def deferred_with_validations = phase_audit_inputs.deferred_with_validations sig { returns(T::Array[DeferredRecoveryValidation]) } def deferred_recovery_validations = phase_audit_inputs.deferred_recovery_validations + sig { returns(T::Array[DeferredGiveValidation]) } + def deferred_give_validations = phase_audit_inputs.deferred_give_validations + + sig { returns(T::Array[DeferredCopyRetainedValidation]) } + def deferred_copy_retained_validations = phase_audit_inputs.deferred_copy_retained_validations + sig { returns(CapabilityAudit::BindingAuditStore) } def capability_audit = phase_audit_inputs.capability_audit diff --git a/compiler/ruby/annotator/phases/capability_evidence.rb b/compiler/ruby/annotator/phases/capability_evidence.rb index 43f1bdbd5..9b5467e26 100644 --- a/compiler/ruby/annotator/phases/capability_evidence.rb +++ b/compiler/ruby/annotator/phases/capability_evidence.rb @@ -32,6 +32,27 @@ class DeferredRecoveryValidation < T::Struct const :value_type, Type end + # GIVE into a non-TAKES param is legal only when keep-analysis proves the + # param kept (retained identity v4). Keep-ness is a whole-program fact, + # so the check replays after the keep fixpoint. + class DeferredGiveValidation < T::Struct + const :arg_node, AST::Node + const :callee_name, String + const :param_index, Integer + const :param_name, String + end + + # Retained-identity v5 (V5-3b): a COPY of a retained (@multiowned/@shared) + # value at a non-UNIQUE call edge. Replayed after keep-analysis so the + # temporary v4 kept-edge exception can consult kept_identity. + class DeferredCopyRetainedValidation < T::Struct + const :arg_node, AST::Node + const :name, String + const :carrier, String + const :callee_name, String + const :param_index, Integer + end + # Facts gathered while typing but consumed only by capability auditing. # This immutable schema is owned by the phase boundary, not by either # executor on its two sides. @@ -45,6 +66,12 @@ class CapabilityAuditInputs < T::Struct prop :deferred_recovery_validations, T::Array[DeferredRecoveryValidation], factory: -> { [] } + prop :deferred_give_validations, + T::Array[DeferredGiveValidation], + factory: -> { [] } + prop :deferred_copy_retained_validations, + T::Array[DeferredCopyRetainedValidation], + factory: -> { [] } prop :predicate_call_sites, T::Array[CapabilityHelper::PredicateCallSite], factory: -> { [] } prop :async_body_facts, T::Array[AsyncBodyFact], factory: -> { [] } prop :capability_audit, CapabilityAudit::BindingAuditStore, factory: -> { {} } diff --git a/compiler/ruby/annotator/phases/conformance_registration.rb b/compiler/ruby/annotator/phases/conformance_registration.rb index 49fdd1e82..f72776583 100644 --- a/compiler/ruby/annotator/phases/conformance_registration.rb +++ b/compiler/ruby/annotator/phases/conformance_registration.rb @@ -171,9 +171,9 @@ def infer_conformance_owner_application!(declaration, owner_name, owner_params) binders = declaration.binders if binders.empty? names = declaration.protocol_type.generic_args.filter_map do |argument| - expression = argument.shape.expression - expression.name.to_s if expression.is_a?(NamedTypeExpression) && expression.arguments.empty? && - !ResolutionSession::BUILTIN_TYPE_PARAMETER_NAMES.include?(expression.name) + kind = argument.shape.expression.kind + kind.name.to_s if kind.is_a?(NamedTypeExpression) && kind.arguments.empty? && + !ResolutionSession::BUILTIN_TYPE_PARAMETER_NAMES.include?(kind.name) end.uniq if names.length != owner_params.length error!(declaration, :CONFORMANCE_BINDERS_CANNOT_INFER, @@ -189,11 +189,10 @@ def infer_conformance_owner_application!(declaration, owner_name, owner_params) expected: owner_params.length, got: binders.length) end - declaration.owner_type = Type.new(NamedTypeExpression.new( + declaration.owner_type = Type.new(TypeExpression.new(kind: NamedTypeExpression.new( name: owner_name.to_sym, - arguments: binders.map { |binder| NamedTypeExpression.new(name: binder.name.to_sym) }, - capabilities: declaration.owner_type.capabilities, - )) + arguments: binders.map { |binder| TypeExpression.of(NamedTypeExpression.new(name: binder.name.to_sym)) }, + ), capabilities: declaration.owner_type.capabilities)) end private :infer_conformance_owner_application! diff --git a/compiler/ruby/annotator/phases/declaration_index.rb b/compiler/ruby/annotator/phases/declaration_index.rb index 9d7c1a615..62cd5c115 100644 --- a/compiler/ruby/annotator/phases/declaration_index.rb +++ b/compiler/ruby/annotator/phases/declaration_index.rb @@ -92,10 +92,20 @@ def self.collect_error_type_registrations(program) registrations = T.let([], T::Array[ErrorTypeRegistration]) AST.each_locatable(program, descend_functions: true) do |node| case node - when AST::Raise, AST::OrElseExit + when AST::Raise kind = node.kind type_name = node.error_name - next unless kind && type_name + next if kind.nil? || type_name.nil? + + registrations << ErrorTypeRegistration.new( + kind: kind, + type_name: type_name, + token: node.token + ) + when AST::OrElseExit + kind = node.kind + type_name = node.error_name + next if kind.nil? || type_name.nil? registrations << ErrorTypeRegistration.new( kind: kind, diff --git a/compiler/ruby/annotator/phases/deferred_validation.rb b/compiler/ruby/annotator/phases/deferred_validation.rb index 2680269df..6bfd7f3ee 100644 --- a/compiler/ruby/annotator/phases/deferred_validation.rb +++ b/compiler/ruby/annotator/phases/deferred_validation.rb @@ -34,6 +34,17 @@ def record_deferred_recovery_validation!(node, left, callee_name, value_type) ) end + sig { params(facts: FunctionAnalysis::CallArgumentFacts).void } + def record_deferred_give_validation!(facts) + T.bind(self, Annotator::Phases::TypeAnalysisSession) + deferred_give_validations << DeferredGiveValidation.new( + arg_node: facts.arg_node, + callee_name: facts.site.node.name.to_s, + param_index: facts.index, + param_name: facts.param.name.to_s, + ) + end + end module DeferredCapabilityAudit @@ -45,10 +56,45 @@ def run_deferred_validations! flush_deferred_recovery_validations! flush_deferred_with_validations! + flush_deferred_give_validations! + flush_deferred_copy_retained_validations! finalize_capability_audit! end private :run_deferred_validations! + # Replay GIVE-into-borrow-param checks after the keep fixpoint: GIVE at + # a kept edge is a legal relinquishment assertion, everywhere else it + # remains an error. + sig { void } + def flush_deferred_give_validations! + T.bind(self, Annotator::Phases::CapabilityAuditSession) + + deferred_give_validations.each do |d| + param = function_node_for(d.callee_name)&.params&.fetch(d.param_index, nil) + next if param&.symbol&.kept_identity + error!(d.arg_node, :GIVE_TO_BORROW_PARAM, param: d.param_name) + end + deferred_give_validations.clear + end + private :flush_deferred_give_validations! + + # Retained-identity v5 (V5-3b): a COPY of a retained carrier at a + # non-UNIQUE edge is illegal (design "Parameter contracts"), UNLESS the + # callee param is a v4 kept-identity edge -- that temporary exception is + # removed when the v4 kept machinery is retired (Phase 6b). + sig { void } + def flush_deferred_copy_retained_validations! + T.bind(self, Annotator::Phases::CapabilityAuditSession) + + deferred_copy_retained_validations.each do |d| + param = function_node_for(d.callee_name)&.params&.fetch(d.param_index, nil) + next if param&.symbol&.kept_identity # v4 kept-edge exception (Phase 6b removes this) + error!(d.arg_node, :COPY_RETAINED_NEEDS_UNIQUE, name: d.name, carrier: d.carrier) + end + deferred_copy_retained_validations.clear + end + private :flush_deferred_copy_retained_validations! + sig { void } def flush_deferred_recovery_validations! T.bind(self, Annotator::Phases::CapabilityAuditSession) diff --git a/compiler/ruby/annotator/phases/expression_domains.rb b/compiler/ruby/annotator/phases/expression_domains.rb index 54c289b2d..e949f1deb 100644 --- a/compiler/ruby/annotator/phases/expression_domains.rb +++ b/compiler/ruby/annotator/phases/expression_domains.rb @@ -2,7 +2,7 @@ require "sorbet-runtime" require_relative "../../ast/ast" -require_relative "../../ast/schemas" +require_relative "../../ast/type" require_relative "../helpers/function_signature" require_relative "../helpers/with_match_check" @@ -303,11 +303,11 @@ def protocol_requirement_signature(protocol, requirement, receiver) T.bind(self, Annotator::Phases::TypeAnalysisSession) substitutions = protocol.associated_types.each_with_object({Self: receiver}) do |associated, table| - table[associated.name.to_sym] = Type.new(TypeProjectionExpression.new( + table[associated.name.to_sym] = Type.new(TypeExpression.of(TypeProjectionExpression.new( owner: receiver.resolved, member: associated.name.to_sym, protocol: protocol.name.to_sym, - )) + ))) end FunctionSignature.new( params: requirement.params.map do |param| @@ -492,7 +492,16 @@ def resolve_inherent_static_call!(node, owner) call.mark_explicit_mutable_argument!(index, token) end resolve_call(call, node.args) + # The metadata copy assigns type_object wholesale, so it clears the + # type resolve_call just computed. Keep the resolved type first. + resolved_return = call.full_type!(context: "inherent static call") AST.copy_pipeline_rewrite_metadata!(call, node, include_call_metadata: true) + stamp_type!(call, resolved_return) + # The StaticCall is what survives in the AST, so it has to carry the + # resolved return type too. Without this, a binding whose value is a + # static call reaches full_type! with nothing stamped and raises + # "unresolved type info for AST::StaticCall". + stamp_type!(node, resolved_return) node.inherent_call = call record_predicate_call_site!(call) record_named_call_site!(call) diff --git a/compiler/ruby/annotator/phases/import_resolution.rb b/compiler/ruby/annotator/phases/import_resolution.rb index 4fa26f56d..a70b27910 100644 --- a/compiler/ruby/annotator/phases/import_resolution.rb +++ b/compiler/ruby/annotator/phases/import_resolution.rb @@ -2,7 +2,7 @@ require "sorbet-runtime" require_relative "../../ast/ast" -require_relative "../../ast/schemas" +require_relative "../../ast/type" require_relative "../../compiler/entrypoint" module Annotator diff --git a/compiler/ruby/annotator/phases/signature_registration.rb b/compiler/ruby/annotator/phases/signature_registration.rb index 19e1949c3..1b2e378b4 100644 --- a/compiler/ruby/annotator/phases/signature_registration.rb +++ b/compiler/ruby/annotator/phases/signature_registration.rb @@ -2,7 +2,7 @@ require "sorbet-runtime" require_relative "../../ast/ast" -require_relative "../../ast/schemas" +require_relative "../../ast/type" require_relative "../helpers/function_signature" require_relative "declaration_index" require_relative "signature_registry" diff --git a/compiler/ruby/annotator/phases/signature_registry.rb b/compiler/ruby/annotator/phases/signature_registry.rb index 94bfbde3d..d78ae3d62 100644 --- a/compiler/ruby/annotator/phases/signature_registry.rb +++ b/compiler/ruby/annotator/phases/signature_registry.rb @@ -19,7 +19,7 @@ def self.function_signature(node, return_lifetime:) fn_type_params: node.type_params.map(&:to_sym), type_params: node.type_params.map(&:to_sym), generic_bounds: generic_bounds(node.generic_params), - reentrant: node.declared_plain_reentrant?, + reentrant: node.reentrance_kind == :reentrant || node.effects_decl == :reentrant, requires: node.requires ) end @@ -34,8 +34,9 @@ def self.generic_bounds(params) sig { params(node: AST::ExternFnDecl).returns(FunctionSignature) } def self.extern_function_signature(node) + params = node.params.nil? ? [] : node.params FunctionSignature.new( - params: node.params.map { |param| extern_param(param) }, + params: params.map { |param| extern_param(param) }, return_type: node.annotation_return_type, return_lifetime: extern_lifetime_paths(node), visibility: :pub, @@ -43,10 +44,10 @@ def self.extern_function_signature(node) module_alias: node.from_module, extern_effects: extern_effects(node), extern_source: node.extern_source, - fn_type_params: fn_type_params(node), - type_params: fn_type_params(node), + fn_type_params: node.fn_type_params.dup, + type_params: node.fn_type_params.dup, owner_type: node.owner_type, - owner_type_params: owner_type_params(node) + owner_type_params: node.owner_type_params.dup ) end @@ -56,9 +57,14 @@ def self.extern_lifetime_paths(node) return [] if lifetime.nil? return [:wildcard] if lifetime == :wildcard - T.cast(lifetime, T::Array[AST::Node]).filter_map do |source| - source.respond_to?(:name) ? T.unsafe(source).name.to_s : nil + paths = T.let([], T::Array[FunctionSignature::LifetimeSource]) + T.cast(lifetime, T::Array[AST::Node]).each do |source| + next unless source.is_a?(AST::Identifier) + + identifier = source + paths << identifier.name.to_s end + paths end private_class_method :extern_lifetime_paths @@ -94,17 +100,6 @@ def self.extern_effects(node) end private_class_method :extern_effects - sig { params(node: AST::ExternFnDecl).returns(T::Array[Symbol]) } - def self.fn_type_params(node) - node.fn_type_params - end - private_class_method :fn_type_params - - sig { params(node: AST::ExternFnDecl).returns(T::Array[Symbol]) } - def self.owner_type_params(node) - node.owner_type_params - end - private_class_method :owner_type_params end end end diff --git a/compiler/ruby/annotator/phases/type_analysis_session.rb b/compiler/ruby/annotator/phases/type_analysis_session.rb index 02bf94f3f..bb6636975 100644 --- a/compiler/ruby/annotator/phases/type_analysis_session.rb +++ b/compiler/ruby/annotator/phases/type_analysis_session.rb @@ -10,6 +10,9 @@ require_relative "../../ast/async_result_shape" require_relative "../../semantic/ownership_graph" require_relative "../../semantic/ownership_transport" +require_relative "../../semantic/keep_analysis" +require_relative "../../semantic/escape_analysis" +require_relative "../../semantic/ownership_edge_planner" require_relative "body_analysis" require_relative "builtin_environment" require_relative "declaration_index" @@ -135,6 +138,7 @@ class TraversalState < T::Struct prop :value_type_refinements, T::Hash[Integer, Type], factory: -> { {} } prop :branch_terminated, T::Boolean, default: false prop :next_synthetic_body_ordinal, Integer, default: -1 + prop :deferred_copy_wrapper_bindings, T::Array[T.any(AST::VarDecl, AST::BindExpr)], factory: -> { [] } end class Config < T::Struct @@ -239,6 +243,16 @@ def deferred_recovery_validations @audit_inputs.deferred_recovery_validations end + sig { returns(T::Array[Annotator::Phases::DeferredGiveValidation]) } + def deferred_give_validations + @audit_inputs.deferred_give_validations + end + + sig { returns(T::Array[Annotator::Phases::DeferredCopyRetainedValidation]) } + def deferred_copy_retained_validations + @audit_inputs.deferred_copy_retained_validations + end + sig { returns(T.nilable(FunctionContext)) } def current_fn_ctx @traversal_state.function_contexts.last @@ -369,19 +383,31 @@ def inside_snapshot_transaction_body? end def with_conditional_context(&blk) fn_ctx = current_fn_ctx + enter_conditional_context!(fn_ctx) + blk.call + ensure + leave_conditional_context!(fn_ctx) + end + + sig { params(fn_ctx: T.nilable(FunctionContext)).void } + def enter_conditional_context!(fn_ctx) if fn_ctx fn_ctx.enter_conditional! else @traversal_state.conditional_depth += 1 end - blk.call - ensure + end + private :enter_conditional_context! + + sig { params(fn_ctx: T.nilable(FunctionContext)).void } + def leave_conditional_context!(fn_ctx) if fn_ctx fn_ctx.exit_conditional! else @traversal_state.conditional_depth -= 1 end end + private :leave_conditional_context! sig do type_parameters(:Result) @@ -390,19 +416,32 @@ def with_conditional_context(&blk) end def with_loop_context(&blk) fn_ctx = current_fn_ctx + enter_loop_context!(fn_ctx) + blk.call + ensure + leave_loop_context!(fn_ctx) + end + + sig { params(fn_ctx: T.nilable(FunctionContext)).void } + def enter_loop_context!(fn_ctx) if fn_ctx fn_ctx.enter_loop! else @traversal_state.loop_depth += 1 end - blk.call - ensure + end + private :enter_loop_context! + + sig { params(fn_ctx: T.nilable(FunctionContext)).void } + def leave_loop_context!(fn_ctx) if fn_ctx fn_ctx.exit_loop! else @traversal_state.loop_depth -= 1 end end + private :leave_loop_context! + private :with_loop_context sig do @@ -651,10 +690,20 @@ def execute_type_analysis!(resolution) ownership = analyze_resolution!(resolution) inventory = Annotator::Phases::AnnotationTypeInventory.scan(resolution.program) inventory.verify_resolved! + # Keep-analysis and its placement run before the lifecycle inventory and + # its derived linear-resource closure are built and frozen: born-as-Rc + # bindings must be inventoried as Rc, not as their pre-keep payload. + apply_keep_analysis!(resolution) + schema_lookup = ->(name) { lookup_type_schema(name) } + lifecycle_inventory = Semantic::LifecycleRegistry.type_inventory(resolution.program, schema_lookup) + linear_resource_facts = Semantic::LinearResourceFacts.build_types(lifecycle_inventory.values, schema_lookup) + validate_copy_linear_resource_facts!(resolution.program, linear_resource_facts) lifecycle_registry = Semantic::LifecycleRegistry.build( resolution.program, - ->(name) { lookup_type_schema(name) }, + schema_lookup, binding_nodes: resolution.function_registry.body_summaries.values.flat_map(&:binding_nodes), + linear_resource_facts: linear_resource_facts, + inventory: lifecycle_inventory, ) typed_program = Annotator::Phases::TypedProgramFacts.new( resolution: resolution, @@ -663,6 +712,7 @@ def execute_type_analysis!(resolution) unresolved_node_count: inventory.unresolved_node_count, ownership_graph: ownership, lifecycle_registry: lifecycle_registry, + linear_resource_facts: linear_resource_facts, ) Annotator::Phases::TypeAnalysisHandoff.new( typed_program: typed_program, @@ -670,6 +720,108 @@ def execute_type_analysis!(resolution) ) end + sig { params(program: AST::Program, facts: Semantic::LinearResourceFacts).void } + def validate_copy_linear_resource_facts!(program, facts) + AST.each_locatable(program, descend_functions: true) do |node| + next unless node.is_a?(AST::CopyNode) + + type_info = node.value.full_type!(context: "post-annotation COPY resource validation") + error!(node, :COPY_NON_COPYABLE, type: type_info.to_s) if facts.contains?(type_info) + end + @traversal_state.deferred_copy_wrapper_bindings.each do |binding| + reject_implicit_wrapper_binding!(binding, process_deferred_copy: true) + end + end + private :validate_copy_linear_resource_facts! + + sig { params(resolution: Annotator::Phases::ResolutionFacts).void } + def apply_keep_analysis!(resolution) + fn_nodes = resolution.function_registry.nodes + body_summaries = resolution.function_registry.body_summaries + KeepAnalysis.propagate_kept_identity!(fn_nodes, body_summaries) + EscapeAnalysis.apply_kept_identity_placement!( + fn_nodes, + body_summaries, + on_mutable_violation: lambda { |entry, arg, callee_name| + sink = entry.kept_identity&.sink || + fn_nodes[callee_name]&.params&.find { |p| p.symbol&.kept_identity }&.symbol&.kept_identity&.sink || + "an @multiowned destination" + anchor = kept_identity_declaration_anchor(body_summaries, entry) || arg + error!(anchor, :KEPT_IDENTITY_NEEDS_MODEL, + name: arg.name, keeper: callee_name, sink: sink) + }, + on_family_violation: lambda { |arg, source_family, dest_family, callee_name| + sink = fn_nodes[callee_name]&.params&.find { |p| p.symbol&.kept_identity }&.symbol&.kept_identity&.sink || + "an @#{dest_family} destination" + name = arg.is_a?(AST::Identifier) ? arg.name : "the argument" + error!(arg, :KEPT_IDENTITY_FAMILY_MISMATCH, + name: name, keeper: callee_name, sink: sink, + source_family: source_family, dest_family: dest_family) + } + ) + reject_kept_function_values!(resolution, fn_nodes) + end + private :apply_keep_analysis! + + # A retaining function's compiled signature takes an owned handle, but an + # ordinary FN type carries no retained-parameter contract. Until retention + # is representable in function types, using a kept function as a VALUE + # must fail closed here - never as invalid Zig downstream. A call spells + # the name on the call node itself, so any Identifier carrying a kept + # function's name is a value use. + sig { params(resolution: Annotator::Phases::ResolutionFacts, fn_nodes: T::Hash[String, AST::FunctionDef]).void } + def reject_kept_function_values!(resolution, fn_nodes) + kept_fns = fn_nodes.each_with_object(T.let({}, T::Hash[String, AST::Param])) do |(name, fn), map| + kept_param = fn.params.find { |p| p.symbol&.kept_identity } + map[name] = kept_param if kept_param + end + return if kept_fns.empty? + + fn_nodes.each_value { |fn| fn.body&.each { |stmt| scan_kept_fn_value_use!(stmt, kept_fns) } } + resolution.program.statements.each do |stmt| + next if stmt.is_a?(AST::FunctionDef) + scan_kept_fn_value_use!(stmt, kept_fns) + end + end + private :reject_kept_function_values! + + sig { params(node: AST::Node, kept_fns: T::Hash[String, AST::Param]).void } + def scan_kept_fn_value_use!(node, kept_fns) + if node.is_a?(AST::Identifier) && kept_fns.key?(node.name) + sym = node.symbol + shadowed = sym.is_a?(SymbolEntry) && FunctionSignature.unwrap(sym.type).nil? && !sym.type.nil? && + sym.type.resolved != :Any + unless shadowed + param = T.must(kept_fns[node.name]) + contract = T.must(param.symbol&.kept_identity) + error!(node, :KEPT_FN_VALUE_ABI, + name: node.name, + param: param.name.to_s, + sink: contract.sink, + fn_type: "FN(...)") + end + end + # each_child_node yields body arrays as direct members; adding a + # child_bodies leg would revisit every nested body exponentially. + AST.each_child_node(node) { |child| scan_kept_fn_value_use!(child, kept_fns) } + end + private :scan_kept_fn_value_use! + + + # The keep diagnostic is declaration-sited: point at the binding that must + # declare a model, not at the call that exposed the conflict. + sig { params(body_summaries: T::Hash[String, T.untyped], entry: SymbolEntry).returns(T.nilable(AST::Node)) } + def kept_identity_declaration_anchor(body_summaries, entry) + body_summaries.each_value do |summary| + decl = summary.binding_nodes.find do |node| + node.respond_to?(:symbol) && T.unsafe(node).symbol.equal?(entry) + end + return decl if decl + end + nil + end + private :kept_identity_declaration_anchor + sig { returns(Symbol) } def language_mode @resolution.program.language_mode @@ -854,6 +1006,7 @@ def dispatch_visit(node) when AST::BreakNode then visit_BreakNode(node) when AST::ContinueNode then visit_ContinueNode(node) when AST::PassStmt then visit_PassStmt(node) + when AST::DeferStmt then visit_DeferStmt(node) when AST::SyncPolicyDecl then visit_SyncPolicyDecl(node) when AST::Assert then visit_Assert(node) when AST::DieNode then visit_DieNode(node) @@ -888,11 +1041,11 @@ def dispatch_visit(node) when AST::StaticCall then visit_StaticCall(node) when AST::MoveNode then visit_MoveNode(node) when AST::CopyNode then visit_CopyNode(node) + when AST::KeepNode then visit_KeepNode(node) when AST::Copy then visit_Copy(node) when AST::LinkNode then visit_LinkNode(node) when AST::ResolveNode then visit_ResolveNode(node) when AST::FreezeNode then visit_FreezeNode(node) - when AST::CloneNode then visit_CloneNode(node) when AST::ShareNode then visit_ShareNode(node) when AST::GetIndex then visit_GetIndex(node) when AST::GetField then visit_GetField(node) @@ -1081,7 +1234,12 @@ def og_move(from, to, at_token: nil, action: :move) = ownership_graph.transfer(f sig { params(name: String, at_token: T.nilable(Lexer::Token), action: Symbol, consumer_param_type: OwnershipGraph::MoveConsumerParamType).returns(T.nilable(T::Set[String])) } def og_set_moved(name, at_token: nil, action: :move, consumer_param_type: nil) = ownership_graph.mark_moved(name, at_token: at_token, action: action, consumer_param_type: consumer_param_type) sig { params(name: String).returns(T.nilable(Symbol)) } - def og_set_live(name) = (ownership_graph[name]&.state = :live) + def og_set_live(name) + graph_node = ownership_graph[name] + return nil unless graph_node + + graph_node.state = :live + end sig { params(name: String).returns(T::Array[String]) } def og_drop(name) = ownership_graph.drop(name) sig { returns(Integer) } @@ -1105,6 +1263,7 @@ def og_pop_scope(archive: false) private :current_stream_yield_frame private :deferred_with_validations private :deferred_recovery_validations + private :deferred_give_validations private :function_node_for private :function_node_map private :handle_prefixed_int_overflow! diff --git a/compiler/ruby/annotator/phases/type_registration.rb b/compiler/ruby/annotator/phases/type_registration.rb index fcc89e849..452ee008e 100644 --- a/compiler/ruby/annotator/phases/type_registration.rb +++ b/compiler/ruby/annotator/phases/type_registration.rb @@ -3,7 +3,7 @@ require_relative "../protocol_projection_resolver" require_relative "../../ast/ast" -require_relative "../../ast/schemas" +require_relative "../../ast/type" require_relative "declaration_index" module Annotator @@ -161,6 +161,50 @@ def inline_recursive_target(type, names) end private :inline_recursive_target + sig do + params( + node: Symbol, + adjacency: T::Hash[Symbol, T::Array[Symbol]], + index: Integer, + indices: T::Hash[Symbol, Integer], + low: T::Hash[Symbol, Integer], + stack: T::Array[Symbol], + on_stack: T::Set[Symbol], + result: T::Array[T::Set[Symbol]] + ).returns(Integer) + end + def visit_recursive_component(node, adjacency, index, indices, low, stack, on_stack, result) + indices[node] = index + low[node] = index + next_index = index + 1 + stack << node + on_stack.add(node) + + T.must(adjacency[node]).each do |target| + unless indices.key?(target) + next_index = visit_recursive_component( + target, adjacency, next_index, indices, low, stack, on_stack, result + ) + low[node] = [T.must(low[node]), T.must(low[target])].min + else + low[node] = [T.must(low[node]), T.must(indices[target])].min if on_stack.include?(target) + end + end + return next_index unless low[node] == indices[node] + + component = T.let(Set.new, T::Set[Symbol]) + while true + member = T.must(stack.pop) + on_stack.delete(member) + component.add(member) + break if member == node + end + self_loop = T.must(adjacency[node]).include?(node) + result << component if component.length > 1 || self_loop + next_index + end + private :visit_recursive_component + sig { params(names: T::Set[Symbol], edges: T::Array[RecursiveFieldEdge]).returns(T::Array[T::Set[Symbol]]) } def recursive_components(names, edges) adjacency = T.let(Hash.new { |h, k| h[k] = T.let([], T::Array[Symbol]) }, T::Hash[Symbol, T::Array[Symbol]]) @@ -172,34 +216,13 @@ def recursive_components(names, edges) on_stack = T.let(Set.new, T::Set[Symbol]) result = T.let([], T::Array[T::Set[Symbol]]) - visit = T.let(nil, T.nilable(T.proc.params(node: Symbol).void)) - visit = Kernel.lambda do |node| - indices[node] = index - low[node] = index - index += 1 - stack << node - on_stack.add(node) - T.must(adjacency[node]).each do |target| - unless indices.key?(target) - T.must(visit).call(target) - low[node] = [T.must(low[node]), T.must(low[target])].min - else - low[node] = [T.must(low[node]), T.must(indices[target])].min if on_stack.include?(target) - end - end - next unless low[node] == indices[node] - - component = T.let(Set.new, T::Set[Symbol]) - Kernel.loop do - member = T.must(stack.pop) - on_stack.delete(member) - component.add(member) - break if member == node - end - self_loop = T.must(adjacency[node]).include?(node) - result << component if component.length > 1 || self_loop + names.each do |name| + next if indices.key?(name) + + index = visit_recursive_component( + name, adjacency, index, indices, low, stack, on_stack, result + ) end - names.each { |name| visit.call(name) unless indices.key?(name) } result end private :recursive_components @@ -265,6 +288,18 @@ def register_struct_declaration(node) node.field_decls.each_value do |field| error!(node, :COLLECTION_HINT_VALUE_ONLY) if field.type.preallocation_hint? end + # Fail closed: identity capabilities on generic type parameters are + # not implemented (no combination compiles today - the wrap and the + # substituted binding double-apply or under-apply). Reject at the + # declaration until generic keep-analysis lands. + tp_names = type_params(node.type_params) + node.field_decls.each do |field_name, field| + ft = field.type + next unless ft.is_a?(Type) && tp_names.include?(ft.resolved) + next unless ft.multiowned? || ft.shared? + error!(node, :GENERIC_IDENTITY_FIELD_UNSUPPORTED, + struct: node.name.to_s, field: field_name.to_s, param: ft.resolved.to_s) + end stamp_field_defaults!(node.field_decls) declare_type_schema!(node, node.name.to_sym, Schemas::StructSchema.new( diff --git a/compiler/ruby/annotator/phases/whole_program_semantics.rb b/compiler/ruby/annotator/phases/whole_program_semantics.rb index 2a521c8f2..a3f861b0f 100644 --- a/compiler/ruby/annotator/phases/whole_program_semantics.rb +++ b/compiler/ruby/annotator/phases/whole_program_semantics.rb @@ -89,7 +89,9 @@ def restamp_requires_on_signatures! root_scope = whole_program_root_scope whole_program_fn_nodes.each do |name, fn| signature = FunctionSignature.unwrap(root_scope.resolve_entry(name)&.type) - signature.sync_from_function_def!(fn) if signature + next unless signature + signature.sync_from_function_def!(fn) + signature.adopt_param_symbols!(fn) end end private :restamp_requires_on_signatures! diff --git a/compiler/ruby/annotator/protocol_projection_resolver.rb b/compiler/ruby/annotator/protocol_projection_resolver.rb index 26dc77eb2..b71ceef55 100644 --- a/compiler/ruby/annotator/protocol_projection_resolver.rb +++ b/compiler/ruby/annotator/protocol_projection_resolver.rb @@ -69,18 +69,25 @@ def initialize(protocols) end def resolve(expression, parameters) issues = T.let([], T::Array[ProtocolProjectionIssue]) - parameter_map = parameters.to_h { |parameter| [parameter.name.to_sym, parameter] } + parameter_map = T.let({}, T::Hash[String, AST::GenericParamDecl]) + parameters.each { |parameter| parameter_map[parameter.name] = parameter } resolved = TypeExpressionTree.transform(expression) do |candidate| - next candidate unless candidate.is_a?(TypeProjectionExpression) - next candidate if candidate.protocol + kind = candidate.kind + next candidate unless kind.is_a?(TypeProjectionExpression) + projection = kind + next candidate if projection.protocol - protocol = projection_protocol(candidate, parameter_map, issues) + protocol = projection_protocol(projection, parameter_map, issues) next candidate unless protocol + protocol_value = protocol - TypeProjectionExpression.new( - owner: candidate.owner, - member: candidate.member, - protocol: protocol.to_sym, + projection_kind = TypeProjectionExpression.new( + owner: projection.owner, + member: projection.member, + protocol: protocol_value.to_sym, + ) + TypeExpression.new( + kind: projection_kind, capabilities: candidate.capabilities, ) end @@ -92,12 +99,12 @@ def resolve(expression, parameters) sig do params( projection: TypeProjectionExpression, - parameters: T::Hash[Symbol, AST::GenericParamDecl], + parameters: T::Hash[String, AST::GenericParamDecl], issues: T::Array[ProtocolProjectionIssue], ).returns(T.nilable(String)) end def projection_protocol(projection, parameters, issues) - parameter = parameters[projection.owner] + parameter = parameters[projection.owner.to_s] unless parameter issues << issue(:GENERIC_PROJECTION_UNKNOWN_OWNER, owner: projection.owner, member: projection.member) @@ -128,14 +135,21 @@ def projection_protocol(projection, parameters, issues) owner: projection.owner, member: projection.member, protocols: matching.join(", ")) return nil end - matching.first + result = matching.first + return nil unless result + + result.dup end sig { params(code: Symbol, values: T.untyped).returns(ProtocolProjectionIssue) } def issue(code, **values) + arguments = T.let({}, T::Hash[Symbol, String]) + values.each do |key, value| + arguments[key] = value.to_s + end ProtocolProjectionIssue.new( code: code, - arguments: values.transform_values(&:to_s), + arguments: arguments, ) end diff --git a/compiler/ruby/ast/ast.rb b/compiler/ruby/ast/ast.rb index 4a9274bd1..aab298de1 100644 --- a/compiler/ruby/ast/ast.rb +++ b/compiler/ruby/ast/ast.rb @@ -5,7 +5,6 @@ require_relative "param" require_relative "struct_field" require_relative "type" -require_relative "schemas" require_relative "lexer" require_relative "function_signature_forward" @@ -26,7 +25,20 @@ module TensePlanValue end RawBody = T.type_alias { T::Array[AST::Node] } - HashLitPairs = T.type_alias { T::Hash[AST::Node, AST::Node] } + HashLitPairs = T.type_alias { T::Hash[AST::Locatable, AST::Locatable] } + AssignmentName = T.type_alias { T.any(String, AST::Identifier, AST::GetField, AST::GetIndex) } + PipelineRewriteNode = T.type_alias do + T.any( + AST::Identifier, AST::FuncCall, AST::MethodCall, AST::BinaryOp, + AST::GetField, AST::GetIndex, AST::BindExpr, AST::Assignment, + AST::UnaryOp, AST::CopyNode, AST::MoveNode, AST::KeepNode, + AST::ShareNode, AST::WithBlock, AST::StructLit, AST::HashLit, + AST::ListLit, AST::BlockExpr, AST::Assert, AST::IfStatement, + # A static call carries the same call metadata: annotation copies it onto + # the synthetic FuncCall it resolves through. + AST::StaticCall, + ) + end BgNode = T.type_alias { T.any(AST::BgBlock, AST::BgStreamBlock) } BindingNode = T.type_alias { T.any(AST::VarDecl, AST::BindExpr, AST::DestructureTarget) } ScalarLiteralCandidate = T.type_alias do @@ -97,48 +109,49 @@ def self.stamp_synthetic_type!(node, value, context:) sig do params( - src: AST::Locatable, - dst: AST::Locatable, + dst: AST::PipelineRewriteNode, + src: AST::PipelineRewriteNode, include_call_metadata: T::Boolean, - ).returns(AST::Locatable) + ).returns(AST::PipelineRewriteNode) end - def self.copy_pipeline_rewrite_metadata!(src, dst, include_call_metadata: false) - src.full_type!(context: "pipeline rewrite type copy") - copy_pipeline_base_metadata!(src, dst) - copy_pipeline_call_metadata!(src, dst) if include_call_metadata - dst - end - - sig { params(src: AST::Locatable, dst: AST::Locatable).void } - def self.copy_pipeline_base_metadata!(src, dst) - dst.full_type = src.full_type - dst.coerced_type = src.coerced_type_object if src.coerced_type_object - dst.storage = src.storage_override if src.storage_override + # ruby-to-clear: data-api + # ruby-to-clear: pub + def self.copy_pipeline_rewrite_metadata!(dst, src, include_call_metadata: false) + dst.type_object = src.type_object + dst.coerced_type_object = src.coerced_type_object + dst.storage_override = src.storage_override dst.var_used = src.var_used unless src.var_used.nil? + # INV-13: `arg.was_moved` is the single source of truth for "callee + # takes". A rewritten pipeline placeholder must keep the annotator's + # stamp or call lowering re-derives (and wrongly copies) the argument. + dst.was_moved = src.was_moved unless src.was_moved.nil? dst.slot_size = src.slot_size unless src.slot_size.nil? dst.container_borrow = src.container_borrow unless src.container_borrow.nil? - dst.tense_plan = T.must(src.tense_plan) if src.tense_plan - if src.respond_to?(:retain_error_channel) && dst.respond_to?(:retain_error_channel=) - retained = T.unsafe(src).retain_error_channel - T.unsafe(dst).retain_error_channel = retained unless retained.nil? + dst.tense_plan = src.tense_plan + case src + when AST::BinaryOp + dst.retain_error_channel = src.retain_error_channel if dst.is_a?(AST::BinaryOp) + when AST::FuncCall + dst.retain_error_channel = src.retain_error_channel if dst.is_a?(AST::FuncCall) + when AST::MethodCall + dst.retain_error_channel = src.retain_error_channel if dst.is_a?(AST::MethodCall) + end + + if include_call_metadata + dst.zig_pattern = src.zig_pattern if src.zig_pattern + dst.matched_stdlib_def = src.matched_stdlib_def if src.matched_stdlib_def + dst.matched_signature = src.matched_signature if src.matched_signature + dst.stdlib_allocates = src.stdlib_allocates unless src.stdlib_allocates.nil? + dst.mutates_receiver = src.mutates_receiver unless src.mutates_receiver.nil? + dst.implicit_layout_cost = src.implicit_layout_cost unless src.implicit_layout_cost.nil? + dst.layout_transport = src.layout_transport unless src.layout_transport.nil? + dst.can_fail = src.can_fail unless src.can_fail.nil? + dst.error_kind = src.error_kind if src.error_kind + dst.error_type = src.error_type if src.error_type end - end - private_class_method :copy_pipeline_base_metadata! - sig { params(src: AST::Locatable, dst: AST::Locatable).void } - def self.copy_pipeline_call_metadata!(src, dst) - dst.zig_pattern = src.zig_pattern if src.zig_pattern - dst.matched_stdlib_def = src.matched_stdlib_def if src.matched_stdlib_def - dst.matched_signature = src.matched_signature if src.matched_signature - dst.stdlib_allocates = src.stdlib_allocates unless src.stdlib_allocates.nil? - dst.mutates_receiver = src.mutates_receiver unless src.mutates_receiver.nil? - dst.implicit_layout_cost = src.implicit_layout_cost unless src.implicit_layout_cost.nil? - dst.layout_transport = src.layout_transport unless src.layout_transport.nil? - dst.can_fail = src.can_fail unless src.can_fail.nil? - dst.error_kind = src.error_kind if src.error_kind - dst.error_type = src.error_type if src.error_type + dst end - private_class_method :copy_pipeline_call_metadata! # A node's value-type is, for these kinds, a pure function of its # structure — so it is DERIVED, never stamped. The full_type getter @@ -164,7 +177,7 @@ def full_type end Capture = Struct.new(:name, :type, :default, :mutable, :takes, - :comptime, :name_token, :storage, + :comptime, :name_token, :storage, :carrier_contract, keyword_init: true) do extend T::Sig @@ -176,6 +189,9 @@ def initialize(**kw) self[:mutable] = !!self[:mutable] self[:takes] = !!self[:takes] self[:comptime] = !!self[:comptime] + # Retained-identity v5 parameter carrier contract: :polymorphic + # (default), :unique, or :shared. + self[:carrier_contract] ||= :polymorphic t = self[:type] self[:type] = Type.new(t || :Any) end @@ -231,9 +247,9 @@ def storage=(val); self[:storage] = val; end :indirect_payload_as, keyword_init: true) do extend T::Sig - # ruby-to-clear: field-type value=Node - # ruby-to-clear: field-type body=Node[] - # ruby-to-clear: field-type extra_values=Node[] + # ruby-to-clear: field-type value=Locatable + # ruby-to-clear: field-type body=[]Locatable + # ruby-to-clear: field-type extra_values=[]Locatable sig { params(kw: StructKwargs).void } def initialize(**kw) @@ -254,6 +270,8 @@ def value end sig { returns(T::Array[AST::Locatable]) } + # ruby-to-clear: data-api + # ruby-to-clear: pub def body self[:body] end @@ -294,7 +312,7 @@ def indirect_payload_as=(val) keyword_init: true) do extend T::Sig attr_accessor :mir_binding_entry - # ruby-to-clear: field-type expr=Node + # ruby-to-clear: field-type expr=Locatable sig { params(kw: StructKwargs).void } def initialize(**kw) @@ -367,6 +385,11 @@ def resolved_type self[:resolved_type] end + sig { returns(T.nilable(Symbol)) } + def capability + T.must(T.cast(self[:capability], T.nilable(Symbol))) + end + sig { params(val: Type).void } def resolved_type=(val) self[:resolved_type] = val @@ -486,11 +509,13 @@ def self.each_locatable(root, descend_functions: false, &visitor) # resolution, capability source naming, and placeholder-root detection # each hand-rolled the same `case node; GetField/GetIndex -> .target` # recursion (decomplex Missing-Abstraction, scatter=7). + # ruby-to-clear: data-api sig { params(node: AST::Node).returns(T.nilable(AST::Identifier)) } + # ruby-to-clear: effects reentrant-tail-call def self.root_identifier(node) case node - when AST::MutableBorrow then root_identifier(node.target) - when AST::GetField, AST::GetIndex then root_identifier(node.target) + when AST::MutableBorrow then AST.root_identifier(node.target) + when AST::GetField, AST::GetIndex then AST.root_identifier(node.target) when AST::Identifier then node end end @@ -515,6 +540,9 @@ def self.call?(node) node.is_a?(AST::FuncCall) || node.is_a?(AST::MethodCall) end + # This data-only unit emits only methods which form its cross-package API. + # ruby-to-clear: data-api + # ruby-to-clear: pub sig { params(node: T.nilable(AST::Node)).returns(T::Boolean) } def self.container_borrow?(node) return false unless node @@ -530,6 +558,8 @@ def self.container_borrow?(node) # manufacture an owned value. Keeping the rule here prevents annotation, # cleanup classification, and lowering from independently guessing which # wrappers preserve borrow provenance. + # ruby-to-clear: data-api + # ruby-to-clear: pub sig { params(node: AST::Node).returns(T.nilable(AST::Node)) } def self.borrow_transparent_operand(node) return node.target if node.is_a?(AST::OptionalUnwrap) || node.is_a?(AST::TenseNavigation) @@ -539,10 +569,12 @@ def self.borrow_transparent_operand(node) nil end + # ruby-to-clear: data-api + # ruby-to-clear: pub sig { params(node: T.nilable(AST::Node)).returns(T::Boolean) } def self.borrowed_ownership_view?(node) return false unless node - return false if node.is_a?(AST::CopyNode) || node.is_a?(AST::CloneNode) + return false if node.is_a?(AST::CopyNode) || node.is_a?(AST::KeepNode) return true if node.is_a?(AST::Identifier) && node.symbol&.borrowed_alias return true if container_borrow?(node) return true if node.is_a?(AST::GetIndex) @@ -572,7 +604,7 @@ def self.capture_expr_owns_result?(node) end call?(node) || node.is_a?(AST::NextExpr) || node.is_a?(AST::ResolveNode) || - node.is_a?(AST::CopyNode) || node.is_a?(AST::CloneNode) || + node.is_a?(AST::CopyNode) || node.is_a?(AST::KeepNode) || node.is_a?(AST::MoveNode) || node.is_a?(AST::ShareNode) end @@ -660,7 +692,7 @@ def self.ownership_transfer_stmt?(node) sig { params(node: T.nilable(AST::Node)).returns(T::Boolean) } def self.ownership_wrapper?(node) node.is_a?(AST::MoveNode) || node.is_a?(AST::CopyNode) || - node.is_a?(AST::CloneNode) || node.is_a?(AST::ShareNode) || + node.is_a?(AST::KeepNode) || node.is_a?(AST::ShareNode) || node.is_a?(AST::FreezeNode) || node.is_a?(AST::CapabilityWrap) end @@ -706,25 +738,35 @@ def self.inline_union_constructor_target?(node) !!(target.name[0] =~ /[A-Z]/) end + # ruby-to-clear: data-api sig { params(node: T.nilable(AST::Node)).returns(T::Boolean) } def self.soa_placeholder_field?(node) return false unless node.is_a?(AST::GetField) - target = node.target + target = T.cast(node.target, AST::Node) !!(target.is_a?(AST::Identifier) && target.name == "_") end + # ruby-to-clear: data-api sig { params(node: T.nilable(AST::Node)).returns(T::Boolean) } def self.soa_placeholder_assignment?(node) - return false unless node.is_a?(AST::BindExpr) || node.is_a?(AST::Assignment) + if node.is_a?(AST::BindExpr) + bind = node + return soa_placeholder_field?(bind.name) + end + if node.is_a?(AST::Assignment) + assignment = node + return soa_placeholder_field?(assignment.name) + end - soa_placeholder_field?(node.name) + false end # Explicit ownership transfer marker stamped by annotation. This is a # predicate over the AST contract, not an ad hoc respond_to? check. + # ruby-to-clear: data-api sig { params(node: T.nilable(AST::Node)).returns(T::Boolean) } def self.moved?(node) - !!(node && node.respond_to?(:was_moved) && node.was_moved == true) + !!(node && node.was_moved == true) end # Statement-position body traversal is an AST fact. MIR passes may attach @@ -764,6 +806,8 @@ def self.body_slots(node) node.branches.each do |branch| slots << BodySlot.new(branch.body, ->(body) { branch.body = body }) end + when DeferStmt + slots << BodySlot.new(node.body, ->(body) { node.body = body }) if node.body end slots end @@ -798,7 +842,7 @@ def self.wrapped_children(expr) (expr.fields&.values || []).compact when ListLit (expr.items || []).compact - when Cast, MoveNode, CopyNode, CloneNode, ShareNode, LinkNode, ResolveNode, + when Cast, MoveNode, CopyNode, KeepNode, ShareNode, LinkNode, ResolveNode, MutableBorrow, FreezeNode, CapabilityWrap child = expr.is_a?(MutableBorrow) ? expr.target : expr.value @@ -815,7 +859,7 @@ def self.expression_children(node, skip_copy: false) return [] unless node case node - when CopyNode, CloneNode, FreezeNode + when CopyNode, KeepNode, FreezeNode skip_copy ? [] : [node.value].compact when TenseNavigation [node.target].compact @@ -1022,15 +1066,6 @@ def child_bodies end end - module HasExpression - extend T::Sig - - sig { returns(AST::Node) } - def expression - T.unsafe(self)[:expression] - end - end - # ruby-to-clear: skip # ruby-to-clear: no-expand module Locatable @@ -1045,7 +1080,7 @@ def tense_plan @tense_plan end - sig { params(value: AST::TensePlanValue).returns(AST::TensePlanValue) } + sig { params(value: T.nilable(AST::TensePlanValue)).returns(T.nilable(AST::TensePlanValue)) } def tense_plan=(value) @tense_plan = value value @@ -1086,11 +1121,21 @@ def coerced_type_object @coerced_type_object = T.let(@coerced_type_object, T.nilable(Type)) end + sig { params(value: T.nilable(Type)).void } + def coerced_type_object=(value) + @coerced_type_object = T.let(value, T.nilable(Type)) + end + sig { returns(T.nilable(Type)) } def type_object @type_object = T.let(@type_object, T.nilable(Type)) end + sig { params(value: T.nilable(Type)).void } + def type_object=(value) + @type_object = T.let(value, T.nilable(Type)) + end + sig { returns(T.nilable(T.any(String, Symbol))) } def zig_pattern @zig_pattern = T.let(@zig_pattern, T.nilable(T.any(String, Symbol))) @@ -1142,6 +1187,26 @@ def mutates_receiver=(val) @mutates_receiver = T.let(val, T.nilable(T::Boolean)) end + sig { returns(T.nilable(T::Hash[Integer, CallEdgeOwnershipPlan])) } + def kept_edge_plans + @kept_edge_plans = T.let(@kept_edge_plans, T.nilable(T::Hash[Integer, CallEdgeOwnershipPlan])) + end + + sig { params(val: T.nilable(T::Hash[Integer, CallEdgeOwnershipPlan])).returns(T.nilable(T::Hash[Integer, CallEdgeOwnershipPlan])) } + def kept_edge_plans=(val) + @kept_edge_plans = T.let(val, T.nilable(T::Hash[Integer, CallEdgeOwnershipPlan])) + end + + sig { returns(T.nilable(CallEdgeOwnershipPlan)) } + def kept_edge_plan + @kept_edge_plan = T.let(@kept_edge_plan, T.nilable(CallEdgeOwnershipPlan)) + end + + sig { params(val: T.nilable(CallEdgeOwnershipPlan)).returns(T.nilable(CallEdgeOwnershipPlan)) } + def kept_edge_plan=(val) + @kept_edge_plan = T.let(val, T.nilable(CallEdgeOwnershipPlan)) + end + sig { returns(T.nilable(T::Boolean)) } def was_moved @was_moved = T.let(@was_moved, T.nilable(T::Boolean)) @@ -1311,11 +1376,24 @@ def full_type=(val) end # :Untyped sentinel (not nil) so no caller branches on nil; PreMirTypeCheck rejects it at the AST->MIR boundary. + # These three methods are the common typed-node interface consumed by + # semantic and MIR packages. The data-only AST package must export their + # concrete implementations for closed-union dispatch. + # ruby-to-clear: data-api + # ruby-to-clear: pub sig { returns(Type) } def full_type @type_object ||= Type.new(:Untyped) end + # Read annotation storage without manufacturing the Untyped sentinel. + sig { returns(T.nilable(Type)) } + def type_object + @type_object + end + + # ruby-to-clear: data-api + # ruby-to-clear: pub sig { params(context: String).returns(Type) } def full_type!(context: "post-annotation AST") ft = full_type @@ -1325,6 +1403,8 @@ def full_type!(context: "post-annotation AST") # True when the node carries a real (stamped) type, i.e. full_type # is not the :Untyped sentinel. + # ruby-to-clear: data-api + # ruby-to-clear: pub sig { returns(T::Boolean) } def typed? !full_type.untyped? @@ -1494,6 +1574,11 @@ def storage_override @storage_override = T.let(@storage_override, T.nilable(Symbol)) end + sig { params(value: T.nilable(Symbol)).void } + def storage_override=(value) + @storage_override = T.let(value, T.nilable(Symbol)) + end + # Canonical "is this expression's value heap-allocated?" — SIMP-13f. # Reads sym.storage when a symbol is attached (binding-level), else the # node's @storage_override (expression-level, stamped by annotator). @@ -1683,8 +1768,11 @@ class ReturnFact < T::Struct sig { params(node: Node, blk: T.proc.params(arg0: Node).void).void } def self.each_child_node(node, &blk) - node.class.members.each do |member| - value = node[member] + # Most AST nodes are Ruby Structs; ProtocolRequirement is a T::Struct, + # which enumerates fields via props instead of members. + members = node.is_a?(T::Struct) ? node.class.props.keys : node.class.members + members.each do |member| + value = node.is_a?(T::Struct) ? node.public_send(member) : node[member] if value.is_a?(Array) value.each { |child| yield child if child.is_a?(Locatable) } elsif value.is_a?(Hash) @@ -1700,6 +1788,7 @@ def self.each_child_node(node, &blk) end Program = Struct.new(:token, :statements) do + # ruby-to-clear: field-type statements=[]Locatable extend T::Sig include Locatable @@ -1778,7 +1867,8 @@ def child_bodies = [] :tight_reentrance, :requires_clauses, :return_type_token, :pre_clauses, :post_clauses, :is_method) do # ruby-to-clear: field-type return_type=?Type - # ruby-to-clear: field-type type_params=String[] + # ruby-to-clear: field-type body=[]Locatable + # ruby-to-clear: field-type type_params=[]String # ruby-to-clear: field-type arrow_token=Token # ruby-to-clear: field-type name_token=?Token # ruby-to-clear: field-type return_type_token=?Token @@ -1827,6 +1917,8 @@ def declared_return_type end sig { returns(Type) } + # ruby-to-clear: data-api + # ruby-to-clear: pub def annotation_return_type self[:return_type] || Type.new(:Any) end @@ -2110,6 +2202,10 @@ def borrowed; self[:borrowed]; end end StructDef = Struct.new(:token, :name, :field_decls, :visibility, :type_params) do + # Struct.new deliberately exposes these members as T.untyped to Sorbet. + # Supply the missing source contract instead of teaching the transpiler a + # name-based StructDef exception. + # ruby-to-clear: field-type name=String extend T::Sig include Locatable @@ -2156,10 +2252,22 @@ def token = self[:token] include Locatable attr_accessor :mir_binding_entry # stamped by CleanupClassifier: per-node cleanup entry (avoids same-name collision) attr_accessor :ownership_transport_plan + # set by parse_const_decl: a top-level CONST binding (immutable, comptime, container-scope) + sig { returns(T.nilable(T::Boolean)) } + def module_const = @module_const + sig { params(value: T::Boolean).void } + def module_const=(value); @module_const = value; end + # :pub / :private / :package - export visibility for a module CONST + sig { returns(T.nilable(Symbol)) } + def const_visibility = @const_visibility + sig { params(value: Symbol).void } + def const_visibility=(value); @const_visibility = value; end sig { params(args: InitArgs).void } def initialize(*args) super + @module_const = T.let(nil, T.nilable(T::Boolean)) + @const_visibility = T.let(nil, T.nilable(Symbol)) t = self[:type] self[:type] = Type.new(t) unless t.nil? end @@ -2189,6 +2297,8 @@ def eql?(other) def hash = [var, sync].hash end Assignment = Struct.new(:token, :name, :value, :compound_op) do + # ruby-to-clear: field-type name=AssignmentName + # ruby-to-clear: field-type value=Locatable include Locatable include StatementVoidType attr_accessor :auto_lock # AutoLockPlan set by annotator for inline @locked/@writeLocked guards. @@ -2281,16 +2391,21 @@ def full_type end end attr_accessor :string_concat # true when this is string + (stamped by annotator) + sig { returns(T::Boolean) } + def string_concat? + string_concat == true + end attr_accessor :or_fallback_dupe # true when OR_ELSE fallback struct needs string-field heap dupe attr_accessor :error_union_type # recoverable result preserved through pipeline composition sig { returns(T.nilable(T::Boolean)) } + # ruby-to-clear: data-api def retain_error_channel - @retain_error_channel = T.let(nil, T.nilable(T::Boolean)) unless defined?(@retain_error_channel) - @retain_error_channel + T.let(@retain_error_channel, T.nilable(T::Boolean)) end sig { params(value: T.nilable(T::Boolean)).returns(T.nilable(T::Boolean)) } + # ruby-to-clear: data-api def retain_error_channel=(value) - @retain_error_channel = value + @retain_error_channel = T.let(value, T.nilable(T::Boolean)) end # Lazy positions: fields whose lowering must NOT leak @pending_stmts to # outer scope. The lowering's `descend` helper consults this and wraps @@ -2362,6 +2477,7 @@ def true_boolean? end end ListLit = Struct.new(:token, :items, :storage, :constructor_options) { + # ruby-to-clear: field-type items=[]Locatable extend T::Sig include Locatable @@ -2458,9 +2574,18 @@ def coerce!(declared_type) [target.resolved, nil] end end - HashLit = Struct.new(:token, :pairs, :storage) { include Locatable } + HashLit = Struct.new(:token, :pairs, :storage) do + # ruby-to-clear: field-type pairs=HashLitPairs + include Locatable + end DefaultLit = Struct.new(:token) { include Locatable } StructLit = Struct.new(:token, :name, :fields, :storage, :type_args) do + # Struct.new leaves these fields untyped even though the parser contract + # is concrete. Keep that information at the declaration boundary so every + # consumer (including generic zip) receives the same fact. + # ruby-to-clear: field-type name=String + # ruby-to-clear: field-type fields={String}Locatable + # ruby-to-clear: field-type type_args=?[]Type extend T::Sig include Locatable @@ -2517,6 +2642,9 @@ def body=(val) end end IfStatement = Struct.new(:token, :condition, :then_branch, :else_branch, :then_drops, :else_drops, :comptime) do + # ruby-to-clear: field-type condition=Locatable + # ruby-to-clear: field-type then_branch=[]Locatable + # ruby-to-clear: field-type else_branch=?[]Locatable extend T::Sig include Locatable include StatementVoidType @@ -2566,6 +2694,8 @@ def bindings=(val) end end WhileLoop = Struct.new(:token, :condition, :do_branch, :deferred_drops, :tight) do + # ruby-to-clear: field-type condition=Locatable + # ruby-to-clear: field-type do_branch=[]Locatable extend T::Sig include Locatable include StatementVoidType @@ -2591,6 +2721,8 @@ def tight=(value) attr_accessor :mark_per_iter end WhileBindLoop = Struct.new(:token, :condition, :binding_name, :binding_token, :do_branch, :deferred_drops) do + # ruby-to-clear: field-type condition=Locatable + # ruby-to-clear: field-type do_branch=[]Locatable extend T::Sig include Locatable include StatementVoidType @@ -2655,9 +2787,29 @@ def explicit_mutable_argument_tokens end FuncCall = Struct.new(:token, :name, :args) do + # ruby-to-clear: field-type args=[]Locatable extend T::Sig include Locatable include ExplicitMutableArguments + + sig { params(index: Integer).returns(T::Boolean) } + # ruby-to-clear: data-api + # ruby-to-clear: pub + def explicit_mutable_argument?(index) + tokens = T.let(@explicit_mutable_argument_tokens, T.nilable(T::Hash[Integer, Lexer::Token])) + return false unless tokens + tokens.key?(index) + end + + sig { params(index: Integer).returns(T.nilable(Lexer::Token)) } + # ruby-to-clear: data-api + # ruby-to-clear: pub + def explicit_mutable_argument_token(index) + tokens = T.let(@explicit_mutable_argument_tokens, T.nilable(T::Hash[Integer, Lexer::Token])) + return nil unless tokens + tokens[index] + end + # ruby-to-clear: field-type name=Any attr_accessor :module_alias attr_accessor :extern_call # true when calling a native EXTERN FN (no rt, no try) @@ -2682,7 +2834,16 @@ def explicit_mutable_argument_tokens # original `!T` is stashed here for OR_ELSE consumers # that need to know whether to emit `catch fallback` # (error union) or `orelse fallback` (optional). - attr_accessor :retain_error_channel # explicit `x:!` / `x:!?` binding keeps the call result wrapped + sig { returns(T.nilable(T::Boolean)) } + # ruby-to-clear: data-api + def retain_error_channel + T.let(@retain_error_channel, T.nilable(T::Boolean)) + end + sig { params(value: T.nilable(T::Boolean)).returns(T.nilable(T::Boolean)) } + # ruby-to-clear: data-api + def retain_error_channel=(value) + @retain_error_channel = T.let(value, T.nilable(T::Boolean)) + end sig { returns(T.nilable(Symbol)) } def protocol_operation @protocol_operation = T.let(@protocol_operation, T.nilable(Symbol)) @@ -2714,9 +2875,30 @@ def name; self[:name].to_s end end MethodCall = Struct.new(:token, :object, :name, :args) do + # ruby-to-clear: field-type object=Locatable + # ruby-to-clear: field-type args=[]Locatable extend T::Sig include Locatable include ExplicitMutableArguments + + sig { params(index: Integer).returns(T::Boolean) } + # ruby-to-clear: data-api + # ruby-to-clear: pub + def explicit_mutable_argument?(index) + tokens = T.let(@explicit_mutable_argument_tokens, T.nilable(T::Hash[Integer, Lexer::Token])) + return false unless tokens + tokens.key?(index) + end + + sig { params(index: Integer).returns(T.nilable(Lexer::Token)) } + # ruby-to-clear: data-api + # ruby-to-clear: pub + def explicit_mutable_argument_token(index) + tokens = T.let(@explicit_mutable_argument_tokens, T.nilable(T::Hash[Integer, Lexer::Token])) + return nil unless tokens + tokens[index] + end + attr_accessor :pool_method # :insert, :get, :remove — set by annotator for Pool dispatch attr_accessor :set_method # :insert, :contains, :remove, :count — set by annotator for Set dispatch attr_accessor :map_method # :delete, :contains, :count, :keys, :values — set by annotator for HashMap dispatch @@ -2727,12 +2909,23 @@ def name; self[:name].to_s end attr_accessor :heap_dupe_result # true when result must be heap-duped (frame string escaping to outer container) attr_accessor :safe_nav_chain # implicit continuation of an earlier ?. over non-optional members attr_accessor :error_union_type # full !T requirement result before expression-level propagation unwraps it - attr_accessor :retain_error_channel + sig { returns(T.nilable(T::Boolean)) } + # ruby-to-clear: data-api + def retain_error_channel + T.let(@retain_error_channel, T.nilable(T::Boolean)) + end + sig { params(value: T.nilable(T::Boolean)).returns(T.nilable(T::Boolean)) } + # ruby-to-clear: data-api + def retain_error_channel=(value) + @retain_error_channel = T.let(value, T.nilable(T::Boolean)) + end sig { params(token: Lexer::Token).void } def mark_explicit_mutable_receiver!(token) @explicit_mutable_receiver_token = T.let(token, T.nilable(Lexer::Token)) end sig { returns(T::Boolean) } + # ruby-to-clear: data-api + # ruby-to-clear: pub def explicit_mutable_receiver? !explicit_mutable_receiver_token.nil? end @@ -2740,6 +2933,12 @@ def explicit_mutable_receiver? def explicit_mutable_receiver_token @explicit_mutable_receiver_token = T.let(@explicit_mutable_receiver_token, T.nilable(Lexer::Token)) end + sig { returns(T.nilable(Lexer::Token)) } + # ruby-to-clear: data-api + # ruby-to-clear: pub + def explicit_mutable_receiver_token_value + explicit_mutable_receiver_token + end sig { returns(T.nilable(Symbol)) } def protocol_operation @protocol_operation = T.let(@protocol_operation, T.nilable(Symbol)) @@ -2771,6 +2970,7 @@ def wildcard?; false end def name; self[:name].to_s end end GetField = Struct.new(:token, :target, :field) do + # ruby-to-clear: field-type target=Locatable extend T::Sig include Locatable # Set by visit_assignment_field before visiting this node so @@ -2831,17 +3031,51 @@ def name; target.name end include Locatable end Require = Struct.new(:token, :path) { include Locatable } + # DEFER / DEFER { ... } — body runs at scope exit (success AND error + # paths), lowered directly to MIR::DeferStmt (Zig defer: zero runtime cost). + DeferStmt = Struct.new(:token, :body) { include Locatable } class WithMatchArm < T::Struct + extend T::Sig + const :family, Symbol prop :body, RawBody, factory: -> { [] } prop :lock_error_clauses, T::Array[ErrorClause], factory: -> { [] } const :token, T.nilable(Lexer::Token), default: nil + + sig { returns(RawBody) } + # ruby-to-clear: data-api + # ruby-to-clear: pub + def body_nodes + body + end + + sig { returns(Symbol) } + # ruby-to-clear: data-api + # ruby-to-clear: pub + def family_value + family + end + + sig { returns(T::Array[ErrorClause]) } + # ruby-to-clear: data-api + # ruby-to-clear: pub + def lock_error_clauses_value + lock_error_clauses + end + + sig { returns(T.nilable(Lexer::Token)) } + # ruby-to-clear: data-api + # ruby-to-clear: pub + def token_value + token + end end # lock_error_clause: optional ErrorClause describing ON TIMEOUT / RETRY # handling for EXCLUSIVE / write_locked_read captures. # retries > 0 means RETRY(N) THEN ; retries nil/0 means plain ON TIMEOUT . WithBlock = Struct.new(:token, :capabilities, :body, :deferred_drops, :capability_plan) do + # ruby-to-clear: field-type body=[]Locatable extend T::Sig include Locatable include HasBodies @@ -2921,15 +3155,14 @@ def semantic_with_blocks=(blocks) SelectOp = Struct.new(:token, :expression, :effect_mode, :stream_mode, :modifier_order, :capture_analysis) do extend T::Sig include Locatable - include HasExpression end - WhereOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } - IndexOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } - ReduceOp = Struct.new(:token, :initial_value, :expression) { include Locatable; include HasExpression } - OrderByOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } + WhereOp = Struct.new(:token, :expression) { include Locatable } + IndexOp = Struct.new(:token, :expression) { include Locatable } + ReduceOp = Struct.new(:token, :initial_value, :expression) { include Locatable } + OrderByOp = Struct.new(:token, :expression) { include Locatable } LimitOp = Struct.new(:token, :count) { include Locatable } - UnnestOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } - DistinctOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } + UnnestOp = Struct.new(:token, :expression) { include Locatable } + DistinctOp = Struct.new(:token, :expression) { include Locatable } # EachOp: side-effect iteration over a collection. # Uses `_` as the implicit item binding. Body is a list of statements. # Syntax: collection |> EACH { _.field = value; }; @@ -2946,28 +3179,28 @@ def semantic_with_blocks=(blocks) # to `lhs.next()` (same shape as NEXT for ~T promises). CollectOp = Struct.new(:token) { include Locatable } # TAKE_WHILE: take elements from the front while predicate is true. - TakeWhileOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } + TakeWhileOp = Struct.new(:token, :expression) { include Locatable } # WINDOW(size): sliding window of `size` elements. _ is the sub-slice. - WindowOp = Struct.new(:token, :size, :expression) { include Locatable; include HasExpression } + WindowOp = Struct.new(:token, :size, :expression) { include Locatable } # WINDOW(size: N, time: 'Xms'): batch/tumbling window. _ is a T[] batch. # options = { "size" => size_node, "time" => time_node } (at least one required) - BatchWindowOp = Struct.new(:token, :options, :expression) { include Locatable; include HasExpression } + BatchWindowOp = Struct.new(:token, :options, :expression) { include Locatable } # JOIN(right_source) key_expr_or_lambda # Equi-join: shared key applied to both sides, or lambda(a, b) -> Bool. # Result: anonymous struct { left: L, right: ?R } for each left element. JoinOp = Struct.new(:token, :right_source, :key_expr) { include Locatable } # Phase 3 predicate query operators — return scalar values (not new lists). # All use `_` as the implicit item binding (like SELECT/WHERE). - FindOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } # ?ElemType - AnyOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } # Bool - AllOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } # Bool - CountOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } # Int64 + FindOp = Struct.new(:token, :expression) { include Locatable } # ?ElemType + AnyOp = Struct.new(:token, :expression) { include Locatable } # Bool + AllOp = Struct.new(:token, :expression) { include Locatable } # Bool + CountOp = Struct.new(:token, :expression) { include Locatable } # Int64 # Phase 4 numeric aggregation operators — expression must be numeric. # SUM/AVERAGE return 0 for empty list; MIN/MAX panic on empty list. - SumOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } # Number - AverageOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } # Number - MinOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } # Number (panics on empty) - MaxOp = Struct.new(:token, :expression) { include Locatable; include HasExpression } # Number (panics on empty) + SumOp = Struct.new(:token, :expression) { include Locatable } # Number + AverageOp = Struct.new(:token, :expression) { include Locatable } # Number + MinOp = Struct.new(:token, :expression) { include Locatable } # Number (panics on empty) + MaxOp = Struct.new(:token, :expression) { include Locatable } # Number (panics on empty) # ShardOp: route items to owning schedulers by key hash. # Syntax: collection |> SHARD(key_expr, target_map) |> CONCURRENT EACH { body } # key_expr uses `_` as the implicit item binding (consistent with SELECT/WHERE). @@ -2984,7 +3217,7 @@ def semantic_with_blocks=(blocks) Placeholder = Struct.new(:token) { include Locatable } Copy = Struct.new(:token, :value) { include Locatable } OptionalUnwrap = Struct.new(:token, :target) do - # ruby-to-clear: field-type target=Node + # ruby-to-clear: field-type target=Locatable extend T::Sig include Locatable attr_accessor :error_union_type @@ -3005,6 +3238,9 @@ def safe_navigation? # the member against the payload while retaining the receiver envelope for # the authoritative tense planner and MIR handoff. TenseNavigation = Struct.new(:token, :target, :markers) do + # Struct.new leaves Sorbet field readers untyped; this is the source type + # contract consumed by the data-only CLEAR declaration. + # ruby-to-clear: field-type target=Locatable extend T::Sig include Locatable @@ -3109,8 +3345,8 @@ def filter_messages=(val); self[:filter_messages] = val; end # CATCH block: error handler at function bottom. Multiple CATCH clauses + optional DEFAULT. # catch_clauses: [AST::CatchClause]; default_body: [ASTNode] or nil CatchBlock = Struct.new(:token, :catch_clauses, :default_body) do - # ruby-to-clear: field-type catch_clauses=CatchClause[] - # ruby-to-clear: field-type default_body=?(Node[]) + # ruby-to-clear: field-type catch_clauses=[]CatchClause + # ruby-to-clear: field-type default_body=?([]Locatable) include Locatable end # RECOVER(default): pipeline operator that replaces errors with a default value. @@ -3184,13 +3420,18 @@ def self.pipeline_complex_op?(node) # Transpiles to Zig labeled block: blk: { stmts; break :blk result; } # body: Array of statement AST nodes # result: AST node whose value is the block's result - BlockExpr = Struct.new(:token, :body, :result) { include Locatable } + BlockExpr = Struct.new(:token, :body, :result) do + # ruby-to-clear: field-type body=[]Locatable + # ruby-to-clear: field-type result=?Locatable + include Locatable + end # StringConcat: flattened string concatenation. # parts: Array of AST nodes (strings, identifiers, expressions) # Rewritten from chained BinaryOp(:ADD) on string types. # Any backend emits a single allocation covering all parts. StringConcat = Struct.new(:token, :parts) do + # ruby-to-clear: field-type parts=[]Locatable include Locatable end @@ -3233,7 +3474,10 @@ def locked_sync? = locked? || write_locked? sig { returns(T::Boolean) } def local_storage_wrap? = local? || (indirect? && !sync && !ownership) end - MoveNode = Struct.new(:token, :value) { include Locatable } # MOVE expr -> transfer Rc/Arc handle without retain + MoveNode = Struct.new(:token, :value) do + # ruby-to-clear: field-type value=Locatable + include Locatable + end # MOVE expr -> transfer Rc/Arc handle without retain # CopyNode -- explicit COPY expr (deep copy of value). # deep_copy: true for unions with heap variants. # alloc: :heap (default) | :frame -- the allocator the duped buffer @@ -3244,16 +3488,43 @@ def local_storage_wrap? = local? || (indirect? && !sync && !ownership) # provenance" bug that forced cleanupAlloc. lower_copy reads # this; hard-coded :heap pre-policy. CopyNode = Struct.new(:token, :value) do + # ruby-to-clear: field-type value=Locatable extend T::Sig include Locatable attr_accessor :deep_copy + # Retained-identity v5: the physical op the OwnershipEdgePlanner selected + # for this COPY -- :payload_copy for a plain source, or + # :shared_to_unique_copy for a retained source detached via OWN COPY. + attr_accessor :carrier_op + # true when written `OWN COPY x`: the explicit handle->owned-RawT downgrade + # (deref the @multiowned/@shared payload, deep-copy it into a fresh + # uniquely-owned value). Bare `COPY x` is a memcpy and is illegal on a live + # handle; only OWN COPY may copy a retained payload out. + attr_accessor :own sig { params(val: Symbol).returns(Symbol) } def alloc=(val); T.must(@alloc = T.let(val, T.nilable(Symbol))); end sig { returns(Symbol) } def alloc; @alloc = T.let(@alloc, T.nilable(Symbol)); @alloc || :heap; end end - CloneNode = Struct.new(:token, :value) { include Locatable } # CLONE expr -> explicit handle retain for non-affine replay/shared futures - ShareNode = Struct.new(:token, :value) { include Locatable } # SHARE expr -> promote/retain as T@shared (semantic lowering follows) + # KeepNode -- KEEP expr (retained-identity v5): the unified carrier- + # preserving fan-out (formerly CLONE, which was the narrow Rc/Arc-retain + # case, and COPY_OR_CLONE). Preserves the caller's carrier -- @multiowned + # Rc retain, @shared Arc retain, or plain payload copy -- chosen at + # placement, never at runtime. Does NOT promise independent identity. + KeepNode = Struct.new(:token, :value) do + # ruby-to-clear: field-type value=Locatable + include Locatable + # Retained-identity v5: the physical ownership operation the + # OwnershipEdgePlanner selected for this fan-out (one of the 7 ops), or + # :deferred_specialization when the source is a carrier-polymorphic + # parameter whose carrier is resolved per specialization (Phase 4). MIR + # lowering consumes this; it does NOT re-derive from the operand type. + attr_accessor :carrier_op + end + ShareNode = Struct.new(:token, :value) do + # ruby-to-clear: field-type value=Locatable + include Locatable + end # SHARE expr -> promote/retain as T@shared (semantic lowering follows) LinkNode = Struct.new(:token, :value) { include Locatable } # LINK expr -> downgrade Rc/Arc to WeakRc/WeakArc ResolveNode = Struct.new(:token, :value) { include Locatable } # RESOLVE expr -> upgrade WeakRc/WeakArc to ?Rc/?Arc FreezeNode = Struct.new(:token, :value) { include Locatable } # FREEZE expr -> compact @multiowned tree into contiguous buffer @@ -3280,6 +3551,8 @@ def fields=(val) # RangeLit: a range expression (start..(params) RETURNS type [EFFECTS :alloc] FROM "module" # Or method: EXTERN FN TypeName.method(params) RETURNS type FROM "module" # Declares a native Zig/C function importable via @import("module"). @@ -3288,8 +3561,8 @@ def fields=(val) :return_lifetime) do # ruby-to-clear: field-type return_type=?Type # ruby-to-clear: field-type owner_type=?String - # ruby-to-clear: field-type owner_type_params=String[]@symbol - # ruby-to-clear: field-type fn_type_params=String[]@symbol + # ruby-to-clear: field-type owner_type_params=[]String@symbol + # ruby-to-clear: field-type fn_type_params=[]String@symbol # ruby-to-clear: field-type return_lifetime=Any extend T::Sig include Locatable @@ -3305,6 +3578,11 @@ def fn_type_params self[:fn_type_params] end + sig { returns(T.nilable(ExternEffects)) } + def effects + self[:effects] + end + sig { params(args: InitArgs).void } def initialize(*args) super @@ -3340,6 +3618,8 @@ def return_type=(val) end sig { returns(Type) } + # ruby-to-clear: data-api + # ruby-to-clear: pub def annotation_return_type self[:return_type] || Type.new(:Any) end @@ -3350,7 +3630,7 @@ def annotation_return_type ExternStructDecl = Struct.new(:token, :name, :field_decls, :from_module, :type_params, :close_method, :as_type, :extern_source) do # ruby-to-clear: field-type from_module=?String - # ruby-to-clear: field-type type_params=String[]@symbol + # ruby-to-clear: field-type type_params=[]String@symbol # ruby-to-clear: field-type close_method=?String # ruby-to-clear: field-type as_type=?String extend T::Sig @@ -3413,8 +3693,12 @@ class UnionMethodRequirement < T::Struct # methods: Array of UnionMethodRequirement records; empty when absent. # — compile-time constraints verified after function registration. UnionDef = Struct.new(:token, :name, :variants, :visibility, :type_params, :methods) do - # ruby-to-clear: field-type type_params=String[] - # ruby-to-clear: field-type methods=UnionMethodRequirement[] + # Struct.new exposes untyped readers to Sorbet. Declare the storage + # contract once so ProgramIndex can type every consumer without inferring + # a Hash shape from whichever method happens to use this field first. + # ruby-to-clear: field-type variants={String}?SchemasUnionSchemaVariantInput + # ruby-to-clear: field-type type_params=[]String + # ruby-to-clear: field-type methods=[]UnionMethodRequirement extend T::Sig include Locatable # Array of type param name strings, e.g. ["T"] @@ -3616,6 +3900,9 @@ def expr # case_drops: Array of drop-arrays (parallel to cases), filled by annotator # default_drops: drop-array for default branch (or nil), filled by annotator MatchStatement = Struct.new(:token, :expr, :cases, :default_case, :case_drops, :default_drops, :exhaustive, :takes) do + # ruby-to-clear: field-type expr=Locatable + # ruby-to-clear: field-type cases=[]MatchCase + # ruby-to-clear: field-type default_case=?([]Locatable) extend T::Sig include Locatable include HasBodies @@ -3782,16 +4069,16 @@ def expr end # ASSERT_RAISES Kind, expr OR ASSERT_RAISES Kind, ErrorName, expr - AssertRaises = Struct.new(:token, :kind, :error_name, :expression) { include Locatable; include HasExpression } + AssertRaises = Struct.new(:token, :kind, :error_name, :expression) { include Locatable } # BENCHMARK expr x - BenchmarkStmt = Struct.new(:token, :expression, :iterations) { include Locatable; include HasExpression } + BenchmarkStmt = Struct.new(:token, :expression, :iterations) { include Locatable } # SMASH expr - SmashStmt = Struct.new(:token, :expression) { include Locatable; include HasExpression } + SmashStmt = Struct.new(:token, :expression) { include Locatable } # PROFILE expr - ProfileStmt = Struct.new(:token, :expression) { include Locatable; include HasExpression } + ProfileStmt = Struct.new(:token, :expression) { include Locatable } class SelectOp extend T::Sig diff --git a/compiler/ruby/ast/async_result_shape.rb b/compiler/ruby/ast/async_result_shape.rb index 5a079f5fa..9c0e9b5e3 100644 --- a/compiler/ruby/ast/async_result_shape.rb +++ b/compiler/ruby/ast/async_result_shape.rb @@ -37,6 +37,7 @@ def boxes_fallible_payload? end sig { returns(String) } + # ruby-to-clear: fallible def payload_zig_type TypeZigRenderer.render_async_payload(payload_type) end diff --git a/compiler/ruby/ast/diagnostic_buckets.rb b/compiler/ruby/ast/diagnostic_buckets.rb index 9a003829a..15d3e48e3 100644 --- a/compiler/ruby/ast/diagnostic_buckets.rb +++ b/compiler/ruby/ast/diagnostic_buckets.rb @@ -321,7 +321,7 @@ module DiagnosticBuckets BG_STREAM_NO_YIELD BG_STREAM_INCONSISTENT_YIELD BG_STREAM_YIELDS_REQUIRED YIELD_OUTSIDE_BG_STREAM CLOSE_OUTSIDE_BG_STREAM NEXT_NEEDS_FUTURE - ATSPLIT_STREAM_ONLY ATSPLIT_NEEDS_OPEN_STREAM + ATSPLIT_STREAM_ONLY ATSPLIT_NEEDS_OPEN_STREAM RETIRED_OPTIONAL_STREAM_SYNTAX INF_STREAM_SELECT_MOVES_ITEM RC_PROMISE_NEEDS_SHARED SOA_TO_EXTERN_FN C_EXTERN_UNSUPPORTED_TYPE ], }, @@ -559,25 +559,29 @@ module DiagnosticBuckets }, ].freeze, T::Array[T::Hash[Symbol, T.untyped]]) - COVERED_CODES = T.let(BUCKETS.flat_map { |bucket| - T.cast(bucket[:codes], T::Array[Symbol]) - }.to_set.freeze, T::Set[Symbol]) - - BUCKETS_BY_CATEGORY = T.let(BUCKETS.group_by { |bucket| - T.cast(bucket[:category], Symbol) - }.transform_values { |buckets| buckets.freeze }.freeze, T::Hash[Symbol, T::Array[T::Hash[Symbol, T.untyped]]]) - # All codes referenced by any bucket — used by the audit to confirm # bucket assignments are exhaustive for their category. sig { returns(T::Set[Symbol]) } def self.covered_codes - COVERED_CODES + codes = T.let(Set.new, T::Set[Symbol]) + DiagnosticRegistry.codes.each do |code| + entry = DiagnosticRegistry.lookup(code) + next unless entry + + category = entry[:category] + codes.add(code) if category == :type || category == :capability + end + codes end # Buckets for a specific category (e.g. `:type`). sig { params(cat: Symbol).returns(T::Array[T::Hash[Symbol, T.untyped]]) } def self.for_category(cat) - BUCKETS_BY_CATEGORY.fetch(cat, []) + matches = T.let([], T::Array[T::Hash[Symbol, T.untyped]]) + BUCKETS.each do |bucket| + matches << bucket if T.cast(bucket[:category], Symbol) == cat + end + matches end # Status of a single code: diff --git a/compiler/ruby/ast/diagnostic_examples.rb b/compiler/ruby/ast/diagnostic_examples.rb index 9f8042364..bd8ff41eb 100644 --- a/compiler/ruby/ast/diagnostic_examples.rb +++ b/compiler/ruby/ast/diagnostic_examples.rb @@ -189,20 +189,21 @@ def self.extract_first_heredoc_in_it(block_lines, expecting_raise:) # non-whitespace is the matching marker name. sig { params(body: String).returns(T.nilable(String)) } def self.extract_heredoc(body) - return nil unless body =~ /<<~(CLEAR|FLUX)\b/ - marker = $~[1] - after = $~.post_match + heredoc_match = body.match(/<<~(CLEAR|FLUX)\b/) + return nil unless heredoc_match + marker = heredoc_match[1] + after = heredoc_match.post_match # Skip the rest of the marker line (e.g. ` ) }.to raise_error(...)`). nl = after.index("\n") return nil unless nl - after_lines = after[(nl + 1)..].lines + after_lines = T.must(after[(nl + 1)..]).lines end_idx = after_lines.index { |l| l =~ /^\s*#{marker}\s*$/ } return nil unless end_idx raw_lines = after_lines[0...end_idx] - nonempty = raw_lines.reject { |l| l.strip.empty? } - return raw_lines.join if nonempty.empty? - min_indent = nonempty.map { |l| l[/\A( *)/].length }.min - raw_lines.map { |l| l.sub(/\A {0,#{min_indent}}/, "") }.join + nonempty = T.must(raw_lines).reject { |l| l.strip.empty? } + return T.must(raw_lines).join if nonempty.empty? + min_indent = nonempty.map { |l| T.must(l[/\A( *)/]).length }.min + T.must(raw_lines).map { |l| l.sub(/\A {0,#{min_indent}}/, "") }.join end private_class_method :extract_first_heredoc_in_it diff --git a/compiler/ruby/ast/diagnostic_registry.rb b/compiler/ruby/ast/diagnostic_registry.rb index 3c9ad6ef4..01b8556ff 100644 --- a/compiler/ruby/ast/diagnostic_registry.rb +++ b/compiler/ruby/ast/diagnostic_registry.rb @@ -188,6 +188,26 @@ def self.entry(severity:, category:, template:, summary:, cause: nil, fix_hint: template: "MUTABLE bare declaration requires an explicit type annotation.", summary: "`MUTABLE x;` (no `=` initializer) needs an explicit `: T[N]` so the parser can synthesize the default-zero list.", }, + CONST_NEEDS_TYPE: { + severity: :error, category: :syntax, + template: "CONST '%{name}' requires an explicit type annotation: `CONST %{name}: T = ...`.", + summary: "A top-level CONST has no enclosing frame to infer from, so its type must be written explicitly.", + }, + CONST_NEEDS_VALUE: { + severity: :error, category: :syntax, + template: "CONST '%{name}' requires an initializer: `CONST %{name}: T = expr`.", + summary: "A CONST is an immutable comptime binding; it must be initialized at declaration.", + }, + CONST_NEEDS_CAPS: { + severity: :error, category: :syntax, + template: "CONST name '%{name}' must be SCREAMING_CASE (all uppercase).", + summary: "Constants are written in all-caps so a bare all-caps identifier reads as a constant reference; TitleCase names denote types.", + }, + CONST_INIT_FALLIBLE: { + severity: :error, category: :ownership, + template: "CONST '%{name}' has a fallible initializer but a CONST has no error channel.", + summary: "A runtime-initialized CONST is assigned once at program start; it cannot propagate a RAISE. Make the initializer infallible (allocation faults are still allowed) or compute the value inside a function.", + }, MUTABLE_BARE_NEEDS_FIXED: { severity: :error, category: :syntax, template: "MUTABLE bare declaration requires a fixed-size array type T[N]; got %{type}.", @@ -210,12 +230,12 @@ def self.entry(severity:, category:, template:, summary:, cause: nil, fix_hint: cause: "A struct field assignment (`p.field = v`) requires the receiver binding `p` to be declared MUTABLE. CLEAR is immutable-by-default; without `MUTABLE p = ...`, no field of `p` can be reassigned.", fix_hint: "Add `MUTABLE` at the receiver's declaration. Capability-wrapped bindings (`@locked`, `@alwaysMutable`) also permit field writes through their unwrapping rules.", }, - ILLEGAL_FIELD_LOOKUP: entry( + ILLEGAL_FIELD_LOOKUP: { severity: :error, category: :type, template: "Type Error: Cannot determine struct type for field access '%{field}'. Receiver is '%{type}'.", summary: "Field access on a non-struct (or unresolved-type) target.", - ), + }, OPTIONAL_FIELD_REQUIRES_SAFE_NAV: { severity: :error, category: :type, template: "Type Error: Cannot access field '%{field}' on optional '%{type}' without safe navigation.", @@ -784,6 +804,18 @@ def self.entry(severity:, category:, template:, summary:, cause: nil, fix_hint: summary: "CLEAR no longer encodes mutation in function or method names.", fix_hint: "Remove the `!` suffix; the call-site checker will insert `&` on values passed to MUTABLE parameters.", }, + INF_STREAM_SELECT_MOVES_ITEM: { + severity: :error, category: :type, + template: "SELECT over an infinite stream cannot MOVE the item (GIVE / TAKES): an [~INF]T rendezvous stream never drains, so moved payloads could be left in flight at teardown. Observe the item (borrow / COPY) or bound the stream with LIMIT into a finite pipeline first.", + summary: "Ownership-moving selectors are rejected on infinite stream sources.", + fix_hint: "Use a borrowing or COPY selector, or apply LIMIT before the moving SELECT.", + }, + RETIRED_OPTIONAL_STREAM_SYNTAX: { + severity: :error, category: :type, + template: "Retired stream type syntax '%{got}'. Streams put the tense/cardinality layer first: use '%{replacement}'. ([~]T = unbound-finite, [~N]T = bound-finite, [~INF]T = infinite; NEXT on [~]T returns a StreamStep unwrapped with EXISTS.)", + summary: "The '~?T[]' open-stream alias is retired; the accepted spellings are [~]T, [~N]T, and [~INF]T.", + fix_hint: "Replace the annotation with the suggested cardinality-first spelling.", + }, RETURN_MISMATCH: { severity: :error, category: :type, template: "Type Error: Function expected to return '%{expected}', but returned '%{got}'", @@ -1160,6 +1192,12 @@ def self.entry(severity:, category:, template:, summary:, cause: nil, fix_hint: }, # =================================================================== + DEFER_NO_CONTROL_FLOW: { + severity: :error, category: :syntax, + template: "%{kind} is not allowed inside a DEFER body — deferred code runs during scope teardown, where redirecting control flow is meaningless.", + summary: "DEFER bodies cannot contain RETURN/BREAK/CONTINUE/YIELD (same restriction as Zig defer).", + fix_hint: "Move the control-flow statement outside the DEFER body.", + }, # PARSER — generic Expected/Unexpected token # =================================================================== PARSER_EXPECTED: { @@ -2393,15 +2431,10 @@ def self.entry(severity:, category:, template:, summary:, cause: nil, fix_hint: cause: "The value is, or transitively contains, a linear resource with a CLOSE contract. Duplicating the handle would make two owners close the same underlying resource.", fix_hint: "Use `CLONE` for shared / refcounted handles. For linear resources, transfer ownership via `GIVE` or pass through a borrow.", }, - CLONE_WITH_SCOPED: { + KEEP_WITH_SCOPED: { severity: :error, category: :escape, - template: "Cannot CLONE WITH-scoped '%{name}'. WITH bindings are protected borrows; use COPY to return owned data.", - summary: "CLONE on a WITH-scoped binding would create another reference that outlives the WITH.", - }, - CLONE_BAD_TARGET: { - severity: :error, category: :ownership, - template: "CLONE is only supported on @split streams, @shared promises, and owned shared handles, got '%{got}'", - summary: "CLONE has a narrow set of supported targets.", + template: "Cannot KEEP WITH-scoped '%{name}'. WITH bindings are protected borrows; use COPY to return owned data.", + summary: "KEEP on a WITH-scoped binding would create another owner that outlives the WITH.", }, SHARE_NEEDS_TYPED: { severity: :error, category: :ownership, @@ -2835,6 +2868,83 @@ def self.entry(severity:, category:, template:, summary:, cause: nil, fix_hint: cause: "Rc/Arc/Weak handles own a control-block count. Copying their pointer fields without a retain fabricates an uncounted owner and causes premature release, leaks, or use-after-free.", fix_hint: "Lowering bug — emit RcRetain/WeakUpgrade/RcDowngrade for direct handles. Aggregate copies must route RC fields through the runtime retain-aware dupeValue path.", }, + GENERIC_IDENTITY_FIELD_UNSUPPORTED: { + severity: :error, category: :ownership, + template: "STRUCT '%{struct}' declares identity field '%{field}: %{param} @multiowned', but identity capabilities on generic type parameters are not yet supported", + summary: "A generic type parameter cannot yet carry an identity capability.", + cause: "Substituting a concrete type into an identity-capable generic field either double-wraps an already-managed binding or fails to wrap a payload; generic keep-analysis has not been implemented.", + fix_hint: "Use a concrete identity type for the field, or drop the capability and wrap at the binding site.", + }, + KEPT_FN_VALUE_ABI: { + severity: :error, category: :ownership, + template: "'%{name}' keeps parameter '%{param}' (%{sink}), so its compiled signature takes an owned handle and is incompatible with the plain function type '%{fn_type}'. Retention is not yet representable in function values", + summary: "A retaining function cannot be used as a plain function value.", + cause: "Keep-analysis changes the kept parameter's ABI to the ownership family's handle. An ordinary FN type carries no retained-parameter contract, so assigning the function to it would produce an incompatible function pointer.", + fix_hint: "Call the function directly, or wrap it: a non-retaining adapter that constructs/receives the identity explicitly can be passed as a function value.", + }, + KEEP_ON_KNOWN_CARRIER: { + severity: :error, category: :ownership, + template: "KEEP on '%{name}' is redundant: its carrier is statically known (%{carrier}), so use COPY for an independent copy", + summary: "KEEP is only for carrier-polymorphic values.", + cause: "KEEP preserves a caller-chosen carrier. A plain local or a UNIQUE parameter has a statically known carrier, so the polymorphic form carries no information and hides whether an independent copy was intended.", + fix_hint: "Use COPY to create an independent copy.", + }, + RETAINED_NEEDS_OWN_COPY: { + severity: :error, category: :ownership, + template: "'%{name}' is a retained %{carrier} handle and cannot fill the plain parameter '%{param}': a handle can't be moved into a plain slot without an explicit carrier decision. Passing it bare would silently deep-copy the payload out of the shared handle", + summary: "A retained handle can't silently fill a plain parameter.", + cause: "The parameter is a plain (RawT) owned slot, but the argument is a @multiowned/@shared handle whose identity other owners may share. Crossing that carrier boundary must be explicit -- never a hidden allocation and deep copy.", + fix_hint: "Use OWN COPY %{name} to own an independent copy of the payload, or declare the parameter '%{param}: SHARED T' (or MONOMORPHIC T) to keep the handle.", + }, + OWN_ALONE_UNSUPPORTED: { + severity: :error, category: :ownership, + template: "OWN must be followed by COPY. OWN COPY x owns an independent copy of the payload (dereferencing a @multiowned/@shared handle); bare OWN x (moving the payload out of a handle) is not yet supported", + summary: "Bare OWN is not yet supported; write OWN COPY.", + cause: "OWN x alone would extract the payload out of a handle, which requires proving the handle is uniquely owned; that path is deferred. OWN COPY x always works because it copies the payload rather than moving it.", + fix_hint: "Write OWN COPY x to own an independent copy of the payload.", + }, + UNIQUE_NEEDS_EXCLUSIVE: { + severity: :error, category: :ownership, + template: "'%{name}' is a retained %{carrier} value and is passed to the UNIQUE parameter '%{param}' without OWN COPY. A UNIQUE parameter requires exactly one owner; a live @multiowned/@shared handle is (or may be) multi-owned", + summary: "A UNIQUE parameter cannot receive a live multi-owned handle.", + cause: "UNIQUE means the callee owns the value exclusively. A retained handle shares its identity with other owners, so handing it over bare would give the callee a non-exclusive value.", + fix_hint: "Use OWN COPY %{name} to detach an independent payload for the UNIQUE parameter, or pass a plain/uniquely-owned value.", + }, + COPY_RETAINED_NEEDS_UNIQUE: { + severity: :error, category: :ownership, + template: "COPY of '%{name}' is not allowed: it is a retained %{carrier} handle, and COPY is a memcpy -- copying the handle bits would duplicate a shared owner without touching the reference count", + summary: "COPY (a memcpy) cannot copy a live @multiowned/@shared handle.", + cause: "COPY performs a bitwise copy of the value. For a retained carrier the value IS a handle, so a memcpy would create a second owner that never incremented the reference count -- a double free. Retaining another handle or detaching an independent payload are different operations.", + fix_hint: "Use KEEP %{name} to retain another handle (carrier-preserving), or OWN COPY %{name} to own an independent copy of the payload.", + }, + COPY_ON_POLYMORPHIC_PARAM: { + severity: :error, category: :ownership, + template: "COPY on '%{name}' is not allowed: it is a carrier-polymorphic parameter, so COPY cannot guarantee independent identity when the caller passed a retained (@multiowned/@shared) value", + summary: "COPY needs a local or UNIQUE value, not a carrier-polymorphic parameter.", + cause: "A carrier-polymorphic parameter may be backed by a retained handle whose identity other owners share; COPY there would either retain (not independent) or require a hidden deep copy. The correctness choice must be explicit.", + fix_hint: "Use KEEP to preserve the caller's carrier, or declare the parameter UNIQUE if independent identity is required.", + }, + CARRIER_POLYMORPHIC_FANOUT: { + severity: :error, category: :ownership, + template: "%{detail}", + summary: "A carrier-polymorphic parameter is consumed then used again.", + cause: "A TAKES parameter with no UNIQUE/SHARED constraint preserves the caller's ownership carrier. Consuming it once and using it again creates a second owner, which requires an explicit correctness choice: KEEP preserves the carrier; UNIQUE + COPY forces independent identity.", + fix_hint: "Wrap the first consuming use as KEEP, or declare the parameter UNIQUE and use COPY if independent identity is required.", + }, + KEPT_IDENTITY_FAMILY_MISMATCH: { + severity: :error, category: :ownership, + template: "'%{name}' is @%{source_family} but is kept by '%{keeper}' (%{sink}), which requires an @%{dest_family} identity. %{source_family} and %{dest_family} use incompatible reference-counting (Arc is atomic, Rc is not), so the handle cannot be retained across the call without breaking the other owners' accounting", + summary: "A kept edge cannot convert between reference-counting families.", + cause: "Keep-analysis retains the caller's handle in the callee's identity field. @shared (Arc) and @multiowned (Rc) maintain their counts differently, so passing one where the other is retained would corrupt the surviving owners' counts.", + fix_hint: "Match the families (declare the source @%{dest_family}), or break identity with COPY %{name} so the keeper constructs an independent handle.", + }, + KEPT_IDENTITY_NEEDS_MODEL: { + severity: :error, category: :ownership, + template: "'%{name}' is kept by '%{keeper}' (%{sink}) and is a plain MUTABLE binding. Unique ownership cannot satisfy a keep; declare the cost model at the declaration: '@multiowned' shares one identity (callers observe mutations), '@value' gives every keeper an independent copy, or pass 'GIVE %{name}' at the last use so the handle moves", + summary: "A plain MUTABLE binding is kept, and no cost model is sound by default.", + cause: "A kept destination stores the value's identity beyond the call. For an immutable binding the mechanism is unobservable, but a MUTABLE binding used after the call makes sharing-vs-copying observable, so the declaration must choose.", + fix_hint: "Append @multiowned or @value to the declaration, or relinquish with GIVE at the call.", + }, SHARDED_ELEMENT_REQUIRES_SHARED: { severity: :error, category: :ownership, template: "@sharded collections cannot store %{got}; cross-scheduler reference-counted elements must use @shared", @@ -3154,7 +3264,7 @@ def self.entry(severity:, category:, template:, summary:, cause: nil, fix_hint: }, ATSPLIT_NEEDS_OPEN_STREAM: { severity: :error, category: :type, - template: "@split is currently only valid on open streams (~?T[]).", + template: "@split is currently only valid on open streams ([~]@split T).", summary: "@split applies only to open streams.", }, SOA_TO_EXTERN_FN: { @@ -3605,7 +3715,7 @@ def self.entry(severity:, category:, template:, summary:, cause: nil, fix_hint: cause: "Capabilities.validate! rejected the binding's capability stack — either an unsupported sigil combination (`@local:atomic`), a capability on an incompatible type (capability on a primitive), or a missing required capability.", fix_hint: "Read the message for the specific rejection. Common fixes: drop a contradictory sigil, wrap a primitive in a struct, add a missing wrapper (`@shared` for cross-fiber sharing).", }, - }.freeze, T::Hash[Symbol, T::Hash[Symbol, T.untyped]]) + }.freeze, T::Hash[Symbol, DiagnosticEntry]) FIX_DESCRIPTIONS = T.let({ INSERT_SELECT_EFFECT_COLON: "Change the legacy SELECT effect spelling to %{selector}.", @@ -3679,8 +3789,8 @@ def self.entry(severity:, category:, template:, summary:, cause: nil, fix_hint: WRAP_FOREIGN_INDEX_UNSAFE_VIEW: "Wrap this access in `WITH UNSAFE VIEW %{name} LENGTH %{length} AS %{alias_name} { ... }`; verify that the C allocation really contains that many elements.", CONVERT_FOREIGN_VIEW_TO_UNSAFE: "Change this to `WITH UNSAFE VIEW %{name} LENGTH 0 AS ...`; then replace 0 with the element count guaranteed by the C API.", WRAP_DIRECT_VIEW_ACCESS: "Wrap this access in `WITH %{permission} %{name} AS %{alias_name} { ... }` and use the scoped alias.", - WRAP_CONSUMER_WITH_CLONE: "Wrap the consuming reference with CLONE at line %{line} (bumps the refcount; both bindings stay live).", WRAP_CONSUMER_WITH_COPY: "Wrap the consuming reference with COPY at line %{line} (the original survives for the later use).", + WRAP_CONSUMER_WITH_KEEP: "Wrap the consuming reference with KEEP at line %{line} (preserves the caller's carrier: refcount retain for @multiowned/@shared/@split, payload copy for a plain value; both bindings stay live).", WRAP_RETURN_WITH_COPY: "Wrap the returned value with `COPY ` so it doesn't borrow from the parameter.", WRAP_VALUE_WITH_CAST: "Wrap value with `CAST(... AS %{type})` (narrowing -- verify it can't lose data).", REPLACE_AUTO_WITH_INFERRED: "Replace `Auto` with the inferred type `%{type}`.", @@ -3708,8 +3818,15 @@ def self.codes # unimplemented compiler check. sig { params(code: Symbol).returns(T::Boolean) } def self.pending?(code) - entry = DIAGNOSTICS[code] - !entry.nil? && entry[:pending] == true + return true if code == :PRIMITIVE_PASSED_AS_MUTABLE || code == :GIVE_BAD_TARGET + + # Ruby tests and extensions may replace DIAGNOSTICS at runtime. The + # self-hosted registry is closed, so it uses the exhaustive fast path + # above and omits this dynamic Hash lookup. + # ruby-to-clear: skip + return DIAGNOSTICS.dig(code, :pending) == true + + false end # Format a registered code's template against `args`. Returns nil @@ -3717,7 +3834,7 @@ def self.pending?(code) # nil — the legacy helper raises an internal-compiler-error there. sig { params(code: Symbol, args: DiagnosticArgs, kwargs: DiagnosticKwValue).returns(T.nilable(String)) } def self.format(code, args = [], **kwargs) - format_from_hash(code, args, kwargs) + DiagnosticRegistry.format_from_hash(code, args, kwargs) end sig { params(code: Symbol, args: DiagnosticArgs, kwargs: DiagnosticKwargs).returns(T.nilable(String)) } @@ -3725,30 +3842,46 @@ def self.format_from_hash(code, args, kwargs) entry = DIAGNOSTICS[code] return nil unless entry - format_template(T.cast(entry[:template], String), args, kwargs) + DiagnosticRegistry.format_template(DiagnosticRegistry.template_from_entry(entry), args, kwargs) + end + + # Entries preserve insertion order and always declare severity/category + # before template. Walking values avoids an optional nested Hash index that + # the self-host type annotator cannot stamp yet. + sig { params(entry: DiagnosticEntry).returns(String) } + def self.template_from_entry(entry) + entry.values.each do |value| + return value if value.is_a?(String) + end + Kernel.raise "Internal Compiler Error: diagnostic entry has no template" end sig { params(template: String, args: DiagnosticArgs, kwargs: DiagnosticKwargs).returns(String) } def self.format_template(template, args = [], kwargs = {}) if !kwargs.empty? || template.include?("%{") - return template % kwargs if !template.include?("%{") || named_template_args_complete?(template, kwargs) + return template % kwargs if !template.include?("%{") || DiagnosticRegistry.named_template_args_complete?(template, kwargs) return "#{template} [Internal Args Error: #{kwargs.inspect}]" end - return template % args if positional_template_args_complete?(template, args) + return template % args if DiagnosticRegistry.positional_template_args_complete?(template, args) "#{template} [Internal Args Error: #{args.inspect}]" end sig { params(template: String, kwargs: DiagnosticKwargs).returns(T::Boolean) } def self.named_template_args_complete?(template, kwargs) - keys = named_template_keys(template) - i = T.let(0, Integer) - while i < keys.length - return false unless kwargs.key?(keys.fetch(i)) + offset = T.let(0, Integer) + loop do + start_index = template.index("%{", offset) + break unless start_index - i += 1 + end_index = template.index("}", start_index + 2) + break unless end_index + + key = T.must(template[(start_index + 2)...end_index]).to_sym + return false unless kwargs.key?(key) + offset = end_index + 1 end true end @@ -3764,7 +3897,7 @@ def self.named_template_keys(template) end_index = template.index("}", start_index + 2) break unless end_index - keys << T.unsafe(template[(start_index + 2)...end_index]).to_sym + keys << T.must(template[(start_index + 2)...end_index]).to_sym offset = end_index + 1 end keys @@ -3772,7 +3905,7 @@ def self.named_template_keys(template) sig { params(template: String, args: DiagnosticArgs).returns(T::Boolean) } def self.positional_template_args_complete?(template, args) - positional_placeholder_count(template) <= args.length + DiagnosticRegistry.positional_placeholder_count(template) <= args.length end sig { params(template: String).returns(Integer) } @@ -3800,29 +3933,33 @@ def self.fix_description_from_hash(code, kwargs) template = FIX_DESCRIPTIONS[code] Kernel.raise "Internal Compiler Error: Unknown fix description code :#{code}" unless template - return template % kwargs if named_template_args_complete?(template, kwargs) + return template % kwargs if DiagnosticRegistry.named_template_args_complete?(template, kwargs) - missing = missing_named_template_key(template, kwargs) + missing = DiagnosticRegistry.missing_named_template_key(template, kwargs) detail = missing ? "key{#{missing}} not found kwargs=#{kwargs.inspect}" : "kwargs=#{kwargs.inspect}" "#{template} [Internal Args Error: #{detail}]" end sig { params(template: String, kwargs: DiagnosticKwargs).returns(T.nilable(Symbol)) } def self.missing_named_template_key(template, kwargs) - keys = named_template_keys(template) - i = T.let(0, Integer) - while i < keys.length - key = keys.fetch(i) - return T.unsafe(key) unless kwargs.key?(key) + offset = T.let(0, Integer) + loop do + start_index = template.index("%{", offset) + break unless start_index - i += 1 + end_index = template.index("}", start_index + 2) + break unless end_index + + key = T.must(template[(start_index + 2)...end_index]).to_sym + return T.unsafe(key) unless kwargs.key?(key) + offset = end_index + 1 end nil end sig { params(code: Symbol, kwargs: DiagnosticKwValue).returns(String) } def self.fix_description(code, **kwargs) - fix_description_from_hash(code, kwargs) + DiagnosticRegistry.fix_description_from_hash(code, kwargs) end # Self-check: every entry is well-formed. Returns an array of diff --git a/compiler/ruby/ast/error_registry.rb b/compiler/ruby/ast/error_registry.rb index 68f8cc670..62a71d43f 100644 --- a/compiler/ruby/ast/error_registry.rb +++ b/compiler/ruby/ast/error_registry.rb @@ -1,5 +1,6 @@ # typed: strict require "sorbet-runtime" +require_relative "lexer" module AST extend T::Sig @@ -68,7 +69,7 @@ module AST # first_site: , # for collision diagnostics # } # Not thread-safe; the compiler is single-threaded per-program. - ERROR_TYPES = T.let({ + BASE_ERROR_TYPES = T.let({ LockTimeout: { kind: :Transient, zig_name: "LockTimeout", id: ERROR_NAME_LOCK_TIMEOUT, first_site: nil }, LockCycle: { kind: :Transient, zig_name: "LockCycle", id: ERROR_NAME_LOCK_CYCLE, first_site: nil }, Deadlock: { kind: :System, zig_name: "Deadlock", id: ERROR_NAME_DEADLOCK, first_site: nil }, @@ -79,19 +80,32 @@ module AST GuardFail: { kind: :Transient, zig_name: "GuardFail", id: ERROR_NAME_GUARD_FAIL, first_site: nil }, PreconditionFail: { kind: :Input, zig_name: "PreconditionFail", id: ERROR_NAME_PRECONDITION_FAIL, first_site: nil }, OutOfMemory: { kind: :System, zig_name: "OutOfMemory", id: ERROR_NAME_OUT_OF_MEMORY, first_site: nil }, - }, T::Hash[Symbol, T::Hash[Symbol, T.untyped]]) + }.freeze, T::Hash[Symbol, T::Hash[Symbol, T.untyped]]) # Counter for the next user-type id. Reset on a per-program basis via # reset_user_types! (called at the start of each SemanticAnnotator run # so parallel rspec runs don't bleed state). @next_user_id = T.let(ERROR_NAME_USER_FIRST, Integer) - @stdlib_frozen = T.let(ERROR_TYPES.keys.to_set, T::Set[Symbol]) + # The mutable per-program registry is initialized lazily so the native + # self-host has no heap-owning module initializer. + @error_types = T.let(nil, T.nilable(T::Hash[Symbol, T::Hash[Symbol, T.untyped]])) class << self extend T::Sig sig { returns(Integer) } attr_reader :next_user_id - sig { returns(T::Set[Symbol]) } - attr_reader :stdlib_frozen + end + + sig { returns(T::Hash[Symbol, T::Hash[Symbol, T.untyped]]) } + def self.error_types + return @error_types unless @error_types.nil? + + @error_types = BASE_ERROR_TYPES.dup + @error_types + end + + sig { returns(T::Array[Symbol]) } + def self.error_type_names + AST.error_types.keys end sig { params(sym: T.nilable(Symbol)).returns(T::Boolean) } @@ -101,22 +115,22 @@ def self.error_kind?(sym) sig { params(sym: T.nilable(Symbol)).returns(T::Boolean) } def self.error_type?(sym) - sym ? ERROR_TYPES.key?(sym) : false + sym ? AST.error_types.key?(sym) : false end sig { params(sym: Symbol).returns(T.nilable(Symbol)) } def self.kind_of_type(sym) - T.cast(ERROR_TYPES.dig(sym, :kind), T.nilable(Symbol)) + T.cast(AST.error_types.dig(sym, :kind), T.nilable(Symbol)) end sig { params(sym: Symbol).returns(T.nilable(String)) } def self.zig_name_of_type(sym) - T.cast(ERROR_TYPES.dig(sym, :zig_name), T.nilable(String)) + T.cast(AST.error_types.dig(sym, :zig_name), T.nilable(String)) end sig { params(sym: Symbol).returns(T.nilable(Integer)) } def self.id_of_type(sym) - T.cast(ERROR_TYPES.dig(sym, :id), T.nilable(Integer)) + T.cast(AST.error_types.dig(sym, :id), T.nilable(Integer)) end # Register a user-defined type with its kind. First call for a given @@ -130,11 +144,12 @@ def self.id_of_type(sym) # or nil when registration succeeded (or was a no-op re-use). sig { params(type_sym: Symbol, kind_sym: Symbol, site_token: T.nilable(Lexer::Token)).returns([T::Boolean, T.nilable(ErrorTypeConflict)]) } def self.register_type!(type_sym, kind_sym, site_token: nil) - entry = ERROR_TYPES[type_sym] + registry = AST.error_types + entry = registry[type_sym] if entry.nil? id = @next_user_id @next_user_id += 1 - ERROR_TYPES[type_sym] = { + registry[type_sym] = { kind: kind_sym, zig_name: type_sym.to_s, id: id, @@ -147,7 +162,7 @@ def self.register_type!(type_sym, kind_sym, site_token: nil) existing_kind: entry[:kind], given_kind: kind_sym, first_site: entry[:first_site], - is_stdlib: @stdlib_frozen.include?(type_sym), + is_stdlib: BASE_ERROR_TYPES.key?(type_sym), }] end @@ -156,9 +171,7 @@ def self.register_type!(type_sym, kind_sym, site_token: nil) # from one parsed program into the next. Stdlib entries are preserved. sig { returns(Integer) } def self.reset_user_types! - ERROR_TYPES.keys.each do |sym| - ERROR_TYPES.delete(sym) unless @stdlib_frozen.include?(sym) - end + @error_types = BASE_ERROR_TYPES.dup @next_user_id = ERROR_NAME_USER_FIRST end @@ -168,14 +181,21 @@ def self.reset_user_types! # deterministic across runs. sig { returns(T::Array[[Symbol, Integer]]) } def self.enum_entries + entries = AST.error_types.map do |sym, meta| + [T.unsafe(sym), T.cast(meta[:id], Integer)] + end [[T.unsafe(:None), T.unsafe(ERROR_NAME_NONE)]] + - ERROR_TYPES.map { |sym, meta| [T.unsafe(sym), T.cast(meta[:id], Integer)] }.sort_by(&:last) + entries.sort_by { |_, id| id } end # Returns the Array of error-type Symbols whose :kind == kind. Used by # the annotator to expand kind selectors into their member types. sig { params(kind: Symbol).returns(T::Array[Symbol]) } def self.types_for_kind(kind) - ERROR_TYPES.select { |_, meta| meta[:kind] == kind }.keys + types = T.let([], T::Array[Symbol]) + AST.error_types.each do |sym, meta| + types << sym if meta[:kind] == kind + end + types end end diff --git a/compiler/ruby/ast/fixable_error.rb b/compiler/ruby/ast/fixable_error.rb index e84e23f53..199ab0a24 100644 --- a/compiler/ruby/ast/fixable_error.rb +++ b/compiler/ruby/ast/fixable_error.rb @@ -25,6 +25,8 @@ # Synthetic token used for fixable spans whose AST node carries a # line/column but not a lexer token for the exact identifier. AnchorToken = Struct.new(:line, :column) do + # ruby-to-clear: field-type line=Int64 + # ruby-to-clear: field-type column=Int64 extend T::Sig sig { returns(Symbol) } def type; :ANCHOR; end @@ -35,6 +37,7 @@ def value; nil; end TypoToken = T.type_alias { T.any(Lexer::Token, AnchorToken) } # ruby-to-clear: pub +# ruby-to-clear: value class Span extend T::Sig @@ -65,6 +68,7 @@ def to_h end # ruby-to-clear: pub +# ruby-to-clear: value class Edit extend T::Sig @@ -78,6 +82,7 @@ def initialize(span:, replacement:) end # ruby-to-clear: pub +# ruby-to-clear: value class Fix extend T::Sig @@ -98,6 +103,7 @@ def initialize(description:, edits:, confidence: :interactive) end # ruby-to-clear: pub +# ruby-to-clear: value class FixableFinding extend T::Sig @@ -141,18 +147,27 @@ def fatal?; @level == :error; end module FixCollector extend T::Sig - @findings = T.let([], T::Array[FixableFinding]) + # Initialize the owned buffer on first use. Keeping an allocated Array in a + # module binding gives native self-host builds no lifetime in which to free + # it; the nil sentinel is a comptime-safe module value. + @findings = T.let(nil, T.nilable(T::Array[FixableFinding])) + @fatal_count = T.let(0, Integer) @enabled = T.let(false, T::Boolean) @type_migrations_enabled = T.let(false, T::Boolean) @fallibility_propagation_enabled = T.let(false, T::Boolean) sig { returns(T::Array[FixableFinding]) } def self.enable! - @findings.clear + FixCollector.ensure_findings! + buffer = T.let(@findings, T.nilable(T::Array[FixableFinding])) + raise "BUG: fix collector storage was not initialized" if buffer.nil? + buffer.clear + @findings = buffer + @fatal_count = 0 @enabled = true @type_migrations_enabled = false @fallibility_propagation_enabled = false - @findings + FixCollector.findings end sig { void } @@ -177,7 +192,12 @@ def self.fallibility_propagation_enabled? sig { void } def self.disable! - @findings.clear + FixCollector.ensure_findings! + buffer = T.let(@findings, T.nilable(T::Array[FixableFinding])) + raise "BUG: fix collector storage was not initialized" if buffer.nil? + buffer.clear + @findings = buffer + @fatal_count = 0 @enabled = false @type_migrations_enabled = false @fallibility_propagation_enabled = false @@ -188,26 +208,59 @@ def self.enabled? @enabled end - sig { params(finding: FixableFinding).returns(T::Array[FixableFinding]) } - def self.push(finding) - @findings << finding if @enabled - @findings + sig { params(finding: FixableFinding, fatal: T::Boolean).returns(T::Array[FixableFinding]) } + def self.push(finding, fatal = false) + FixCollector.ensure_findings! + buffer = T.let(@findings, T.nilable(T::Array[FixableFinding])) + raise "BUG: fix collector storage was not initialized" if buffer.nil? + if @enabled + buffer << finding + @fatal_count += 1 if fatal + end + @findings = buffer + FixCollector.findings end sig { returns(T::Array[FixableFinding]) } def self.drain - out = @findings.dup - @findings.clear if @enabled + FixCollector.ensure_findings! + buffer = T.let(@findings, T.nilable(T::Array[FixableFinding])) + raise "BUG: fix collector storage was not initialized" if buffer.nil? + out = buffer.dup + buffer.clear if @enabled + @findings = buffer out end sig { returns(T::Boolean) } def self.has_fatal? - @enabled && @findings.any?(&:fatal?) + FixCollector.ensure_findings! + buffer = T.let(@findings, T.nilable(T::Array[FixableFinding])) + raise "BUG: fix collector storage was not initialized" if buffer.nil? + @enabled && @fatal_count.positive? end sig { returns(Integer) } def self.fatal_count - @enabled ? @findings.count(&:fatal?) : 0 + FixCollector.ensure_findings! + buffer = T.let(@findings, T.nilable(T::Array[FixableFinding])) + raise "BUG: fix collector storage was not initialized" if buffer.nil? + @enabled ? @fatal_count : 0 + end + + sig { void } + def self.ensure_findings! + return unless @findings.nil? + + @findings = [] + end + + sig { returns(T::Array[FixableFinding]) } + def self.findings + FixCollector.ensure_findings! + buffer = T.let(@findings, T.nilable(T::Array[FixableFinding])) + raise "BUG: fix collector storage was not initialized" if buffer.nil? + out = buffer.dup + out end end diff --git a/compiler/ruby/ast/frontend_resource_budget.rb b/compiler/ruby/ast/frontend_resource_budget.rb index 98b936570..ee049d886 100644 --- a/compiler/ruby/ast/frontend_resource_budget.rb +++ b/compiler/ruby/ast/frontend_resource_budget.rb @@ -54,7 +54,20 @@ def source_violation(source) sig { params(count: Integer).void } def check_tokens!(count) - raise Exceeded.new(:tokens, @max_tokens) if count > @max_tokens + violation = tokens_violation(count) + raise violation if violation + end + + sig { params(count: Integer).returns(T.nilable(Exceeded)) } + def tokens_violation(count) + return Exceeded.new(:tokens, @max_tokens) if count > @max_tokens + + nil + end + + sig { returns(Integer) } + def nesting_limit + @max_nesting end sig { void } diff --git a/compiler/ruby/ast/lexer.rb b/compiler/ruby/ast/lexer.rb index 55432fdb4..784e88b15 100644 --- a/compiler/ruby/ast/lexer.rb +++ b/compiler/ruby/ast/lexer.rb @@ -60,6 +60,16 @@ def float! raise TokenPayloadError, payload_error("float", "Float") end + sig { returns(String) } + def display_value + payload = value + return payload if payload.is_a?(String) + return payload.to_s if payload.is_a?(Integer) + return payload.to_s if payload.is_a?(Float) + + "" + end + sig { returns(Integer) } def start_line = line @@ -100,16 +110,16 @@ def payload_error(accessor, expected_class) # We use a hash for O(1) lookups KEYWORDS = T.let(%w[ MUTABLE - FN METHOD RETURN RETURNS USE + FN METHOD RETURN RETURNS USE CONST IF THEN ELSE ELSE_IF END COMPTIME IS_A EXISTS IS_OK IS_READY WHILE DO FOR IN BG NEXT BREAK CONTINUE CAST AS TRY UNWRAP STRUCT ENUM UNION PROTOCOL IMPLEMENTATION TRUE FALSE NIL Auto - ASSERT RAISE CATCH EXIT DIE PASS PRUNE + ASSERT RAISE CATCH EXIT DIE PASS PRUNE DEFER MOD AND OR OR_ELSE XOR BIT_AND BIT_OR REQUIRE SELECT WHERE INDEX REDUCE ORDER_BY LIMIT SKIP UNNEST DISTINCT EACH TAP FIND ANY ALL COUNT SUM AVERAGE MIN MAX CONCURRENT SHARD TAKE_WHILE WINDOW JOIN RECOVER COLLECT - GIVE TAKES COPY MOVE CLONE SHARE LINK RESOLVE FREEZE + GIVE TAKES COPY KEEP MOVE SHARE LINK RESOLVE FREEZE UNIQUE MONOMORPHIC OWN WITH EXCLUSIVE RESTRICT BORROWED ON RETRY POSSIBLE_DEADLOCK POSSIBLE_LOCK_CYCLE VIEW MATERIALIZED UNSAFE LENGTH SNAPSHOT GUARD PRE DEBUG_POST POLYMORPHIC SHARED SYNC POLICY REQUIRES @@ -223,25 +233,25 @@ def tokenize # `0xff_u32` is hex + suffix. The suffix-bearing regex runs before # the plain form so the suffix is captured when present. when @s.scan(/0x[0-9a-fA-F]+(?:_[0-9a-fA-F]+)*_(#{NUMERIC_SUFFIX_RE})\b/o) - hex_value = strip_digit_separators(@s.matched, @s[1]).to_i(16) + hex_value = strip_digit_separators(@s.matched, @s[1]).delete_prefix('0x').to_i(16) add_prefixed_int(hex_value, @s[1], start_col) when @s.scan(/0x[0-9a-fA-F]+(?:_[0-9a-fA-F]+)*/) - add(:PREFIXED_INT, @s.matched.tr('_', '').to_i(16), start_col) + add(:PREFIXED_INT, check_based_literal(@s.matched.tr('_', '').delete_prefix('0x').to_i(16)), start_col) when @s.scan(/0o[0-7]+(?:_[0-7]+)*_(#{NUMERIC_SUFFIX_RE})\b/o) - octal_value = strip_digit_separators(@s.matched, @s[1]).to_i(8) + octal_value = strip_digit_separators(@s.matched, @s[1]).delete_prefix('0o').to_i(8) add_prefixed_int(octal_value, @s[1], start_col) when @s.scan(/0o[0-7]+(?:_[0-7]+)*/) - add(:PREFIXED_INT, @s.matched.tr('_', '').to_i(8), start_col) + add(:PREFIXED_INT, check_based_literal(@s.matched.tr('_', '').delete_prefix('0o').to_i(8)), start_col) when @s.scan(/0b[0-1]+(?:_[0-1]+)*_(#{NUMERIC_SUFFIX_RE})\b/o) - binary_value = strip_digit_separators(@s.matched, @s[1]).to_i(2) + binary_value = strip_digit_separators(@s.matched, @s[1]).delete_prefix('0b').to_i(2) add_prefixed_int(binary_value, @s[1], start_col) when @s.scan(/0b[0-1]+(?:_[0-1]+)*/) - add(:PREFIXED_INT, @s.matched.tr('_', '').to_i(2), start_col) + add(:PREFIXED_INT, check_based_literal(@s.matched.tr('_', '').delete_prefix('0b').to_i(2)), start_col) when @s.scan(/\d+(?:_\d+)*\.\d+(?:_\d+)*_(#{NUMERIC_SUFFIX_RE})\b/o) float_value = strip_digit_separators(@s.matched, @s[1]).to_f @@ -257,10 +267,14 @@ def tokenize when @s.scan(/\d+(?:_\d+)*_(#{NUMERIC_SUFFIX_RE})\b/o) body = strip_digit_separators(@s.matched, @s[1]) - add_prefixed_int(body.to_i, @s[1], start_col) + add_prefixed_int(body.to_i(10), @s[1], start_col) when @s.scan(/\d+(?:_\d+)*/) - add(:INT64, @s.matched.tr('_', '').to_i, start_col) + decimal_value = @s.matched.tr('_', '').to_i + if decimal_value > 9_223_372_036_854_775_807 + raise Error, "Lexer Error: Literal #{@s.matched} overflows i64 (range #{integer_suffix_range_text('i64')})" + end + add(:INT64, decimal_value, start_col) when @s.scan(/"/) advance_pos(@s.matched) # Advance past the opening quote @@ -519,6 +533,11 @@ def advance_pos(str) # Closed set of numeric type suffixes. Matches a word boundary so the # suffix can't absorb a following identifier. + # Lexer integer literals are never negative; their full domain is u64 + # (self-hosted translation types TokenInt as UInt64 via the helper + # config; Ruby keeps its arbitrary-precision Integer). + TokenInt = T.type_alias { Integer } + NUMERIC_SUFFIX_RE = T.let(/i8|i16|i32|i64|u8|u16|u32|u64|f32|f64/.freeze, Regexp) # Strip digit-group separators (underscores) from a numeric literal's @@ -531,7 +550,7 @@ def strip_digit_separators(matched, suffix) body.tr('_', '') end - sig { params(val: Integer, suffix: String, start_col: Integer).returns(T.nilable(Integer)) } + sig { params(val: TokenInt, suffix: String, start_col: Integer).returns(T.nilable(Integer)) } def add_prefixed_int(val, suffix, start_col) unless numeric_suffix?(suffix) raise Error, "Lexer Error: Unknown numeric suffix '_#{suffix}' at line #{@line}:#{@column}" @@ -563,21 +582,38 @@ def integer_suffix?(suffix) %w[u8 i8 i16 u16 i32 u32 i64 u64].include?(suffix) end - sig { params(suffix: String, value: Integer).returns(T::Boolean) } + sig { params(suffix: String, value: TokenInt).returns(T::Boolean) } def integer_suffix_contains?(suffix, value) + # TokenInt is non-negative by construction (a literal carries no sign; + # minus is an operator), so only upper bounds apply. This keeps the + # checks valid in the self-hosted UInt64 domain. case suffix - when 'u8' then value >= 0 && value <= 255 - when 'i8' then value >= -128 && value <= 127 - when 'i16' then value >= -32_768 && value <= 32_767 - when 'u16' then value >= 0 && value <= 65_535 - when 'i32' then value >= -2_147_483_648 && value <= 2_147_483_647 - when 'u32' then value >= 0 && value <= 4_294_967_295 - when 'i64' then value >= -9_223_372_036_854_775_808 && value <= 9_223_372_036_854_775_807 - when 'u64' then value >= 0 && value.bit_length <= 64 + when 'u8' then value <= 255 + when 'i8' then value <= 127 + when 'i16' then value <= 32_767 + when 'u16' then value <= 65_535 + when 'i32' then value <= 2_147_483_647 + when 'u32' then value <= 4_294_967_295 + when 'i64' then value <= 9_223_372_036_854_775_807 + # Sorbet cannot parse literals above i64 max, so "fits in u64" is + # spelled as a shift: (value >> 63) >> 1 is zero iff value < 2**64. + # The split shift keeps the u64 domain defined (a single >> 64 would + # be an over-wide shift there). + when 'u64' then ((value >> 63) >> 1) == 0 else false end end + # Bare based literals (0x/0o/0b, no suffix) live in the u64 token domain; + # the self-hosted parse rejects larger digits at the parseUInt boundary. + sig { params(value: TokenInt).returns(TokenInt) } + def check_based_literal(value) + if ((value >> 63) >> 1) != 0 + raise Error, "Lexer Error: Literal #{@s.matched} overflows u64 (range #{integer_suffix_range_text('u64')})" + end + value + end + sig { params(suffix: String).returns(String) } def integer_suffix_range_text(suffix) case suffix diff --git a/compiler/ruby/ast/param.rb b/compiler/ruby/ast/param.rb index 9d27dfca4..2417d9f48 100644 --- a/compiler/ruby/ast/param.rb +++ b/compiler/ruby/ast/param.rb @@ -4,20 +4,70 @@ require "sorbet-runtime" require_relative "type" +# Retained identity v4 (docs/agents/retained-identity-design.md). +# +# KeptIdentityContract is the retained-parameter fact: which ownership +# family the sink demands and which sink created the keep. Stamped on the +# param's SymbolEntry by keep-analysis; consumed by signature lowering +# (handle ABI) and edge planning. Never a bare boolean: the family is what +# downstream code dispatches on, and the sink names the keeper in +# declaration-sited diagnostics. +class KeptIdentityContract < T::Struct + extend T::Sig + + # :multiowned (Rc) today; :shared (Arc) and :value (independent copy) + # are reserved for the B/C phases of the design. + const :family, Symbol + const :sink, String + + sig { returns(T::Boolean) } + def rc? = family == :multiowned +end + +# One call edge's ownership decision, produced by placement (the single +# writer, which sees the caller's declared model, the callee's contract, +# and caller liveness) and consumed by MIR lowering without re-derivation. +# +# op is exactly one of: +# :retain_handle - live Rc source; the edge retains (+1) +# :move_handle - last use or GIVE; the handle bits move +# :move_payload_wrap - owned payload expression; move it and rcCreate +# :copy_wrap - COPY override; deep-copy payload and rcCreate +# :pass_null - omitted/NIL optional edge +class CallEdgeOwnershipPlan < T::Struct + const :op, Symbol + const :family, Symbol +end + + module AST StructKwargs = T.type_alias { BasicObject } # ruby-to-clear: pub Param = Struct.new(:name, :type, :default, :mutable, :takes, :comptime, :name_token, :required, :sync, :symbol, + :carrier_contract, keyword_init: true) do extend T::Sig + # Ruby Struct members default to nil, while Sorbet cannot attach a type to + # the generated readers. Keep the self-hosted record faithful to that + # contract instead of collapsing these boolean fields to Any. + # ruby-to-clear: field-type mutable=?Bool + # ruby-to-clear: field-type takes=?Bool + # ruby-to-clear: field-type comptime=?Bool + # ruby-to-clear: field-type required=?Bool + sig { params(kw: StructKwargs).void } def initialize(**kw) super t = T.let(self[:type], T.nilable(Type::TypeInput)) self[:type] = Type.new(t || Type.type_input_symbol_or_any(nil)) + # Retained-identity v5 parameter carrier contract: :polymorphic + # (default, carrier-preserving), :unique (exclusively owned), :shared + # (requires a retained-identity family). + contract = T.let(self[:carrier_contract], T.nilable(Symbol)) + self[:carrier_contract] = contract || :polymorphic end # Mirror of Type#atomic? (Param has :sync but no :layout, so no diff --git a/compiler/ruby/ast/parser.rb b/compiler/ruby/ast/parser.rb index c58298d8d..16d274919 100644 --- a/compiler/ruby/ast/parser.rb +++ b/compiler/ruby/ast/parser.rb @@ -3,9 +3,8 @@ require_relative "./ast" require_relative "./param" -require_relative "./schemas" -require_relative "./struct_field" require_relative "./type" +require_relative "./struct_field" require_relative "./lexer" require_relative "./parser_rules" require_relative "./error_registry" @@ -21,7 +20,10 @@ class ClearParser extend T::Sig + # Reopened parser classes in required files use these constants. + # ruby-to-clear: data-api OPEN_DELIMITERS = T.let('([{'.freeze, String) + # ruby-to-clear: data-api CLOSE_DELIMITERS = T.let(')]}'.freeze, String) class CapabilityParseResult < T::Struct @@ -250,8 +252,32 @@ def parse_type_syntax_document sig { returns(AST::Program) } def parse - @budget.check_source!(@source_code) - @budget.check_tokens!(@tokens.length) + parse_with_budget + rescue SystemStackError + raise ParserError.new(current, "Frontend nesting resource limit exceeded", @source_code) + end + + sig { returns(AST::Program) } + def parse_with_budget + source_violation = @budget.source_violation(@source_code) + if source_violation + raise ParserError.new(current, "Frontend #{source_violation.kind} resource limit exceeded (limit #{source_violation.limit})", @source_code) + end + token_violation = @budget.tokens_violation(@tokens.length) + if token_violation + raise ParserError.new(current, "Frontend #{token_violation.kind} resource limit exceeded (limit #{token_violation.limit})", @source_code) + end + parse_program + rescue FrontendResourceBudget::Exceeded + raise ParserError.new( + current, + "Frontend nesting resource limit exceeded (limit #{@budget.nesting_limit})", + @source_code, + ) + end + + sig { returns(AST::Program) } + def parse_program stmts = [] stmts << parse_statement() while current.type != :EOF program = AST::Program.new(current, stmts) @@ -259,10 +285,6 @@ def parse stamp_source_range!(program, first, current) program.language_mode = @gradual ? :easy : self.class.ownership_mode program - rescue FrontendResourceBudget::Exceeded => e - raise ParserError.new(current, "Frontend #{e.kind} resource limit exceeded (limit #{e.limit})", @source_code) - rescue SystemStackError - raise ParserError.new(current, "Frontend nesting resource limit exceeded", @source_code) end private @@ -271,6 +293,7 @@ def parse rule(:KEYWORD, 'REQUIRE', action: :parse_require), rule(:KEYWORD, 'EXTERN', action: :parse_extern_decl), rule(:KEYWORD, 'MUTABLE', action: :parse_mutable_var_decl), + rule(:KEYWORD, 'CONST', action: :parse_const_decl), rule(:KEYWORD, 'FN', action: :parse_function_def), rule(:KEYWORD, 'METHOD', action: :parse_method_function_def), rule(:KEYWORD, 'PUB', action: :parse_pub_visibility), @@ -287,6 +310,7 @@ def parse rule(:KEYWORD, 'TIGHT', action: :parse_tight_stmt), rule(:KEYWORD, 'RETURN', action: :parse_return), rule(:KEYWORD, 'ASSERT', action: :parse_assert), + rule(:KEYWORD, 'DEFER', action: :parse_defer), rule(:KEYWORD, 'ASSERT_RAISES', action: :parse_assert_raises), rule(:KEYWORD, 'TEST', action: :parse_test_block), rule(:KEYWORD, 'STUB', action: :parse_stub), @@ -337,7 +361,8 @@ def parse rule(:KEYWORD, 'MOVE', action: :parse_move_node), rule(:KEYWORD, 'GIVE', action: :parse_move_node), rule(:KEYWORD, 'COPY', action: :parse_copy_node), - rule(:KEYWORD, 'CLONE', action: :parse_clone_node), + rule(:KEYWORD, 'OWN', action: :parse_own_node), + rule(:KEYWORD, 'KEEP', action: :parse_keep_node), rule(:KEYWORD, 'SHARE', action: :parse_share_node), rule(:KEYWORD, 'LINK', action: :parse_link_node), rule(:KEYWORD, 'RESOLVE', action: :parse_resolve_node), diff --git a/compiler/ruby/ast/parser/declarations_and_definitions.rb b/compiler/ruby/ast/parser/declarations_and_definitions.rb index 0523faee8..4afdc3ac5 100644 --- a/compiler/ruby/ast/parser/declarations_and_definitions.rb +++ b/compiler/ruby/ast/parser/declarations_and_definitions.rb @@ -26,6 +26,7 @@ def parse_argument_specs p_name = name_tok.text! if name_tok p_type = T.let(nil, T.nilable(ArgumentType)) default_val = nil + carrier_contract = T.let(:polymorphic, Symbol) if is_comptime consume(:CHAR, ':') @@ -33,7 +34,14 @@ def parse_argument_specs p_name = "comptime" else if match!(:CHAR, ':') + carrier_contract = parse_carrier_contract p_type = parse_type_annotation + # The existing `SHARED T` spelling produces a polymorphic-shared + # type; that IS the v5 :shared parameter contract (requires a + # retained-identity family). Derive it rather than intercepting + # the SHARED keyword, which belongs to the type annotation. + carrier_contract = :shared if carrier_contract == :polymorphic && + p_type.is_a?(Type) && p_type.polymorphic_shared? default_val = parse_expression if match!(:CHAR, '=') elsif match!(:CHAR, '=') # Compatibility with the original spelling, `name=default: Type`. @@ -49,17 +57,32 @@ def parse_argument_specs AST::Capture.new(name: p_name, type: p_type, default: default_val, mutable: is_mutable, takes: takes, - comptime: is_comptime, name_token: name_tok) + comptime: is_comptime, name_token: name_tok, + carrier_contract: carrier_contract) end .last # always ignore the first token end + # Retained-identity v5: an optional UNIQUE keyword between the `:` and the + # parameter type constrains the carrier to exclusively-owned. (The :shared + # contract is spelled `SHARED T`, which the type annotation already parses + # as a polymorphic-shared type; the caller derives :shared from that.) + # Absence is the carrier-polymorphic default. + sig { returns(Symbol) } + def parse_carrier_contract + return :unique if match!(:KEYWORD, 'UNIQUE') + return :monomorphic if match!(:KEYWORD, 'MONOMORPHIC') + + :polymorphic + end + sig { returns(T::Array[AST::Param]) } def parse_argument_list parse_argument_specs.map do |spec| AST::Param.new(name: spec.name, type: spec.type, default: spec.default, mutable: spec.mutable, takes: spec.takes, - comptime: spec.comptime, name_token: spec.name_token) + comptime: spec.comptime, name_token: spec.name_token, + carrier_contract: spec.carrier_contract) end end @@ -102,6 +125,38 @@ def parse_mutable_var_decl error!(start_token, :MUTABLE_BARE_NEEDS_TYPE) end + # CONST NAME: Type = expr; + # A top-level immutable, comptime-initialized binding emitted at Zig container + # scope. Unlike a local, a CONST requires both an explicit type and an + # initializer: it has no enclosing frame to infer from or default into, and + # its value must be comptime-known (a heap-owning initializer is rejected + # downstream by MODULE_SCOPE_OWNED_VALUE). + sig { params(visibility: Symbol).returns(AST::VarDecl) } + def parse_const_decl(visibility = :package) + start_token = consume(:KEYWORD, 'CONST') + # A CONST name is SCREAMING_CASE, so it lexes as a TYPE_ID (leading capital). + # Enforce all-caps to keep constants visually distinct from types at every + # reference site (a bare all-caps identifier reads as "constant"). + name_tok = match?(:TYPE_ID) ? consume(:TYPE_ID) : consume(:VAR_ID) + name = name_tok.text! + unless name.match?(/\A[A-Z][A-Z0-9_]*\z/) + error!(name_tok, :CONST_NEEDS_CAPS, name: name) + end + unless match!(:CHAR, ':') + error!(start_token, :CONST_NEEDS_TYPE, name: name) + end + type_annotation = parse_inferred_wrapper_annotation || parse_type_annotation + unless match!(:CHAR, '=') + error!(start_token, :CONST_NEEDS_VALUE, name: name) + end + value = parse_expression + consume(:CHAR, ';') + decl = AST::VarDecl.new(start_token, name, type_annotation, value, false) + decl.module_const = true + decl.const_visibility = visibility + decl + end + # Build a compact default-initialized AST value for a `T[N]` annotation. # Used by `parse_mutable_var_decl` when no `= expr` was given. Restricted to # fixed-size raw arrays of element types with an obvious zero (primitives @@ -164,6 +219,8 @@ def parse_visibility_decl(visibility) parse_enum_def(visibility) elsif match?(:KEYWORD, 'UNION') parse_union_def(visibility) + elsif match?(:KEYWORD, 'CONST') + parse_const_decl(visibility) else error!(current, :VISIBILITY_BAD_KIND, got: current.value) end diff --git a/compiler/ruby/ast/parser/expressions_and_postfix.rb b/compiler/ruby/ast/parser/expressions_and_postfix.rb index f7ec8cd8c..d950663de 100644 --- a/compiler/ruby/ast/parser/expressions_and_postfix.rb +++ b/compiler/ruby/ast/parser/expressions_and_postfix.rb @@ -19,13 +19,14 @@ def dispatch_primary_rule(rule) when :parse_nil_literal then parse_nil_literal when :parse_default_literal then parse_default_literal when :parse_cast then parse_cast - when :parse_move_node then AST::MoveNode.new(consume(:KEYWORD), parse_expression) - when :parse_copy_node then AST::CopyNode.new(consume(:KEYWORD, 'COPY'), parse_expression) - when :parse_clone_node then AST::CloneNode.new(consume(:KEYWORD, 'CLONE'), parse_expression) - when :parse_share_node then AST::ShareNode.new(consume(:KEYWORD, 'SHARE'), parse_expression) - when :parse_link_node then AST::LinkNode.new(consume(:KEYWORD, 'LINK'), parse_expression) - when :parse_resolve_node then AST::ResolveNode.new(consume(:KEYWORD, 'RESOLVE'), parse_expression) - when :parse_freeze_node then AST::FreezeNode.new(consume(:KEYWORD, 'FREEZE'), parse_expression) + when :parse_move_node then AST::MoveNode.new(consume(:KEYWORD), parse_expression(wrapper_operand_precedence)) + when :parse_copy_node then AST::CopyNode.new(consume(:KEYWORD, 'COPY'), parse_expression(wrapper_operand_precedence)) + when :parse_own_node then parse_own_node + when :parse_keep_node then AST::KeepNode.new(consume(:KEYWORD, 'KEEP'), parse_expression(wrapper_operand_precedence)) + when :parse_share_node then AST::ShareNode.new(consume(:KEYWORD, 'SHARE'), parse_expression(wrapper_operand_precedence)) + when :parse_link_node then AST::LinkNode.new(consume(:KEYWORD, 'LINK'), parse_expression(wrapper_operand_precedence)) + when :parse_resolve_node then AST::ResolveNode.new(consume(:KEYWORD, 'RESOLVE'), parse_expression(wrapper_operand_precedence)) + when :parse_freeze_node then AST::FreezeNode.new(consume(:KEYWORD, 'FREEZE'), parse_expression(wrapper_operand_precedence)) when :parse_bg_block then parse_bg_block when :parse_next_expr then parse_next_expr when :parse_sigil_construct then parse_sigil_construct @@ -65,6 +66,21 @@ def dispatch_primary_rule(rule) T.must(result) end + # OWN COPY x -- the sole handle->owned-RawT downgrade: deref the retained + # payload and deep-copy it into a fresh uniquely-owned value. Modeled as a + # CopyNode with `own` set, reusing the payload-detach lowering. Bare `OWN x` + # (extract/move the payload out of a handle) is deferred, so OWN currently + # requires a following COPY. + sig { returns(AST::CopyNode) } + def parse_own_node + token = consume(:KEYWORD, 'OWN') + error!(token, :OWN_ALONE_UNSUPPORTED) unless match?(:KEYWORD, 'COPY') + consume(:KEYWORD, 'COPY') + node = AST::CopyNode.new(token, parse_expression(wrapper_operand_precedence)) + node.own = true + node + end + sig { returns(AST::SelectOp) } def parse_select_op token = consume(:KEYWORD, 'SELECT') @@ -145,8 +161,13 @@ def parse_partial_match_expr sig { returns(AST::Literal) } def parse_symbol_literal colon_tok = consume(:CHAR, ':') - error!(colon_tok, :EXPECTED_SYMBOL_AFTER_COLON) unless match?(:VAR_ID) || match?(:TYPE_ID) - ident_tok = current.type == :TYPE_ID ? consume(:TYPE_ID) : consume(:VAR_ID) + # After ':' the word is a symbol NAME, so a keyword spelling like :EXISTS + # or :RETURN is unambiguous - there is no expression position here for the + # keyword to mean anything else. + unless match?(:VAR_ID) || match?(:TYPE_ID) || match?(:KEYWORD) + error!(colon_tok, :EXPECTED_SYMBOL_AFTER_COLON) + end + ident_tok = consume(current.type) AST::Literal.new(colon_tok, :SYMBOL, ident_tok.text!, :stack) end @@ -472,9 +493,22 @@ def literal_token_type(val) sig { params(precedence: Integer).returns(AST::Node) } def parse_expression(precedence = 0) @budget.enter! + # Value wrappers (COPY/MOVE/KEEP/...) parse their operand at the AMBIENT + # minimum precedence: inside a pipeline-stage element (parsed at 1) the + # operand stops before |>, while at statement level (parsed at 0) + # `COPY xs |> SELECT ...` keeps wrapping the whole pipeline. + prev_ambient = T.let(@wrapper_operand_precedence, T.nilable(Integer)) + @wrapper_operand_precedence = T.let(precedence, T.nilable(Integer)) expression = parse_expression_body(precedence) @budget.leave! expression + ensure + @wrapper_operand_precedence = prev_ambient + end + + sig { returns(Integer) } + def wrapper_operand_precedence + @wrapper_operand_precedence || 0 end sig { params(precedence: Integer).returns(AST::Node) } diff --git a/compiler/ruby/ast/parser/predicates_and_refinements.rb b/compiler/ruby/ast/parser/predicates_and_refinements.rb index 169a108eb..d43f11aa2 100644 --- a/compiler/ruby/ast/parser/predicates_and_refinements.rb +++ b/compiler/ruby/ast/parser/predicates_and_refinements.rb @@ -1,5 +1,7 @@ # typed: strict +require_relative "state" + class ClearParser extend T::Sig @@ -126,8 +128,8 @@ def conditional_capture_ahead? token = peek_at(offset) return false unless token if token.type == :CHAR - depth += 1 if OPEN_DELIMITERS.include?(token.text!) - depth -= 1 if CLOSE_DELIMITERS.include?(token.text!) + depth += 1 if "([{".include?(token.text!) + depth -= 1 if ")]}".include?(token.text!) end return false if depth == 0 && ((token.type == :KEYWORD && %w[THEN ELSE END].include?(token.value)) || token.type == :ARROW || token.type == :EOF) if token.type == :KEYWORD && %w[EXISTS IS_OK].include?(token.value) diff --git a/compiler/ruby/ast/parser/state.rb b/compiler/ruby/ast/parser/state.rb index 3bbec52df..3fe117fcf 100644 --- a/compiler/ruby/ast/parser/state.rb +++ b/compiler/ruby/ast/parser/state.rb @@ -1,8 +1,27 @@ # typed: strict +require_relative "../source_error" class ClearParser extend T::Sig + sig { returns(T::Boolean) } + def parser_error_host? + true + end + + include ErrorHelper + + SYNTAX_TOKENS_AT_STATEMENT_END = T.let( + %w[; THEN DO ->].freeze, + T::Array[String], + ) + + # Partial-class files are compiled as separate CLEAR packages during + # self-hosting, so restate the storage types they read from parser.rb. + # ruby-to-clear: field-type pos=Int64 + # ruby-to-clear: field-type source_code=String + # ruby-to-clear: field-type tokens=[]Token + private # Pair structural delimiters once so token lookahead can skip nested forms in @@ -18,12 +37,12 @@ def index_delimiter_closings(tokens) closings << nil next unless token.type == :CHAR value = token.text! - if OPEN_DELIMITERS.include?(value) + if "([{".include?(value) opening_chars << value opening_indices << index elsif !opening_chars.empty? opening = T.must(opening_chars.last) - closing = CLOSE_DELIMITERS[T.must(OPEN_DELIMITERS.index(opening))] + closing = ")]}"[T.must("([{".index(opening))] next unless closing == value opening_chars.pop opening_index = opening_indices.pop @@ -35,7 +54,10 @@ def index_delimiter_closings(tokens) sig { returns(Lexer::Token) } def peek - @tokens[@pos + 1] || Lexer::Token.new(:EOF, "", current.line, current.column) + token = T.let(@tokens[@pos + 1], T.nilable(Lexer::Token)) + return token if token + + Lexer::Token.new(:EOF, "", current.line, current.column) end sig { params(n: Integer).returns(T.nilable(Lexer::Token)) } @@ -47,12 +69,12 @@ def peek_at(n) sig { returns(Lexer::Token) } def current - T.must(@tokens[@pos]) + @tokens.fetch(@pos) end sig { returns(Lexer::Token) } def previous - T.must(@tokens[@pos-1]) + @tokens.fetch(@pos - 1) end # Consume a numeric literal (either :NUMBER float or :INT64 integer). @@ -71,9 +93,16 @@ def consume_number def consume(type, value=nil) # Return the consumed token rather than `current`, which advances to the next token. token = current + matches_value = T.let(false, T::Boolean) + if value + expected_value = value + if token.value.is_a?(String) + matches_value = token.text! == expected_value + end + end - if (token.type == type) || (value && token.value == value) - if value && token.value != value + if (token.type == type) || matches_value + if value && !matches_value emit_consume_error_with_fix(token, type, value) end @@ -102,7 +131,10 @@ def consume(type, value=nil) sig { params(token: Lexer::Token, expected_type: Symbol, expected_value: T.nilable(String)).returns(T.noreturn) } def emit_consume_error_with_fix(token, expected_type, expected_value) - prev_tok = @pos > 0 ? @tokens[@pos - 1] : nil + prev_tok = T.let(nil, T.nilable(Lexer::Token)) + if @pos > 0 + prev_tok = @tokens[@pos - 1] + end if expected_value && SYNTAX_TOKENS_AT_STATEMENT_END.include?(expected_value) && prev_tok if prev_tok.line < token.line @@ -113,6 +145,14 @@ def emit_consume_error_with_fix(token, expected_type, expected_value) end end + # Retired bang-mutation syntax: `name!` fails whatever the parser expected + # next, but the useful diagnostic is the registered migration, not the raw + # token expectation. + if token.type == :CHAR && token.text! == '!' && prev_tok && + %i[VAR_ID TYPE_ID].include?(prev_tok.type) && prev_tok.end_offset == token.start_offset + error!(token, :LEGACY_MUTATION_NAME_SUFFIX) + end + error!(token, :PARSER_EXPECTED, expected: expected_value || expected_type, got: token.value, type: token.type, line: token.line) end @@ -150,7 +190,7 @@ def emit_syntax_insert_end_of_line!(prev_tok, next_tok, expected_value) sig { params(token: Lexer::Token, expected_value: String).returns(T.noreturn) } def emit_syntax_insert_before_token!(token, expected_value) fix = Fix.new( - description: fix_description(:INSERT_EXPECTED_BEFORE_TOKEN, expected: expected_value, got: token.value, line: token.line), + description: fix_description(:INSERT_EXPECTED_BEFORE_TOKEN, expected: expected_value, got: token.display_value, line: token.line), confidence: :auto, edits: [Edit.new( span: Span.new(file: nil, line: token.line, col: token.column, length: 0), @@ -168,7 +208,12 @@ def emit_syntax_insert_before_token!(token, expected_value) sig { params(type: Symbol, val: T.nilable(String)).returns(T::Boolean) } def match?(type, val=nil) - current.type == type && (val.nil? || current.value == val) + token = current + return false unless token.type == type + return true if val.nil? + + expected_value = val + token.text! == expected_value end # `>>` is a shift in expression context, but it is also two adjacent generic @@ -201,7 +246,11 @@ def consume_generic_close def match_at?(n, type, val=nil) tok = peek_at(n) return false unless tok - tok.type == type && (val.nil? || tok.value == val) + return false unless tok.type == type + return true if val.nil? + + expected_value = val + tok.text! == expected_value end # Used by `parse_match_*` to decide whether the `,` at `current` is a @@ -215,8 +264,8 @@ def multi_pattern_continues? nxt = peek_at(1) return false unless nxt return false if nxt.type == :ARROW || nxt.type == :EOF - return false if nxt.type == :KEYWORD && %w[AS WHEN DEFAULT END].include?(nxt.value) - return false if nxt.type == :CHAR && nxt.value == '{' + return false if nxt.type == :KEYWORD && %w[AS WHEN DEFAULT END].include?(nxt.text!) + return false if nxt.type == :CHAR && nxt.text! == '{' true end @@ -234,7 +283,7 @@ def match!(type, value=nil) sig { params(node: AST::Node, first: Lexer::Token, last: Lexer::Token).returns(AST::Node) } def stamp_source_range!(node, first, last) start_offset = first.start_offset || 0 - end_offset = last.end_offset || (start_offset + last.value.to_s.bytesize) + end_offset = last.end_offset || (start_offset + last.display_value.bytesize) node.source_range = AST::SourceRange.new( file: first.file || last.file, start_offset: start_offset, @@ -253,7 +302,9 @@ def stamp_source_range!(node, first, last) # need the latter. sig { params(node: AST::Node, first: AST::Locatable, last: Lexer::Token).returns(AST::Node) } def stamp_source_range_from_node!(node, first, last) - range = first.source_range + source_range = first.source_range + raise "Internal: source range missing from postfix receiver" unless source_range + range = source_range end_offset = last.end_offset || ((last.start_offset || range.end_offset) + last.value.to_s.bytesize) node.source_range = AST::SourceRange.new( file: range.file || last.file, diff --git a/compiler/ruby/ast/parser/statements_and_control_flow.rb b/compiler/ruby/ast/parser/statements_and_control_flow.rb index a26686bd6..347466136 100644 --- a/compiler/ruby/ast/parser/statements_and_control_flow.rb +++ b/compiler/ruby/ast/parser/statements_and_control_flow.rb @@ -11,6 +11,7 @@ def dispatch_stmt_rule(rule) when :parse_require then parse_require when :parse_extern_decl then parse_extern_decl when :parse_mutable_var_decl then parse_mutable_var_decl + when :parse_const_decl then parse_const_decl when :parse_function_def then parse_function_def when :parse_method_function_def then parse_function_def(:package, is_method: true) when :parse_pub_visibility then parse_visibility_decl(:pub) @@ -27,6 +28,7 @@ def dispatch_stmt_rule(rule) when :parse_tight_stmt then parse_tight_stmt when :parse_return then parse_return when :parse_assert then parse_assert + when :parse_defer then parse_defer when :parse_assert_raises then parse_assert_raises when :parse_test_block then parse_test_block when :parse_stub then parse_stub @@ -65,6 +67,53 @@ def parse_assert AST::Assert.new(token, condition, message) end + # DEFER | DEFER { stmt* } + # The body runs at scope exit on BOTH the success and error paths (Zig + # defer semantics — compile-time control flow, zero runtime cost). + # RETURN/BREAK/CONTINUE/YIELD cannot appear inside a DEFER body: the + # deferred code runs during scope teardown, where redirecting control + # flow is meaningless (same restriction as Zig). + sig { returns(AST::DeferStmt) } + def parse_defer + token = consume(:KEYWORD, 'DEFER') + body = if match?(:CHAR, '{') + consume(:CHAR, '{') + stmts = T.let([], T::Array[AST::Node]) + stmts << parse_statement until match?(:CHAR, '}') + consume(:CHAR, '}') + stmts + else + [parse_statement] + end + reject_defer_control_flow!(token, body) + AST::DeferStmt.new(token, body) + end + + sig { params(token: Lexer::Token, body: T::Array[AST::Node]).void } + def reject_defer_control_flow!(token, body) + stack = T.let(body.dup, T::Array[AST::Node]) + until stack.empty? + node = stack.pop + next unless node.is_a?(AST::Locatable) + if node.is_a?(AST::ReturnNode) || node.is_a?(AST::BreakNode) || + node.is_a?(AST::ContinueNode) || node.is_a?(AST::YieldExpr) + kind = node.class.name.to_s.split("::").last + error!(token, :DEFER_NO_CONTROL_FLOW, kind: kind) + end + # FN/lambda bodies are their own control-flow scopes. + next if node.is_a?(AST::FunctionDef) || node.is_a?(AST::LambdaLit) + + node.class.members.each do |member| + value = T.unsafe(node)[member] + if value.is_a?(Array) + value.each { |child| stack << child if child.is_a?(AST::Locatable) } + elsif value.is_a?(AST::Locatable) + stack << value + end + end + end + end + sig { returns(AST::BreakNode) } def parse_break token = consume(:KEYWORD, 'BREAK') @@ -92,8 +141,6 @@ def parse_pass_statement AST::PassStmt.new(tok) end - SYNTAX_TOKENS_AT_STATEMENT_END = %w[; THEN DO ->].freeze - sig { returns(AST::Node) } def parse_statement @budget.enter! @@ -426,7 +473,7 @@ def parse_value_block_expr end VALUE_BLOCK_STATEMENT_KEYWORDS = T.let(Set[ - 'ASSERT', 'ASSERT_RAISES', 'BENCHMARK', 'BREAK', 'CONTINUE', 'DIE', + 'ASSERT', 'ASSERT_RAISES', 'BENCHMARK', 'BREAK', 'CONTINUE', 'DEFER', 'DIE', 'DO', 'ENUM', 'EXIT', 'EXTERN', 'FN', 'FOR', 'METHOD', 'MUTABLE', 'IF', 'MATCH', 'PARTIAL', 'PASS', 'PRIVATE', 'PROFILE', 'PUB', 'RAISE', 'RETURN', 'SMASH', 'STRUCT', 'STUB', 'SYNC', 'TEST', 'TIGHT', 'UNION', 'WHILE', 'WITH', diff --git a/compiler/ruby/ast/parser/types.rb b/compiler/ruby/ast/parser/types.rb index f6fd6d284..2f0b77e90 100644 --- a/compiler/ruby/ast/parser/types.rb +++ b/compiler/ruby/ast/parser/types.rb @@ -1,5 +1,7 @@ # typed: strict +require_relative "state" + class ClearParser extend T::Sig @@ -224,6 +226,15 @@ def parse_type_annotation_body end end + # Retired open-stream alias: ~?T[] (and ~?T[N] / ~?T[INF]) accepted-then- + # leaked through the alias compat path. Fail closed with the migration. + if tense_prefix == "~" && optional_prefix == "?" && error_prefix == "" && + (legacy_card = inner[/\A\[(\d*|INF)\]\z/, 1]) + error!(current, :RETIRED_OPTIONAL_STREAM_SYNTAX, + got: "~?#{base}#{inner}", + replacement: "[~#{legacy_card}]#{base}") + end + # Capability suffix: T @shared, T[]@list:soa, T[N]@soa:shared:locked, HashMap@sharded(N), etc. # ClearParser only does token consumption and duplicate detection. Semantic validation # (e.g., "@list requires array", "@soa requires fixed array") is in the annotator. @@ -266,7 +277,7 @@ def emit_legacy_type_migration(start_token, end_token, type) return unless FixCollector.type_migrations_enabled? return unless start_token.line == end_token.line return unless TypeExpressionTree.each_node(type.shape.expression).any? do |node| - node.is_a?(LinearTypeExpression) || node.is_a?(MapTypeExpression) || node.is_a?(StreamTypeExpression) + node.kind.is_a?(LinearTypeExpression) || node.kind.is_a?(MapTypeExpression) || node.kind.is_a?(StreamTypeExpression) end replacement = Type.inline_migration_name(type) @@ -349,7 +360,7 @@ def parse_inline_type_expression parse_inline_atom_expression end - sig { returns(StreamTypeExpression) } + sig { returns(TypeExpression) } def parse_inline_stream_expression consume(:CHAR, '[') consume(:CHAR, '~') @@ -362,11 +373,10 @@ def parse_inline_stream_expression end consume(:CHAR, ']') caps = parse_inline_capabilities - StreamTypeExpression.new( + TypeExpression.new(kind: StreamTypeExpression.new( cardinality: cardinality, item: parse_inline_type_expression, - capabilities: caps, - ) + ), capabilities: caps) end sig { returns(TypeExpression) } @@ -375,16 +385,16 @@ def parse_inline_prefixed_expression prefix = prefix_token.text! inner = parse_inline_type_expression if prefix == "?" - if inner.is_a?(StreamTypeExpression) + if inner.kind.is_a?(StreamTypeExpression) error!(prefix_token, :PARSER_EXPECTED, expected: "an optional stream item such as [~]?T", got: "?[~]T", type: prefix_token.type, line: prefix_token.line) end - return OptionalTypeExpression.new(inner: inner) + return TypeExpression.of(OptionalTypeExpression.new(inner: inner)) end - return FallibleTypeExpression.new(inner: inner) if prefix == "!" + return TypeExpression.of(FallibleTypeExpression.new(inner: inner)) if prefix == "!" - FutureTypeExpression.new(inner: inner) + TypeExpression.of(FutureTypeExpression.new(inner: inner)) end sig { returns(TypeExpression) } @@ -398,8 +408,8 @@ def parse_inline_atom_expression if match?(:DOUBLE_COLON) consume(:DOUBLE_COLON) member = consume(:TYPE_ID).text! - expression = TypeProjectionExpression.new(owner: name.to_sym, member: member.to_sym) - return TypeExpressionTree.with_root_capabilities(expression, parse_inline_capabilities) + projection = TypeProjectionExpression.new(owner: name.to_sym, member: member.to_sym) + return TypeExpression.new(kind: projection, capabilities: parse_inline_capabilities) end arguments = T.let([], T::Array[TypeExpression]) if match?(:CHAR, '<') @@ -410,22 +420,22 @@ def parse_inline_atom_expression end consume_generic_close end - expression = if name == "Tuple" + node = if name == "Tuple" TupleTypeExpression.new(items: arguments) else NamedTypeExpression.new(name: name.to_sym, arguments: arguments) end - TypeExpressionTree.with_root_capabilities(expression, parse_inline_capabilities) + TypeExpression.new(kind: node, capabilities: parse_inline_capabilities) end - sig { returns(LinearTypeExpression) } + sig { returns(TypeExpression) } def parse_inline_linear_expression consume(:CHAR, '[') if match!(:CHAR, ']') caps = parse_inline_capabilities(collection: :list) - return LinearTypeExpression.new(kind: :list, dimensions: [:LIST], - item: parse_inline_type_expression, capabilities: caps) + return TypeExpression.new(kind: LinearTypeExpression.new(kind: :list, dimensions: [:LIST], + item: parse_inline_type_expression), capabilities: caps) end kind = T.let(:array, Symbol) @@ -472,20 +482,19 @@ def parse_inline_linear_expression kind = :rank if dimensions.length > 1 collection = %i[list set pool].include?(kind) ? kind : nil caps = parse_inline_capabilities(collection: collection) - LinearTypeExpression.new( + TypeExpression.new(kind: LinearTypeExpression.new( kind: kind, dimensions: dimensions, item: parse_inline_type_expression, allocation_hint: allocation_hint, - capabilities: caps - ) + ), capabilities: caps) end - sig { returns(MapTypeExpression) } + sig { returns(TypeExpression) } def parse_inline_map_expression consume(:CHAR, '{') key = if match?(:CHAR, '}') - NamedTypeExpression.new(name: :Symbol) + TypeExpression.of(NamedTypeExpression.new(name: :Symbol)) else parsed_key = parse_inline_type_expression if match?(:CHAR, ',') @@ -495,7 +504,7 @@ def parse_inline_map_expression end consume(:CHAR, '}') caps = parse_inline_capabilities - MapTypeExpression.new(key: key, value: parse_inline_type_expression, capabilities: caps) + TypeExpression.new(kind: MapTypeExpression.new(key: key, value: parse_inline_type_expression), capabilities: caps) end sig { params(collection: T.nilable(Symbol)).returns(TypeCapabilities) } diff --git a/compiler/ruby/ast/schemas.rb b/compiler/ruby/ast/schemas.rb deleted file mode 100644 index c8a6981b1..000000000 --- a/compiler/ruby/ast/schemas.rb +++ /dev/null @@ -1,530 +0,0 @@ -# typed: strict -# Typed schemas for declared types stored in Scope. -# -# A declared type's schema is ALWAYS one of the typed classes below — -# never a raw Hash. Producers (the annotator's visit_*Def) construct -# these directly; consumers use the typed accessors (`.fields`, -# `.variants`, `.kind`, `.struct?`, ...). There is exactly one -# representation. -require "sorbet-runtime" -require "set" -require_relative "struct_field" - -module Schemas - extend T::Sig - - class ExternSource < T::Struct - const :dependency, String - const :abi, Symbol, default: :zig - const :symbol, T.nilable(String), default: nil - const :callconv, Symbol, default: :c - const :header, T.nilable(String), default: nil - end - - # Plain classes (not Data.define) so Sorbet's 4010 doesn't fire on - # the kwarg-only initialize signatures we need for default values. - # Frozen at the end of initialize so callers see immutable shapes - # (the methods table is mutable in place — see StructSchema#methods). - - class EnumSchema - extend T::Sig - - VariantInput = T.type_alias { T::Enumerable[T.any(String, Symbol)] } - - sig { returns(T::Set[String]) } - attr_reader :variants - sig { returns(Symbol) } - attr_reader :visibility - sig { params(variants: VariantInput, visibility: Symbol).void } - def initialize(variants:, visibility: :package) - @variants = T.let(normalize_variants(variants).freeze, T::Set[String]) - @visibility = T.let(visibility, Symbol) - freeze - end - - sig { params(variants: VariantInput).returns(T::Set[String]) } - def normalize_variants(variants) - normalized = T.let(Set.new, T::Set[String]) - variants.each do |variant| - normalized << variant.to_s - end - normalized - end - private :normalize_variants - - sig { returns(T.nilable(Symbol)) } - def kind = :enum - sig { returns(T::Boolean) } - def enum? = true - sig { returns(T::Boolean) } - def union? = false - sig { returns(T::Boolean) } - def struct? = false - sig { returns(T::Boolean) } - def resource? = false - end - - class ResourceCloseCallKind < T::Enum - enums do - Method = new("method") - Function = new("function") - CFunction = new("c_function") - end - end - - class ResourceCloseAction < T::Struct - extend T::Sig - - const :call_kind, ResourceCloseCallKind - const :name, String - const :field_path, T::Array[String], default: [] - const :runtime_heap_alloc_args, Integer, default: 0 - - sig { params(field: String).returns(ResourceCloseAction) } - def for_field(field) - ResourceCloseAction.new( - call_kind: call_kind, - name: name, - field_path: [field] + field_path, - runtime_heap_alloc_args: runtime_heap_alloc_args, - ) - end - end - - class ResourceClosePlan < T::Struct - extend T::Sig - - const :actions, T::Array[ResourceCloseAction] - - sig { params(name: String, runtime_heap_alloc_args: Integer).returns(ResourceClosePlan) } - def self.method(name, runtime_heap_alloc_args: 0) - new(actions: [ - ResourceCloseAction.new( - call_kind: ResourceCloseCallKind::Method, - name: name, - runtime_heap_alloc_args: runtime_heap_alloc_args, - ), - ]) - end - - sig { params(name: String, runtime_heap_alloc_args: Integer).returns(ResourceClosePlan) } - def self.function(name, runtime_heap_alloc_args: 0) - new(actions: [ - ResourceCloseAction.new( - call_kind: ResourceCloseCallKind::Function, - name: name, - runtime_heap_alloc_args: runtime_heap_alloc_args, - ), - ]) - end - - sig { params(name: String).returns(ResourceClosePlan) } - def self.c_function(name) - new(actions: [ - ResourceCloseAction.new( - call_kind: ResourceCloseCallKind::CFunction, - name: name, - ), - ]) - end - - sig { params(actions: T::Array[ResourceCloseAction]).returns(ResourceClosePlan) } - def self.composite(actions) - new(actions: actions) - end - - sig { params(field: String).returns(ResourceClosePlan) } - def for_field(field) - mapped_actions = actions.map { |action| action.for_field(field) } - ResourceClosePlan.new(actions: mapped_actions) - end - - sig { returns(T::Boolean) } - def empty? - actions.empty? - end - end - - # Resource type schema — types with RAII cleanup (CLOSE method). - # - # Used for the 3 hand-written runtime types (File, TCPServer, TCPClient) - # and EXTERN STRUCT ... CLOSE forms, which can carry generic type params, - # an extern module name, and an AS alias. - class ResourceSchema - extend T::Sig - - FieldMetadataValue = T.type_alias { T.any(Type::TypeInput, AST::Locatable, T::Boolean) } - FieldMetadata = T.type_alias { T::Hash[T.any(Symbol, String), FieldMetadataValue] } - FieldInput = T.type_alias { T.any(Type::TypeInput, AST::StructField, FieldMetadata) } - FieldInputMap = T.type_alias { T::Hash[T.any(Symbol, String), FieldInput] } - StaticMethodValue = T.type_alias { T.any(T::Array[Symbol], Symbol, String, T::Boolean) } - StaticMethodSpec = T.type_alias { T::Hash[Symbol, StaticMethodValue] } - StaticMethodsMap = T.type_alias { T::Hash[String, StaticMethodSpec] } - MethodsMap = T.type_alias { T::Hash[T.any(Symbol, String), FunctionSignature] } - - attr_reader :close_plan, :static_methods, :fields, :type_params, :extern_module, :as_type, :visibility, :methods - sig { params(close_plan: Schemas::ResourceClosePlan, static_methods: Schemas::ResourceSchema::StaticMethodsMap, fields: FieldInputMap, type_params: T::Array[Symbol], extern_module: T.nilable(String), as_type: T.nilable(String), visibility: Symbol, methods: Schemas::ResourceSchema::MethodsMap).void } - def initialize(close_plan:, static_methods: {}, fields: {}, type_params: [], extern_module: nil, as_type: nil, visibility: :package, methods: {}) - @close_plan = T.let(close_plan.dup, Schemas::ResourceClosePlan) - @static_methods = T.let(static_methods, Schemas::ResourceSchema::StaticMethodsMap) - @fields = T.let(normalize_fields(fields), T::Hash[String, AST::StructField]) - @type_params = T.let(type_params.dup, T::Array[Symbol]) - @extern_module = T.let(extern_module, T.nilable(String)) - @as_type = T.let(as_type, T.nilable(String)) - @visibility = T.let(visibility, Symbol) - @methods = T.let(methods, Schemas::ResourceSchema::MethodsMap) - freeze - end - - sig { returns(T.nilable(Symbol)) } - def kind = :resource - sig { returns(T::Boolean) } - def resource? = true - sig { returns(T::Boolean) } - def union? = false - sig { returns(T::Boolean) } - def enum? = false - sig { returns(T::Boolean) } - def struct? = false - sig { returns(T::Array[Symbol]) } - def type_params = @type_params - - sig { params(fields: FieldInputMap).returns(T::Hash[String, AST::StructField]) } - def normalize_fields(fields) - out = T.let({}, T::Hash[String, AST::StructField]) - keys = fields.keys - i = T.let(0, Integer) - while i < keys.length - out[keys[i].to_s] = normalize_field(T.must(fields[T.unsafe(keys[i])])) - i += 1 - end - out - end - private :normalize_fields - - sig { params(field: FieldInput).returns(AST::StructField) } - def normalize_field(field) - return field if field.is_a?(AST::StructField) - if field.is_a?(Hash) - raw_type_value = T.let(field[:type], T.nilable(FieldMetadataValue)) - raw_type_value = field["type"] if raw_type_value.nil? - default_value = T.let(field[:default], T.untyped) - default_value = field["default"] if default_value.nil? - borrowed_value = T.let(field[:borrowed], T.untyped) - borrowed_value = field["borrowed"] if borrowed_value.nil? - if raw_type_value.nil? - return AST::StructField.new( - type: :Any, - default: default_value, - borrowed: !borrowed_value.nil? - ) - end - return AST::StructField.new( - type: raw_type_value, - default: default_value, - borrowed: !borrowed_value.nil? - ) - end - - AST::StructField.new(type: field.dup) - end - private :normalize_field - end - - class InlineStructDeinitEntry < T::Struct - extend T::Sig - - const :field, String - const :kind, Symbol - const :zig_type, T.nilable(String) - const :elem_zig_type, T.nilable(String) - - sig { params(field: String, zig_type: String).returns(Schemas::InlineStructDeinitEntry) } - def self.indirect(field:, zig_type:) - new(field: field, kind: :indirect, zig_type: zig_type, elem_zig_type: nil) - end - - sig { params(field: String, zig_type: String).returns(Schemas::InlineStructDeinitEntry) } - def self.uniform(field:, zig_type:) - new(field: field, kind: :uniform, zig_type: zig_type, elem_zig_type: nil) - end - - sig { params(field: String, elem_zig_type: String).returns(Schemas::InlineStructDeinitEntry) } - def self.array(field:, elem_zig_type:) - new(field: field, kind: :array, zig_type: nil, elem_zig_type: elem_zig_type) - end - end - - # One union variant whose payload is an anonymous inline struct - # (`UNION Shape { Circle { radius: Float64 } }`). `fields` maps field - # name (String) to its declared type input. `deinit_entries` is filled in by - # the annotator after parse (which fields need @boxed / array - # cleanup) and is intentionally mutable in place, like - # StructSchema#methods. - # ruby-to-clear: pub - class InlineStructVariant - extend T::Sig - - FieldMap = T.type_alias { T::Hash[T.any(String, Symbol), Type::TypeInput] } - FieldInputMap = T.type_alias { T::Hash[T.any(String, Symbol), Type::TypeInput] } - - attr_reader :fields - sig { params(fields: FieldInputMap, deinit_entries: T::Array[Schemas::InlineStructDeinitEntry]).void } - def initialize(fields:, deinit_entries: []) - @fields = T.let(normalize_fields(fields), Schemas::InlineStructVariant::FieldMap) - @deinit_entries = T.let(deinit_entries, T::Array[Schemas::InlineStructDeinitEntry]) - end - - sig { returns(T::Array[Schemas::InlineStructDeinitEntry]) } - def deinit_entries - @deinit_entries - end - - sig { params(entries: T::Array[Schemas::InlineStructDeinitEntry]).returns(T::Array[Schemas::InlineStructDeinitEntry]) } - def deinit_entries=(entries) - @deinit_entries = entries - end - - # ruby-to-clear: skip - sig { returns(T::Hash[String, Type]) } - def typed_fields - out = T.let({}, T::Hash[String, Type]) - keys = @fields.keys - i = T.let(0, Integer) - while i < keys.length - out[keys[i].to_s] = Type.new(T.must(@fields[T.unsafe(keys[i])])) - i += 1 - end - out - end - - # Value equality on the field shape (not deinit_entries, which is - # derived). The multi-arm shared-destructure check compares two - # variants' payloads structurally — this used to be Hash `==`. - sig { params(other: InlineStructVariant).returns(T::Boolean) } - def ==(other) - other.fields == @fields - end - - sig { params(other: InlineStructVariant).returns(T::Boolean) } - def eql?(other) - self == other - end - # ruby-to-clear: skip - sig { returns(Integer) } - def hash = @fields.hash - - sig { params(fields: FieldInputMap).returns(FieldMap) } - def normalize_fields(fields) - out = T.let({}, Schemas::InlineStructVariant::FieldMap) - keys = fields.keys - i = T.let(0, Integer) - while i < keys.length - key = keys.fetch(i) - out[key] = T.must(fields[key]) - i += 1 - end - out - end - private :normalize_fields - end - - # Union (sum-type) schema. `variants` is a Hash[Symbol => value] where - # the value is `nil` for payload-less variants, a type input for single-type - # payloads, or an InlineStructVariant for inline struct variants. - class UnionSchema - extend T::Sig - - VariantValue = T.type_alias { T.nilable(T.any(Type::TypeInput, Schemas::InlineStructVariant)) } - VariantMap = T.type_alias { T::Hash[T.any(String, Symbol), VariantValue] } - VariantInput = T.type_alias { T.nilable(T.any(Type::TypeInput, Schemas::InlineStructVariant)) } - VariantInputMap = T.type_alias { T::Hash[T.any(String, Symbol), VariantInput] } - - attr_reader :variants, :type_params, :visibility - sig { params(variants: VariantInputMap, type_params: T::Array[Symbol], visibility: Symbol).void } - def initialize(variants:, type_params: [], visibility: :package) - @variants = T.let(normalize_variants(variants), Schemas::UnionSchema::VariantMap) - @type_params = T.let(type_params.dup, T::Array[Symbol]) - @visibility = T.let(visibility, Symbol) - freeze - end - - sig { params(variants: VariantInputMap).returns(VariantMap) } - def normalize_variants(variants) - out = T.let({}, Schemas::UnionSchema::VariantMap) - keys = variants.keys - i = T.let(0, Integer) - while i < keys.length - key = keys.fetch(i) - out[key] = normalize_variant(variants[key]) - i += 1 - end - out - end - private :normalize_variants - - sig { params(variant: VariantInput).returns(VariantValue) } - def normalize_variant(variant) - return nil if variant.nil? - return variant if variant.is_a?(Schemas::InlineStructVariant) - - variant - end - private :normalize_variant - - sig { returns(T.nilable(Symbol)) } - def kind = :union - sig { returns(T::Boolean) } - def union? = true - sig { returns(T::Boolean) } - def enum? = false - sig { returns(T::Boolean) } - def struct? = false - sig { returns(T::Boolean) } - def resource? = false - sig { returns(T::Array[Symbol]) } - def type_params = @type_params - end - - # Struct/record schema. `fields` maps String field names to Type/Symbol - # representations of field types. Metadata (defaults, borrowed-set, - # generic type params, methods, EXTERN module, AS alias type, - # visibility) live as named attrs. `methods` is intentionally mutable - # in place: method signatures are registered after the struct is - # declared (when the method's FunctionDef is visited). - class StructSchema - extend T::Sig - - # `fields` is ALWAYS Hash[String => AST::StructField]. Per-field - # default value and borrowed-ness live on the StructField, so - # `field_defaults` / `borrowed_fields` are derived, not stored. - FieldMetadataValue = T.type_alias { T.any(Type::TypeInput, AST::Locatable, T::Boolean) } - FieldMetadata = T.type_alias { T::Hash[T.any(Symbol, String), FieldMetadataValue] } - FieldInput = T.type_alias { T.any(Type::TypeInput, AST::StructField, FieldMetadata) } - FieldInputMap = T.type_alias { T::Hash[T.any(Symbol, String), FieldInput] } - MethodsMap = T.type_alias { T::Hash[T.any(Symbol, String), FunctionSignature] } - - attr_reader :fields, :type_params, :methods, :visibility, :extern_module, :as_type - sig { returns(T::Array[AST::GenericParamDecl]) } - attr_reader :generic_params - sig { returns(MethodsMap) } - attr_reader :static_methods - sig { params(fields: FieldInputMap, type_params: T::Array[Symbol], generic_params: T::Array[AST::GenericParamDecl], methods: MethodsMap, static_methods: MethodsMap, visibility: Symbol, extern_module: T.nilable(String), as_type: T.nilable(String)).void } - def initialize(fields: {}, type_params: [], generic_params: [], methods: {}, static_methods: {}, visibility: :package, extern_module: nil, as_type: nil) - @fields = T.let(normalize_fields(fields), T::Hash[String, AST::StructField]) - @type_params = T.let(type_params.dup, T::Array[Symbol]) - @generic_params = T.let(generic_params.dup, T::Array[AST::GenericParamDecl]) - @methods = T.let(methods, MethodsMap) - @static_methods = T.let(static_methods, MethodsMap) - @visibility = T.let(visibility, Symbol) - @extern_module = T.let(extern_module, T.nilable(String)) - @as_type = T.let(as_type, T.nilable(String)) - freeze - end - - sig { returns(T::Array[Symbol]) } - def type_params = @type_params - - sig { returns(T::Hash[String, AST::Locatable]) } - def field_defaults - out = T.let({}, T::Hash[String, AST::Locatable]) - keys = @fields.keys - i = T.let(0, Integer) - while i < keys.length - default = T.must(@fields[T.unsafe(keys[i])]).default - out[T.unsafe(keys[i])] = T.must(default) unless default.nil? - i += 1 - end - out - end - - sig { returns(T::Set[String]) } - def borrowed_fields - out = T.let(Set.new, T::Set[String]) - keys = @fields.keys - i = T.let(0, Integer) - while i < keys.length - out << T.unsafe(keys[i]) if T.must(@fields[T.unsafe(keys[i])]).borrowed - i += 1 - end - out - end - - sig { returns(T.nilable(Symbol)) } - def kind = nil - sig { returns(T::Boolean) } - def struct? = true - sig { returns(T::Boolean) } - def union? = false - sig { returns(T::Boolean) } - def enum? = false - sig { returns(T::Boolean) } - def resource? = false - - sig { params(fields: FieldInputMap).returns(T::Hash[String, AST::StructField]) } - def normalize_fields(fields) - out = T.let({}, T::Hash[String, AST::StructField]) - keys = fields.keys - i = T.let(0, Integer) - while i < keys.length - out[keys[i].to_s] = normalize_field(T.must(fields[T.unsafe(keys[i])])) - i += 1 - end - out - end - private :normalize_fields - - sig { params(field: FieldInput).returns(AST::StructField) } - def normalize_field(field) - return field if field.is_a?(AST::StructField) - if field.is_a?(Hash) - raw_type_value = T.let(field[:type], T.nilable(FieldMetadataValue)) - raw_type_value = field["type"] if raw_type_value.nil? - default_value = T.let(field[:default], T.untyped) - default_value = field["default"] if default_value.nil? - borrowed_value = T.let(field[:borrowed], T.untyped) - borrowed_value = field["borrowed"] if borrowed_value.nil? - if raw_type_value.nil? - return AST::StructField.new( - type: :Any, - default: default_value, - borrowed: !borrowed_value.nil? - ) - end - return AST::StructField.new( - type: raw_type_value, - default: default_value, - borrowed: !borrowed_value.nil? - ) - end - - AST::StructField.new(type: field.dup) - end - private :normalize_field - end - - SchemaValue = T.type_alias { T.nilable(T.any(EnumSchema, StructSchema, UnionSchema, ResourceSchema)) } - FieldBearingSchema = T.type_alias { T.nilable(T.any(StructSchema, ResourceSchema)) } - - # Nil-safe kind predicates. Single representation: a schema is always - # one of the typed classes above (or nil for an unknown type name). - sig { params(s: SchemaValue).returns(T::Boolean) } - def self.struct?(s) = s.is_a?(StructSchema) - - sig { params(s: SchemaValue).returns(T::Boolean) } - def self.union?(s) = s.is_a?(UnionSchema) - - sig { params(s: SchemaValue).returns(T::Boolean) } - def self.enum?(s) = s.is_a?(EnumSchema) - - sig { params(s: SchemaValue).returns(T::Boolean) } - def self.resource?(s) = s.is_a?(ResourceSchema) - - sig { params(v: Object).returns(T::Boolean) } - def self.inline_struct?(v) = v.is_a?(InlineStructVariant) - - # Field-bearing schema: StructSchema or ResourceSchema (EXTERN STRUCT - # ... CLOSE carries fields too), so `.fields` is safe to read. - sig { params(s: SchemaValue).returns(T::Boolean) } - def self.field_bearing?(s) = s.is_a?(StructSchema) || s.is_a?(ResourceSchema) -end diff --git a/compiler/ruby/ast/scope.rb b/compiler/ruby/ast/scope.rb index 315fc193c..9c9a52ba7 100644 --- a/compiler/ruby/ast/scope.rb +++ b/compiler/ruby/ast/scope.rb @@ -3,13 +3,12 @@ require "set" require_relative "./symbol_entry" -require_relative "./schemas" +require_relative "./type" class Scope extend T::Sig EMPTY_CAPABILITIES = T.let(Set.new.freeze, T::Set[Symbol]) - RegInput = T.type_alias { T.nilable(T.any(AST::Node, String, Symbol)) } MutabilityInput = T.type_alias { T.nilable(T.any(T::Boolean, Lexer::Token)) } ScopeTypeSchema = T.type_alias do T.any(Schemas::EnumSchema, Schemas::ResourceSchema, Schemas::StructSchema, Schemas::UnionSchema) @@ -100,12 +99,13 @@ def keys sig { returns(T::Hash[String, String]) } attr_reader :dependencies attr_accessor :depth # stack depth at scope creation; 0 for root - attr_reader :types, :parent + attr_reader :binding_entries, :types, :parent sig { void } def initialize @parent = T.let(nil, T.nilable(Scope)) @bindings = T.let(ScopeBindings.new, ScopeBindings) + @binding_entries = T.let(@bindings.entries, T::Hash[String, SymbolEntry]) @dependencies = T.let({}, T::Hash[String, String]) @type_store = T.let(ScopeTypes.new, ScopeTypes) @types = T.let(@type_store.entries, T::Hash[Symbol, Scope::ScopeTypeEntry]) @@ -113,7 +113,8 @@ def initialize @depth = T.let(0, Integer) end - sig { params(name: String, reg: RegInput, type: SymbolEntry::TypeInput, is_mutable: MutabilityInput, is_rebindable: T::Boolean, size: T.nilable(Integer), storage: Symbol, capabilities: T::Set[Symbol], _borrowed_paths: T::Array[SymbolEntry], sync: T.nilable(Symbol), layout: T.nilable(Symbol), resource: T.nilable(T::Boolean), close_plan: T.nilable(Schemas::ResourceClosePlan)).returns(SymbolEntry) } + sig { params(name: String, reg: SymbolEntry::RegInput, type: SymbolEntry::TypeInput, is_mutable: MutabilityInput, is_rebindable: T::Boolean, size: T.nilable(Integer), storage: Symbol, capabilities: T::Set[Symbol], _borrowed_paths: T::Array[SymbolEntry], sync: T.nilable(Symbol), layout: T.nilable(Symbol), resource: T.nilable(T::Boolean), close_plan: T.nilable(Schemas::ResourceClosePlan)).returns(SymbolEntry) } + # ruby-to-clear: fallible def declare(name, reg, type, is_mutable = true, is_rebindable = false, size = nil, storage = :stack, capabilities = Set.new, _borrowed_paths = [], sync: nil, layout: nil, resource: nil, close_plan: nil) @owned_names.add(name) entry = SymbolEntry.new( @@ -160,6 +161,7 @@ def initialize_copy(original) @parent = original @bindings = ScopeBindings.new + @binding_entries = @bindings.entries @dependencies = original.dependencies.dup @type_store = ScopeTypes.new @types = @type_store.entries @@ -198,13 +200,32 @@ def resolve_type_definition(name) sig { params(name: Symbol).returns(T.nilable(ScopeTypeEntry)) } def resolve_type_entry(name) - @type_store[name] || @parent&.resolve_type_entry(name) + local = @type_store[name] + return local if local + + cursor = T.let(@parent, T.nilable(Scope)) + until cursor.nil? + ancestor = cursor + inherited = ancestor.types[name] + return inherited if inherited + + cursor = ancestor.parent + end + nil end sig { returns(T::Hash[Symbol, ScopeTypeEntry]) } def visible_types - inherited = @parent ? @parent.visible_types : {} - inherited.merge(@types) + visible = @types.dup + cursor = T.let(@parent, T.nilable(Scope)) + until cursor.nil? + ancestor = cursor + ancestor.types.each do |name, entry| + visible[name] = entry unless visible.key?(name) + end + cursor = ancestor.parent + end + visible end sig { params(name: String).returns(T.nilable(SymbolEntry)) } @@ -219,7 +240,18 @@ def local_entry!(name) sig { params(name: String).returns(T.nilable(SymbolEntry)) } def resolve_entry(name) - @bindings[name] || @parent&.resolve_entry(name) + local = @bindings[name] + return local if local + + cursor = T.let(@parent, T.nilable(Scope)) + until cursor.nil? + ancestor = cursor + inherited = ancestor.binding_entries[name] + return inherited if inherited + + cursor = ancestor.parent + end + nil end sig { params(name: String).returns(SymbolEntry) } @@ -234,7 +266,7 @@ def local_entry?(name) sig { params(name: String).returns(T::Boolean) } def entry?(name) - local_entry?(name) || !!@parent&.entry?(name) + !resolve_entry(name).nil? end sig { params(name: String).returns(T.nilable(SymbolEntry)) } @@ -242,7 +274,7 @@ def entry_for_write(name) entry = @bindings[name] return entry if entry - inherited = @parent&.resolve_entry(name) + inherited = resolve_entry(name) return nil unless inherited materialized = clone_entry_for_scope(inherited) @@ -277,26 +309,41 @@ def local_entry_count end sig { returns(Integer) } + # ruby-to-clear: fallible def visible_entry_count count_visible_entries!(Set.new) end sig { params(seen: T::Set[String]).returns(Integer) } + # ruby-to-clear: fallible def count_visible_entries!(seen) - count = @parent&.count_visible_entries!(seen) || 0 - @bindings.keys.each do |name| - next if seen.include?(name) - - seen << name - count += 1 + count = T.let(0, Integer) + cursor = T.let(self, T.nilable(Scope)) + until cursor.nil? + current = cursor + current.binding_entries.each_key do |name| + next if seen.include?(name) + + seen << name + count += 1 + end + cursor = current.parent end count end sig { returns(T::Hash[String, SymbolEntry]) } def visible_entries - inherited = @parent ? @parent.visible_entries : {} - inherited.merge(@bindings.entries) + visible = @binding_entries.dup + cursor = T.let(@parent, T.nilable(Scope)) + until cursor.nil? + ancestor = cursor + ancestor.binding_entries.each do |name, entry| + visible[name] = entry unless visible.key?(name) + end + cursor = ancestor.parent + end + visible end sig { returns(T::Array[String]) } @@ -309,8 +356,7 @@ def visible_names sig { params(names: T::Array[String], seen: T::Set[String]).void } def append_visible_names!(names, seen) - @parent&.append_visible_names!(names, seen) - @bindings.keys.each do |name| + visible_entries.each_key do |name| next if seen.include?(name) seen << name @@ -335,10 +381,11 @@ def resolve_full_type(name) base_type = Type.new(entry.type) - value_sync = if entry.locked? - :locked + value_sync = T.let(nil, T.nilable(Symbol)) + if entry.locked? + value_sync = :locked elsif entry.write_locked? - :write_locked + value_sync = :write_locked end base_type.apply_symbol_overlay!( storage: entry.storage, @@ -390,8 +437,10 @@ def mark_read(name) # in the cap's old_scope (caller is responsible for emitting a diagnostic). sig { params(capability: AST::Capability).returns(T.nilable(SymbolEntry)) } def declare_with_new_capability(capability) - name = capability[:var_node].name - local = capability[:old_scope].resolve_entry(name) + var_node = T.cast(capability[:var_node], AST::Node) + old_scope = T.cast(capability[:old_scope], Scope) + name = T.must(AST.root_identifier(var_node)).name + local = old_scope.resolve_entry(name) return nil if local.nil? local = local.dup local.capabilities = local.capabilities.dup @@ -404,18 +453,30 @@ def declare_with_new_capability(capability) sig { params(node: AST::Node).returns(T::Array[Symbol]) } def get_path_to_root(node) - path = [] - curr = T.let(node, T.untyped) - while curr.is_a?(AST::GetField) || curr.is_a?(AST::GetIndex) - if curr.is_a?(AST::GetField) - path.unshift(curr.field.to_sym) - elsif curr.is_a?(AST::GetIndex) - path.unshift(:*) + path = T.let([], T::Array[Symbol]) + curr = T.let(node, AST::Node) + while true + next_curr = T.let(nil, T.nilable(AST::Node)) + case curr + when AST::GetField + path << curr.field.to_sym + next_curr = curr.target + when AST::GetIndex + path << :* + next_curr = curr.target + else + break end - curr = curr.target + curr = T.must(next_curr) + end + path << T.cast(curr, AST::Identifier).name.to_sym + reversed = T.let([], T::Array[Symbol]) + i = path.length - 1 + while i >= 0 + reversed << T.must(path[i]) + i -= 1 end - path.unshift(curr.name.to_sym) - path + reversed end sig { params(name: String).void } diff --git a/compiler/ruby/ast/source_error.rb b/compiler/ruby/ast/source_error.rb index f56e35583..c32cd1546 100644 --- a/compiler/ruby/ast/source_error.rb +++ b/compiler/ruby/ast/source_error.rb @@ -30,41 +30,45 @@ def source_code; end # `%{name}` interpolation against the hash. Legacy positional args # against `%s`/`%d` still work for the (shrinking) set of templates # that haven't been migrated to named form yet. - sig { params(node_or_token: T.untyped, code_or_message: T.any(String, Symbol), args: String, kwargs: T.untyped).returns(T.noreturn) } + sig { params(node_or_token: T.untyped, code_or_message: T.any(String, Symbol), args: DiagnosticRegistry::DiagnosticKwValue, kwargs: T.untyped).returns(T.noreturn) } def error!(node_or_token, code_or_message, *args, **kwargs) T.bind(self, T.untyped) rescue nil token = diagnostic_token(node_or_token) # 2. Determine Message + message = T.let("", String) if code_or_message.is_a?(Symbol) - message = DiagnosticRegistry.format_from_hash(code_or_message, args, kwargs) + message = DiagnosticRegistry.format_from_hash(code_or_message, args, kwargs) || "" raise "Internal Compiler Error: Unknown error code :#{code_or_message}" unless message else # C. Legacy Support (Raw String) message = code_or_message end - # 3. Raise the specific error class - err_class = self.class.name&.include?("Parser") ? ParserError : CompilerError source_token = source_error_token(token) - - raise err_class.new( - source_token, - T.unsafe(message), - diagnostic_source_code, - code: code_or_message.is_a?(Symbol) ? code_or_message : nil - ) + diagnostic_code = T.let(nil, T.nilable(Symbol)) + if code_or_message.is_a?(Symbol) + diagnostic_code = code_or_message.dup + end + raise_source_error!( + source_token, + T.unsafe(message), + # Keep the narrowed symbol in a typed optional local. This avoids + # materializing the original String-or-Symbol parameter as a union in + # an optional ternary slot at the ownership boundary. + code: diagnostic_code, + ) end # Try the hash form first when applicable; fall back to positional; # surface any internal mismatch as an "Internal Args Error" suffix. - sig { params(template: String, args: T::Array[String], kwargs: T::Hash[Symbol, T::Array[Symbol]]).returns(String) } + sig { params(template: String, args: DiagnosticRegistry::DiagnosticArgs, kwargs: DiagnosticRegistry::DiagnosticKwargs).returns(String) } def format_diagnostic_template(template, args, kwargs) T.bind(self, T.untyped) rescue nil DiagnosticRegistry.format_template(template, args, kwargs) end - sig { params(code: Symbol, args: String, kwargs: T.untyped).returns(String) } + sig { params(code: Symbol, args: DiagnosticRegistry::DiagnosticKwValue, kwargs: T.untyped).returns(String) } def diagnostic_message(code, *args, **kwargs) message = DiagnosticRegistry.format_from_hash(code, args, kwargs) Kernel.raise "Internal Compiler Error: Unknown error code :#{code}" unless message @@ -88,7 +92,7 @@ def fix_description_from_hash(code, kwargs) sig { params(node_or_token: AST::Node, message: String).returns(NilClass) } def note!(node_or_token, message) T.bind(self, T.untyped) rescue nil - token = diagnostic_token(node_or_token) + token = source_error_token(diagnostic_token(node_or_token)) loc = token ? " (line #{token.line})" : "" diagnostic_output("\e[36m[Note]\e[0m #{message}#{loc}") nil @@ -97,7 +101,7 @@ def note!(node_or_token, message) sig { params(node_or_token: AST::Node, message: String).returns(NilClass) } def warning!(node_or_token, message) T.bind(self, T.untyped) rescue nil - token = diagnostic_token(node_or_token) + token = source_error_token(diagnostic_token(node_or_token)) loc = token ? " (line #{token.line})" : "" diagnostic_output("\e[33m[Warning]\e[0m #{message}#{loc}") nil @@ -137,36 +141,25 @@ def fixable!(node_or_token, category:, fixes:, message: nil, code: nil, level: : T.bind(self, T.untyped) rescue nil rendered_message = message || diagnostic_message(T.must(code), **kwargs) token = diagnostic_token(node_or_token) + source_token = source_error_token(token) finding = FixableFinding.new( level: level, message: rendered_message, token: token, category: category, fixes: fixes ) if FixCollector.enabled? - FixCollector.push(finding) + FixCollector.push(finding, level == :error) return unless raise_in_collector - err_class = self.class.name&.include?("Parser") ? ParserError : CompilerError - source_token = source_error_token(token) - raise err_class.new( - source_token, - rendered_message, - diagnostic_source_code - ) + raise_source_error!(source_token, rendered_message) end case level when :hint, :info, :warning - loc = token ? " (line #{token.line})" : "" + loc = source_token ? " (line #{source_token.line})" : "" tag = level == :warning ? "\e[33m[Warning]\e[0m" : "\e[36m[#{level.to_s.capitalize}]\e[0m" diagnostic_output("#{tag} #{rendered_message}#{loc}") when :error - err_class = self.class.name&.include?("Parser") ? ParserError : CompilerError - source_token = source_error_token(token) - raise err_class.new( - source_token, - rendered_message, - diagnostic_source_code - ) + raise_source_error!(source_token, rendered_message) end end @@ -175,6 +168,26 @@ def diagnostic_source_code source_code end + sig { returns(T::Boolean) } + def parser_error_host? + false + end + + sig do + params( + token: T.nilable(Lexer::Token), + message: String, + code: T.nilable(Symbol), + ).returns(T.noreturn) + end + def raise_source_error!(token, message, code: nil) + if parser_error_host? + Kernel.raise ParserError.new(token, message, diagnostic_source_code, code: code) + end + + Kernel.raise CompilerError.new(token, message, diagnostic_source_code, code: code) + end + sig { params(node_or_token: T.untyped).returns(DiagnosticToken) } def diagnostic_token(node_or_token) token = node_or_token.respond_to?(:token) ? node_or_token.token : node_or_token @@ -195,7 +208,8 @@ def diagnostic_output(message) nil end - private :format_diagnostic_template, :diagnostic_source_code, :diagnostic_token, :source_error_token, :diagnostic_output + private :format_diagnostic_template, :diagnostic_source_code, :diagnostic_token, + :source_error_token, :diagnostic_output, :raise_source_error! end diff --git a/compiler/ruby/ast/std_lib.rb b/compiler/ruby/ast/std_lib.rb index a8fa85e4b..8f857a68e 100644 --- a/compiler/ruby/ast/std_lib.rb +++ b/compiler/ruby/ast/std_lib.rb @@ -254,6 +254,9 @@ { args: [:Int64], return: :Float64, zig: "@as(f64, @floatFromInt({0}))", bc: true, is_method: true, }, + { args: [:UInt64], return: :Float64, zig: "@as(f64, @floatFromInt({0}))", bc: true, + is_method: true, + }, { args: [:Float64], return: :Float64, zig: "{0}", bc: true, is_method: true, } @@ -1340,11 +1343,10 @@ class StdLibTypeBinding < T::Struct args: [:"String{}", :String, { type: :Any, takes: true }], numeric_zig: "try CheatLib.numericMapPut({key_zig}, {val_zig}, {alloc}, &{0}, {1}, {2})", validate: ->(node, args, obj_type, error_fn) { - key_type = Type.new(args[0].resolved_type) - if obj_type.numeric_map? - error_fn.call(node, "HashMap.put: key must be a numeric type, got #{args[0].resolved_type}") unless key_type.numeric? - else - error_fn.call(node, "HashMap.put: key must be a String, got #{args[0].resolved_type}") unless key_type.string? + actual_key = args[0].full_type!(context: "HashMap.put key") + expected_key = obj_type.key_type + unless obj_type.accepts_map_key?(actual_key) + error_fn.call(node, "HashMap.put: key must be #{Type.surface_name(expected_key)}, got #{Type.surface_name(actual_key)}") end }, return_type: :Void, @@ -1358,11 +1360,10 @@ class StdLibTypeBinding < T::Struct mutates_receiver: true, numeric_zig: "CheatLib.numericMapDelete({key_zig}, {val_zig}, {alloc}, &{0}, {1})", validate: ->(node, args, obj_type, error_fn) { - arg_type = Type.new(args[0].resolved_type) - if obj_type.numeric_map? - error_fn.call(node, "HashMap.delete: key must be a numeric type, got #{args[0].resolved_type}") unless arg_type.numeric? - else - error_fn.call(node, "HashMap.delete: key must be a String, got #{args[0].resolved_type}") unless arg_type.string? + actual_key = args[0].full_type!(context: "HashMap.delete key") + expected_key = obj_type.key_type + unless obj_type.accepts_map_key?(actual_key) + error_fn.call(node, "HashMap.delete: key must be #{Type.surface_name(expected_key)}, got #{Type.surface_name(actual_key)}") end }, return_type: :Void, @@ -1375,11 +1376,10 @@ class StdLibTypeBinding < T::Struct bc: true, numeric_zig: "CheatLib.numericMapContains({key_zig}, {val_zig}, {0}, {1})", validate: ->(node, args, obj_type, error_fn) { - arg_type = Type.new(args[0].resolved_type) - if obj_type.numeric_map? - error_fn.call(node, "HashMap.contains?: key must be a numeric type, got #{args[0].resolved_type}") unless arg_type.numeric? - else - error_fn.call(node, "HashMap.contains?: key must be a String, got #{args[0].resolved_type}") unless arg_type.string? + actual_key = args[0].full_type!(context: "HashMap.contains? key") + expected_key = obj_type.key_type + unless obj_type.accepts_map_key?(actual_key) + error_fn.call(node, "HashMap.contains?: key must be #{Type.surface_name(expected_key)}, got #{Type.surface_name(actual_key)}") end }, return_type: :Bool, diff --git a/compiler/ruby/ast/struct_field.rb b/compiler/ruby/ast/struct_field.rb index 1b97aadea..96b0fcfce 100644 --- a/compiler/ruby/ast/struct_field.rb +++ b/compiler/ruby/ast/struct_field.rb @@ -4,15 +4,13 @@ require "sorbet-runtime" module AST + # Acyclic foundation record: field-declaration metadata with no dependency + # on semantic Type. The Type-returning `type` accessor is defined alongside + # Type itself (type.rb) so this file's generated package stays leaf-level. # ruby-to-clear: pub StructField = Struct.new(:type, :default, :borrowed, keyword_init: true) do extend T::Sig - sig { returns(Type) } - def type - T.cast(self[:type], Type) - end - sig { returns(T.untyped) } def default self[:default] diff --git a/compiler/ruby/ast/symbol_entry.rb b/compiler/ruby/ast/symbol_entry.rb index f22f174ab..44106b50c 100644 --- a/compiler/ruby/ast/symbol_entry.rb +++ b/compiler/ruby/ast/symbol_entry.rb @@ -31,8 +31,8 @@ # Load its implementation here so isolated tooling such as Mutant can evaluate # these signatures without relying on a broader compiler require order. require_relative "type" +require_relative "param" require_relative "../annotator/helpers/function_signature" -require_relative "schemas" require_relative "async_result_shape" # Scope and SymbolEntry are a mutual back-reference: scope.rb requires @@ -50,6 +50,7 @@ class SymbolEntry @next_binding_id = T.let(0, Integer) TypeInput = T.type_alias { T.nilable(T.any(Type::TypeInput, FunctionSignature)) } + RegInput = T.type_alias { T.nilable(T.any(AST::Node, String, Symbol)) } LifetimeSourceInput = T.type_alias { T.any(SymbolEntry, Symbol) } LifetimeInput = T.type_alias { T.nilable(T.any(Symbol, T::Array[LifetimeSourceInput], T::Hash[Symbol, T::Array[LifetimeSourceInput]])) } @@ -89,9 +90,26 @@ class BindingLifecycleFacts < T::Struct # value capture does not provide a writable source slot, so moving this # payload would leave the enclosing optional cleanup owning the same data. prop :owned_optional_capture, T::Boolean, default: false - end - - attr_accessor :reg, :mutable, :rebindable, + # Keep-analysis (retained identity v4): this param's value flows into an + # identity destination, so its ABI is the family's handle and every call + # edge consumes a CallEdgeOwnershipPlan derived from the caller's + # declared model. nil = not kept. + prop :kept_identity, T.nilable(KeptIdentityContract), default: nil + # Retained-identity v5 parameter carrier contract: :polymorphic (default, + # carrier-preserving), :unique (exclusively owned, enables COPY), :shared + # (requires a retained-identity family). Written once at param binding. + prop :carrier_contract, Symbol, default: :polymorphic + # True when this binding's CARRIER is statically unknown -- an + # unconstrained TAKES parameter, or a local directly aliasing one. COPY + # (which needs independent identity) is rejected on such bindings; the + # default :polymorphic contract on an ordinary local does NOT set this. + prop :carrier_polymorphic, T::Boolean, default: false + end + + sig { returns(RegInput) } + attr_accessor :reg + + attr_accessor :mutable, :rebindable, :size, :capabilities, :scope, # Back-reference to owning Scope (set by Scope#declare) :scope_depth, # declaring scope depth (0 = root) @@ -106,40 +124,128 @@ class BindingLifecycleFacts < T::Struct sig { returns(BindingLifecycleFacts) } attr_reader :lifecycle - class << self - extend T::Sig + sig { returns(T.nilable(AsyncResultShape)) } + def async_result_shape = lifecycle.async_result_shape - sig { params(name: Symbol).void } - def lifecycle_attr(name) - define_method(name) do - T.bind(self, SymbolEntry).lifecycle.public_send(name) - end - define_method(:"#{name}=") do |value| - T.bind(self, SymbolEntry).lifecycle.public_send(:"#{name}=", value) - end - end + sig { params(value: T.nilable(AsyncResultShape)).void } + def async_result_shape=(value) + @lifecycle.async_result_shape = value + end - sig { params(name: Symbol).void } - def flow_attr(name) - define_method(name) do - T.bind(self, SymbolEntry).flow_facts.public_send(name) - end - end + sig { returns(Type) } + def type = lifecycle.type + + sig { returns(Symbol) } + def storage = lifecycle.storage + + sig { params(value: Symbol).void } + def storage=(value) + @lifecycle.storage = value + end + + sig { returns(T.nilable(Symbol)) } + def sync = lifecycle.sync + + sig { params(value: T.nilable(Symbol)).void } + def sync=(value) + @lifecycle.sync = value + end + + sig { returns(T.nilable(Symbol)) } + def layout = lifecycle.layout + + sig { params(value: T.nilable(Symbol)).void } + def layout=(value) + @lifecycle.layout = value + end + + sig { returns(T.nilable(T::Boolean)) } + def resource = lifecycle.resource + + sig { params(value: T.nilable(T::Boolean)).void } + def resource=(value) + @lifecycle.resource = value + end + + sig { returns(T.nilable(Schemas::ResourceClosePlan)) } + def close_plan = lifecycle.close_plan + + sig { params(value: T.nilable(Schemas::ResourceClosePlan)).void } + def close_plan=(value) + @lifecycle.close_plan = value + end + + sig { returns(T.nilable(Symbol)) } + def ownership_kind = lifecycle.ownership_kind + + sig { params(value: T.nilable(Symbol)).void } + def ownership_kind=(value) + @lifecycle.ownership_kind = value end - lifecycle_attr :async_result_shape - lifecycle_attr :type - lifecycle_attr :storage - lifecycle_attr :sync - lifecycle_attr :layout - lifecycle_attr :resource - lifecycle_attr :close_plan - lifecycle_attr :ownership_kind - lifecycle_attr :takes - lifecycle_attr :is_param - lifecycle_attr :link_source - lifecycle_attr :foreign_out_owner - lifecycle_attr :owned_optional_capture + sig { returns(T::Boolean) } + def takes = lifecycle.takes + + sig { params(value: T::Boolean).void } + def takes=(value) + @lifecycle.takes = value + end + + sig { returns(T::Boolean) } + def is_param = lifecycle.is_param + + sig { params(value: T::Boolean).void } + def is_param=(value) + @lifecycle.is_param = value + end + + sig { returns(T.nilable(KeptIdentityContract)) } + def kept_identity = lifecycle.kept_identity + + sig { params(value: T.nilable(KeptIdentityContract)).void } + def kept_identity=(value) + @lifecycle.kept_identity = value + end + + sig { returns(Symbol) } + def carrier_contract = lifecycle.carrier_contract + + sig { params(value: Symbol).void } + def carrier_contract=(value) + @lifecycle.carrier_contract = value + end + + sig { returns(T::Boolean) } + def carrier_polymorphic = lifecycle.carrier_polymorphic + + sig { params(value: T::Boolean).void } + def carrier_polymorphic=(value) + @lifecycle.carrier_polymorphic = value + end + + sig { returns(T.nilable(Symbol)) } + def link_source = lifecycle.link_source + + sig { params(value: T.nilable(Symbol)).void } + def link_source=(value) + @lifecycle.link_source = value + end + + sig { returns(T::Boolean) } + def foreign_out_owner = lifecycle.foreign_out_owner + + sig { params(value: T::Boolean).void } + def foreign_out_owner=(value) + @lifecycle.foreign_out_owner = value + end + + sig { returns(T::Boolean) } + def owned_optional_capture = lifecycle.owned_optional_capture + + sig { params(value: T::Boolean).void } + def owned_optional_capture=(value) + @lifecycle.owned_optional_capture = value + end sig { returns(T.nilable(Integer)) } def semantic_place_id @@ -148,23 +254,37 @@ def semantic_place_id sig { params(value: Integer).void } def adopt_semantic_place_id!(value) - @lifecycle.semantic_place_id ||= value + return unless @lifecycle.semantic_place_id.nil? + + @lifecycle.semantic_place_id = value end - flow_attr :non_escaping - flow_attr :borrowed_alias - flow_attr :valid - flow_attr :invalid_reason - flow_attr :read - flow_attr :mutated - flow_attr :mutable_ref_target - flow_attr :poly_borrow_target - flow_attr :init_contents_heap + sig { returns(T::Boolean) } + def non_escaping = flow_facts.non_escaping - class << self - undef_method :lifecycle_attr - undef_method :flow_attr - end + sig { returns(T::Boolean) } + def borrowed_alias = flow_facts.borrowed_alias + + sig { returns(T::Boolean) } + def valid = flow_facts.valid + + sig { returns(T.nilable(String)) } + def invalid_reason = flow_facts.invalid_reason + + sig { returns(T::Boolean) } + def read = flow_facts.read + + sig { returns(T::Boolean) } + def mutated = flow_facts.mutated + + sig { returns(T::Boolean) } + def mutable_ref_target = flow_facts.mutable_ref_target + + sig { returns(T::Boolean) } + def poly_borrow_target = flow_facts.poly_borrow_target + + sig { returns(T::Boolean) } + def init_contents_heap = flow_facts.init_contents_heap sig { returns(T::Array[SymbolEntry]) } attr_reader :lifetime @@ -187,7 +307,17 @@ def inherit_ownership_identity!(source) sig { params(value: LifetimeInput).void } def lifetime=(value) @lifetime = normalize_lifetime(value) - @flow.non_escaping = @lifetime.length == 1 && @lifetime.first.equal?(self) + @flow.non_escaping = lifetime_self_only? + end + + sig { returns(T::Boolean) } + def lifetime_self_only? + return false unless @lifetime.length == 1 + + source = @lifetime.first + return false unless source + + source.binding_id == @binding_id end # A function binding is a Type whose @raw is its FunctionSignature @@ -199,7 +329,13 @@ def lifetime=(value) # require ordering. sig { returns(T.nilable(FunctionSignature)) } def fn_signature - T.unsafe(type.function_signature) + function_type = type.function_type + return nil unless function_type + + signature = function_type.source_signature + return nil unless signature + + T.cast(signature, FunctionSignature) end # Backward-compat alias for `lifetime == :current_scope`. @@ -537,23 +673,17 @@ def self.tied_lifetime(sources) sources.uniq end - sig { params(reg: T.untyped, type: TypeInput, mutable: T::Boolean, storage: Symbol, sync: T.nilable(Symbol), layout: T.nilable(Symbol), rebindable: T::Boolean, size: Integer, capabilities: T::Set[Symbol], valid: T::Boolean, invalid_reason: T.nilable(String), resource: T.nilable(T::Boolean), close_plan: T.nilable(Schemas::ResourceClosePlan)).void } + sig { params(reg: RegInput, type: TypeInput, mutable: T::Boolean, storage: Symbol, sync: T.nilable(Symbol), layout: T.nilable(Symbol), rebindable: T::Boolean, size: Integer, capabilities: T::Set[Symbol], valid: T::Boolean, invalid_reason: T.nilable(String), resource: T.nilable(T::Boolean), close_plan: T.nilable(Schemas::ResourceClosePlan)).void } + # ruby-to-clear: fallible def initialize(reg:, type:, mutable:, storage:, sync: nil, layout: nil, rebindable: false, size: 0, capabilities: Set.new, valid: true, invalid_reason: nil, resource: nil, close_plan: nil) @binding_id = T.let(self.class.next_binding_id, Integer) @ownership_binding_id = T.let(@binding_id, Integer) @reg = reg - normalized_type = if type.nil? - Type.new(:Untyped) - elsif type.is_a?(FunctionSignature) - Type.from_function_signature(type) - else - Type.new(type) - end @lifecycle = T.let( BindingLifecycleFacts.new( - type: normalized_type, + type: self.class.normalize_type_input(type), storage: storage, sync: sync, layout: layout, @@ -581,18 +711,28 @@ def initialize(reg:, type:, mutable:, storage:, sync: nil, layout: nil, rebindab # single Type. The runtime sig now enforces the accepted domain -- # anything outside it is a compiler bug, surfaced here. sig { params(val: TypeInput).void } + # ruby-to-clear: fallible def type=(val) - @lifecycle.type = if val.nil? - Type.new(:Untyped) - elsif val.is_a?(FunctionSignature) - Type.from_function_signature(val) - else - Type.new(val) - end + @lifecycle.type = self.class.normalize_type_input(val) end private + sig { params(value: TypeInput).returns(Type) } + # ruby-to-clear: fallible + def self.normalize_type_input(value) + return Type.new(:Untyped) if value.nil? + return type_from_function_signature(value) if value.is_a?(FunctionSignature) + + Type.new(value) + end + + sig { params(signature: FunctionSignature).returns(Type) } + def self.type_from_function_signature(signature) + param_types = signature.params.map(&:type) + Type.function_type_from_parts(param_types, signature.return_type, signature.reentrant, signature) + end + sig { returns(Integer) } def self.next_binding_id id = @next_binding_id @@ -602,19 +742,26 @@ def self.next_binding_id sig { params(value: LifetimeInput).returns(T::Array[SymbolEntry]) } def normalize_lifetime(value) - return [self] if value == :current_scope + if value.is_a?(Symbol) + return [self] if value == :current_scope + end - sources = if value.is_a?(Hash) - value[:sources] - else - value + sources = T.let([], T::Array[LifetimeSourceInput]) + if value.is_a?(Hash) + sources = value[:sources] || [] + elsif value.is_a?(Array) + sources = value + elsif value + sources = [value] end - Array(sources).map do |source| + normalized = T.let([], T::Array[SymbolEntry]) + sources.each do |source| unless source.is_a?(SymbolEntry) raise TypeError, "SymbolEntry#lifetime sources must be SymbolEntry instances" end - source - end.uniq + normalized << source + end + normalized.uniq end end diff --git a/compiler/ruby/ast/syntax_typo_scanner.rb b/compiler/ruby/ast/syntax_typo_scanner.rb index 5d290e79b..14886bb57 100644 --- a/compiler/ruby/ast/syntax_typo_scanner.rb +++ b/compiler/ruby/ast/syntax_typo_scanner.rb @@ -107,7 +107,9 @@ def self.scan!(source) next end RULES.each do |r| - pat = r.match + # Keep the matched text independently owned: emitting a finding below + # may mutably borrow the rule while `pat` remains live. + pat = r.match.dup next unless source[i, pat.length] == pat if pat[0] =~ /[A-Za-z_]/ && i > 0 && source[i - 1] =~ /[A-Za-z0-9_]/ next @@ -148,17 +150,17 @@ def self.emit_legacy_mutation_suffix_finding!(line, col) replacement: '' )] ) - anchor = Struct.new(:line, :column).new(line, col) + anchor = AnchorToken.new(line, col) FixCollector.push(FixableFinding.new( level: :error, - message: T.must(DiagnosticRegistry.format(:LEGACY_MUTATION_NAME_SUFFIX)), + message: T.must(DiagnosticRegistry.format(:LEGACY_MUTATION_NAME_SUFFIX, [])), token: anchor, category: :mutability, fixes: [fix] - )) + ), true) end - sig { params(source: String, i: Integer, line: Integer, col: Integer).returns(T::Array[Integer]) } + sig { params(source: String, i: Integer, line: Integer, col: Integer).returns([Integer, Integer, Integer]) } def self.advance(source, i, line, col) if source[i] == "\n" [i + 1, line + 1, 1] @@ -183,7 +185,7 @@ def self.emit_typo_finding!(line, col, rule) )] ) - anchor = Struct.new(:line, :column).new(line, col) + anchor = AnchorToken.new(line, col) message = T.must(DiagnosticRegistry.format( :OPERATOR_TYPO_SUGGESTION, match: rule.match, @@ -196,6 +198,6 @@ def self.emit_typo_finding!(line, col, rule) category: :type, fixes: [fix] ) - FixCollector.push(finding) + FixCollector.push(finding, true) end end diff --git a/compiler/ruby/ast/type.rb b/compiler/ruby/ast/type.rb index 2d659839e..346fe5b7e 100644 --- a/compiler/ruby/ast/type.rb +++ b/compiler/ruby/ast/type.rb @@ -1,222 +1,9 @@ # typed: strict require "sorbet-runtime" +require "set" require_relative "lexer" require_relative "struct_field" - -class TypeCapabilitySuffix < T::Struct - const :base, String - const :ownership, T.nilable(Symbol) - const :sync, T.nilable(Symbol) -end - -class TypeCapabilityUnset < T::Struct -end - -# ruby-to-clear: value -class TypeCapabilities - extend T::Sig - - UNSET = T.let(TypeCapabilityUnset.new.freeze, TypeCapabilityUnset) - MaybeSymbol = T.type_alias { T.any(TypeCapabilityUnset, Symbol, NilClass) } - MaybeInteger = T.type_alias { T.any(TypeCapabilityUnset, Integer, NilClass) } - MaybeBoolean = T.type_alias { T.any(TypeCapabilityUnset, T::Boolean) } - MaybeToken = T.type_alias { T.any(TypeCapabilityUnset, Lexer::Token, NilClass) } - - sig { returns(T.nilable(Symbol)) } - attr_reader :ownership, :sync, :layout, :collection, :elem_ownership, :elem_sync, - :elem_layout, :link_source, :observable_terminal - sig { returns(T.nilable(Integer)) } - attr_reader :lock_rank, :shard_count - sig { returns(T::Boolean) } - attr_reader :ownership_set, :soa, :observable, :polymorphic_shared - sig { returns(T.nilable(Lexer::Token)) } - attr_reader :observable_token - - sig do - params( - ownership: T.nilable(Symbol), - ownership_set: T::Boolean, - sync: T.nilable(Symbol), - layout: T.nilable(Symbol), - lock_rank: T.nilable(Integer), - collection: T.nilable(Symbol), - shard_count: T.nilable(Integer), - soa: T::Boolean, - elem_ownership: T.nilable(Symbol), - elem_sync: T.nilable(Symbol), - elem_layout: T.nilable(Symbol), - link_source: T.nilable(Symbol), - observable: T::Boolean, - observable_terminal: T.nilable(Symbol), - observable_token: T.nilable(Lexer::Token), - polymorphic_shared: T::Boolean - ).void - end - def initialize( - ownership: nil, - ownership_set: false, - sync: nil, - layout: nil, - lock_rank: nil, - collection: nil, - shard_count: nil, - soa: false, - elem_ownership: nil, - elem_sync: nil, - elem_layout: nil, - link_source: nil, - observable: false, - observable_terminal: nil, - observable_token: nil, - polymorphic_shared: false - ) - @ownership = ownership - @ownership_set = ownership_set - @sync = sync - @layout = layout - @lock_rank = lock_rank - @collection = collection - @shard_count = shard_count - @soa = soa - @elem_ownership = elem_ownership - @elem_sync = elem_sync - @elem_layout = elem_layout - @link_source = link_source - @observable = observable - @observable_terminal = observable_terminal - @observable_token = observable_token - @polymorphic_shared = polymorphic_shared - freeze - end - - sig { returns(TypeCapabilities) } - def copy - self - end - - sig do - params( - ownership: MaybeSymbol, - ownership_set: MaybeBoolean, - sync: MaybeSymbol, - layout: MaybeSymbol, - lock_rank: MaybeInteger, - collection: MaybeSymbol, - shard_count: MaybeInteger, - soa: MaybeBoolean, - elem_ownership: MaybeSymbol, - elem_sync: MaybeSymbol, - elem_layout: MaybeSymbol, - link_source: MaybeSymbol, - observable: MaybeBoolean, - observable_terminal: MaybeSymbol, - observable_token: MaybeToken, - polymorphic_shared: MaybeBoolean - ).returns(TypeCapabilities) - end - def with( - ownership: UNSET, - ownership_set: UNSET, - sync: UNSET, - layout: UNSET, - lock_rank: UNSET, - collection: UNSET, - shard_count: UNSET, - soa: UNSET, - elem_ownership: UNSET, - elem_sync: UNSET, - elem_layout: UNSET, - link_source: UNSET, - observable: UNSET, - observable_terminal: UNSET, - observable_token: UNSET, - polymorphic_shared: UNSET - ) - next_ownership = T.let(ownership.equal?(UNSET) ? self.ownership : T.cast(ownership, T.nilable(Symbol)), T.nilable(Symbol)) - next_ownership_set = T.let( - ownership_set.equal?(UNSET) ? (!ownership.equal?(UNSET) || self.ownership_set) : T.cast(ownership_set, T::Boolean), - T::Boolean - ) - next_sync = T.let(sync.equal?(UNSET) ? self.sync : T.cast(sync, T.nilable(Symbol)), T.nilable(Symbol)) - next_layout = T.let(layout.equal?(UNSET) ? self.layout : T.cast(layout, T.nilable(Symbol)), T.nilable(Symbol)) - next_lock_rank = T.let(lock_rank.equal?(UNSET) ? self.lock_rank : T.cast(lock_rank, T.nilable(Integer)), T.nilable(Integer)) - next_collection = T.let(collection.equal?(UNSET) ? self.collection : T.cast(collection, T.nilable(Symbol)), T.nilable(Symbol)) - next_shard_count = T.let(shard_count.equal?(UNSET) ? self.shard_count : T.cast(shard_count, T.nilable(Integer)), T.nilable(Integer)) - next_soa = T.let(soa.equal?(UNSET) ? self.soa : T.cast(soa, T::Boolean), T::Boolean) - next_elem_ownership = T.let(elem_ownership.equal?(UNSET) ? self.elem_ownership : T.cast(elem_ownership, T.nilable(Symbol)), T.nilable(Symbol)) - next_elem_sync = T.let(elem_sync.equal?(UNSET) ? self.elem_sync : T.cast(elem_sync, T.nilable(Symbol)), T.nilable(Symbol)) - next_elem_layout = T.let(elem_layout.equal?(UNSET) ? self.elem_layout : T.cast(elem_layout, T.nilable(Symbol)), T.nilable(Symbol)) - next_link_source = T.let(link_source.equal?(UNSET) ? self.link_source : T.cast(link_source, T.nilable(Symbol)), T.nilable(Symbol)) - next_observable = T.let(observable.equal?(UNSET) ? self.observable : T.cast(observable, T::Boolean), T::Boolean) - next_observable_terminal = T.let(observable_terminal.equal?(UNSET) ? self.observable_terminal : T.cast(observable_terminal, T.nilable(Symbol)), T.nilable(Symbol)) - next_observable_token = T.let(observable_token.equal?(UNSET) ? self.observable_token : T.cast(observable_token, T.nilable(Lexer::Token)), T.nilable(Lexer::Token)) - next_polymorphic_shared = T.let(polymorphic_shared.equal?(UNSET) ? self.polymorphic_shared : T.cast(polymorphic_shared, T::Boolean), T::Boolean) - - return self if next_ownership == self.ownership && next_ownership_set == self.ownership_set && - next_sync == self.sync && next_layout == self.layout && next_lock_rank == self.lock_rank && - next_collection == self.collection && next_shard_count == self.shard_count && next_soa == self.soa && - next_elem_ownership == self.elem_ownership && next_elem_sync == self.elem_sync && - next_elem_layout == self.elem_layout && next_link_source == self.link_source && - next_observable == self.observable && next_observable_terminal == self.observable_terminal && - next_observable_token == self.observable_token && next_polymorphic_shared == self.polymorphic_shared - - TypeCapabilities.new( - ownership: next_ownership, - ownership_set: next_ownership_set, - sync: next_sync, - layout: next_layout, - lock_rank: next_lock_rank, - collection: next_collection, - shard_count: next_shard_count, - soa: next_soa, - elem_ownership: next_elem_ownership, - elem_sync: next_elem_sync, - elem_layout: next_elem_layout, - link_source: next_link_source, - observable: next_observable, - observable_terminal: next_observable_terminal, - observable_token: next_observable_token, - polymorphic_shared: next_polymorphic_shared - ) - end - - sig { returns(TypeCapabilities) } - def without_runtime_wrappers - with( - ownership: :affine, - ownership_set: false, - sync: nil, - layout: nil, - elem_ownership: nil, - elem_sync: nil, - elem_layout: nil - ) - end - - sig { returns(T::Boolean) } - def inline_migration_safe? - return false unless lock_rank.nil? && elem_ownership.nil? && elem_sync.nil? && - elem_layout.nil? && link_source.nil? && observable_terminal.nil? - return false if polymorphic_shared - - collection.nil? || collection == :list || collection == :set || collection == :pool - end - - sig { returns(T::Boolean) } - def explicit_layer_capability? - (!ownership.nil? && ownership != :affine) || !sync.nil? || !layout.nil? || !lock_rank.nil? || - !shard_count.nil? || soa || !elem_ownership.nil? || !elem_sync.nil? || - !elem_layout.nil? || !link_source.nil? || observable || - !observable_terminal.nil? || polymorphic_shared - end - - sig { params(ownership: MaybeSymbol, sync: MaybeSymbol, layout: MaybeSymbol).returns(T::Boolean) } - def element_update_requested?(ownership:, sync:, layout:) - !ownership.equal?(UNSET) || !sync.equal?(UNSET) || !layout.equal?(UNSET) || - !elem_ownership.nil? || !elem_sync.nil? || !elem_layout.nil? - end -end - +require_relative "type_capabilities" class TypePlacementUnset < T::Struct end @@ -273,6 +60,8 @@ def alloc # ruby-to-clear: value class Type + extend T::Sig + ArrayCapacity = T.type_alias { T.nilable(T.any(Integer, Symbol)) } # ruby-to-clear: pub @@ -288,16 +77,69 @@ class FunctionType < T::Struct const :source_signature, T.nilable(BasicObject), default: nil const :abi, Symbol, default: :clear end + + # Boundary converters between the semantic FunctionType and the foundation's + # FunctionSignatureExpression. The foundation spells signatures as + # TypeExpressions and never sees FunctionType; the original semantic object + # rides along as the opaque semantic_payload so a round-trip is lossless + # (source_signature included). + sig { params(signature: FunctionType).returns(TypeExpressionKind) } + def self.function_type_expression_for(signature) + FunctionTypeExpression.new(signature: FunctionSignatureExpression.new( + params: signature.params.map { |param| FunctionParamExpression.new(expression: param.type.shape.expression) }, + return_expression: signature.return_type.shape.expression, + reentrant: signature.reentrant, + abi: signature.abi, + semantic_payload: signature, + )) + end + + sig { params(expression: FunctionTypeExpression).returns(FunctionType) } + # ruby-to-clear: fallible + # ruby-to-clear: effects reentrant + def self.function_type_for_expression(expression) + payload = T.cast(expression.signature.semantic_payload, T.nilable(FunctionType)) + return payload unless payload.nil? + + FunctionType.new( + params: expression.signature.params.map { |param| FunctionTypeParam.new(type: Type.new(param.expression)) }, + return_type: Type.new(expression.signature.return_expression), + reentrant: expression.signature.reentrant, + abi: expression.signature.abi, + ) + end end require_relative "type_expression" +class Type + extend T::Sig + + sig { params(kind: TypeExpressionKind).returns(TypeExpressionKind) } + def self.unwrap_fallible_kind(kind) + return kind unless kind.is_a?(FallibleTypeExpression) + + kind.inner.kind + end + + sig { params(kind: TypeExpressionKind).returns(TypeExpressionKind) } + def self.unwrap_optional_kind(kind) + return kind unless kind.is_a?(OptionalTypeExpression) + + kind.inner.kind + end +end + # ruby-to-clear: value class TypeShape < T::Struct extend T::Sig Raw = T.type_alias { T.any(Type::FunctionType, Symbol, String) } CORE_CACHE_LIMIT = 4096 + # The host Ruby compiler interns common shapes for throughput. CLEAR has no + # module teardown lifetime yet, so its self-hosted path constructs the same + # immutable value directly instead of retaining a process-global heap map. + # ruby-to-clear: skip CORE_CACHE = T.let({}, T::Hash[String, TypeShape]) const :auto, T::Boolean @@ -315,6 +157,8 @@ class TypeShape < T::Struct wrapped_function_type_raw: T.nilable(Type::FunctionType) ).returns(TypeShape) end + # ruby-to-clear: fallible + # ruby-to-clear: effects reentrant def self.from_raw( raw:, auto: false, @@ -323,12 +167,16 @@ def self.from_raw( wrapped_type_raw: nil, wrapped_function_type_raw: nil ) - parsed = T.let(expression || TypeExpressionParser.parse(raw), TypeExpression) + parsed = T.let( + expression || + (raw.is_a?(Type::FunctionType) ? TypeExpression.of(Type.function_type_expression_for(raw)) : TypeExpressionParser.parse(raw)), + TypeExpression + ) if optional if wrapped_function_type_raw - parsed = OptionalTypeExpression.new(inner: TypeExpressionParser.parse(wrapped_function_type_raw)) + parsed = TypeExpression.of(OptionalTypeExpression.new(inner: TypeExpression.of(Type.function_type_expression_for(wrapped_function_type_raw)))) elsif wrapped_type_raw - parsed = OptionalTypeExpression.new(inner: TypeExpressionParser.parse(wrapped_type_raw)) + parsed = TypeExpression.of(OptionalTypeExpression.new(inner: TypeExpressionParser.parse(wrapped_type_raw))) end end TypeShape.new( @@ -340,16 +188,23 @@ def self.from_raw( end sig { params(core_str: String, auto: T::Boolean).returns(TypeShape) } + # ruby-to-clear: fallible + # ruby-to-clear: effects reentrant def self.from_core(core_str, auto: false) + # ruby-to-clear: skip key = T.let("#{auto}:#{core_str}", String) + # ruby-to-clear: skip cached = CORE_CACHE[key] + # ruby-to-clear: skip return cached if cached shape = TypeShape.from_raw(raw: core_str.to_sym, auto: auto) + # ruby-to-clear: skip if CORE_CACHE.length >= CORE_CACHE_LIMIT key_to_evict = CORE_CACHE.keys.first CORE_CACHE.delete(key_to_evict) if key_to_evict end + # ruby-to-clear: skip CORE_CACHE[key] = shape shape end @@ -360,6 +215,7 @@ def copy end sig { params(auto_value: T::Boolean).returns(TypeShape) } + # ruby-to-clear: effects reentrant def copy_with_auto(auto_value) return self if auto_value == auto @@ -367,6 +223,7 @@ def copy_with_auto(auto_value) end sig { params(next_expression: TypeExpression).returns(TypeShape) } + # ruby-to-clear: effects reentrant def with_expression(next_expression) TypeShape.from_raw(raw: :Any, auto: auto, expression: next_expression) end @@ -384,8 +241,12 @@ def semantic_key private sig { params(current: TypeExpression).returns(Raw) } + # ruby-to-clear: effects reentrant def self.render_legacy_raw(current) - return current.signature if current.is_a?(FunctionTypeExpression) + current_kind = current.kind + if current_kind.is_a?(FunctionTypeExpression) + return Type.function_type_for_expression(current_kind) + end root_caps = TypeExpressionTree.root_capabilities(current) shape_only = TypeExpressionTree.with_root_capabilities( @@ -400,17 +261,15 @@ def self.render_legacy_raw(current) sig { returns(Symbol) } def resolved raw_value = raw - case raw_value - when Type::FunctionType then :Any - when Symbol then raw_value - when String then raw_value.to_sym - else T.absurd(raw_value) - end + return :Any if raw_value.is_a?(Type::FunctionType) + return raw_value if raw_value.is_a?(Symbol) + + raw_value.to_sym end sig { returns(T::Boolean) } def fn_type? - expression.is_a?(FunctionTypeExpression) + expression.kind.is_a?(FunctionTypeExpression) end sig { returns(T::Boolean) } @@ -420,32 +279,39 @@ def array sig { returns(T::Boolean) } def map - structural_expression.is_a?(MapTypeExpression) + structural_expression.kind.is_a?(MapTypeExpression) end sig { returns(T::Boolean) } def optional current = expression - current = current.inner if current.is_a?(FallibleTypeExpression) - current.is_a?(OptionalTypeExpression) || - (current.is_a?(LinearTypeExpression) && current.item.is_a?(OptionalTypeExpression)) + current_kind = current.kind + current = current_kind.inner if current_kind.is_a?(FallibleTypeExpression) + kind = current.kind + return true if kind.is_a?(OptionalTypeExpression) + return false unless kind.is_a?(LinearTypeExpression) + + kind.item.kind.is_a?(OptionalTypeExpression) end sig { returns(T::Boolean) } def error_union - expression.is_a?(FallibleTypeExpression) + expression.kind.is_a?(FallibleTypeExpression) end sig { returns(T::Boolean) } def tense - expression.is_a?(FutureTypeExpression) || expression.is_a?(StreamTypeExpression) + kind = expression.kind + kind.is_a?(FutureTypeExpression) || kind.is_a?(StreamTypeExpression) end sig { returns(T::Boolean) } def generic_instance - structural = structural_expression - structural.is_a?(TupleTypeExpression) || - (structural.is_a?(NamedTypeExpression) && !structural.arguments.empty?) + structural = structural_expression.kind + return true if structural.is_a?(TupleTypeExpression) + return false unless structural.is_a?(NamedTypeExpression) + + !structural.arguments.empty? end sig { returns(Type::ArrayCapacity) } @@ -470,29 +336,31 @@ def allocation_hint sig { returns(T.nilable(Symbol)) } def payload_type_raw - current = expression - return nil unless current.is_a?(FallibleTypeExpression) + kind = expression.kind + return nil unless kind.is_a?(FallibleTypeExpression) - TypeExpressionPrinter.legacy(current.inner).to_sym + TypeExpressionPrinter.legacy(kind.inner).to_sym end sig { returns(T.nilable(Symbol)) } def wrapped_type_raw - current = expression - return nil unless current.is_a?(OptionalTypeExpression) - return nil if current.inner.is_a?(FunctionTypeExpression) + kind = expression.kind + return nil unless kind.is_a?(OptionalTypeExpression) + optional_kind = kind + return nil if optional_kind.inner.kind.is_a?(FunctionTypeExpression) - TypeExpressionPrinter.legacy(current.inner).to_sym + TypeExpressionPrinter.legacy(optional_kind.inner).to_sym end sig { returns(T.nilable(Type::FunctionType)) } + # ruby-to-clear: fallible def wrapped_function_type_raw - current = expression - return nil unless current.is_a?(OptionalTypeExpression) - inner = current.inner - return nil unless inner.is_a?(FunctionTypeExpression) + kind = expression.kind + return nil unless kind.is_a?(OptionalTypeExpression) + inner_kind = kind.inner.kind + return nil unless inner_kind.is_a?(FunctionTypeExpression) - inner.signature + Type.function_type_for_expression(inner_kind) end sig { returns(T.nilable(Symbol)) } @@ -501,13 +369,14 @@ def element_type_raw return nil if linear.nil? item = linear.item - item = item.inner if item.is_a?(OptionalTypeExpression) + item_kind = item.kind + item = item_kind.inner if item_kind.is_a?(OptionalTypeExpression) TypeExpressionPrinter.legacy(item).to_sym end sig { returns(T.nilable(Symbol)) } def key_type_raw - structural = structural_expression + structural = structural_expression.kind return nil unless structural.is_a?(MapTypeExpression) TypeExpressionPrinter.legacy(structural.key).to_sym @@ -515,7 +384,7 @@ def key_type_raw sig { returns(T.nilable(Symbol)) } def value_type_raw - structural = structural_expression + structural = structural_expression.kind return nil unless structural.is_a?(MapTypeExpression) TypeExpressionPrinter.legacy(structural.value).to_sym @@ -523,40 +392,43 @@ def value_type_raw sig { returns(T.nilable(Symbol)) } def generic_base_raw - structural = structural_expression + structural = structural_expression.kind return :Tuple if structural.is_a?(TupleTypeExpression) - return structural.name if structural.is_a?(NamedTypeExpression) && !structural.arguments.empty? + if structural.is_a?(NamedTypeExpression) + named = structural + return named.name unless named.arguments.empty? + end nil end sig { returns(T::Array[Symbol]) } def generic_args_raw - structural = structural_expression - items = if structural.is_a?(TupleTypeExpression) - structural.items + structural = structural_expression.kind + items = T.let([], T::Array[TypeExpression]) + if structural.is_a?(TupleTypeExpression) + structural.items.each { |item| items << item } elsif structural.is_a?(NamedTypeExpression) - structural.arguments - else - [] + structural.arguments.each { |item| items << item } end items.map { |item| TypeExpressionPrinter.legacy(item).to_sym } end sig { returns(T.nilable(Symbol)) } def tense_type_raw - current = expression - if current.is_a?(StreamTypeExpression) + kind = expression.kind + if kind.is_a?(StreamTypeExpression) + stream_kind = kind dimension = T.let( - current.cardinality == :FINITE ? :LIST : current.cardinality, + stream_kind.cardinality == :FINITE ? :LIST : stream_kind.cardinality, TypeExpression::Dimension ) - linear = LinearTypeExpression.new(kind: :array, dimensions: [dimension], item: current.item) + linear = TypeExpression.of(LinearTypeExpression.new(kind: :array, dimensions: [dimension], item: stream_kind.item)) return TypeExpressionPrinter.legacy(linear).to_sym end - return nil unless current.is_a?(FutureTypeExpression) + return nil unless kind.is_a?(FutureTypeExpression) - TypeExpressionPrinter.legacy(current.inner).to_sym + TypeExpressionPrinter.legacy(kind.inner).to_sym end sig { returns(T::Boolean) } @@ -568,7 +440,7 @@ def numeric_map? sig { returns(T.nilable(LinearTypeExpression)) } def linear_expression - structural = structural_expression + structural = structural_expression.kind return structural if structural.is_a?(LinearTypeExpression) nil @@ -577,9 +449,12 @@ def linear_expression sig { returns(TypeExpression) } def structural_expression structural = T.let(expression, TypeExpression) - structural = structural.inner if structural.is_a?(FallibleTypeExpression) - if structural.is_a?(OptionalTypeExpression) && !structural.inner.is_a?(LinearTypeExpression) - structural = structural.inner + fallible_kind = structural.kind + structural = fallible_kind.inner if fallible_kind.is_a?(FallibleTypeExpression) + optional_kind = structural.kind + if optional_kind.is_a?(OptionalTypeExpression) + optional_expression = optional_kind + structural = optional_expression.inner unless optional_expression.inner.kind.is_a?(LinearTypeExpression) end structural end @@ -690,6 +565,7 @@ def self.indirect_type?(value) end sig { params(type: TypeInput).returns(Type) } + # ruby-to-clear: fallible def self.from_input(type) if type.is_a?(Type) return copy_type(type) @@ -705,6 +581,7 @@ def self.from_input(type) end sig { params(expression: TypeExpression).returns(Type) } + # ruby-to-clear: fallible def self.from_child_expression(expression) Type.new(expression) end @@ -716,6 +593,7 @@ def self.function_signature_like?(value) end sig { params(vt: Schemas::UnionSchema::VariantValue).returns(Type) } + # ruby-to-clear: fallible def self.from_variant_input(vt) return Type.new(:Any) unless vt return Type.new(:Any) if vt.is_a?(Schemas::InlineStructVariant) @@ -871,9 +749,11 @@ def self.array_capacity_symbol(capacity) sig { params(expression: TypeExpression).returns(T::Boolean) } def self.preallocation_expression?(expression) - return false unless expression.is_a?(LinearTypeExpression) + kind = expression.kind + return false unless kind.is_a?(LinearTypeExpression) - (expression.list? || expression.set?) && !expression.allocation_hint.nil? + linear_kind = kind + (linear_kind.list? || linear_kind.set?) && !linear_kind.allocation_hint.nil? end sig { params(value: Symbol).returns(T::Boolean) } @@ -915,31 +795,12 @@ def self.resource_type_symbol?(value) sig { params(value: Symbol).returns(T.nilable(String)) } def self.ownership_surface_name_for(value) - return "@multiowned" if value == :multiowned - return "@shared" if value == :shared - return "@node" if value == :node - return "@shared:node" if value == :shared_node - return "@split" if value == :split - return "@link" if value == :link - return "@frozen" if value == :frozen - - nil + TypeCapabilities.ownership_surface_name_for(value) end sig { params(value: Symbol).returns(T.nilable(String)) } def self.sync_surface_name_for(value) - return "@locked" if value == :locked - return "@writeLocked" if value == :write_locked - return "@versioned" if value == :versioned - return "@atomic" if value == :atomic - return "@alwaysMutable" if value == :always_mutable - return "@local" if value == :local - return "@raw" if value == :raw - return "@symbol" if value == :symbol - return "@c" if value == :c - return "@size" if value == :size - - nil + TypeCapabilities.sync_surface_name_for(value) end sig { params(value: Symbol).returns(T.nilable(String)) } @@ -985,7 +846,6 @@ def self.zig_type_name_for(value) value.to_s end - # ruby-to-clear: skip sig { returns(T.untyped) } def self.deinit_resource_close_plan Schemas::ResourceClosePlan.new(actions: [ @@ -998,6 +858,7 @@ def self.deinit_resource_close_plan end sig { params(type: Type).returns(Type) } + # ruby-to-clear: fallible def self.copy_type(type) copy = Type.new(:Any) copy.replace_shape!(type.shape.copy) @@ -1018,13 +879,13 @@ def self.inline_migration_name(type) return nil unless caps.inline_migration_safe? expression = type.shape.expression - return nil if expression.is_a?(FunctionTypeExpression) + return nil if expression.kind.is_a?(FunctionTypeExpression) # Legacy async collection spellings overload the same surface form for # streams, lists of promises, and promises resolving to lists. Migrating # only the annotation can change NEXT's result protocol, so those require # a whole-program migration and are deliberately not auto-fixed here. return nil if TypeExpressionTree.each_node(expression).any? do |node| - node.is_a?(FutureTypeExpression) + node.kind.is_a?(FutureTypeExpression) end # A bare legacy T[] is a slice/view, while Inline Pivot []T is an owned # dynamic list. Only an explicit legacy @list is semantics-preserving. @@ -1043,62 +904,81 @@ def self.inline_migration_name(type) sig { params(node: TypeExpression, type: Type).returns(T::Boolean) } def self.unsafe_inline_linear_migration?(node, type) - return false unless node.is_a?(LinearTypeExpression) - return true if node.dimensions.include?(:INFERRED) + kind = node.kind + return false unless kind.is_a?(LinearTypeExpression) + return true if kind.dimensions.include?(:INFERRED) bare_legacy_slice?(node, type) end - sig { params(node: LinearTypeExpression, type: Type).returns(T::Boolean) } + sig { params(node: TypeExpression, type: Type).returns(T::Boolean) } def self.bare_legacy_slice?(node, type) - type.collection.nil? && node.list? && node.capabilities.collection.nil? + kind = node.kind + return false unless type.collection.nil? && kind.is_a?(LinearTypeExpression) + + kind.list? && node.capabilities.collection.nil? end sig { params(expression: TypeExpression, type: Type).returns(T.nilable(TypeExpression)) } def self.project_inline_collection(expression, type) collection = type.collection return expression if collection.nil? - if expression.is_a?(OptionalTypeExpression) - inner = project_inline_collection(expression.inner, type) - return inner.nil? ? nil : OptionalTypeExpression.new(inner: inner) + kind = expression.kind + cap = expression.capabilities + if kind.is_a?(OptionalTypeExpression) + inner = project_inline_collection(kind.inner, type) + return inner.nil? ? nil : TypeExpression.of(OptionalTypeExpression.new(inner: inner)) end - if expression.is_a?(FallibleTypeExpression) - inner = project_inline_collection(expression.inner, type) - return inner.nil? ? nil : FallibleTypeExpression.new(inner: inner, error_set: expression.error_set) + if kind.is_a?(FallibleTypeExpression) + fallible_kind = kind + inner = project_inline_collection(fallible_kind.inner, type) + return inner.nil? ? nil : TypeExpression.of(FallibleTypeExpression.new(inner: inner, error_set: fallible_kind.error_set)) end - return nil unless expression.is_a?(LinearTypeExpression) + return nil unless kind.is_a?(LinearTypeExpression) - hint = expression.allocation_hint + linear_kind = kind + hint = linear_kind.allocation_hint if hint.nil? && type.pool? - pool_dimension = expression.dimensions.find { |dimension| dimension.is_a?(Integer) } + pool_dimension = linear_kind.dimensions.find { |dimension| dimension.is_a?(Integer) } hint = pool_dimension if pool_dimension.is_a?(Integer) end # Inline Pivot pools require an explicit capacity (`[Pool(N)]T`). A # legacy dynamic `T[]@pool` has no semantics-preserving spelling yet. return nil if type.pool? && hint.nil? - LinearTypeExpression.new( - kind: collection, - dimensions: expression.dimensions, - item: expression.item, - allocation_hint: hint, - capabilities: expression.capabilities + TypeExpression.new( + kind: LinearTypeExpression.new( + kind: collection, + dimensions: linear_kind.dimensions, + item: linear_kind.item, + allocation_hint: hint, + ), + capabilities: cap, ) end private_class_method :project_inline_collection # ruby-to-clear: effects reentrant + # ruby-to-clear: fallible sig { params(t: Type).returns(String) } def self.surface_name_type(t) return "~#{surface_name_type(t.tense_type)}" if t.tense? - return "!#{surface_name_type(T.must(t.payload_type))}" if t.error_union? + if t.error_union? + payload = t.payload_type + return "!#{surface_name_type(T.cast(payload, Type))}" + end if t.optional? - wrapped = T.must(t.wrapped_type) + wrapped = T.cast(t.wrapped_type, Type) inner = surface_name_type(wrapped) - return "?(#{inner})" if wrapped.array? || wrapped.map? return "?#{inner}" end - return "#{surface_name_type(T.must(t.element_type))}#{array_capacity_suffix(t.capacity)}" if t.array? + if t.array? + element = T.cast(t.element_type, Type) + return "#{array_capacity_suffix(t.capacity)}#{surface_name_type(element)}" + end + if t.map? + return "{#{surface_name_type(t.key_type)}}#{surface_name_type(t.value_type)}" + end return function_type_surface_name(t) if t.fn_type? if t.generic_instance? @@ -1121,6 +1001,7 @@ def self.coercion_surface_name_type(t) end sig { params(element_type: TypeInput, capacity: ArrayCapacity).returns(Type) } + # ruby-to-clear: fallible def self.array_of(element_type, capacity: nil) element = from_input(element_type) item_expression = TypeExpressionTree.with_root_capabilities( @@ -1136,10 +1017,12 @@ def self.array_of(element_type, capacity: nil) TypeCapabilities.new(ownership: :affine) end Type.new( - LinearTypeExpression.new( - kind: kind, - dimensions: dimensions, - item: item_expression, + TypeExpression.new( + kind: LinearTypeExpression.new( + kind: kind, + dimensions: dimensions, + item: item_expression, + ), capabilities: collection_capabilities, ) ) @@ -1152,8 +1035,8 @@ def self.array_of(element_type, capacity: nil) def self.promise_list_of(element_type) list = array_of(element_type) Type.new( - FutureTypeExpression.new( - inner: list.shape.expression, + TypeExpression.new( + kind: FutureTypeExpression.new(inner: list.shape.expression), capabilities: TypeCapabilities.new(ownership: :affine, collection: :list), ) ) @@ -1167,11 +1050,13 @@ def self.set_of(element_type, capacity: nil) element.capabilities ) Type.new( - LinearTypeExpression.new( - kind: :set, - dimensions: [:SET], - item: item_expression, - allocation_hint: capacity, + TypeExpression.new( + kind: LinearTypeExpression.new( + kind: :set, + dimensions: [:SET], + item: item_expression, + allocation_hint: capacity, + ), capabilities: TypeCapabilities.new(collection: :set), ) ) @@ -1179,7 +1064,13 @@ def self.set_of(element_type, capacity: nil) sig { params(payload_type: TypeInput).returns(Type) } def self.error_union_of(payload_type) - Type.new("!#{surface_name(payload_type)}") + payload = Type.new(payload_type) + return payload if payload.error_union? + + t = Type.new(TypeExpression.of(FallibleTypeExpression.new(inner: payload.shape.expression))) + t.merge_capabilities_from!(payload, include_affine_ownership: true) + t.copy_placement_from!(payload, preserve_existing: false) + t end sig { params(wrapped_type: TypeInput).returns(Type) } @@ -1188,7 +1079,7 @@ def self.optional_of(wrapped_type) wrapped = Type.new(wrapped_type) return wrapped if wrapped.optional? - t = Type.new(OptionalTypeExpression.new(inner: wrapped.shape.expression)) + t = Type.new(TypeExpression.of(OptionalTypeExpression.new(inner: wrapped.shape.expression))) t.merge_capabilities_from!(wrapped, include_affine_ownership: true) t.copy_placement_from!(wrapped, preserve_existing: false) t @@ -1196,16 +1087,30 @@ def self.optional_of(wrapped_type) sig { params(value_type: TypeInput).returns(Type) } def self.tense_of(value_type) - Type.new("~#{surface_name(value_type)}") + value = Type.new(value_type) + t = Type.new(TypeExpression.of(FutureTypeExpression.new(inner: value.shape.expression))) + t.merge_capabilities_from!(value, include_affine_ownership: true) + t.copy_placement_from!(value, preserve_existing: false) + t end sig { params(base: Symbol, args: T::Array[TypeInput]).returns(Type) } def self.generic_instance_of(base, args) - names = T.let(args.map { |arg| surface_name(arg) }, T::Array[String]) - Type.new("#{base}<#{names.join(",")}>") + arguments = T.let([], T::Array[TypeExpression]) + index = T.let(0, Integer) + while index < args.length + argument = from_input(args.fetch(index)) + arguments << TypeExpressionTree.with_root_capabilities( + argument.shape.expression, + argument.capabilities, + ) + index += 1 + end + Type.new(TypeExpression.of(NamedTypeExpression.new(name: base, arguments: arguments))) end sig { params(item_type: TypeInput).returns(Type) } + # ruby-to-clear: fallible def self.stream_step_of(item_type) generic_instance_of(:StreamStep, [item_type]) end @@ -1277,7 +1182,7 @@ def self.function_type_surface_name(type) BITWISE_OPS = [:XOR, :BIT_AND, :BIT_OR].freeze SHIFT_OPS = [:SHL, :SHR].freeze BOOL_RESULT_OPS = T.let((EQUALITY_OPS + ORDERING_OPS).freeze, T::Array[Symbol]) - NUMBER_RESULT_OPS = [:SUB, :MUL, :DIV, :POW, :MOD, :WRAP_SUB, :WRAP_MUL, :CHECK_SUB, :CHECK_MUL] + NUMBER_RESULT_OPS = [:SUB, :MUL, :DIV, :POW, :MOD, :WRAP_SUB, :WRAP_MUL, :CHECK_SUB, :CHECK_MUL].freeze sig { params(op: Symbol).returns(T::Boolean) } def self.logical_op?(op) @@ -1425,6 +1330,7 @@ def self.resolve_integer_op(op, left_type, right_type, preserve_left:) end sig { params(left_type: Type, right_type: Type).returns(T::Boolean) } + # ruby-to-clear: fallible def self.equality_compatible?(left_type, right_type) return true if left_type.any? || right_type.any? || optional_any_comparable?(left_type, right_type) @@ -1437,17 +1343,24 @@ def self.equality_compatible?(left_type, right_type) sig { params(left_type: Type, right_type: Type).returns(T::Boolean) } def self.optional_any_comparable?(left_type, right_type) - (left_type.optional? && T.must(left_type.wrapped_type).any?) || - (right_type.optional? && T.must(right_type.wrapped_type).any?) + if left_type.optional? + return true if T.cast(left_type.wrapped_type, Type).any? + end + if right_type.optional? + return true if T.cast(right_type.wrapped_type, Type).any? + end + false end sig { params(left_type: Type, right_type: Type).returns(T::Boolean) } + # ruby-to-clear: fallible def self.ordered_compatible?(left_type, right_type) scalar_comparable?(left_type, right_type) || optional_payload_ordered_comparable?(left_type, right_type) end sig { params(left_type: Type, right_type: Type).returns(T::Boolean) } + # ruby-to-clear: fallible def self.scalar_comparable?(left_type, right_type) (left_type.numeric? && right_type.numeric?) || (left_type.string? && right_type.string?) || @@ -1464,12 +1377,13 @@ def self.optional_nil_comparable?(left_type, right_type) def self.optional_payload_comparable?(left_type, right_type) return false unless left_type.optional? != right_type.optional? - return payload_comparable?(T.must(left_type.wrapped_type), right_type) if left_type.optional? + return payload_comparable?(T.cast(left_type.wrapped_type, Type), right_type) if left_type.optional? - payload_comparable?(T.must(right_type.wrapped_type), left_type) + payload_comparable?(T.cast(right_type.wrapped_type, Type), left_type) end sig { params(inner_type: Type, payload_type: Type).returns(T::Boolean) } + # ruby-to-clear: fallible def self.payload_comparable?(inner_type, payload_type) inner_type.resolved == payload_type.resolved || scalar_comparable?(inner_type, payload_type) end @@ -1479,10 +1393,10 @@ def self.optional_payload_ordered_comparable?(left_type, right_type) return false unless left_type.optional? != right_type.optional? if left_type.optional? - return scalar_comparable?(T.must(left_type.wrapped_type), right_type) + return scalar_comparable?(T.cast(left_type.wrapped_type, Type), right_type) end - scalar_comparable?(T.must(right_type.wrapped_type), left_type) + scalar_comparable?(T.cast(right_type.wrapped_type, Type), left_type) end sig { params(left_type: Type, right_type: Type).returns(T::Boolean) } @@ -1613,6 +1527,7 @@ def self.safe_autocast?(from_type, to_type) auto: T::Boolean ).void end + # ruby-to-clear: effects reentrant def initialize(raw_input, ownership: nil, sync: nil, layout: nil, location: nil, collection: nil, shard_count: nil, stripe_count: nil, observable: nil, observable_terminal: nil, auto: false) # stripe_count kept for backwards compat (ignored) @shape = T.let(DEFAULT_SHAPE, TypeShape) @capabilities = T.let(TypeCapabilities.new, TypeCapabilities) @@ -1905,6 +1820,8 @@ def polymorphic_shared polymorphic_shared: TypeCapabilities::MaybeBoolean ).returns(TypeCapabilities) end + # ruby-to-clear: fallible + # ruby-to-clear: effects reentrant def apply_capabilities!( ownership: TypeCapabilities::UNSET, sync: TypeCapabilities::UNSET, @@ -1936,6 +1853,7 @@ def apply_capabilities!( if update_item next_item_capabilities = item_capabilities.with( ownership: Type.prefer_legacy_element_capability(elem_ownership, legacy_elem_ownership), + ownership_set: TypeCapabilities::UNSET, sync: Type.prefer_legacy_element_capability(elem_sync, legacy_elem_sync), layout: Type.prefer_legacy_element_capability(elem_layout, legacy_elem_layout) ) @@ -1947,6 +1865,7 @@ def apply_capabilities!( @capabilities = @capabilities.with( ownership: ownership, + ownership_set: TypeCapabilities::UNSET, sync: sync, layout: layout, lock_rank: lock_rank, @@ -2207,7 +2126,11 @@ def copy_collection_shape_from!(source) # Symbol-only reconstruction cannot represent Inline Pivot markers such # as [Set] (its legacy projection is Int64[SET]). Recover the recursive # node from the authoritative source before merging binding capabilities. - @shape = source.shape.copy if element_type.nil? && !source.element_type.nil? + current_element = T.let(element_type, T.nilable(Type)) + if current_element.nil? + source_element = T.let(source.element_type, T.nilable(Type)) + @shape = source.shape.copy unless source_element.nil? + end apply_capabilities!( collection: Type.capability_symbol_or_unset(collection.nil? && !source_collection.nil? ? source_collection : nil), shard_count: Type.capability_integer_or_unset(shard_count.nil? && !source_shard_count.nil? ? source_shard_count : nil), @@ -2574,19 +2497,19 @@ def accepts?(other_type) if optional? return true if other_type.resolved == :NIL if other_type.optional? - return T.must(wrapped_type).accepts?(T.must(other_type.wrapped_type)) + return T.cast(wrapped_type, Type).accepts?(T.cast(other_type.wrapped_type, Type)) end - return T.must(wrapped_type).accepts?(other_type) + return T.cast(wrapped_type, Type).accepts?(other_type) end # 5. Error union coercion: !T accepts T or !T if error_union? if other_type.error_union? - return T.must(payload_type).accepts?(T.must(other_type.payload_type)) + return T.cast(payload_type, Type).accepts?(T.cast(other_type.payload_type, Type)) end - return T.must(payload_type).accepts?(other_type) + return T.cast(payload_type, Type).accepts?(other_type) end # 6. Primitive widening @@ -2614,8 +2537,15 @@ def accepts?(other_type) ownership == other_type.ownership && sync == other_type.sync && layout == other_type.layout expected_args = generic_args actual_args = other_type.generic_args - return expected_args.length == actual_args.length && - expected_args.each_with_index.all? { |expected, index| expected.accepts?(actual_args.fetch(index)) } + return false unless expected_args.length == actual_args.length + + index = T.let(0, Integer) + while index < expected_args.length + return false unless expected_args.fetch(index).accepts?(actual_args.fetch(index)) + + index += 1 + end + return true end false @@ -2623,6 +2553,7 @@ def accepts?(other_type) # Used specifically to check if assigning an array too large to a fixed array sig { params(other_type: Type).returns(T.nilable(T::Boolean)) } + # ruby-to-clear: fallible def array_overflow?(other_type) return false if !other_type.array? || !self.array? return false if self.base_type != other_type.base_type @@ -2775,20 +2706,20 @@ def array? sig { returns(T::Boolean) } def rank? - expression = shape.expression - expression = expression.inner if expression.is_a?(FallibleTypeExpression) - expression = expression.inner if expression.is_a?(OptionalTypeExpression) && expression.inner.is_a?(LinearTypeExpression) - expression.is_a?(LinearTypeExpression) && expression.dimensions.length > 1 + kind = Type.unwrap_fallible_kind(shape.expression.kind) + if kind.is_a?(OptionalTypeExpression) + optional_inner = kind.inner.kind + kind = optional_inner if optional_inner.is_a?(LinearTypeExpression) + end + kind.is_a?(LinearTypeExpression) && kind.dimensions.length > 1 end sig { returns(T::Array[TypeExpression::Dimension]) } def rank_dimensions return [] unless rank? - expression = shape.expression - expression = expression.inner if expression.is_a?(FallibleTypeExpression) - expression = expression.inner if expression.is_a?(OptionalTypeExpression) - T.cast(expression, LinearTypeExpression).dimensions + kind = Type.unwrap_optional_kind(Type.unwrap_fallible_kind(shape.expression.kind)) + T.cast(kind, LinearTypeExpression).dimensions end sig { returns(Integer) } @@ -2833,6 +2764,7 @@ def fixed_position_type(index) end sig { returns(T::Boolean) } + # ruby-to-clear: fallible def string? resolved == :String || (array? && base_type == :Byte) end @@ -2940,12 +2872,18 @@ def node? sig { returns(T::Boolean) } def node_reference? return true if node? - optional? && !!wrapped_type&.node? + return false unless optional? + + wrapped = wrapped_type + !wrapped.nil? && wrapped.node? end sig { returns(T.nilable(Type)) } def node_payload_type - return T.must(wrapped_type).node_payload_type if optional? && wrapped_type&.node? + if optional? + wrapped = wrapped_type + return wrapped.node_payload_type if !wrapped.nil? && wrapped.node? + end return self if node? nil @@ -3027,11 +2965,13 @@ def symbol? end sig { returns(T::Boolean) } + # ruby-to-clear: fallible def c_string? string? && sync == :c end sig { returns(T::Boolean) } + # ruby-to-clear: fallible def c_array_view? array? && !string? && sync == :c end @@ -3110,11 +3050,27 @@ def map? shape.map end - # True when this is a numeric-keyed map (HashMap or HashMap). - # Backed by AutoHashMapUnmanaged — no key duplication, pure arena-allocated. + # True when this map uses the typed-key representation rather than the + # specialized StringMap representation. The historical name predates + # `{K}V` support for structural keys; NumericMapType is now the generic + # value-key map in the runtime. sig { returns(T::Boolean) } def numeric_map? - map? && key_type.numeric? + map? && !string_key_map? + end + + # String and interned Symbol keys share the specialized string map ABI. + sig { returns(T::Boolean) } + def string_key_map? + map? && (key_type.string? || key_type.symbol? || key_type.resolved == :Symbol) + end + + sig { params(actual: Type).returns(T::Boolean) } + def accepts_map_key?(actual) + return false unless map? + return actual.string? if string_key_map? + + key_type.accepts?(actual) end sig { returns(T::Boolean) } @@ -3124,10 +3080,8 @@ def plain_numeric_map? sig { returns(Type) } def key_type - expression = shape.expression - expression = expression.inner if expression.is_a?(FallibleTypeExpression) - expression = expression.inner if expression.is_a?(OptionalTypeExpression) - return Type.from_child_expression(expression.key) if expression.is_a?(MapTypeExpression) + kind = Type.unwrap_optional_kind(Type.unwrap_fallible_kind(shape.expression.kind)) + return Type.from_child_expression(kind.key) if kind.is_a?(MapTypeExpression) Type.new(:String) end @@ -3165,6 +3119,7 @@ def collection? # collections (HashMap/@pool/@list/@set); plain non-string arrays/slices are # value-shaped but need the same boundary treatment. sig { returns(T::Boolean) } + # ruby-to-clear: fallible def collection_value? collection? || non_string_array? end @@ -3173,7 +3128,7 @@ def collection_value? # outlive its allocator region. sig { returns(T::Boolean) } def heap_ptr? - return T.must(wrapped_type).heap_ptr? if optional? + return T.cast(wrapped_type, Type).heap_ptr? if optional? string? || indirect? || tense_observable? || collection? || (array? && !fixed?) end @@ -3184,6 +3139,7 @@ def associative_collection? end sig { returns(T::Boolean) } + # ruby-to-clear: fallible def linear_collection? pool? || list_collection? || set_collection? || non_string_array? end @@ -3195,16 +3151,19 @@ def ownership_storage end sig { returns(T::Boolean) } + # ruby-to-clear: fallible def direct_indexable_collection? list_collection? || (array? && !string? && !collection?) end sig { returns(T::Boolean) } + # ruby-to-clear: fallible def non_string_array? array? && !string? end sig { returns(T::Boolean) } + # ruby-to-clear: fallible def indexed_container_borrow? map? || pool? || direct_indexable_collection? end @@ -3397,72 +3356,6 @@ def resource? SchemaLookup = T.type_alias { T.proc.params(name: Symbol).returns(SchemaLookupResult) } SchemaResolver = T.type_alias { SchemaLookup } - # COPY may deep-copy ordinary heap data, but it cannot duplicate a linear - # resource handle. This predicate is deliberately transitive: wrapping a - # CLOSE resource in an optional, collection, union, or ordinary struct does - # not make the handle copyable. - sig { params(schema_lookup: T.nilable(SchemaLookup), seen: T.nilable(T::Set[String])).returns(T::Boolean) } - def contains_linear_resource?(schema_lookup = nil, seen = nil) - return true if resource? - - seen_types = seen || Set.new - key = type_id.key - return false if seen_types.include?(key) - - next_seen = seen_types.dup.add(key) - - if optional? - inner = wrapped_type - return inner ? inner.contains_linear_resource?(schema_lookup, next_seen) : false - end - - if error_union? - return success_type.contains_linear_resource?(schema_lookup, next_seen) - end - - if tuple? - return generic_args.any? { |arg| arg.contains_linear_resource?(schema_lookup, next_seen) } - end - - if map? - return key_type.contains_linear_resource?(schema_lookup, next_seen) || - value_type.contains_linear_resource?(schema_lookup, next_seen) - end - - if array? || collection? - element = element_type - return element ? element.contains_linear_resource?(schema_lookup, next_seen) : false - end - - return false unless schema_lookup - - schema = schema_lookup.call(resolved) - return true if schema.is_a?(Schemas::ResourceSchema) - - if schema.is_a?(Schemas::StructSchema) - return schema.fields.values.any? do |field| - next false if field.borrowed - - substitute_generic_schema_field_type(field.type, schema) - .contains_linear_resource?(schema_lookup, next_seen) - end - end - - if schema.is_a?(Schemas::UnionSchema) - return schema.variants.values.any? do |variant| - if variant.is_a?(Schemas::InlineStructVariant) - variant.fields.values.any? do |field| - Type.from_input(field).contains_linear_resource?(schema_lookup, next_seen) - end - else - Type.from_variant_input(variant).contains_linear_resource?(schema_lookup, next_seen) - end - end - end - - false - end - # Resolve the close/deinit plan for resource types. # Returns a named result so the generated Clear has a concrete type. # @@ -3484,7 +3377,6 @@ def self.schema_resolver(lookup_arg, lookup_block) lookup_block end - # ruby-to-clear: skip sig { params(schema_lookup: T.nilable(SchemaLookup)).returns(ResourceCloseResult) } def resolve_resource_close(schema_lookup = nil) return ResourceCloseResult.new(is_resource: false, close_plan: nil) if node? @@ -3559,11 +3451,13 @@ def dynamic_field_array? end sig { returns(T::Boolean) } + # ruby-to-clear: fallible def borrowed_array_argument? non_string_array? && !pool? end sig { params(other: T.nilable(Type)).returns(T::Boolean) } + # ruby-to-clear: fallible def string_comparable_with?(other) return false unless string? return true if other.nil? @@ -3587,10 +3481,8 @@ def striped? sig { returns(Type) } def value_type - expression = shape.expression - expression = expression.inner if expression.is_a?(FallibleTypeExpression) - expression = expression.inner if expression.is_a?(OptionalTypeExpression) - return Type.from_child_expression(expression.value) if expression.is_a?(MapTypeExpression) + kind = Type.unwrap_optional_kind(Type.unwrap_fallible_kind(shape.expression.kind)) + return Type.from_child_expression(kind.value) if kind.is_a?(MapTypeExpression) Type.new(:Any) end @@ -3603,7 +3495,7 @@ def generic_instance? sig { returns(T::Boolean) } def projection? - shape.expression.is_a?(TypeProjectionExpression) + shape.expression.kind.is_a?(TypeProjectionExpression) end # Generic projections and aggregates cannot settle their cleanup shape @@ -3613,27 +3505,33 @@ def projection? sig { returns(T::Boolean) } def specialization_may_need_cleanup? return true if projection? || generic_instance? - return payload_type&.specialization_may_need_cleanup? == true if error_union? + if error_union? + payload = payload_type + return !payload.nil? && payload.specialization_may_need_cleanup? + end - optional? && wrapped_type&.specialization_may_need_cleanup? == true + return false unless optional? + + wrapped = wrapped_type + !wrapped.nil? && wrapped.specialization_may_need_cleanup? end sig { returns(T.nilable(Symbol)) } def projection_owner - expression = shape.expression - expression.is_a?(TypeProjectionExpression) ? expression.owner : nil + kind = shape.expression.kind + kind.is_a?(TypeProjectionExpression) ? kind.owner : nil end sig { returns(T.nilable(Symbol)) } def projection_member - expression = shape.expression - expression.is_a?(TypeProjectionExpression) ? expression.member : nil + kind = shape.expression.kind + kind.is_a?(TypeProjectionExpression) ? kind.member : nil end sig { returns(T.nilable(Symbol)) } def projection_protocol - expression = shape.expression - expression.is_a?(TypeProjectionExpression) ? expression.protocol : nil + kind = shape.expression.kind + kind.is_a?(TypeProjectionExpression) ? kind.protocol : nil end # The base type name of a generic instance: :"Pair" → :Pair @@ -3679,20 +3577,24 @@ def sync_family_name # The type arguments as Type objects: [Type(:Float64), Type(:String)] sig { returns(T::Array[Type]) } def generic_args - expression = shape.expression - expression = expression.inner if expression.is_a?(FallibleTypeExpression) - expression = expression.inner if expression.is_a?(OptionalTypeExpression) - items = if expression.is_a?(TupleTypeExpression) - expression.items - elsif expression.is_a?(NamedTypeExpression) - expression.arguments - else - [] + kind = Type.unwrap_optional_kind(Type.unwrap_fallible_kind(shape.expression.kind)) + items = T.let([], T::Array[TypeExpression]) + if kind.is_a?(TupleTypeExpression) + kind.items.each { |item| items << item } + elsif kind.is_a?(NamedTypeExpression) + kind.arguments.each { |item| items << item } + end + args = T.let([], T::Array[Type]) + index = T.let(0, Integer) + while index < items.length + args << Type.from_child_expression(items.fetch(index)) + index += 1 end - items.map { |item| Type.from_child_expression(item) } + args end sig { returns(T::Boolean) } + # ruby-to-clear: fallible def struct? !primitive? && !any? && !void? && !string? && !array? && !map? && !tuple? && !optional? && !error_union? && !tense? && !fn_type? end @@ -3715,11 +3617,10 @@ def optional_element_array? def wrapped_type return nil unless optional? - expression = shape.expression - expression = expression.inner if expression.is_a?(FallibleTypeExpression) - return nil unless expression.is_a?(OptionalTypeExpression) + kind = Type.unwrap_fallible_kind(shape.expression.kind) + return nil unless kind.is_a?(OptionalTypeExpression) - inner = Type.from_child_expression(expression.inner) + inner = Type.from_child_expression(kind.inner) inner.merge_capabilities_from!(self) inner.copy_placement_from!(self) inner @@ -3734,10 +3635,10 @@ def error_union? sig { returns(T.nilable(Type)) } def payload_type return nil unless error_union? - expression = shape.expression - return nil unless expression.is_a?(FallibleTypeExpression) + kind = shape.expression.kind + return nil unless kind.is_a?(FallibleTypeExpression) - Type.from_child_expression(expression.inner) + Type.from_child_expression(kind.inner) end sig { returns(Type) } @@ -3749,7 +3650,7 @@ def success_type sig { returns(Type) } def error_union_payload_with_outer_capabilities - payload = Type.new(T.must(payload_type)) + payload = Type.new(T.cast(payload_type, Type)) payload.merge_capabilities_from!(self) payload.copy_placement_from!(self) payload @@ -3758,12 +3659,12 @@ def error_union_payload_with_outer_capabilities sig { returns(Type) } def value_payload_type t = success_type - t.optional? ? T.must(t.wrapped_type) : t + t.optional? ? T.cast(t.wrapped_type, Type) : t end sig { returns(Type) } def non_optional_type - optional? ? T.must(wrapped_type) : self + optional? ? T.cast(wrapped_type, Type) : self end # Tense (Promise) types: ~T — a background task that will produce T @@ -3916,7 +3817,8 @@ def observable_wrapper_zig(tense_type) # this should be unreachable in practice, but if it does fire we # need a clear compiler-level message rather than the previous # internal "BYPASS at " debug raise. - if observable_terminal.nil? + terminal = observable_terminal + if terminal.nil? raise CompilerError.new( nil, "Internal: Type#observable_wrapper_zig called on `#{self.to_s}` " \ @@ -3928,12 +3830,12 @@ def observable_wrapper_zig(tense_type) nil, ) end - terminal = T.must(observable_terminal) - wrapper = Type.observable_wrapper_for_terminal(terminal, tense_type) + terminal_value = terminal + wrapper = Type.observable_wrapper_for_terminal(terminal_value, tense_type) if wrapper.nil? raise CompilerError.new( nil, - "Internal: unknown observable terminal kind #{terminal.to_s}. " \ + "Internal: unknown observable terminal kind #{terminal_value.to_s}. " \ "Add an entry to Type.observable_terminals in src/ast/type.rb.", nil, ) @@ -3951,12 +3853,13 @@ def future? sig { returns(Type) } def tense_type - expression = shape.expression - return Type.from_child_expression(expression.inner) if expression.is_a?(FutureTypeExpression) - if expression.is_a?(StreamTypeExpression) - dimension = expression.cardinality == :FINITE ? :LIST : expression.cardinality + kind = shape.expression.kind + return Type.from_child_expression(kind.inner) if kind.is_a?(FutureTypeExpression) + if kind.is_a?(StreamTypeExpression) + stream_kind = kind + dimension = T.let(stream_kind.cardinality == :FINITE ? :LIST : stream_kind.cardinality, TypeExpression::Dimension) return Type.from_child_expression( - LinearTypeExpression.new(kind: :array, dimensions: [dimension], item: expression.item) + TypeExpression.of(LinearTypeExpression.new(kind: :array, dimensions: [dimension], item: stream_kind.item)) ) end @@ -3965,7 +3868,7 @@ def tense_type sig { returns(T::Boolean) } def canonical_stream? - shape.expression.is_a?(StreamTypeExpression) + shape.expression.kind.is_a?(StreamTypeExpression) end # True when this value is a canonical stream, possibly behind the one legal @@ -3977,7 +3880,9 @@ def canonical_stream_result? return false unless error_union? payload = payload_type - payload&.canonical_stream? == true + return false if payload.nil? + + payload.canonical_stream? end # Preserve all wrappers on the item (`?T`, `!T`, `!?T`) instead of @@ -3986,12 +3891,14 @@ def canonical_stream_result? sig { returns(T.nilable(Type)) } def canonical_stream_item_type stream = error_union? ? payload_type : self - return nil unless stream&.canonical_stream? + return nil if stream.nil? + stream_type = stream + return nil unless stream_type.canonical_stream? - expression = stream.shape.expression - return nil unless expression.is_a?(StreamTypeExpression) + kind = stream_type.shape.expression.kind + return nil unless kind.is_a?(StreamTypeExpression) - Type.from_child_expression(expression.item) + Type.from_child_expression(kind.item) end sig { returns(T::Boolean) } @@ -4010,8 +3917,10 @@ def stream_step_item_type # Used for lazy finite producers like ranges. NEXT returns ?T until exhausted. sig { returns(T::Boolean) } def dynamic_stream? - expression = shape.expression - return true if expression.is_a?(StreamTypeExpression) && expression.cardinality == :FINITE + kind = shape.expression.kind + if kind.is_a?(StreamTypeExpression) + return true if kind.cardinality == :FINITE + end !!(future? && tense_type.dynamic? && !tense_type.optional? && !tense_type.optional_element_array? && !list_collection?) @@ -4030,12 +3939,14 @@ def optional_stream_shape_type return nil unless element.optional? - return Type.array_of(T.must(element.wrapped_type), capacity: stream_shape.capacity) + return Type.array_of(T.cast(element.wrapped_type, Type), capacity: stream_shape.capacity) end return nil unless stream_shape.optional? wrapped = T.let(stream_shape.wrapped_type, T.nilable(Type)) - return wrapped if wrapped&.array? + return nil if wrapped.nil? + wrapped_type = wrapped + return wrapped_type if wrapped_type.array? nil end @@ -4083,15 +3994,22 @@ def single_future? sig { returns(T::Boolean) } def split_open_stream? - split? && open_stream? + return false unless split? + return true if open_stream? + + # Canonical cardinality-first spelling: [~]T @split. + kind = shape.expression.kind + kind.is_a?(StreamTypeExpression) && kind.cardinality == :FINITE end # Bounded stream: ~T[N] or ~?T[N] — a fixed stream of N elements consumed via NEXT. # Distinct from a single promise (~T): NEXT can be called N times, not exactly once. sig { returns(T::Boolean) } def bounded_stream? - expression = shape.expression - return true if expression.is_a?(StreamTypeExpression) && expression.cardinality.is_a?(Integer) + kind = shape.expression.kind + if kind.is_a?(StreamTypeExpression) + return true if kind.cardinality.is_a?(Integer) + end # ~T[N] is a bounded stream of N elements. ~String is NOT a bounded stream # even though String is internally []const u8 (a fixed array) - it's a Promise. @@ -4123,11 +4041,11 @@ def open_stream? future? && (tense_type.open_stream_marker? || open_stream_alias?) end - # The element type T in ~?T[] / ~T[?]. + # The element type T of an open stream ([~]T; legacy ~?T[] / ~T[?]). sig { returns(T.nilable(Type)) } def open_stream_element_type - return nil unless open_stream? - if open_stream_alias? + return nil unless open_stream? || split_open_stream? + if open_stream? && open_stream_alias? shape = T.let(optional_stream_shape_type, T.nilable(Type)) return nil if shape.nil? @@ -4141,8 +4059,10 @@ def open_stream_element_type # Resource semantics: call deinit() to free the heap-allocated Inner. sig { returns(T::Boolean) } def inf_stream? - expression = shape.expression - return true if expression.is_a?(StreamTypeExpression) && expression.cardinality == :INF + kind = shape.expression.kind + if kind.is_a?(StreamTypeExpression) + return true if kind.cardinality == :INF + end future? && tense_type.inf_stream_marker? end @@ -4185,12 +4105,14 @@ def stream_capacity # ruby-to-clear: effects reentrant def element_type return nil unless array? - expression = shape.expression - expression = expression.inner if expression.is_a?(FallibleTypeExpression) - expression = expression.inner if expression.is_a?(OptionalTypeExpression) && expression.inner.is_a?(LinearTypeExpression) - return nil unless expression.is_a?(LinearTypeExpression) + kind = Type.unwrap_fallible_kind(shape.expression.kind) + if kind.is_a?(OptionalTypeExpression) + optional_inner = kind.inner.kind + kind = optional_inner if optional_inner.is_a?(LinearTypeExpression) + end + return nil unless kind.is_a?(LinearTypeExpression) - Type.from_child_expression(expression.item) + Type.from_child_expression(kind.item) end sig { params(lookup_arg: T.nilable(SchemaResolver), lookup_block: T.nilable(SchemaLookup)).returns(Integer) } @@ -4358,7 +4280,8 @@ def implicitly_copyable?(lookup_arg = nil, &lookup_block) # schema here makes recursive structs overflow the compiler stack. return false if indirect? || any_rc? || link? if optional? - return lookup_block ? T.must(wrapped_type).implicitly_copyable?(&lookup_block) : T.must(wrapped_type).implicitly_copyable?(lookup_arg) + wrapped = T.cast(wrapped_type, Type) + return lookup_block ? wrapped.implicitly_copyable?(&lookup_block) : wrapped.implicitly_copyable?(lookup_arg) end # Pool Id handles are u64 indices — always Copy. return true if id_handle? @@ -4431,7 +4354,14 @@ def recursive_cleanup_shape?(schema_lookup = nil, seen = nil) return true if string? || any_rc? || any_sync? || frozen? || link? || collection? || indirect? || future? if tuple? - return generic_args.any? { |arg| arg.recursive_cleanup_shape?(schema_lookup, seen_set) } + args = generic_args + index = T.let(0, Integer) + while index < args.length + return true if args.fetch(index).recursive_cleanup_shape?(schema_lookup, seen_set) + + index += 1 + end + return false end if array? @@ -4448,11 +4378,11 @@ def recursive_cleanup_shape?(schema_lookup = nil, seen = nil) return true if schema.is_a?(Schemas::ResourceSchema) if schema.is_a?(Schemas::UnionSchema) values = T.let(schema.variants.values, T::Array[Schemas::UnionSchema::VariantValue]) - i = T.let(0, Integer) - while i < values.length - vt = values.fetch(i) + union_index = T.let(0, Integer) + while union_index < values.length + vt = values.fetch(union_index) unless vt - i += 1 + union_index += 1 next end if Schemas.inline_struct?(vt) @@ -4465,37 +4395,22 @@ def recursive_cleanup_shape?(schema_lookup = nil, seen = nil) else return true if Type.from_variant_input(vt).recursive_cleanup_shape?(lookup, seen_set) end - i += 1 + union_index += 1 end return false end if schema.is_a?(Schemas::StructSchema) - fields = schema.fields.values - i = T.let(0, Integer) - while i < fields.length - field = fields.fetch(i) - if field.borrowed - i += 1 + struct_fields = schema.fields.values + struct_index = T.let(0, Integer) + while struct_index < struct_fields.length + struct_field = struct_fields.fetch(struct_index) + if struct_field.borrowed + struct_index += 1 next end - return true if substitute_generic_schema_field_type(field.type, schema).recursive_cleanup_shape?(lookup, seen_set) - i += 1 - end - return false - end - - if schema.is_a?(Schemas::ResourceSchema) - fields = schema.fields.values - i = T.let(0, Integer) - while i < fields.length - field = fields.fetch(i) - if field.borrowed - i += 1 - next - end - return true if substitute_generic_schema_field_type(field.type, schema).recursive_cleanup_shape?(lookup, seen_set) - i += 1 + return true if substitute_generic_schema_field_type(struct_field.type, schema).recursive_cleanup_shape?(lookup, seen_set) + struct_index += 1 end return false end @@ -4537,6 +4452,15 @@ def needs_promotion?(schema_lookup = nil, seen = nil) # heap-allocated data that must be freed at scope exit. # Same as needs_promotion? but excludes bare strings (freed by # StringMap.freeUnionPayload inside collections, not at top level). + # A String that owns its bytes. Symbols are interned and raw strings are + # borrowed, so neither is owning -- wrapping one in an optional or an error + # union must not make it owning. Mirrors the test the cleanup classifier + # already uses for struct fields. + sig { returns(T::Boolean) } + def owning_string? + string? && !symbol? && !raw? + end + # Plus: RC, NumericMap, Pool, Set. sig { params(schema_lookup: T.nilable(SchemaLookup), seen: T.nilable(T::Set[String])).returns(T::Boolean) } # ruby-to-clear: effects reentrant @@ -4549,13 +4473,16 @@ def needs_cleanup?(schema_lookup = nil, seen = nil) inner = wrapped_type return false unless inner - return inner.needs_cleanup?(schema_lookup, seen) || inner.string? + return inner.needs_cleanup?(schema_lookup, seen) || inner.owning_string? end if error_union? payload = payload_type return false unless payload - return payload.needs_cleanup?(schema_lookup, seen) || payload.string? + # payload_type drops the @symbol/@raw modifier, so consult this type for + # the non-owning markers -- they survive on the error union itself. + return payload.needs_cleanup?(schema_lookup, seen) || + (payload.string? && !symbol? && !raw?) end return non_string_array_needs_cleanup?(schema_lookup, seen) if non_string_array? @@ -4568,7 +4495,14 @@ def needs_cleanup?(schema_lookup = nil, seen = nil) seen_set << key if tuple? - return generic_args.any? { |arg| arg.needs_cleanup?(schema_lookup, seen_set) } + args = generic_args + index = T.let(0, Integer) + while index < args.length + return true if args.fetch(index).needs_cleanup?(schema_lookup, seen_set) + + index += 1 + end + return false end if schema_lookup @@ -4630,9 +4564,12 @@ def parallel_boundary_forbidden_reason(schema_lookup = nil, seen = nil) child_types << value_type end generic_args.each { |arg| child_types << arg } - child_types.each do |child| - reason = child.parallel_boundary_forbidden_reason(schema_lookup, seen) + child_index = T.let(0, Integer) + while child_index < child_types.length + reason = child_types.fetch(child_index).parallel_boundary_forbidden_reason(schema_lookup, seen) return reason if reason + + child_index += 1 end return nil unless schema_lookup @@ -4647,25 +4584,25 @@ def parallel_boundary_forbidden_reason(schema_lookup = nil, seen = nil) T.unsafe(next_seen) << schema_key if schema.is_a?(Schemas::StructSchema) schema.fields.each_value do |field| - reason = Type.from_input(field.type).parallel_boundary_forbidden_reason(schema_lookup, next_seen) - return reason if reason + struct_reason = Type.from_input(field.type).parallel_boundary_forbidden_reason(schema_lookup, next_seen) + return struct_reason if struct_reason end elsif schema.is_a?(Schemas::ResourceSchema) schema.fields.each_value do |field| - reason = Type.from_input(field.type).parallel_boundary_forbidden_reason(schema_lookup, next_seen) - return reason if reason + resource_reason = Type.from_input(field.type).parallel_boundary_forbidden_reason(schema_lookup, next_seen) + return resource_reason if resource_reason end elsif schema.is_a?(Schemas::UnionSchema) schema.variants.each_value do |variant| next if variant.nil? if variant.is_a?(Schemas::InlineStructVariant) variant.fields.each_value do |field_type| - reason = Type.from_input(field_type).parallel_boundary_forbidden_reason(schema_lookup, next_seen) - return reason if reason + inline_reason = Type.from_input(field_type).parallel_boundary_forbidden_reason(schema_lookup, next_seen) + return inline_reason if inline_reason end else - reason = Type.from_input(variant).parallel_boundary_forbidden_reason(schema_lookup, next_seen) - return reason if reason + union_reason = Type.from_input(variant).parallel_boundary_forbidden_reason(schema_lookup, next_seen) + return union_reason if union_reason end end end @@ -4869,7 +4806,11 @@ def nested_zig_type(is_param: false, is_field: false) sig { params(is_param: T::Boolean, is_field: T::Boolean).returns(String) } # ruby-to-clear: effects reentrant def render_zig_type(is_param: false, is_field: false) - compute_zig_type(is_param: is_param, is_field: is_field) + mutable_is_param = T.let(is_param, T::Boolean) + mutable_is_field = T.let(is_field, T::Boolean) + mutable_is_param = is_param + mutable_is_field = is_field + compute_zig_type(is_param: mutable_is_param, is_field: mutable_is_field) end # Determines the appropriate storage location based on type characteristics and size. @@ -4910,19 +4851,47 @@ def finalize_storage(size, current_storage = nil) # Types that require moves need frame or heap based on size if requires_move? - if current_storage.nil? || current_storage == :stack + if current_storage.nil? + return size > 128 ? :frame : :stack + end + if current_storage == :stack return size > 128 ? :frame : :stack end end - # Default to current or stack - current_storage || :stack + # Default to current or stack. + return :stack if current_storage.nil? + + current_storage end private sig { params(field_type: TypeInput, schema: T.any(Schemas::StructSchema, Schemas::ResourceSchema)).returns(Type) } # ruby-to-clear: effects reentrant + # Type.from_input DEEP-COPIES an existing Type so callers may safely mutate + # the result. Recursive cleanup queries only inspect what they get back, so + # that copy is pure overhead - and it dominated: 10511 of 20039 wall samples + # (51%) were this substitution allocating Types, each paying sorbet-runtime + # prop checking, which is what made the frontend appear to hang. Reuse the + # existing Type when no generic substitution is needed. Never hand the + # result to anything that mutates. + def schema_field_type_for_query(field_type, schema) + return substitute_generic_schema_field_type(field_type, schema) if generic_instance? + + Type.for_query(field_type) + end + + # Read-only counterpart to from_input: skips the defensive copy for a value + # that is already a Type. + sig { params(type: TypeInput).returns(Type) } + def self.for_query(type) + return type if type.is_a?(Type) + + Type.from_input(type) + end + + sig { params(field_type: TypeInput, schema: T.any(Schemas::StructSchema, Schemas::ResourceSchema)).returns(Type) } def substitute_generic_schema_field_type(field_type, schema) normalized_field_type = Type.from_input(field_type) return normalized_field_type unless generic_instance? @@ -4950,7 +4919,7 @@ def substitute_generic_schema_field_type(field_type, schema) sig { params(schema: Schemas::UnionSchema, schema_lookup: SchemaLookup, seen: T::Set[String]).returns(T::Boolean) } # ruby-to-clear: effects reentrant def schema_union_needs_promotion?(schema, schema_lookup, seen) - values = schema.variants.values + values = T.let(schema.variants.values, T::Array[Schemas::UnionSchema::VariantValue]) i = T.let(0, Integer) while i < values.length vt = values.fetch(i) @@ -5003,7 +4972,7 @@ def schema_struct_needs_promotion?(schema, schema_lookup, seen) sig { params(schema: Schemas::UnionSchema, schema_lookup: SchemaLookup, seen: T::Set[String]).returns(T::Boolean) } # ruby-to-clear: effects reentrant def schema_union_needs_cleanup?(schema, schema_lookup, seen) - values = schema.variants.values + values = T.let(schema.variants.values, T::Array[Schemas::UnionSchema::VariantValue]) i = T.let(0, Integer) while i < values.length vt = values.fetch(i) @@ -5100,7 +5069,7 @@ def schema_struct_any?(schema, &blk) sig { params(schema: Schemas::UnionSchema, blk: T.proc.params(t: Type).returns(T::Boolean)).returns(T::Boolean) } # ruby-to-clear: effects reentrant def schema_union_any?(schema, &blk) - values = schema.variants.values + values = T.let(schema.variants.values, T::Array[Schemas::UnionSchema::VariantValue]) i = T.let(0, Integer) while i < values.length vt = values.fetch(i) @@ -5252,18 +5221,24 @@ def parse_raw_input!(raw_input, auto: false) end sig { params(raw_input: FunctionType, auto: T::Boolean).void } + # ruby-to-clear: fallible + # ruby-to-clear: effects reentrant def parse_function_type_input!(raw_input, auto: false) @shape = TypeShape.from_raw(raw: raw_input, auto: auto) @capabilities = TypeCapabilities.new(ownership: :affine) end sig { params(expression: TypeExpression, auto: T::Boolean).void } + # ruby-to-clear: fallible + # ruby-to-clear: effects reentrant def parse_expression_input!(expression, auto: false) @shape = TypeShape.from_raw(raw: :Any, auto: auto, expression: expression) @capabilities = TypeExpressionTree.root_capabilities(expression) end sig { params(raw_str: String, auto: T::Boolean).void } + # ruby-to-clear: fallible + # ruby-to-clear: effects reentrant def parse_raw_string!(raw_str, auto: false) normalized_str = T.let(raw_str, String) if raw_str == "Number" @@ -5273,8 +5248,9 @@ def parse_raw_string!(raw_str, auto: false) end suffix = T.let(TypeCapabilitySuffix.new(base: normalized_str, ownership: nil, sync: nil), TypeCapabilitySuffix) - capability_marker = normalized_str.rindex("@") - unless normalized_str.include?("<") || capability_marker.nil? || capability_marker.zero? + if !normalized_str.include?("<") && + normalized_str.include?("@") && + !normalized_str.start_with?("@") suffix = Type.strip_capability_suffix_from(normalized_str) end @shape = TypeShape.from_core(suffix.base, auto: auto) @@ -5364,10 +5340,12 @@ def tense_zig_type(is_param:, is_field:) return "std.ArrayListUnmanaged(CheatLib.Promise(#{elem_zig}))" end if canonical_stream? - expression = T.cast(shape.expression, StreamTypeExpression) + expression = T.cast(shape.expression.kind, StreamTypeExpression) elem_zig = Type.from_child_expression(expression.item) .nested_zig_type(is_param: is_param, is_field: is_field) - return "CheatLib.Stream(#{elem_zig})" if expression.cardinality == :FINITE + if expression.cardinality == :FINITE + return split_open_stream? ? "CheatLib.SplitStream(#{elem_zig})" : "CheatLib.Stream(#{elem_zig})" + end return "CheatLib.InfStream(#{elem_zig})" if expression.cardinality == :INF return "CheatLib.BoundedStream(#{elem_zig}, #{expression.cardinality})" @@ -5376,6 +5354,10 @@ def tense_zig_type(is_param:, is_field:) elem_zig = T.must(stream_element_type).nested_zig_type(is_param: is_param, is_field: is_field) return "CheatLib.BoundedStream(#{elem_zig}, #{stream_capacity})" end + if split_open_stream? + elem_zig = T.must(open_stream_element_type).nested_zig_type(is_param: is_param, is_field: is_field) + return "CheatLib.SplitStream(#{elem_zig})" + end if dynamic_stream? inner_t = T.let(T.must(tense_type.element_type), Type) return case inner_t.resolved @@ -5417,6 +5399,11 @@ def capability_wrapped_zig_type(is_param:, is_field:) return "*#{inner_zig}" end + # Capability layers are idempotent: CLEAR has no nested handles, so a + # generic substitution that binds T to an already-managed type must not + # double-wrap (T @multiowned with T = Budget@multiowned is one Rc). + already_handle = inner_zig.start_with?("CheatLib.Rc(", "CheatLib.Arc(") + return inner_zig if already_handle && (ownership == :multiowned || ownership == :shared) return "CheatLib.Rc(#{inner_zig})" if ownership == :multiowned return "CheatLib.Arc(#{inner_zig})" if ownership == :shared return link_zig_type(inner_zig) if ownership == :link @@ -5518,7 +5505,11 @@ def map_zig_type def compute_zig_type(is_param: false, is_field: false) if projection? protocol = projection_protocol - facts = protocol.nil? || protocol == :Map ? "CheatLib.MapFacts" : "__clearProtocolFacts_#{protocol}" + facts = T.let("CheatLib.MapFacts", String) + unless protocol.nil? + protocol_value = protocol + facts = "__clearProtocolFacts_#{protocol_value}" unless protocol_value == :Map + end return "#{facts}(#{T.must(projection_owner)}).#{T.must(projection_member)}" end @@ -5582,11 +5573,7 @@ def compute_zig_type(is_param: false, is_field: false) while i < fn_raw.params.length p = fn_raw.params.fetch(i) t = p.type - if t.is_a?(Type) - param_types_zig << t.zig_type(is_param: true) - else - param_types_zig << Type.new(t).zig_type(is_param: true) - end + param_types_zig << t.zig_type(is_param: true) i += 1 end ret_zig = fn_raw.return_type.zig_type @@ -5629,7 +5616,8 @@ def compute_zig_type(is_param: false, is_field: false) if rank? base_zig = T.must(element_type).nested_zig_type(is_param: is_param, is_field: is_field) if dynamic_rank? - return "CheatLib.Grid(#{base_zig}, #{rank})" + rank_value = rank + return "CheatLib.Grid(#{base_zig}, #{rank_value})" end return "[#{T.must(capacity)}]#{base_zig}" @@ -5650,7 +5638,12 @@ def compute_zig_type(is_param: false, is_field: false) # 3d. Handle @set collection if set_collection? - base_zig = T.must(element_type).nested_zig_type(is_param: is_param, is_field: is_field) + elem = T.must(element_type) + # Interned symbols are rodata/intern-table handles the set never + # owns; the owned-string Set would free them on duplicate insert + # and at deinit (misaligned free of rodata). + return "CheatLib.InternedStringSet()" if elem.symbol? + base_zig = elem.nested_zig_type(is_param: is_param, is_field: is_field) return "CheatLib.Set(#{base_zig})" end @@ -5817,3 +5810,575 @@ def self.from_function_signature(signature) ) end end + +# StructField itself is an acyclic foundation record (struct_field.rb); the +# semantic-Type view of its `type` slot lives here, next to Type, so the +# foundation never depends on this file. +class AST::StructField + extend T::Sig + + sig { returns(Type) } + def type + T.cast(self[:type], Type) + end +end + +# --------------------------------------------------------------------------- +# Typed schemas for declared types stored in Scope. +# +# A declared type's schema is ALWAYS one of the typed classes below — +# never a raw Hash. Producers (the annotator's visit_*Def) construct +# these directly; consumers use the typed accessors (`.fields`, +# `.variants`, `.kind`, `.struct?`, ...). There is exactly one +# representation. +# +# Schemas lives in this file because Type and Schemas are one strongly +# connected component: schema field maps STORE resolved Type values, and +# Type's ownership classification (needs_cleanup?, slot_size, storage +# finalization) READS schemas. A module boundary between them is a cycle — +# Ruby tolerates it via load order, but CLEAR's module system (and any +# sound separate compilation) rejects it. One SCC, one unit. +# --------------------------------------------------------------------------- +module Schemas + extend T::Sig + + class ExternSource < T::Struct + const :dependency, String + const :abi, Symbol, default: :zig + const :symbol, T.nilable(String), default: nil + const :callconv, Symbol, default: :c + const :header, T.nilable(String), default: nil + end + + # Plain classes (not Data.define) so Sorbet's 4010 doesn't fire on + # the kwarg-only initialize signatures we need for default values. + # Frozen at the end of initialize so callers see immutable shapes + # (the methods table is mutable in place — see StructSchema#methods). + + class EnumSchema + extend T::Sig + + VariantInput = T.type_alias { T::Enumerable[T.any(String, Symbol)] } + + sig { returns(T::Set[String]) } + attr_reader :variants + sig { returns(Symbol) } + attr_reader :visibility + sig { params(variants: VariantInput, visibility: Symbol).void } + # ruby-to-clear: fallible + def initialize(variants:, visibility: :package) + @variants = T.let(normalize_variants(variants).freeze, T::Set[String]) + @visibility = T.let(visibility, Symbol) + freeze + end + + sig { params(variants: VariantInput).returns(T::Set[String]) } + def normalize_variants(variants) + normalized = T.let(Set.new, T::Set[String]) + variants.each do |variant| + normalized << variant.to_s + end + normalized + end + private :normalize_variants + + sig { returns(T.nilable(Symbol)) } + def kind = :enum + sig { returns(T::Boolean) } + def enum? = true + sig { returns(T::Boolean) } + def union? = false + sig { returns(T::Boolean) } + def struct? = false + sig { returns(T::Boolean) } + def resource? = false + end + + class ResourceCloseCallKind < T::Enum + enums do + Method = new("method") + Function = new("function") + CFunction = new("c_function") + end + end + + class ResourceCloseAction < T::Struct + extend T::Sig + + const :call_kind, ResourceCloseCallKind + const :name, String + const :field_path, T::Array[String], default: [] + const :runtime_heap_alloc_args, Integer, default: 0 + + sig { params(field: String).returns(ResourceCloseAction) } + def for_field(field) + ResourceCloseAction.new( + call_kind: call_kind, + name: name, + field_path: [field] + field_path, + runtime_heap_alloc_args: runtime_heap_alloc_args, + ) + end + end + + class ResourceClosePlan < T::Struct + extend T::Sig + + const :actions, T::Array[ResourceCloseAction] + + sig { params(name: String, runtime_heap_alloc_args: Integer).returns(ResourceClosePlan) } + def self.method(name, runtime_heap_alloc_args: 0) + new(actions: [ + ResourceCloseAction.new( + call_kind: ResourceCloseCallKind::Method, + name: name, + runtime_heap_alloc_args: runtime_heap_alloc_args, + ), + ]) + end + + sig { params(name: String, runtime_heap_alloc_args: Integer).returns(ResourceClosePlan) } + def self.function(name, runtime_heap_alloc_args: 0) + new(actions: [ + ResourceCloseAction.new( + call_kind: ResourceCloseCallKind::Function, + name: name, + runtime_heap_alloc_args: runtime_heap_alloc_args, + ), + ]) + end + + sig { params(name: String).returns(ResourceClosePlan) } + def self.c_function(name) + new(actions: [ + ResourceCloseAction.new( + call_kind: ResourceCloseCallKind::CFunction, + name: name, + ), + ]) + end + + sig { params(actions: T::Array[ResourceCloseAction]).returns(ResourceClosePlan) } + def self.composite(actions) + new(actions: actions) + end + + sig { params(field: String).returns(ResourceClosePlan) } + def for_field(field) + mapped_actions = actions.map { |action| action.for_field(field) } + ResourceClosePlan.new(actions: mapped_actions) + end + + sig { returns(T::Boolean) } + def empty? + actions.empty? + end + end + + # Resource type schema — types with RAII cleanup (CLOSE method). + # + # Used for the 3 hand-written runtime types (File, TCPServer, TCPClient) + # and EXTERN STRUCT ... CLOSE forms, which can carry generic type params, + # an extern module name, and an AS alias. + class ResourceSchema + extend T::Sig + + FieldMetadataValue = T.type_alias { T.any(Type::TypeInput, AST::Locatable, T::Boolean) } + FieldMetadata = T.type_alias { T::Hash[T.any(Symbol, String), FieldMetadataValue] } + FieldInput = T.type_alias { T.any(Type::TypeInput, AST::StructField, FieldMetadata) } + FieldInputMap = T.type_alias { T::Hash[T.any(Symbol, String), FieldInput] } + StaticMethodValue = T.type_alias { T.any(T::Array[Symbol], Symbol, String, T::Boolean) } + StaticMethodSpec = T.type_alias { T::Hash[Symbol, StaticMethodValue] } + StaticMethodsMap = T.type_alias { T::Hash[String, StaticMethodSpec] } + MethodsMap = T.type_alias { T::Hash[T.any(Symbol, String), FunctionSignature] } + + sig { returns(Schemas::ResourceClosePlan) } + attr_reader :close_plan + + sig { returns(Schemas::ResourceSchema::StaticMethodsMap) } + attr_reader :static_methods + + sig { returns(T::Hash[String, AST::StructField]) } + attr_reader :fields + + sig { returns(T.nilable(String)) } + attr_reader :extern_module + + sig { returns(T.nilable(String)) } + attr_reader :as_type + + sig { returns(Symbol) } + attr_reader :visibility + + sig { returns(Schemas::ResourceSchema::MethodsMap) } + attr_reader :methods + sig { returns(T::Array[Symbol]) } + attr_reader :type_params + sig { params(close_plan: Schemas::ResourceClosePlan, static_methods: Schemas::ResourceSchema::StaticMethodsMap, fields: FieldInputMap, type_params: T::Array[Symbol], extern_module: T.nilable(String), as_type: T.nilable(String), visibility: Symbol, methods: Schemas::ResourceSchema::MethodsMap).void } + def initialize(close_plan:, static_methods: {}, fields: {}, type_params: [], extern_module: nil, as_type: nil, visibility: :package, methods: {}) + @close_plan = T.let(close_plan.dup, Schemas::ResourceClosePlan) + @static_methods = T.let(static_methods, Schemas::ResourceSchema::StaticMethodsMap) + @fields = T.let(normalize_fields(fields), T::Hash[String, AST::StructField]) + @type_params = T.let(type_params.dup, T::Array[Symbol]) + @extern_module = T.let(extern_module, T.nilable(String)) + @as_type = T.let(as_type, T.nilable(String)) + @visibility = T.let(visibility, Symbol) + @methods = T.let(methods, Schemas::ResourceSchema::MethodsMap) + freeze + end + + sig { returns(T.nilable(Symbol)) } + def kind = :resource + sig { returns(T::Boolean) } + def resource? = true + sig { returns(T::Boolean) } + def union? = false + sig { returns(T::Boolean) } + def enum? = false + sig { returns(T::Boolean) } + def struct? = false + + sig { params(fields: FieldInputMap).returns(T::Hash[String, AST::StructField]) } + def normalize_fields(fields) + out = T.let({}, T::Hash[String, AST::StructField]) + fields.each do |key, field| + out[key.to_s] = normalize_resource_field(field) + end + out + end + private :normalize_fields + + # ruby-to-clear: fallible + sig { params(field: FieldInput).returns(AST::StructField) } + def normalize_resource_field(field) + return field if field.is_a?(AST::StructField) + if field.is_a?(Hash) + raw_type_value = T.let(field[:type], T.nilable(FieldMetadataValue)) + raw_type_value = field["type"] if raw_type_value.nil? + default_value = T.let(field[:default], T.untyped) + default_value = field["default"] if default_value.nil? + borrowed_value = T.let(field[:borrowed], T.untyped) + borrowed_value = field["borrowed"] if borrowed_value.nil? + if raw_type_value.nil? + return AST::StructField.new( + type: :Any, + default: default_value, + borrowed: !borrowed_value.nil? + ) + end + return AST::StructField.new( + type: raw_type_value, + default: default_value, + borrowed: !borrowed_value.nil? + ) + end + + AST::StructField.new(type: field.dup) + end + private :normalize_resource_field + end + + class InlineStructDeinitEntry < T::Struct + extend T::Sig + + const :field, String + const :kind, Symbol + const :zig_type, T.nilable(String) + const :elem_zig_type, T.nilable(String) + + sig { params(field: String, zig_type: String).returns(Schemas::InlineStructDeinitEntry) } + def self.indirect(field:, zig_type:) + new(field: field, kind: :indirect, zig_type: zig_type, elem_zig_type: nil) + end + + sig { params(field: String, zig_type: String).returns(Schemas::InlineStructDeinitEntry) } + def self.uniform(field:, zig_type:) + new(field: field, kind: :uniform, zig_type: zig_type, elem_zig_type: nil) + end + + sig { params(field: String, elem_zig_type: String).returns(Schemas::InlineStructDeinitEntry) } + def self.array(field:, elem_zig_type:) + new(field: field, kind: :array, zig_type: nil, elem_zig_type: elem_zig_type) + end + end + + # One union variant whose payload is an anonymous inline struct + # (`UNION Shape { Circle { radius: Float64 } }`). `fields` maps field + # name (String) to its declared type input. `deinit_entries` is filled in by + # the annotator after parse (which fields need @boxed / array + # cleanup) and is intentionally mutable in place, like + # StructSchema#methods. + # ruby-to-clear: pub + class InlineStructVariant + extend T::Sig + + FieldMap = T.type_alias { T::Hash[T.any(String, Symbol), Type::TypeInput] } + FieldInputMap = T.type_alias { T::Hash[T.any(String, Symbol), Type::TypeInput] } + + sig { returns(Schemas::InlineStructVariant::FieldMap) } + attr_reader :fields + sig { params(fields: FieldInputMap, deinit_entries: T::Array[Schemas::InlineStructDeinitEntry]).void } + # ruby-to-clear: fallible + def initialize(fields:, deinit_entries: []) + @fields = T.let(normalize_fields(fields), Schemas::InlineStructVariant::FieldMap) + @deinit_entries = T.let(deinit_entries, T::Array[Schemas::InlineStructDeinitEntry]) + end + + sig { returns(T::Array[Schemas::InlineStructDeinitEntry]) } + def deinit_entries + @deinit_entries + end + + sig { params(entries: T::Array[Schemas::InlineStructDeinitEntry]).returns(T::Array[Schemas::InlineStructDeinitEntry]) } + def deinit_entries=(entries) + @deinit_entries = entries + end + + # ruby-to-clear: skip + sig { returns(T::Hash[String, Type]) } + def typed_fields + out = T.let({}, T::Hash[String, Type]) + keys = @fields.keys + i = T.let(0, Integer) + while i < keys.length + out[keys[i].to_s] = Type.new(T.must(@fields[T.unsafe(keys[i])])) + i += 1 + end + out + end + + # Value equality on the field shape (not deinit_entries, which is + # derived). The multi-arm shared-destructure check compares two + # variants' payloads structurally — this used to be Hash `==`. + sig { params(other: InlineStructVariant).returns(T::Boolean) } + def ==(other) + other.fields == @fields + end + + sig { params(other: InlineStructVariant).returns(T::Boolean) } + def eql?(other) + self == other + end + # ruby-to-clear: skip + sig { returns(Integer) } + def hash = @fields.hash + + sig { params(fields: FieldInputMap).returns(FieldMap) } + def normalize_fields(fields) + out = T.let({}, Schemas::InlineStructVariant::FieldMap) + fields.each do |key, field| + out[key] = field + end + out + end + private :normalize_fields + end + + # Union (sum-type) schema. `variants` is a Hash[Symbol => value] where + # the value is `nil` for payload-less variants, a type input for single-type + # payloads, or an InlineStructVariant for inline struct variants. + class UnionSchema + extend T::Sig + + VariantValue = T.type_alias { T.nilable(T.any(Type::TypeInput, Schemas::InlineStructVariant)) } + VariantMap = T.type_alias { T::Hash[T.any(String, Symbol), VariantValue] } + VariantInput = T.type_alias { T.nilable(T.any(Type::TypeInput, Schemas::InlineStructVariant)) } + VariantInputMap = T.type_alias { T::Hash[T.any(String, Symbol), VariantInput] } + + sig { returns(Schemas::UnionSchema::VariantMap) } + attr_reader :variants + + sig { returns(Symbol) } + attr_reader :visibility + sig { returns(T::Array[Symbol]) } + attr_reader :type_params + sig { params(variants: VariantInputMap, type_params: T::Array[Symbol], visibility: Symbol).void } + # ruby-to-clear: fallible + def initialize(variants:, type_params: [], visibility: :package) + @variants = T.let(normalize_variants(variants), Schemas::UnionSchema::VariantMap) + @type_params = T.let(type_params.dup, T::Array[Symbol]) + @visibility = T.let(visibility, Symbol) + freeze + end + + sig { params(variants: VariantInputMap).returns(VariantMap) } + def normalize_variants(variants) + out = T.let({}, Schemas::UnionSchema::VariantMap) + keys = variants.keys + i = T.let(0, Integer) + while i < keys.length + key = keys.fetch(i) + out[key] = normalize_variant(variants[key]) + i += 1 + end + out + end + private :normalize_variants + + sig { params(variant: VariantInput).returns(VariantValue) } + def normalize_variant(variant) + return nil if variant.nil? + return variant if variant.is_a?(Schemas::InlineStructVariant) + + variant + end + private :normalize_variant + + sig { returns(T.nilable(Symbol)) } + def kind = :union + sig { returns(T::Boolean) } + def union? = true + sig { returns(T::Boolean) } + def enum? = false + sig { returns(T::Boolean) } + def struct? = false + sig { returns(T::Boolean) } + def resource? = false + end + + # Struct/record schema. `fields` maps String field names to Type/Symbol + # representations of field types. Metadata (defaults, borrowed-set, + # generic type params, methods, EXTERN module, AS alias type, + # visibility) live as named attrs. `methods` is intentionally mutable + # in place: method signatures are registered after the struct is + # declared (when the method's FunctionDef is visited). + class StructSchema + extend T::Sig + + # `fields` is ALWAYS Hash[String => AST::StructField]. Per-field + # default value and borrowed-ness live on the StructField, so + # `field_defaults` / `borrowed_fields` are derived, not stored. + FieldMetadataValue = T.type_alias { T.any(Type::TypeInput, AST::Locatable, T::Boolean) } + FieldMetadata = T.type_alias { T::Hash[T.any(Symbol, String), FieldMetadataValue] } + FieldInput = T.type_alias { T.any(Type::TypeInput, AST::StructField, FieldMetadata) } + FieldInputMap = T.type_alias { T::Hash[T.any(Symbol, String), FieldInput] } + MethodsMap = T.type_alias { T::Hash[T.any(Symbol, String), FunctionSignature] } + + sig { returns(T::Hash[String, AST::StructField]) } + attr_reader :fields + + sig { returns(MethodsMap) } + attr_reader :methods + + sig { returns(Symbol) } + attr_reader :visibility + + sig { returns(T.nilable(String)) } + attr_reader :extern_module + + sig { returns(T.nilable(String)) } + attr_reader :as_type + sig { returns(T::Array[Symbol]) } + attr_reader :type_params + sig { returns(T::Array[AST::GenericParamDecl]) } + attr_reader :generic_params + sig { returns(MethodsMap) } + attr_reader :static_methods + sig { params(fields: FieldInputMap, type_params: T::Array[Symbol], generic_params: T::Array[AST::GenericParamDecl], methods: MethodsMap, static_methods: MethodsMap, visibility: Symbol, extern_module: T.nilable(String), as_type: T.nilable(String)).void } + def initialize(fields: {}, type_params: [], generic_params: [], methods: {}, static_methods: {}, visibility: :package, extern_module: nil, as_type: nil) + @fields = T.let(normalize_fields(fields), T::Hash[String, AST::StructField]) + @type_params = T.let(type_params.dup, T::Array[Symbol]) + @generic_params = T.let(generic_params.dup, T::Array[AST::GenericParamDecl]) + @methods = T.let(methods, MethodsMap) + @static_methods = T.let(static_methods, MethodsMap) + @visibility = T.let(visibility, Symbol) + @extern_module = T.let(extern_module, T.nilable(String)) + @as_type = T.let(as_type, T.nilable(String)) + freeze + end + + sig { returns(T::Hash[String, AST::Locatable]) } + def field_defaults + out = T.let({}, T::Hash[String, AST::Locatable]) + @fields.each do |key, field| + default = field.default + out[key] = T.cast(default, AST::Locatable) unless default.nil? + end + out + end + + sig { returns(T::Set[String]) } + def borrowed_fields + out = T.let(Set.new, T::Set[String]) + @fields.each do |key, field| + out << key if field.borrowed + end + out + end + + sig { returns(T.nilable(Symbol)) } + def kind = nil + sig { returns(T::Boolean) } + def struct? = true + sig { returns(T::Boolean) } + def union? = false + sig { returns(T::Boolean) } + def enum? = false + sig { returns(T::Boolean) } + def resource? = false + + sig { params(fields: FieldInputMap).returns(T::Hash[String, AST::StructField]) } + def normalize_fields(fields) + out = T.let({}, T::Hash[String, AST::StructField]) + fields.each do |key, field| + out[key.to_s] = normalize_struct_field(field) + end + out + end + private :normalize_fields + + # ruby-to-clear: fallible + sig { params(field: FieldInput).returns(AST::StructField) } + def normalize_struct_field(field) + return field if field.is_a?(AST::StructField) + if field.is_a?(Hash) + raw_type_value = T.let(field[:type], T.nilable(FieldMetadataValue)) + raw_type_value = field["type"] if raw_type_value.nil? + default_value = T.let(field[:default], T.untyped) + default_value = field["default"] if default_value.nil? + borrowed_value = T.let(field[:borrowed], T.untyped) + borrowed_value = field["borrowed"] if borrowed_value.nil? + if raw_type_value.nil? + return AST::StructField.new( + type: :Any, + default: default_value, + borrowed: !borrowed_value.nil? + ) + end + return AST::StructField.new( + type: raw_type_value, + default: default_value, + borrowed: !borrowed_value.nil? + ) + end + + AST::StructField.new(type: field.dup) + end + private :normalize_struct_field + end + + SchemaValue = T.type_alias { T.nilable(T.any(EnumSchema, StructSchema, UnionSchema, ResourceSchema)) } + FieldBearingSchema = T.type_alias { T.nilable(T.any(StructSchema, ResourceSchema)) } + + # Nil-safe kind predicates. Single representation: a schema is always + # one of the typed classes above (or nil for an unknown type name). + sig { params(s: SchemaValue).returns(T::Boolean) } + def self.struct?(s) = s.is_a?(StructSchema) + + sig { params(s: SchemaValue).returns(T::Boolean) } + def self.union?(s) = s.is_a?(UnionSchema) + + sig { params(s: SchemaValue).returns(T::Boolean) } + def self.enum?(s) = s.is_a?(EnumSchema) + + sig { params(s: SchemaValue).returns(T::Boolean) } + def self.resource?(s) = s.is_a?(ResourceSchema) + + sig { params(v: UnionSchema::VariantValue).returns(T::Boolean) } + def self.inline_struct?(v) = v.is_a?(InlineStructVariant) + + # Field-bearing schema: StructSchema or ResourceSchema (EXTERN STRUCT + # ... CLOSE carries fields too), so `.fields` is safe to read. + sig { params(s: SchemaValue).returns(T::Boolean) } + def self.field_bearing?(s) = s.is_a?(StructSchema) || s.is_a?(ResourceSchema) +end diff --git a/compiler/ruby/ast/type_capabilities.rb b/compiler/ruby/ast/type_capabilities.rb new file mode 100644 index 000000000..c88580b21 --- /dev/null +++ b/compiler/ruby/ast/type_capabilities.rb @@ -0,0 +1,260 @@ +# typed: strict +# frozen_string_literal: true + +# Acyclic foundation: capability data records and their surface-name tables. +# Pure syntax facts - no dependency on semantic Type. + +require "sorbet-runtime" +require_relative "lexer" + +class TypeCapabilitySuffix < T::Struct + const :base, String + const :ownership, T.nilable(Symbol) + const :sync, T.nilable(Symbol) +end + +class TypeCapabilityUnset < T::Struct +end + +# ruby-to-clear: value +class TypeCapabilities + extend T::Sig + + UNSET = T.let(TypeCapabilityUnset.new.freeze, TypeCapabilityUnset) + MaybeSymbol = T.type_alias { T.any(TypeCapabilityUnset, Symbol, NilClass) } + MaybeInteger = T.type_alias { T.any(TypeCapabilityUnset, Integer, NilClass) } + MaybeBoolean = T.type_alias { T.any(TypeCapabilityUnset, T::Boolean) } + MaybeToken = T.type_alias { T.any(TypeCapabilityUnset, Lexer::Token, NilClass) } + + sig { returns(T.nilable(Symbol)) } + attr_reader :ownership, :sync, :layout, :collection, :elem_ownership, :elem_sync, + :elem_layout, :link_source, :observable_terminal + sig { returns(T.nilable(Integer)) } + attr_reader :lock_rank, :shard_count + sig { returns(T::Boolean) } + attr_reader :ownership_set, :soa, :observable, :polymorphic_shared + sig { returns(T.nilable(Lexer::Token)) } + attr_reader :observable_token + + sig do + params( + ownership: T.nilable(Symbol), + ownership_set: T::Boolean, + sync: T.nilable(Symbol), + layout: T.nilable(Symbol), + lock_rank: T.nilable(Integer), + collection: T.nilable(Symbol), + shard_count: T.nilable(Integer), + soa: T::Boolean, + elem_ownership: T.nilable(Symbol), + elem_sync: T.nilable(Symbol), + elem_layout: T.nilable(Symbol), + link_source: T.nilable(Symbol), + observable: T::Boolean, + observable_terminal: T.nilable(Symbol), + observable_token: T.nilable(Lexer::Token), + polymorphic_shared: T::Boolean + ).void + end + def initialize( + ownership: nil, + ownership_set: false, + sync: nil, + layout: nil, + lock_rank: nil, + collection: nil, + shard_count: nil, + soa: false, + elem_ownership: nil, + elem_sync: nil, + elem_layout: nil, + link_source: nil, + observable: false, + observable_terminal: nil, + observable_token: nil, + polymorphic_shared: false + ) + @ownership = ownership + @ownership_set = ownership_set + @sync = sync + @layout = layout + @lock_rank = lock_rank + @collection = collection + @shard_count = shard_count + @soa = soa + @elem_ownership = elem_ownership + @elem_sync = elem_sync + @elem_layout = elem_layout + @link_source = link_source + @observable = observable + @observable_terminal = observable_terminal + @observable_token = observable_token + @polymorphic_shared = polymorphic_shared + freeze + end + + sig { returns(TypeCapabilities) } + def copy + self + end + + sig do + params( + ownership: MaybeSymbol, + ownership_set: MaybeBoolean, + sync: MaybeSymbol, + layout: MaybeSymbol, + lock_rank: MaybeInteger, + collection: MaybeSymbol, + shard_count: MaybeInteger, + soa: MaybeBoolean, + elem_ownership: MaybeSymbol, + elem_sync: MaybeSymbol, + elem_layout: MaybeSymbol, + link_source: MaybeSymbol, + observable: MaybeBoolean, + observable_terminal: MaybeSymbol, + observable_token: MaybeToken, + polymorphic_shared: MaybeBoolean + ).returns(TypeCapabilities) + end + def with( + ownership: UNSET, + ownership_set: UNSET, + sync: UNSET, + layout: UNSET, + lock_rank: UNSET, + collection: UNSET, + shard_count: UNSET, + soa: UNSET, + elem_ownership: UNSET, + elem_sync: UNSET, + elem_layout: UNSET, + link_source: UNSET, + observable: UNSET, + observable_terminal: UNSET, + observable_token: UNSET, + polymorphic_shared: UNSET + ) + next_ownership = T.let(ownership.equal?(UNSET) ? self.ownership : T.cast(ownership, T.nilable(Symbol)), T.nilable(Symbol)) + next_ownership_set = T.let( + ownership_set.equal?(UNSET) ? (!ownership.equal?(UNSET) || self.ownership_set) : T.cast(ownership_set, T::Boolean), + T::Boolean + ) + next_sync = T.let(sync.equal?(UNSET) ? self.sync : T.cast(sync, T.nilable(Symbol)), T.nilable(Symbol)) + next_layout = T.let(layout.equal?(UNSET) ? self.layout : T.cast(layout, T.nilable(Symbol)), T.nilable(Symbol)) + next_lock_rank = T.let(lock_rank.equal?(UNSET) ? self.lock_rank : T.cast(lock_rank, T.nilable(Integer)), T.nilable(Integer)) + next_collection = T.let(collection.equal?(UNSET) ? self.collection : T.cast(collection, T.nilable(Symbol)), T.nilable(Symbol)) + next_shard_count = T.let(shard_count.equal?(UNSET) ? self.shard_count : T.cast(shard_count, T.nilable(Integer)), T.nilable(Integer)) + next_soa = T.let(soa.equal?(UNSET) ? self.soa : T.cast(soa, T::Boolean), T::Boolean) + next_elem_ownership = T.let(elem_ownership.equal?(UNSET) ? self.elem_ownership : T.cast(elem_ownership, T.nilable(Symbol)), T.nilable(Symbol)) + next_elem_sync = T.let(elem_sync.equal?(UNSET) ? self.elem_sync : T.cast(elem_sync, T.nilable(Symbol)), T.nilable(Symbol)) + next_elem_layout = T.let(elem_layout.equal?(UNSET) ? self.elem_layout : T.cast(elem_layout, T.nilable(Symbol)), T.nilable(Symbol)) + next_link_source = T.let(link_source.equal?(UNSET) ? self.link_source : T.cast(link_source, T.nilable(Symbol)), T.nilable(Symbol)) + next_observable = T.let(observable.equal?(UNSET) ? self.observable : T.cast(observable, T::Boolean), T::Boolean) + next_observable_terminal = T.let(observable_terminal.equal?(UNSET) ? self.observable_terminal : T.cast(observable_terminal, T.nilable(Symbol)), T.nilable(Symbol)) + next_observable_token = T.let(observable_token.equal?(UNSET) ? self.observable_token : T.cast(observable_token, T.nilable(Lexer::Token)), T.nilable(Lexer::Token)) + next_polymorphic_shared = T.let(polymorphic_shared.equal?(UNSET) ? self.polymorphic_shared : T.cast(polymorphic_shared, T::Boolean), T::Boolean) + + return self if next_ownership == self.ownership && next_ownership_set == self.ownership_set && + next_sync == self.sync && next_layout == self.layout && next_lock_rank == self.lock_rank && + next_collection == self.collection && next_shard_count == self.shard_count && next_soa == self.soa && + next_elem_ownership == self.elem_ownership && next_elem_sync == self.elem_sync && + next_elem_layout == self.elem_layout && next_link_source == self.link_source && + next_observable == self.observable && next_observable_terminal == self.observable_terminal && + next_observable_token == self.observable_token && next_polymorphic_shared == self.polymorphic_shared + + TypeCapabilities.new( + ownership: next_ownership, + ownership_set: next_ownership_set, + sync: next_sync, + layout: next_layout, + lock_rank: next_lock_rank, + collection: next_collection, + shard_count: next_shard_count, + soa: next_soa, + elem_ownership: next_elem_ownership, + elem_sync: next_elem_sync, + elem_layout: next_elem_layout, + link_source: next_link_source, + observable: next_observable, + observable_terminal: next_observable_terminal, + observable_token: next_observable_token, + polymorphic_shared: next_polymorphic_shared + ) + end + + sig { returns(TypeCapabilities) } + def without_runtime_wrappers + with( + ownership: :affine, + ownership_set: false, + sync: nil, + layout: nil, + elem_ownership: nil, + elem_sync: nil, + elem_layout: nil + ) + end + + sig { returns(T::Boolean) } + def inline_migration_safe? + return false unless lock_rank.nil? && elem_ownership.nil? && elem_sync.nil? && + elem_layout.nil? && link_source.nil? && observable_terminal.nil? + return false if polymorphic_shared + + collection.nil? || collection == :list || collection == :set || collection == :pool + end + + sig { returns(T::Boolean) } + def explicit_layer_capability? + (!ownership.nil? && ownership != :affine) || !sync.nil? || !layout.nil? || !lock_rank.nil? || + !shard_count.nil? || soa || !elem_ownership.nil? || !elem_sync.nil? || + !elem_layout.nil? || !link_source.nil? || observable || + !observable_terminal.nil? || polymorphic_shared + end + + sig { params(ownership: MaybeSymbol, sync: MaybeSymbol, layout: MaybeSymbol).returns(T::Boolean) } + def element_update_requested?(ownership:, sync:, layout:) + !ownership.equal?(UNSET) || !sync.equal?(UNSET) || !layout.equal?(UNSET) || + !elem_ownership.nil? || !elem_sync.nil? || !elem_layout.nil? + end + + # Shared default for T::Struct `factory:` props. `default:` deep-clones + # non-primitive values per instantiation even when frozen; a factory + # returning this frozen instance is the only T::Props path that shares. + # The .freeze is redundant (initialize freezes) but marks immutability + # where the constant is declared. + AFFINE = T.let(TypeCapabilities.new(ownership: :affine).freeze, TypeCapabilities) + + # Surface-name tables are syntax facts: how a capability symbol is spelled + # in CLEAR source. Semantic Type delegates here, never the reverse. + sig { params(value: Symbol).returns(T.nilable(String)) } + def self.ownership_surface_name_for(value) + return "@multiowned" if value == :multiowned + return "@shared" if value == :shared + return "@node" if value == :node + return "@shared:node" if value == :shared_node + return "@split" if value == :split + return "@link" if value == :link + return "@frozen" if value == :frozen + + nil + end + + sig { params(value: Symbol).returns(T.nilable(String)) } + def self.sync_surface_name_for(value) + return "@locked" if value == :locked + return "@writeLocked" if value == :write_locked + return "@versioned" if value == :versioned + return "@atomic" if value == :atomic + return "@alwaysMutable" if value == :always_mutable + return "@local" if value == :local + return "@raw" if value == :raw + return "@symbol" if value == :symbol + return "@c" if value == :c + return "@size" if value == :size + + nil + end +end diff --git a/compiler/ruby/ast/type_expression.rb b/compiler/ruby/ast/type_expression.rb index 990ffa0c7..87f53a647 100644 --- a/compiler/ruby/ast/type_expression.rb +++ b/compiler/ruby/ast/type_expression.rb @@ -1,12 +1,24 @@ # typed: strict require "sorbet-runtime" - -module TypeExpression +require_relative "type_capabilities" + +# The variant-specific payload of a type expression. Every concrete node kind +# includes this marker. The per-node `capabilities` field is NOT duplicated into +# each variant -- it is hoisted onto the `TypeExpression` wrapper below and +# stored once, so reading it off any node is a direct field access with no +# per-variant dispatch. A node kind is always paired with its capabilities via +# the wrapper (`TypeExpression#kind` / `#capabilities`). +module TypeExpressionKind extend T::Helpers - extend T::Sig include Kernel - interface! + sealed! +end + +# A type expression: a variant payload (`kind`) plus the capabilities that apply +# to this node. Capabilities live here (once) rather than on every variant. +class TypeExpression < T::Struct + extend T::Sig Dimension = T.type_alias { T.any(Integer, Symbol) } VALID_TENSE_ORDERS = T.let( @@ -14,23 +26,35 @@ module TypeExpression T::Array[String], ) - sig { abstract.returns(TypeCapabilities) } - def capabilities; end + const :kind, TypeExpressionKind + const :capabilities, TypeCapabilities, factory: -> { TypeCapabilities::AFFINE } + + # Build a wrapper around a freshly-constructed variant payload, defaulting + # capabilities to AFFINE when the caller does not care. + sig { params(kind: TypeExpressionKind, capabilities: TypeCapabilities).returns(TypeExpression) } + def self.of(kind, capabilities = TypeCapabilities::AFFINE) + new(kind: kind, capabilities: capabilities) + end + + # Replace this node's capabilities, keeping the variant payload. + sig { params(capabilities: TypeCapabilities).returns(TypeExpression) } + def with_capabilities(capabilities) + TypeExpression.new(kind: kind, capabilities: capabilities) + end end class NamedTypeExpression < T::Struct - include TypeExpression + include TypeExpressionKind const :name, Symbol const :arguments, T::Array[TypeExpression], default: [] - const :capabilities, TypeCapabilities, default: TypeCapabilities.new(ownership: :affine), override: true end # A protocol-associated type selected from a generic type parameter, such as # M::Key or M::Value. This remains symbolic while a generic body is checked # and is replaced from the concrete conformance witness at instantiation. class TypeProjectionExpression < T::Struct - include TypeExpression + include TypeExpressionKind const :owner, Symbol const :member, Symbol @@ -38,54 +62,63 @@ class TypeProjectionExpression < T::Struct # this on the immutable syntax node makes `M::Item` unambiguous when M is # constrained by more than one protocol with an Item associated type. const :protocol, T.nilable(Symbol), default: nil - const :capabilities, TypeCapabilities, default: TypeCapabilities.new(ownership: :affine), override: true +end + +class FunctionParamExpression < T::Struct + const :expression, TypeExpression +end + +# Foundation-native function signature: parameters and return spelled as +# TypeExpressions. `semantic_payload` is an opaque backref (the semantic +# Type::FunctionType) owned entirely by type.rb; the foundation never +# inspects it. +class FunctionSignatureExpression < T::Struct + const :params, T::Array[FunctionParamExpression] + const :return_expression, TypeExpression + const :reentrant, T::Boolean, default: false + const :abi, Symbol, default: :clear + const :semantic_payload, T.nilable(BasicObject), default: nil end class FunctionTypeExpression < T::Struct - include TypeExpression + include TypeExpressionKind - const :signature, Type::FunctionType - const :capabilities, TypeCapabilities, default: TypeCapabilities.new(ownership: :affine), override: true + const :signature, FunctionSignatureExpression end class TupleTypeExpression < T::Struct - include TypeExpression + include TypeExpressionKind const :items, T::Array[TypeExpression] - const :capabilities, TypeCapabilities, default: TypeCapabilities.new(ownership: :affine), override: true end class OptionalTypeExpression < T::Struct - include TypeExpression + include TypeExpressionKind const :inner, TypeExpression - const :capabilities, TypeCapabilities, default: TypeCapabilities.new(ownership: :affine), override: true end class FallibleTypeExpression < T::Struct - include TypeExpression + include TypeExpressionKind const :inner, TypeExpression const :error_set, T.nilable(TypeExpression), default: nil - const :capabilities, TypeCapabilities, default: TypeCapabilities.new(ownership: :affine), override: true end class FutureTypeExpression < T::Struct - include TypeExpression + include TypeExpressionKind const :inner, TypeExpression - const :capabilities, TypeCapabilities, default: TypeCapabilities.new(ownership: :affine), override: true end class LinearTypeExpression < T::Struct extend T::Sig - include TypeExpression + include TypeExpressionKind const :kind, Symbol const :dimensions, T::Array[TypeExpression::Dimension] const :item, TypeExpression const :allocation_hint, T.nilable(Integer), default: nil - const :capabilities, TypeCapabilities, default: TypeCapabilities.new(ownership: :affine), override: true sig { returns(T::Boolean) } def list? @@ -104,7 +137,7 @@ def pool? end class MapTypeExpression < T::Struct - include TypeExpression + include TypeExpressionKind const :key, TypeExpression const :value, TypeExpression @@ -113,15 +146,13 @@ class MapTypeExpression < T::Struct # still expected to round-trip both accepted spellings. const :key_implicit, T::Boolean, default: false const :legacy_separator, String, default: "," - const :capabilities, TypeCapabilities, default: TypeCapabilities.new(ownership: :affine), override: true end class StreamTypeExpression < T::Struct - include TypeExpression + include TypeExpressionKind const :cardinality, TypeExpression::Dimension const :item, TypeExpression - const :capabilities, TypeCapabilities, default: TypeCapabilities.new(ownership: :affine), override: true end class TypeExpressionTree @@ -129,88 +160,80 @@ class TypeExpressionTree sig { params(expression: TypeExpression).returns(TypeCapabilities) } def self.root_capabilities(expression) - expression.capabilities + # ruby-to-clear represents the wrapper as its closed kind union and + # distributes wrapper fields onto each payload. Keep the read under a + # kind refinement so the generated frontend can select the concrete + # payload field; Ruby still reads the single wrapper field. + kind = expression.kind + return expression.capabilities if kind.is_a?(NamedTypeExpression) + return expression.capabilities if kind.is_a?(TypeProjectionExpression) + return expression.capabilities if kind.is_a?(FunctionTypeExpression) + return expression.capabilities if kind.is_a?(TupleTypeExpression) + return expression.capabilities if kind.is_a?(OptionalTypeExpression) + return expression.capabilities if kind.is_a?(FallibleTypeExpression) + return expression.capabilities if kind.is_a?(FutureTypeExpression) + return expression.capabilities if kind.is_a?(LinearTypeExpression) + return expression.capabilities if kind.is_a?(MapTypeExpression) + return expression.capabilities if kind.is_a?(StreamTypeExpression) + + # TypeExpressionKind is sealed, so this is unreachable. Keep a concrete + # fallback because the generated CLEAR frontend has no T.absurd helper. + TypeCapabilities::AFFINE end sig { params(expression: TypeExpression, capabilities: TypeCapabilities).returns(TypeExpression) } def self.with_root_capabilities(expression, capabilities) - case expression - when OptionalTypeExpression - OptionalTypeExpression.new(inner: expression.inner, capabilities: capabilities) - when FallibleTypeExpression - FallibleTypeExpression.new(inner: expression.inner, error_set: expression.error_set, capabilities: capabilities) - when FutureTypeExpression - FutureTypeExpression.new(inner: expression.inner, capabilities: capabilities) - when NamedTypeExpression - NamedTypeExpression.new(name: expression.name, arguments: expression.arguments, capabilities: capabilities) - when TypeProjectionExpression - TypeProjectionExpression.new(owner: expression.owner, member: expression.member, - protocol: expression.protocol, capabilities: capabilities) - when FunctionTypeExpression - FunctionTypeExpression.new(signature: expression.signature, capabilities: capabilities) - when TupleTypeExpression - TupleTypeExpression.new(items: expression.items, capabilities: capabilities) - when LinearTypeExpression - LinearTypeExpression.new(kind: expression.kind, dimensions: expression.dimensions, - item: expression.item, allocation_hint: expression.allocation_hint, capabilities: capabilities) - when MapTypeExpression - MapTypeExpression.new(key: expression.key, value: expression.value, - key_implicit: expression.key_implicit, legacy_separator: expression.legacy_separator, - capabilities: capabilities) - when StreamTypeExpression - StreamTypeExpression.new(cardinality: expression.cardinality, item: expression.item, - capabilities: capabilities) - else - expression - end + # Capabilities are hoisted onto the wrapper, so replacing the root's + # capabilities is a single field swap that keeps the variant payload. + expression.with_capabilities(capabilities) end sig { params(expression: TypeExpression).returns(T.nilable(TypeCapabilities)) } def self.linear_item_capabilities(expression) node = T.let(expression, TypeExpression) loop do - case node + kind = node.kind + case kind when FallibleTypeExpression, FutureTypeExpression, OptionalTypeExpression - node = node.inner + node = kind.inner else break end end - return nil unless node.is_a?(LinearTypeExpression) + linear = node.kind + return nil unless linear.is_a?(LinearTypeExpression) - root_capabilities(node.item) + root_capabilities(linear.item) end sig { params(expression: TypeExpression, capabilities: TypeCapabilities).returns(TypeExpression) } def self.with_linear_item_capabilities(expression, capabilities) - if expression.is_a?(FallibleTypeExpression) - return FallibleTypeExpression.new( - inner: with_linear_item_capabilities(expression.inner, capabilities), - error_set: expression.error_set, - capabilities: expression.capabilities - ) + kind = expression.kind + cap = expression.capabilities + if kind.is_a?(FallibleTypeExpression) + return TypeExpression.new(kind: FallibleTypeExpression.new( + inner: with_linear_item_capabilities(kind.inner, capabilities), + error_set: kind.error_set, + ), capabilities: cap) end - if expression.is_a?(FutureTypeExpression) - return FutureTypeExpression.new( - inner: with_linear_item_capabilities(expression.inner, capabilities), - capabilities: expression.capabilities - ) + if kind.is_a?(FutureTypeExpression) + return TypeExpression.new(kind: FutureTypeExpression.new( + inner: with_linear_item_capabilities(kind.inner, capabilities), + ), capabilities: cap) end - if expression.is_a?(OptionalTypeExpression) - return OptionalTypeExpression.new( - inner: with_linear_item_capabilities(expression.inner, capabilities), - capabilities: expression.capabilities - ) + if kind.is_a?(OptionalTypeExpression) + return TypeExpression.new(kind: OptionalTypeExpression.new( + inner: with_linear_item_capabilities(kind.inner, capabilities), + ), capabilities: cap) end - return expression unless expression.is_a?(LinearTypeExpression) + return expression unless kind.is_a?(LinearTypeExpression) - LinearTypeExpression.new( - kind: expression.kind, - dimensions: expression.dimensions, - item: with_root_capabilities(expression.item, capabilities), - allocation_hint: expression.allocation_hint, - capabilities: expression.capabilities - ) + TypeExpression.new(kind: LinearTypeExpression.new( + kind: kind.kind, + dimensions: kind.dimensions, + item: with_root_capabilities(kind.item, capabilities), + allocation_hint: kind.allocation_hint, + ), capabilities: cap) end # Return the item beneath one linear collection while retaining every tense @@ -219,28 +242,26 @@ def self.with_linear_item_capabilities(expression, capabilities) # envelope that aggregate NEXT later reconstructs around the result list. sig { params(expression: TypeExpression).returns(T.nilable(TypeExpression)) } def self.linear_item_envelope(expression) - case expression + kind = expression.kind + cap = expression.capabilities + case kind when FallibleTypeExpression - inner = linear_item_envelope(expression.inner) + inner = linear_item_envelope(kind.inner) return nil unless inner - FallibleTypeExpression.new( - inner: inner, - error_set: expression.error_set, - capabilities: expression.capabilities, - ) + TypeExpression.new(kind: FallibleTypeExpression.new(inner: inner, error_set: kind.error_set), capabilities: cap) when FutureTypeExpression - inner = linear_item_envelope(expression.inner) + inner = linear_item_envelope(kind.inner) return nil unless inner - FutureTypeExpression.new(inner: inner, capabilities: expression.capabilities) + TypeExpression.new(kind: FutureTypeExpression.new(inner: inner), capabilities: cap) when OptionalTypeExpression - inner = linear_item_envelope(expression.inner) + inner = linear_item_envelope(kind.inner) return nil unless inner - OptionalTypeExpression.new(inner: inner, capabilities: expression.capabilities) + TypeExpression.new(kind: OptionalTypeExpression.new(inner: inner), capabilities: cap) when LinearTypeExpression - expression.item + kind.item else nil end @@ -260,59 +281,55 @@ def self.linear_item_envelope(expression) ).returns(TypeExpression) end def self.with_nominal_arguments(expression, name, arguments) - case expression + kind = expression.kind + cap = expression.capabilities + case kind when OptionalTypeExpression - OptionalTypeExpression.new( - inner: with_nominal_arguments(expression.inner, name, arguments), - capabilities: expression.capabilities, - ) + TypeExpression.new(kind: OptionalTypeExpression.new( + inner: with_nominal_arguments(kind.inner, name, arguments), + ), capabilities: cap) when FallibleTypeExpression - FallibleTypeExpression.new( - inner: with_nominal_arguments(expression.inner, name, arguments), - error_set: expression.error_set, - capabilities: expression.capabilities, - ) + TypeExpression.new(kind: FallibleTypeExpression.new( + inner: with_nominal_arguments(kind.inner, name, arguments), + error_set: kind.error_set, + ), capabilities: cap) when FutureTypeExpression - FutureTypeExpression.new( - inner: with_nominal_arguments(expression.inner, name, arguments), - capabilities: expression.capabilities, - ) + TypeExpression.new(kind: FutureTypeExpression.new( + inner: with_nominal_arguments(kind.inner, name, arguments), + ), capabilities: cap) when LinearTypeExpression - LinearTypeExpression.new( - kind: expression.kind, - dimensions: expression.dimensions, - item: with_nominal_arguments(expression.item, name, arguments), - allocation_hint: expression.allocation_hint, - capabilities: expression.capabilities, - ) + TypeExpression.new(kind: LinearTypeExpression.new( + kind: kind.kind, + dimensions: kind.dimensions, + item: with_nominal_arguments(kind.item, name, arguments), + allocation_hint: kind.allocation_hint, + ), capabilities: cap) when StreamTypeExpression - StreamTypeExpression.new( - cardinality: expression.cardinality, - item: with_nominal_arguments(expression.item, name, arguments), - capabilities: expression.capabilities, - ) + TypeExpression.new(kind: StreamTypeExpression.new( + cardinality: kind.cardinality, + item: with_nominal_arguments(kind.item, name, arguments), + ), capabilities: cap) when FunctionTypeExpression expression when TupleTypeExpression return expression unless name == :Tuple - TupleTypeExpression.new(items: arguments, capabilities: expression.capabilities) + TypeExpression.new(kind: TupleTypeExpression.new(items: arguments), capabilities: cap) when NamedTypeExpression - return expression unless expression.name == name + return expression unless kind.name == name - NamedTypeExpression.new(name: name, arguments: arguments, capabilities: expression.capabilities) + TypeExpression.new(kind: NamedTypeExpression.new(name: name, arguments: arguments), capabilities: cap) when MapTypeExpression return expression unless name == :HashMap - key = arguments.length == 1 ? expression.key : arguments.fetch(0) + key = arguments.length == 1 ? kind.key : arguments.fetch(0) value = arguments.length == 1 ? arguments.fetch(0) : arguments.fetch(1) - MapTypeExpression.new( + TypeExpression.new(kind: MapTypeExpression.new( key: key, value: value, key_implicit: arguments.length == 1, - legacy_separator: expression.legacy_separator, - capabilities: expression.capabilities, - ) + legacy_separator: kind.legacy_separator, + ), capabilities: cap) else expression end @@ -330,89 +347,70 @@ def self.with_nominal_arguments(expression, name, arguments) ).returns(TypeExpression) end def self.transform(expression, &visitor) - rebuilt = T.let(case expression + kind = expression.kind + cap = expression.capabilities + rebuilt = T.let(case kind when NamedTypeExpression - NamedTypeExpression.new( - name: expression.name, - arguments: expression.arguments.map { |argument| transform(argument, &visitor) }, - capabilities: expression.capabilities, - ) + TypeExpression.new(kind: NamedTypeExpression.new( + name: kind.name, + arguments: kind.arguments.map { |argument| transform(argument, &visitor) }, + ), capabilities: cap) when TypeProjectionExpression expression when FunctionTypeExpression - signature = expression.signature - FunctionTypeExpression.new( - signature: Type::FunctionType.new( + signature = kind.signature + TypeExpression.new(kind: FunctionTypeExpression.new( + signature: FunctionSignatureExpression.new( params: signature.params.map do |param| - Type::FunctionTypeParam.new(type: transformed_type(param.type, &visitor)) + FunctionParamExpression.new(expression: transform(param.expression, &visitor)) end, - return_type: transformed_type(signature.return_type, &visitor), + return_expression: transform(signature.return_expression, &visitor), reentrant: signature.reentrant, - source_signature: signature.source_signature, abi: signature.abi, ), - capabilities: expression.capabilities, - ) + ), capabilities: cap) when TupleTypeExpression - TupleTypeExpression.new( - items: expression.items.map { |item| transform(item, &visitor) }, - capabilities: expression.capabilities, - ) + TypeExpression.new(kind: TupleTypeExpression.new( + items: kind.items.map { |item| transform(item, &visitor) }, + ), capabilities: cap) when OptionalTypeExpression - OptionalTypeExpression.new(inner: transform(expression.inner, &visitor), - capabilities: expression.capabilities) + TypeExpression.new(kind: OptionalTypeExpression.new(inner: transform(kind.inner, &visitor)), capabilities: cap) when FallibleTypeExpression - error_set = expression.error_set + error_set = kind.error_set transformed_error_set = if error_set transform(error_set, &visitor) end - FallibleTypeExpression.new( - inner: transform(expression.inner, &visitor), + TypeExpression.new(kind: FallibleTypeExpression.new( + inner: transform(kind.inner, &visitor), error_set: transformed_error_set, - capabilities: expression.capabilities, - ) + ), capabilities: cap) when FutureTypeExpression - FutureTypeExpression.new(inner: transform(expression.inner, &visitor), - capabilities: expression.capabilities) + TypeExpression.new(kind: FutureTypeExpression.new(inner: transform(kind.inner, &visitor)), capabilities: cap) when LinearTypeExpression - LinearTypeExpression.new( - kind: expression.kind, - dimensions: expression.dimensions, - item: transform(expression.item, &visitor), - allocation_hint: expression.allocation_hint, - capabilities: expression.capabilities, - ) + TypeExpression.new(kind: LinearTypeExpression.new( + kind: kind.kind, + dimensions: kind.dimensions, + item: transform(kind.item, &visitor), + allocation_hint: kind.allocation_hint, + ), capabilities: cap) when MapTypeExpression - MapTypeExpression.new( - key: transform(expression.key, &visitor), - value: transform(expression.value, &visitor), - key_implicit: expression.key_implicit, - legacy_separator: expression.legacy_separator, - capabilities: expression.capabilities, - ) + TypeExpression.new(kind: MapTypeExpression.new( + key: transform(kind.key, &visitor), + value: transform(kind.value, &visitor), + key_implicit: kind.key_implicit, + legacy_separator: kind.legacy_separator, + ), capabilities: cap) when StreamTypeExpression - StreamTypeExpression.new( - cardinality: expression.cardinality, - item: transform(expression.item, &visitor), - capabilities: expression.capabilities, - ) + TypeExpression.new(kind: StreamTypeExpression.new( + cardinality: kind.cardinality, + item: transform(kind.item, &visitor), + ), capabilities: cap) else expression end, TypeExpression) visitor.call(rebuilt) end - sig do - params( - type: Type, - visitor: T.proc.params(node: TypeExpression).returns(TypeExpression), - ).returns(Type) - end - def self.transformed_type(type, &visitor) - Type.new(transform(type.shape.expression, &visitor)) - end - private_class_method :transformed_type - sig { params(expression: TypeExpression).returns(Integer) } def self.node_count(expression) each_node(expression).length @@ -438,7 +436,7 @@ def self.nested_capabilities?(expression) sig { params(expression: TypeExpression).returns(T::Boolean) } def self.tense_wrapper?(expression) - case expression + case expression.kind when OptionalTypeExpression, FallibleTypeExpression, FutureTypeExpression then true else false end @@ -468,18 +466,19 @@ def self.direct_children(expression) sig { params(expression: TypeExpression).returns(T::Array[TypeExpression]) } def self.children(expression) - case expression - when NamedTypeExpression then expression.arguments + kind = expression.kind + case kind + when NamedTypeExpression then kind.arguments when TypeProjectionExpression then [] when FunctionTypeExpression - expression.signature.params.map { |param| param.type.shape.expression } + - [expression.signature.return_type.shape.expression] - when TupleTypeExpression then expression.items - when OptionalTypeExpression, FutureTypeExpression then [expression.inner] + kind.signature.params.map(&:expression) + + [kind.signature.return_expression] + when TupleTypeExpression then kind.items + when OptionalTypeExpression, FutureTypeExpression then [kind.inner] when FallibleTypeExpression - expression.error_set.nil? ? [expression.inner] : [expression.inner, T.must(expression.error_set)] - when LinearTypeExpression, StreamTypeExpression then [expression.item] - when MapTypeExpression then [expression.key, expression.value] + kind.error_set.nil? ? [kind.inner] : [kind.inner, T.must(kind.error_set)] + when LinearTypeExpression, StreamTypeExpression then [kind.item] + when MapTypeExpression then [kind.key, kind.value] else [] end end @@ -495,10 +494,10 @@ class GenericParts < T::Struct const :arguments, T::Array[String] end - sig { params(raw: T.any(Type::FunctionType, Symbol, String)).returns(TypeExpression) } + # Semantic function-type raws (Type::FunctionType) convert at the type.rb + # boundary (Type.function_type_expression); the foundation parses spellings. + sig { params(raw: T.any(Symbol, String)).returns(TypeExpression) } def self.parse(raw) - return FunctionTypeExpression.new(signature: raw) if raw.is_a?(Type::FunctionType) - source = raw.to_s if source.start_with?("[~") closing = source.index("]") @@ -516,7 +515,7 @@ def self.parse(raw) ) unless cardinality.nil? item_source = source[(closing + 1)..].to_s - return StreamTypeExpression.new(cardinality: cardinality, item: parse(item_source)) unless item_source.empty? + return TypeExpression.of(StreamTypeExpression.new(cardinality: cardinality, item: parse(item_source))) unless item_source.empty? end end end @@ -554,9 +553,9 @@ def self.parse_prefixed_source(source) when "~" raise ArgumentError, "double future type is not allowed" if inner_source.start_with?("~") - FutureTypeExpression.new(inner: parse_source(inner_source)) + TypeExpression.of(FutureTypeExpression.new(inner: parse_source(inner_source))) when "!" - FallibleTypeExpression.new(inner: parse_source(inner_source)) + TypeExpression.of(FallibleTypeExpression.new(inner: parse_source(inner_source))) when "?" parse_optional_source(inner_source) end @@ -566,20 +565,20 @@ def self.parse_prefixed_source(source) def self.parse_optional_source(inner_source) raise ArgumentError, "double optional type is not allowed" if inner_source.start_with?("?") if grouped_type?(inner_source) - return OptionalTypeExpression.new(inner: parse_source(inner_source[1..-2].to_s)) + return TypeExpression.of(OptionalTypeExpression.new(inner: parse_source(inner_source[1..-2].to_s))) end optional_array_parts = split_array_suffix(inner_source) unless optional_array_parts.nil? item_source, dimension = optional_array_parts - return LinearTypeExpression.new( + return TypeExpression.of(LinearTypeExpression.new( kind: dimension.nil? ? :list : :array, dimensions: [dimension || :LIST], - item: OptionalTypeExpression.new(inner: parse_source(item_source)) - ) + item: TypeExpression.of(OptionalTypeExpression.new(inner: parse_source(item_source))), + )) end - OptionalTypeExpression.new(inner: parse_source(inner_source)) + TypeExpression.of(OptionalTypeExpression.new(inner: parse_source(inner_source))) end sig { params(source: String).returns(TypeExpression) } @@ -590,26 +589,26 @@ def self.parse_primary_source(source) parse_generic_or_named_source(source) end - sig { params(source: String).returns(T.nilable(LinearTypeExpression)) } + sig { params(source: String).returns(T.nilable(TypeExpression)) } def self.parse_array_source(source) array_parts = split_array_suffix(source) return nil if array_parts.nil? item_source, dimension = array_parts - LinearTypeExpression.new( + TypeExpression.of(LinearTypeExpression.new( kind: dimension.nil? ? :list : :array, dimensions: [dimension || :LIST], - item: parse_source(item_source) - ) + item: parse_source(item_source), + )) end sig { params(source: String).returns(TypeExpression) } def self.parse_generic_or_named_source(source) if (projection = /\A([A-Z]\w*)::([A-Z]\w*)\z/.match(source)) - return TypeProjectionExpression.new( + return TypeExpression.of(TypeProjectionExpression.new( owner: T.must(projection[1]).to_sym, member: T.must(projection[2]).to_sym, - ) + )) end generic = split_generic(source) @@ -617,31 +616,31 @@ def self.parse_generic_or_named_source(source) arguments = generic.arguments.map { |argument| parse_source(argument) } if generic.base == "HashMap" if arguments.length == 1 - return MapTypeExpression.new( - key: NamedTypeExpression.new(name: :String), + return TypeExpression.of(MapTypeExpression.new( + key: TypeExpression.of(NamedTypeExpression.new(name: :String)), value: T.must(arguments.first), - key_implicit: true - ) + key_implicit: true, + )) end if arguments.length == 2 key = T.must(arguments.first) value = T.must(arguments.last) - return MapTypeExpression.new( + return TypeExpression.of(MapTypeExpression.new( key: key, value: value, - legacy_separator: top_level_argument_separator(source) - ) + legacy_separator: top_level_argument_separator(source), + )) end raise ArgumentError, "HashMap expects one or two type arguments" elsif generic.base == "Tuple" - return TupleTypeExpression.new(items: arguments) + return TypeExpression.of(TupleTypeExpression.new(items: arguments)) end - return NamedTypeExpression.new(name: generic.base.to_sym, arguments: arguments) + return TypeExpression.of(NamedTypeExpression.new(name: generic.base.to_sym, arguments: arguments)) end base, capabilities = split_legacy_capability_suffix(source) - NamedTypeExpression.new(name: base.to_sym, capabilities: capabilities) + TypeExpression.new(kind: NamedTypeExpression.new(name: base.to_sym), capabilities: capabilities) end sig { params(source: String).returns([String, TypeCapabilities]) } @@ -788,141 +787,168 @@ def self.top_level_argument_separator(source) class TypeExpressionPrinter extend T::Sig + # A collection dimension is either an Integer size or a Symbol marker. Branch + # the union variant explicitly instead of matching the union value against + # symbol literals: the typed self-host cannot compare `Integer | Symbol` to a + # bare `:LIST`, so narrowing to Symbol first keeps the inner case well-typed. + sig { params(dimension: TypeExpression::Dimension).returns(String) } + def self.dimension_suffix(dimension) + if dimension.is_a?(Symbol) + case dimension + when :LIST then "[]" + when :STREAM_OPEN then "[?]" + when :INF then "[INF]" + when :INFERRED then "[*]" + else "[#{dimension}]" + end + else + "[#{dimension}]" + end + end + + # Inline Pivot dimension marker: same variant split as `dimension_suffix`. + sig { params(dimension: TypeExpression::Dimension).returns(String) } + def self.dimension_marker(dimension) + if dimension.is_a?(Symbol) + case dimension + when :LIST then "List" + when :STREAM_OPEN then "~" + when :INF then "~INF" + when :INFERRED then "*" + else dimension.to_s + end + else + dimension.to_s + end + end + # Canonical semantic spelling used for type identity. Unlike `legacy`, this # deliberately normalizes stream syntax and omits representation # capabilities, which are keyed separately by Type. sig { params(expression: TypeExpression).returns(String) } def self.semantic(expression) - case expression + kind = expression.kind + case kind when NamedTypeExpression - return expression.name.to_s if expression.arguments.empty? + return kind.name.to_s if kind.arguments.empty? - "#{expression.name}<#{expression.arguments.map { |argument| semantic(argument) }.join(",")}>" + "#{kind.name}<#{kind.arguments.map { |argument| semantic(argument) }.join(",")}>" when TypeProjectionExpression - "#{expression.owner}::#{expression.member}" + "#{kind.owner}::#{kind.member}" when FunctionTypeExpression - params = expression.signature.params.map { |param| semantic(param.type.shape.expression) }.join(",") - "FN(#{params}) -> #{semantic(expression.signature.return_type.shape.expression)}" + params = kind.signature.params.map { |param| semantic(param.expression) }.join(",") + "FN(#{params}) -> #{semantic(kind.signature.return_expression)}" when TupleTypeExpression - "Tuple<#{expression.items.map { |item| semantic(item) }.join(",")}>" + "Tuple<#{kind.items.map { |item| semantic(item) }.join(",")}>" when OptionalTypeExpression - inner = semantic(expression.inner) - grouped = expression.inner.is_a?(LinearTypeExpression) || expression.inner.is_a?(MapTypeExpression) + inner = semantic(kind.inner) + inner_kind = kind.inner.kind + grouped = inner_kind.is_a?(LinearTypeExpression) || inner_kind.is_a?(MapTypeExpression) grouped ? "?(#{inner})" : "?#{inner}" when FallibleTypeExpression - "!#{semantic(expression.inner)}" + "!#{semantic(kind.inner)}" when FutureTypeExpression - "~#{semantic(expression.inner)}" + "~#{semantic(kind.inner)}" when LinearTypeExpression - expression.dimensions.reduce(semantic(expression.item)) do |surface, dimension| - suffix = case dimension - when :LIST then "[]" - when :STREAM_OPEN then "[?]" - when :INF then "[INF]" - when :INFERRED then "[*]" - else "[#{dimension}]" - end - "#{surface}#{suffix}" + kind.dimensions.reduce(semantic(kind.item)) do |surface, dimension| + "#{surface}#{dimension_suffix(dimension)}" end when MapTypeExpression - "HashMap<#{semantic(expression.key)},#{semantic(expression.value)}>" + "HashMap<#{semantic(kind.key)},#{semantic(kind.value)}>" when StreamTypeExpression - suffix = expression.cardinality == :FINITE ? "[]" : "[#{expression.cardinality}]" - "~#{semantic(expression.item)}#{suffix}" + suffix = kind.cardinality == :FINITE ? "[]" : "[#{kind.cardinality}]" + "~#{semantic(kind.item)}#{suffix}" else - raise "unknown type expression #{expression.class}" + raise "unknown type expression #{kind.class}" end end sig { params(expression: TypeExpression).returns(String) } def self.legacy(expression) - case expression + kind = expression.kind + cap = expression.capabilities + case kind when NamedTypeExpression - base = if expression.arguments.empty? - expression.name.to_s + base = if kind.arguments.empty? + kind.name.to_s else - "#{expression.name}<#{expression.arguments.map { |argument| legacy(argument) }.join(",")}>" + "#{kind.name}<#{kind.arguments.map { |argument| legacy(argument) }.join(",")}>" end - "#{base}#{capability_suffix(expression.capabilities)}" + "#{base}#{capability_suffix(cap)}" when TypeProjectionExpression - "#{expression.owner}::#{expression.member}#{capability_suffix(expression.capabilities)}" + "#{kind.owner}::#{kind.member}#{capability_suffix(cap)}" when FunctionTypeExpression - "FN(#{expression.signature.params.map { |param| legacy(TypeExpressionParser.parse(param.type.raw)) }.join(",")}) -> #{legacy(TypeExpressionParser.parse(expression.signature.return_type.raw))}#{capability_suffix(expression.capabilities)}" + "FN(#{kind.signature.params.map { |param| legacy(param.expression) }.join(",")}) -> #{legacy(kind.signature.return_expression)}#{capability_suffix(cap)}" when TupleTypeExpression - "Tuple<#{expression.items.map { |item| legacy(item) }.join(",")}>#{capability_suffix(expression.capabilities)}" + "Tuple<#{kind.items.map { |item| legacy(item) }.join(",")}>#{capability_suffix(cap)}" when OptionalTypeExpression - inner = legacy(expression.inner) - inner_expression = expression.inner - base = inner_expression.is_a?(LinearTypeExpression) || inner_expression.is_a?(MapTypeExpression) ? "?(#{inner})" : "?#{inner}" - "#{base}#{capability_suffix(expression.capabilities)}" + inner = legacy(kind.inner) + inner_kind = kind.inner.kind + base = inner_kind.is_a?(LinearTypeExpression) || inner_kind.is_a?(MapTypeExpression) ? "?(#{inner})" : "?#{inner}" + "#{base}#{capability_suffix(cap)}" when FallibleTypeExpression - "!#{legacy(expression.inner)}#{capability_suffix(expression.capabilities)}" + "!#{legacy(kind.inner)}#{capability_suffix(cap)}" when FutureTypeExpression - "~#{legacy(expression.inner)}#{capability_suffix(expression.capabilities)}" + "~#{legacy(kind.inner)}#{capability_suffix(cap)}" when LinearTypeExpression - surface = expression.dimensions.reduce(legacy(expression.item)) do |item_surface, dimension| - suffix = case dimension - when :LIST then "[]" - when :STREAM_OPEN then "[?]" - when :INF then "[INF]" - when :INFERRED then "[*]" - else "[#{dimension}]" - end - "#{item_surface}#{suffix}" + surface = kind.dimensions.reduce(legacy(kind.item)) do |item_surface, dimension| + "#{item_surface}#{dimension_suffix(dimension)}" end - "#{surface}#{capability_suffix(expression.capabilities, include_collection: false)}" + "#{surface}#{capability_suffix(cap, include_collection: false)}" when MapTypeExpression - key = legacy(expression.key) - value = legacy(expression.value) - base = expression.key_implicit ? "HashMap<#{value}>" : "HashMap<#{key}#{expression.legacy_separator}#{value}>" - "#{base}#{capability_suffix(expression.capabilities)}" + key = legacy(kind.key) + value = legacy(kind.value) + base = kind.key_implicit ? "HashMap<#{value}>" : "HashMap<#{key}#{kind.legacy_separator}#{value}>" + "#{base}#{capability_suffix(cap)}" when StreamTypeExpression - cardinality = expression.cardinality + cardinality = kind.cardinality marker = cardinality == :FINITE ? "" : cardinality.to_s - "[~#{marker}]#{legacy(expression.item)}#{capability_suffix(expression.capabilities)}" + "[~#{marker}]#{legacy(kind.item)}#{capability_suffix(cap)}" else - raise "unknown type expression #{expression.class}" + raise "unknown type expression #{kind.class}" end end sig { params(expression: TypeExpression).returns(String) } def self.inline(expression) - case expression + kind = expression.kind + cap = expression.capabilities + case kind when NamedTypeExpression - base = if expression.arguments.empty? - expression.name.to_s + base = if kind.arguments.empty? + kind.name.to_s else - "#{expression.name}<#{expression.arguments.map { |argument| inline(argument) }.join(", ")}>" + "#{kind.name}<#{kind.arguments.map { |argument| inline(argument) }.join(", ")}>" end - "#{base}#{capability_suffix(expression.capabilities)}" + "#{base}#{capability_suffix(cap)}" when TypeProjectionExpression - "#{expression.owner}::#{expression.member}#{capability_suffix(expression.capabilities)}" + "#{kind.owner}::#{kind.member}#{capability_suffix(cap)}" when FunctionTypeExpression - "FN(#{expression.signature.params.map { |param| inline(TypeExpressionParser.parse(param.type.raw)) }.join(", ")}) -> #{inline(TypeExpressionParser.parse(expression.signature.return_type.raw))}#{capability_suffix(expression.capabilities)}" + "FN(#{kind.signature.params.map { |param| inline(param.expression) }.join(", ")}) -> #{inline(kind.signature.return_expression)}#{capability_suffix(cap)}" when TupleTypeExpression - "Tuple<#{expression.items.map { |item| inline(item) }.join(", ")}>#{capability_suffix(expression.capabilities)}" + "Tuple<#{kind.items.map { |item| inline(item) }.join(", ")}>#{capability_suffix(cap)}" when OptionalTypeExpression - "?#{inline(expression.inner)}#{capability_suffix(expression.capabilities)}" + "?#{inline(kind.inner)}#{capability_suffix(cap)}" when FallibleTypeExpression - "!#{inline(expression.inner)}#{capability_suffix(expression.capabilities)}" + "!#{inline(kind.inner)}#{capability_suffix(cap)}" when FutureTypeExpression - "~#{inline(expression.inner)}#{capability_suffix(expression.capabilities)}" + "~#{inline(kind.inner)}#{capability_suffix(cap)}" when LinearTypeExpression - prefix = inline_linear_prefix(expression) - caps = capability_suffix(expression.capabilities, include_collection: false) - "#{prefix}#{caps.empty? ? "" : "#{caps} "}#{inline(expression.item)}" + prefix = inline_linear_prefix(kind) + caps = capability_suffix(cap, include_collection: false) + "#{prefix}#{caps.empty? ? "" : "#{caps} "}#{inline(kind.item)}" when MapTypeExpression - caps = capability_suffix(expression.capabilities) - "{#{inline(expression.key)}}#{caps.empty? ? "" : "#{caps} "}#{inline(expression.value)}" + caps = capability_suffix(cap) + "{#{inline(kind.key)}}#{caps.empty? ? "" : "#{caps} "}#{inline(kind.value)}" when StreamTypeExpression - cardinality = expression.cardinality + cardinality = kind.cardinality marker = cardinality == :FINITE ? "" : cardinality.to_s - caps = capability_suffix(expression.capabilities) - "[~#{marker}]#{caps.empty? ? "" : "#{caps} "}#{inline(expression.item)}" + caps = capability_suffix(cap) + "[~#{marker}]#{caps.empty? ? "" : "#{caps} "}#{inline(kind.item)}" else - raise "unknown type expression #{expression.class}" + raise "unknown type expression #{kind.class}" end end @@ -940,15 +966,7 @@ def self.inline_linear_prefix(expression) return "[Pool(#{T.must(pool_size)})]" end - dimensions = expression.dimensions.map do |dimension| - case dimension - when :LIST then "List" - when :STREAM_OPEN then "~" - when :INF then "~INF" - when :INFERRED then "*" - else dimension.to_s - end - end + dimensions = expression.dimensions.map { |dimension| dimension_marker(dimension) } "[#{dimensions.join(", ")}]" end private_class_method :inline_linear_prefix @@ -958,13 +976,13 @@ def self.capability_suffix(capabilities, include_collection: true) parts = T.let([], T::Array[String]) ownership = capabilities.ownership if ownership && ownership != :affine - parts << T.must(Type.ownership_surface_name_for(ownership)) + parts << T.must(TypeCapabilities.ownership_surface_name_for(ownership)) end parts << "@boxed" if capabilities.layout == :indirect parts << "@soa" if capabilities.soa parts << "@sharded(#{capabilities.shard_count})" unless capabilities.shard_count.nil? sync = capabilities.sync - parts << T.must(Type.sync_surface_name_for(sync)) unless sync.nil? + parts << T.must(TypeCapabilities.sync_surface_name_for(sync)) unless sync.nil? parts << "@observable" if capabilities.observable if include_collection && !capabilities.collection.nil? parts << "@#{capabilities.collection}" diff --git a/compiler/ruby/backends/mir_emitter.rb b/compiler/ruby/backends/mir_emitter.rb index d3107a68e..d49de5136 100644 --- a/compiler/ruby/backends/mir_emitter.rb +++ b/compiler/ruby/backends/mir_emitter.rb @@ -157,6 +157,8 @@ def emit(node) when MIR::Comment then "// #{node.text}" when MIR::Suppress then "_ = &#{node.name};" when MIR::PubConst then "pub const #{node.name} = #{node.value};" + when MIR::ModuleVar then "#{node.visibility == :pub ? 'pub ' : ''}var #{node.name}: #{node.zig_type} = undefined;" + when MIR::ModuleConstFree then "defer CheatLib.cleanup(@TypeOf(#{node.name}), #{@rt_name}.heapAlloc(), &#{node.name});" when MIR::ProtocolAdapterDef then emit_protocol_adapter_def(node) # --- Memory operations --- @@ -173,6 +175,8 @@ def emit(node) when MIR::CapWrap then emit_cap_wrap(node) when MIR::SharePromote then emit_share_promote(node) when MIR::RcRetain then emit_rc_retain(node) + when MIR::ComptimeCarrierPayload then emit_comptime_carrier_payload(node) + when MIR::MonomorphicKeep then emit_monomorphic_keep(node) when MIR::RcRelease then emit_rc_release(node) when MIR::RcDowngrade then emit_rc_downgrade(node) when MIR::WeakUpgrade then emit_weak_upgrade(node) @@ -605,7 +609,9 @@ def emit_inline_bc_as_zig(node) entry = node.stdlib_def raise "emit_inline_bc_as_zig: node has no stdlib_def (:#{node.op})" unless entry pattern = entry.required_intrinsic_template(IntrinsicTemplateKind::Zig) - node.args.each_with_index { |a, i| pattern = pattern.gsub("{#{i}}") { emit(a) } } + node.args.each_with_index do |a, i| + pattern = pattern.split("{#{i}}").join(T.must(emit(a))) + end node.suppress_try ? pattern.delete_prefix("try ") : pattern end @@ -1219,16 +1225,16 @@ def emit_capability_unwrap(node) inner_t = "@typeInfo(@TypeOf(#{source})).pointer.child" "(if (comptime #{is_ptr}) " \ "(if (comptime @typeInfo(#{inner_t}) == .@\"struct\") " \ - "(if (comptime @hasField(#{inner_t}, \"ctrl\")) #{source}.ctrl.data else #{source}) " \ + "(if (comptime @hasDecl(#{inner_t}, \"__clear_ref_carrier\")) #{source}.ctrl.data else #{source}) " \ "else " \ "(if (comptime @typeInfo(#{inner_t}) == .pointer) " \ "(if (comptime @typeInfo(@typeInfo(#{inner_t}).pointer.child) == .@\"struct\") " \ - "(if (comptime @hasField(@typeInfo(#{inner_t}).pointer.child, \"ctrl\")) #{source}.*.ctrl.data else #{source}.*) " \ + "(if (comptime @hasDecl(@typeInfo(#{inner_t}).pointer.child, \"__clear_ref_carrier\")) #{source}.*.ctrl.data else #{source}.*) " \ "else #{source}.*) " \ "else #{source})) " \ "else " \ "(if (comptime @typeInfo(@TypeOf(#{source})) == .@\"struct\") " \ - "(if (comptime @hasField(@TypeOf(#{source}), \"ctrl\")) #{source}.ctrl.data else &#{source}) " \ + "(if (comptime @hasDecl(@TypeOf(#{source}), \"__clear_ref_carrier\")) #{source}.ctrl.data else &#{source}) " \ "else &#{source}))" end @@ -1252,11 +1258,11 @@ def comptime_arc_unwrap_expr(source) base_t = "@TypeOf(#{source})" "(if (comptime @typeInfo(#{base_t}) == .pointer) " \ "(if (comptime @typeInfo(@typeInfo(#{base_t}).pointer.child) == .@\"struct\") " \ - "(if (comptime @hasField(@typeInfo(#{base_t}).pointer.child, \"ctrl\")) #{source}.*.ctrl.data.* else #{source}.*) " \ + "(if (comptime @hasDecl(@typeInfo(#{base_t}).pointer.child, \"__clear_ref_carrier\")) #{source}.*.ctrl.data.* else #{source}.*) " \ "else #{source}.*) " \ "else " \ "(if (comptime @typeInfo(#{base_t}) == .@\"struct\") " \ - "(if (comptime @hasField(#{base_t}, \"ctrl\")) #{source}.ctrl.data.* else #{source}) " \ + "(if (comptime @hasDecl(#{base_t}, \"__clear_ref_carrier\")) #{source}.ctrl.data.* else #{source}) " \ "else #{source}))" end @@ -1462,6 +1468,17 @@ def with_heap_allocator_cache(cache_name, runtime_name, &blk) @heap_allocator_cache_runtime_name = runtime_name blk.call ensure + restore_heap_allocator_cache(cache_name, previous_name, previous_runtime) + end + + sig do + params( + cache_name: T.nilable(String), + previous_name: T.nilable(String), + previous_runtime: T.nilable(String), + ).void + end + def restore_heap_allocator_cache(cache_name, previous_name, previous_runtime) @heap_allocator_cache_names.pop if cache_name @heap_allocator_cache_name = previous_name @heap_allocator_cache_runtime_name = previous_runtime @@ -1586,7 +1603,8 @@ def emit_snapshot_multi_txn(node) sig { params(node: MIR::WithMatchDispatch).returns(String) } def emit_with_match_dispatch(node) cell_zig = T.must(emit(node.cell)) - arm_strs = node.arms.each_with_index.map { |arm, i| + arms = node.arms + arm_strs = arms.each_with_index.map { |arm, i| probe = emit_with_match_probe(arm.family, cell_zig, node.snapshot_mode) head = i.zero? ? "if (comptime #{probe})" : "else if (comptime #{probe})" body_zig = emit_body(arm.body || []) @@ -1731,6 +1749,7 @@ def zig_byte_string_literal(text) when 0x0D then '\\r' when 0x09 then '\\t' when 0x00 then '\\x00' + when 0x01..0x1F, 0x7F then "\\x#{'%02x' % b}" when 0x80..0xFF then "\\x#{'%02x' % b}" else b.chr end @@ -1966,7 +1985,7 @@ def emit_sorted_lock_handler_switch(action, matched, bubble, rt_name, source_lin sig { params(node: MIR::Program).returns(String) } def emit_program(node) - parts = node.items.filter_map { |item| emit(item) } + parts = node.items.filter_map { |item| emit_container_item(item) } symbol_pool = symbol_pool_declarations parts.unshift(symbol_pool) unless symbol_pool.empty? out = [] @@ -2070,7 +2089,13 @@ def emit_import(node) sig { params(node: MIR::TypeAlias).returns(String) } def emit_type_alias(node) - "const #{node.name} = #{node.target};" + # Aliases are declarative; a unit that keeps a declared-but-unused alias + # must still compile inside block scopes (gen.rb wraps library units in a + # `test { ... }` block, where Zig hard-errors on unused locals). The + # comptime reference placates that check at zero runtime cost. `pub` + # matches CLEAR's scope model, where imported types re-export: a module + # aliasing a foreign type must expose it to its own importers. + "pub const #{node.name} = #{node.target};\ncomptime { _ = @typeName(#{node.name}); }" end sig { params(node: MIR::CExternFnDecl).returns(String) } @@ -2093,30 +2118,96 @@ def emit_c_extern_struct_def(node) sig { params(node: MIR::ModuleNamespace).returns(String) } def emit_module_namespace(node) - body = emit_body(node.items || []) + body = (node.items || []).filter_map { |item| emit_container_item(item) }.join("\n") "const #{node.name} = struct {\n#{indent_block(body, 4)}\n};" end + # Zig container scope accepts declarations but not standalone discard + # statements. A lowering-time suppression belongs to a function body, so + # omit it when a MIR::Let is itself a program/module item. + sig { params(node: MIR::Emittable).returns(T.nilable(String)) } + def emit_container_item(node) + return emit_let(node, include_suppression: false) if node.is_a?(MIR::Let) + + emit(node) + end + private :emit_container_item + sig { params(node: MIR::TestDef).returns(String) } def emit_test_def(node) + return emit_scheduler_test_def(node) if node.needs_scheduler + body = emit_body(node.body) "test \"#{node.name}\" {\n#{body}\n}" end + # Fiber-spawning test bodies (stamped by TestLowering): boot a scheduler + # and drive the body as a task, mirroring how clearMain runs — the bare + # test-runner thread has no scheduler (GPF in submitSpawn/getSched). The + # body's `rt` is the task fiber's runtime, so the plain preamble's rt + # binding is replaced by the entry-wrapper parameter. + sig { params(node: MIR::TestDef).returns(String) } + def emit_scheduler_test_def(node) + stmts = node.body.reject { |stmt| stmt.is_a?(MIR::TestPreamble) } + body = emit_body(stmts) + <<~ZIG.chomp + test "#{node.name}" { + var da = std.heap.DebugAllocator(.{}){}; + defer _ = da.deinit(); + const allocator = da.allocator(); + var global_ctx = EbrContext{}; + defer global_ctx.deinit(allocator); + const fp = @import("runtime/scheduler.zig"); + var sched = try fp.Scheduler.init(allocator, &global_ctx, null); + defer { + fp.scheduler_running = false; + sched.deinit(); + fp.global_registry.deinit(allocator); + } + fp.active_scheduler = &sched; + fp.scheduler_running = true; + const __TestBody = struct { + fn run(raw_rt: *anyopaque, raw_args: ?*anyopaque) anyerror!void { + _ = raw_args; + const rt = @as(*Runtime, @ptrCast(@alignCast(raw_rt))); + _ = &rt; + #{body} + } + }; + try sched.submitSpawn( + @intFromPtr(&Runtime.entryWrapper), + @as(CheatHeader.TaskFn, @ptrCast(&__TestBody.run)), + null, + .{ .stack_size = .Large }, + ); + sched.run(); + } + ZIG + end + # --- Statement emitters --- - sig { params(node: MIR::Let).returns(String) } - def emit_let(node) + sig { params(node: MIR::Let, include_suppression: T::Boolean).returns(String) } + def emit_let(node, include_suppression: true) if node.init.is_a?(MIR::FreezeExpr) buf = "#{node.name}__buf" kw = node.mutable ? "var" : "const" - sup = node.suppression ? " #{node.suppression}" : "" + sup = include_suppression && node.suppression ? " #{node.suppression}" : "" return "const #{buf} = #{emit(node.init)};\n#{kw} #{node.name} = #{buf}._root;#{sup}" end kw = node.mutable ? "var" : "const" ann = node.annotation ? ": #{node.annotation.nested_zig_type}" : "" + if node.module_const + # A comptime CONST initializer runs at container scope with no runtime; any + # `rt` reference in the (pure) initializer binds to `undefined`. Zig folds + # the call at comptime, and a heap-touching initializer (which would deref + # `undefined`) is already rejected upstream as MODULE_SCOPE_OWNED_VALUE. + init = with_ident_overrides("rt" => "undefined") { emit(node.init) } + vis = node.const_visibility == :pub ? "pub " : "" + return "#{vis}const #{node.name}#{ann} = #{init};" + end init = emit(node.init) - sup = node.suppression ? " #{node.suppression}" : "" + sup = include_suppression && node.suppression ? " #{node.suppression}" : "" "#{kw} #{node.name}#{ann} = #{init};#{sup}" end @@ -2317,7 +2408,8 @@ def emit_catch_wrapper(node) return "return #{inner_call} catch {\n#{indent_block(emit_catch_default_body(node), 4)}\n};" end - branch_parts = node.clauses.each_with_index.map do |clause, index| + clauses = node.clauses + branch_parts = clauses.each_with_index.map do |clause, index| emit_catch_clause(clause, node.rt_name, node.snapshot_type, index.zero?) end branch_parts << emit_catch_default(node) @@ -2485,9 +2577,10 @@ def emit_index_insert(node) def emit_sort(node) et = node.elem_type items = emit(node.items_expr) + cmp = node.string_keys ? "std.mem.order(u8, #{emit(node.key_a)}, #{emit(node.key_b)}) == .lt" : "#{emit(node.key_a)} < #{emit(node.key_b)}" "std.mem.sort(#{et}, #{items}, {}, struct {\n" \ " pub fn lessThan(_: void, a: #{et}, b: #{et}) bool {\n" \ - " return #{emit(node.key_a)} < #{emit(node.key_b)};\n" \ + " return #{cmp};\n" \ " }\n" \ "}.lessThan);" end @@ -2935,6 +3028,22 @@ def emit_rc_retain(node) "CheatLib.#{node.func}(#{node.zig_base}, #{emit(node.source)})" end + sig { params(node: MIR::ComptimeCarrierPayload).returns(String) } + def emit_comptime_carrier_payload(node) + src = T.must(emit(node.source)) + "(if (comptime @hasDecl(@TypeOf(#{src}), \"__clear_ref_carrier\")) #{src}.ctrl.data.* else #{src})" + end + + sig { params(node: MIR::MonomorphicKeep).returns(String) } + def emit_monomorphic_keep(node) + src = T.must(emit(node.source)) + alloc = alloc_from_sym(node.alloc || :heap) + # dupeValue is carrier-generic: retainOne for an Rc/Arc handle, a structural + # deep copy for a plain struct payload (heap fields included), and a + # @compileError for a linear plain type (drop glue without clone glue). + "try CheatLib.dupeValue(@TypeOf(#{src}), #{src}, #{alloc})" + end + sig { params(node: MIR::RcRelease).returns(String) } def emit_rc_release(node) "CheatLib.#{node.func}(#{node.zig_base}, #{emit(node.alloc)}, #{emit(node.source)})" @@ -3532,7 +3641,8 @@ def render_resource_close_plan(plan, root_name) sig { params(action: Schemas::ResourceCloseAction, root_name: String).returns(String) } def render_resource_close_action(action, root_name) target = ([root_name] + action.field_path).join(".") - runtime_args = Array.new(action.runtime_heap_alloc_args) { alloc_zig(:heap) } + runtime_args = T.let([], T::Array[String]) + action.runtime_heap_alloc_args.times { runtime_args << alloc_zig(:heap) } case action.call_kind when Schemas::ResourceCloseCallKind::Method "#{target}.#{action.name}(#{runtime_args.join(", ")})" diff --git a/compiler/ruby/backends/transpiler.rb b/compiler/ruby/backends/transpiler.rb index 2423dd316..c1333e363 100644 --- a/compiler/ruby/backends/transpiler.rb +++ b/compiler/ruby/backends/transpiler.rb @@ -3,6 +3,9 @@ require 'bundler/setup' # so `bundle exec` not needed require "sorbet-runtime" +# This configures the Ruby host only; Sorbet runtime checks do not exist in +# the self-hosted CLEAR artifact. +# ruby-to-clear: skip begin T::Configuration.default_checked_level = :never unless ENV["CLEAR_SORBET_RUNTIME"] == "1" rescue RuntimeError @@ -331,25 +334,38 @@ def transpile_as_module(cheat_code, source_dir: @source_dir, pkg_paths: {}) # --- RUN IT --- +# ruby-to-clear: skip $logger = T.let(Logger.new(STDOUT), Logger) +# ruby-to-clear: skip $logger.level = Logger::INFO +# ruby-to-clear: skip $logger.formatter = proc do |severity, datetime, progname, msg| "[#{severity}] #{msg}\n" end +# The self-hosted compiler has its own entrypoint; this is Ruby CLI glue. +# ruby-to-clear: skip if __FILE__ == $0 options = { mode: :standalone, pkg_paths: {} } OptionParser.new do |opts| opts.on('--log-level LEVEL', 'Set log level (DEBUG, INFO, WARN, ERROR)') do |level| - $logger.level = Logger.const_get(level.upcase) + levels = { + 'DEBUG' => Logger::DEBUG, + 'INFO' => Logger::INFO, + 'WARN' => Logger::WARN, + 'ERROR' => Logger::ERROR, + } + resolved = levels[level.upcase] + abort "Unknown log level: #{level} (expected DEBUG, INFO, WARN, or ERROR)" unless resolved + $logger.level = resolved end opts.on('--module', 'Emit as a Zig module (uses @import("cheat_runtime"), no runtime footer)') do options[:mode] = :module end - opts.on('--pkg SPEC', 'Register a package path as "name=/abs/path/to/lib.clear"') do |spec| + opts.on('--pkg SPEC', 'Register a package path as "name=/abs/path/to/lib.clear" (comma list = multi-file package)') do |spec| name, path = spec.split('=', 2) - options[:pkg_paths][name] = File.expand_path(path) + options[:pkg_paths][name] = path.split(',').map { |member| File.expand_path(member.strip) }.join(',') end opts.on('--use-c-allocator', 'Use the C allocator (jemalloc/mimalloc) instead of GPA') do options[:use_c_allocator] = true @@ -386,6 +402,17 @@ def transpile_as_module(cheat_code, source_dir: @source_dir, pkg_paths: {}) script_file = ARGV.first if script_file + if script_file.start_with?('pkg:') + # A multi-file package as the root unit: materialize the merged source. + require 'tmpdir' + require_relative '../compiler/package_source' + pkg_name = script_file.delete_prefix('pkg:') + registered = options[:pkg_paths][pkg_name] + abort "pkg:#{pkg_name}: not registered (pass --pkg #{pkg_name}=a.clear,b.clear)" unless registered + merged = PackageSource.merge(registered.split(','), resolve_pkg: ->(name) { options[:pkg_paths][name] }) + script_file = File.join(Dir.tmpdir, "clear_pkg_#{pkg_name}.clear") + File.write(script_file, merged.source) + end code = File.read(script_file) source_dir = File.dirname(File.expand_path(script_file)) ENV["AUDIT_CURRENT_FILE"] = script_file diff --git a/compiler/ruby/backends/type_zig_renderer.rb b/compiler/ruby/backends/type_zig_renderer.rb index fc25c2418..d1f9e14d8 100644 --- a/compiler/ruby/backends/type_zig_renderer.rb +++ b/compiler/ruby/backends/type_zig_renderer.rb @@ -11,6 +11,7 @@ class TypeZigRenderer extend T::Sig sig { params(type: Type, is_param: T::Boolean, is_field: T::Boolean).returns(String) } + # ruby-to-clear: effects reentrant def self.render_async_payload(type, is_param: false, is_field: false) unless type.error_union? return render(type, is_param: is_param, is_field: is_field, nested: true) @@ -22,6 +23,7 @@ def self.render_async_payload(type, is_param: false, is_field: false) end sig { params(type: Type, is_param: T::Boolean, is_field: T::Boolean, nested: T::Boolean).returns(String) } + # ruby-to-clear: fallible def self.render(type, is_param: false, is_field: false, nested: false) # Zig needs an explicit error set before an optional payload. Its shorthand # accepts `!i64`, but not `!?i64`; the latter must be `anyerror!?i64`. diff --git a/compiler/ruby/backends/zig_type.rb b/compiler/ruby/backends/zig_type.rb index cc72ce3ef..17a0c4873 100644 --- a/compiler/ruby/backends/zig_type.rb +++ b/compiler/ruby/backends/zig_type.rb @@ -8,6 +8,7 @@ class ZigType extend T::Sig RESERVED_IDENTIFIERS = T.let(Set.new(%w[ + _ addrspace align allowzero and anyframe anytype asm async await break callconv catch comptime const continue defer else enum errdefer error export extern false fn for if inline linksection noalias noinline nosuspend null opaque or orelse diff --git a/compiler/ruby/backends/zig_type_mapper.rb b/compiler/ruby/backends/zig_type_mapper.rb index 1774139e3..dfc28087a 100644 --- a/compiler/ruby/backends/zig_type_mapper.rb +++ b/compiler/ruby/backends/zig_type_mapper.rb @@ -34,9 +34,9 @@ module ZigTypeMapper # Special AST nodes you might map to operators #:OR_ELSE => "orelse" - }, T::Hash[Symbol, String]) + }.freeze, T::Hash[Symbol, String]) - ZIG_PRIMITIVES = ["i8", "i16", "i32", "i64", "u8", "u16", "u32", "u64", "f32", "f64", "bool", "void", "[]const u8"] + ZIG_PRIMITIVES = ["i8", "i16", "i32", "i64", "u8", "u16", "u32", "u64", "f32", "f64", "bool", "void", "[]const u8"].freeze # Delegates to Type#zig_type for type-to-Zig conversion. # This keeps the transpiler interface stable while the logic lives in Type. diff --git a/compiler/ruby/compiler/compiler_frontend.rb b/compiler/ruby/compiler/compiler_frontend.rb index 931a5c464..186719c54 100644 --- a/compiler/ruby/compiler/compiler_frontend.rb +++ b/compiler/ruby/compiler/compiler_frontend.rb @@ -112,20 +112,7 @@ def self.compile(cheat_code, importer:, source_dir:, strict_test: false, ownersh end end - fn_sigs = T.let({}, T::Hash[String, FunctionSignature]) - ast.statements.each do |stmt| - next unless stmt.is_a?(AST::FunctionDef) - fn_sigs[stmt.name] = FunctionSignature.from_function_def(stmt) - end - - # Include module-imported function signatures so MIRLowering can - # determine needs_rt/can_fail for cross-module calls. - annotator.semantic_root_scope.visible_entries.each do |name, entry| - next if fn_sigs.key?(name) - sig = entry.fn_signature - next unless sig && sig.module_alias - fn_sigs[name] = sig - end + fn_sigs = FunctionSignature.lowering_signatures(ast, annotator.semantic_root_scope) moved_guard_info = T.let({}, MIRLoweringInput::MovedGuardInfo) fn_nodes.each { |name, fn| moved_guard_info[name] = fn.moved_guard_info if fn.moved_guard_info } diff --git a/compiler/ruby/compiler/module_importer.rb b/compiler/ruby/compiler/module_importer.rb index 3035724ca..6c44ec35f 100644 --- a/compiler/ruby/compiler/module_importer.rb +++ b/compiler/ruby/compiler/module_importer.rb @@ -2,6 +2,7 @@ require "sorbet-runtime" require "set" +require_relative "package_source" class ModuleImportError < StandardError; end class CircularDependencyError < ModuleImportError; end @@ -48,10 +49,24 @@ def initialize(base_dir: Dir.pwd, pkg_paths: {}, use_mir: false, stdlib_root: ST @base_dir = T.let(File.expand_path(base_dir), String) @module_cache = T.let({}, T::Hash[T.untyped, T.untyped]) # abs_path => CompiledModule @compiling = T.let(Set.new, T::Set[T.untyped]) # abs_paths currently being compiled (cycle detection) - # pkg_paths: { "name" => "/abs/path/to/lib.clear" } -- registered package sources. + # pkg_paths: { "name" => "/abs/path/to/lib.clear" } -- registered package + # sources. A comma-separated value registers a MULTI-FILE package: all + # listed files compile together as ONE unit (Go model — the acyclic + # import rule applies between packages, not between a package's files). @pkg_paths = T.let(pkg_paths.transform_keys(&:to_s), T::Hash[T.untyped, T.untyped]) @inline_packages = T.let(inline_packages.map(&:to_s).to_set, T::Set[String]) @stdlib_root = T.let(stdlib_root, String) + # abs member file -> owning multi-file package name. Any compile of a + # member file (directly or via its own single-file pkg name) is aliased + # to the whole package so the unit is never split. + @package_members = T.let({}, T::Hash[String, String]) + @pkg_paths.each do |name, value| + next unless value.to_s.include?(",") + + value.to_s.split(",").each do |member| + @package_members[File.expand_path(member.strip)] = name.to_s + end + end end # Compile a .clear package by name and return a CompiledModule. @@ -68,9 +83,64 @@ def compile_package(pkg_name, caller_dir: @base_dir) "Register it with --pkg #{pkg_name}=/path/to/lib.clear " \ "or place it under #{@stdlib_root}/#{pkg_name}/src/lib.clear" end + return compile_package_group(pkg_name.to_s, path.to_s.split(",").map(&:strip)) if path.to_s.include?(",") + + # A single-file package whose file belongs to a multi-file package + # compiles as that whole package — a unit is never split. + owner = @package_members[File.expand_path(path)] + return compile_package(owner, caller_dir: caller_dir) if owner && owner != pkg_name.to_s + compile_file(path, caller_dir: File.dirname(File.expand_path(path))) end + # Compile a multi-file package: all members merged into ONE compilation + # unit (source-level, sibling REQUIREs dropped, externals deduplicated). + # Cycles BETWEEN packages are still rejected by the ordinary @compiling + # guard; references between members never re-enter the importer at all. + sig { params(pkg_name: String, members: T::Array[String]).returns(T.nilable(ModuleImporter::CompiledModule)) } + def compile_package_group(pkg_name, members) + cache_key = "pkg-group:#{pkg_name}" + return @module_cache[cache_key] if @module_cache.key?(cache_key) + + if @compiling.include?(cache_key) + cycle = @compiling.to_a.map { |p| File.basename(p.to_s) }.join(" -> ") + raise CircularDependencyError, "Circular dependency detected: #{cycle} -> pkg:#{pkg_name}" + end + + members.each do |member| + abs = File.expand_path(member) + raise ModuleImportError, "REQUIRE error: package '#{pkg_name}' member not found: #{abs}" unless File.exist?(abs) + end + + @compiling.add(cache_key) + begin + merged = PackageSource.merge(members, resolve_pkg: ->(name) { @pkg_paths[name] || resolve_stdlib_package(name) }) + source_dir = File.dirname(T.must(merged.member_paths.first)) + + saved_gradual = ClearParser.gradual_mode + ClearParser.gradual_mode = false + ast = begin + budget = FrontendResourceBudget.new + tokens = Lexer.new(merged.source, file: "pkg:#{pkg_name}", budget: budget).tokenize + ClearParser.new(tokens, merged.source, budget: budget).parse + ensure + ClearParser.gradual_mode = saved_gradual + end + + reject_auto_in_public_signatures!(ast, "pkg:#{pkg_name}") + + annotator = SemanticAnnotator.new(importer: self, source_dir: source_dir, source_code: merged.source) + annotator.annotate!(ast) + + mod = compile_module_mir(ast, annotator, source_dir) + + @module_cache[cache_key] = mod + mod + ensure + @compiling.delete(cache_key) + end + end + sig { params(pkg_name: String).returns(T.nilable(String)) } def resolve_stdlib_package(pkg_name) return nil unless @stdlib_root @@ -100,6 +170,10 @@ def stdlib_package?(pkg_name) def compile_file(path, caller_dir: @base_dir) abs_path = File.expand_path(path, caller_dir) + # A member of a multi-file package always compiles as the whole package. + owner = @package_members[abs_path] + return compile_package(owner, caller_dir: caller_dir) if owner + return @module_cache[abs_path] if @module_cache.key?(abs_path) if @compiling.include?(abs_path) @@ -202,11 +276,7 @@ def compile_module_mir(ast, annotator, source_dir) end end - fn_sigs = T.let({}, T::Hash[String, FunctionSignature]) - ast.statements.each do |stmt| - next unless stmt.is_a?(AST::FunctionDef) - fn_sigs[stmt.name] = FunctionSignature.from_function_def(stmt) - end + fn_sigs = FunctionSignature.lowering_signatures(ast, annotator.semantic_root_scope) moved_guard_info = T.let({}, MIRLoweringInput::MovedGuardInfo) fn_nodes.each { |name, fn| moved_guard_info[name] = fn.moved_guard_info if fn.moved_guard_info } diff --git a/compiler/ruby/compiler/package_source.rb b/compiler/ruby/compiler/package_source.rb new file mode 100644 index 000000000..3aedd2adc --- /dev/null +++ b/compiler/ruby/compiler/package_source.rb @@ -0,0 +1,195 @@ +# typed: strict +require "sorbet-runtime" + +# Source-level merge for multi-file packages (Go model): a package is a set +# of .clear files compiled TOGETHER as one unit. Files of one package +# reference each other's declarations directly — REQUIREs between members +# are dropped during the merge, so the acyclic-import rule applies only +# BETWEEN packages, never between files of the same package. +# +# The merge is textual and deliberately simple: +# - every member's REQUIRE lines are classified: a require that resolves +# into the member set is a SIBLING require and is dropped; anything else +# is EXTERNAL and is hoisted to the top of the merged unit, deduplicated +# by (path, alias). +# - member bodies follow in the given order, each behind a `# FILE:` +# marker comment so merged-unit diagnostics can be traced back. +# +# Constraints (documented, checked where cheap): +# - REQUIRE statements must be single-line (they are, in both hand-written +# and translated CLEAR). +# - members living in different directories must use `pkg:` requires for +# externals; a path-relative external require is resolved against ITS +# member's directory and re-emitted as an absolute-path require so the +# merged unit (whose source_dir is the first member's directory) still +# finds it. +module PackageSource + extend T::Sig + + REQUIRE_LINE = T.let(/\A\s*REQUIRE\s+"([^"]+)"(\s+AS\s+([A-Za-z_][A-Za-z0-9_]*))?\s*;?\s*\z/, Regexp) + REQUIRE_ALIAS = T.let(/\s+AS\s+([A-Za-z_][A-Za-z0-9_]*)\s*;?\s*\z/, Regexp) + + class MergedPackage < T::Struct + const :source, String + const :member_paths, T::Array[String] + end + + # Merge `member_paths` (absolute paths, deterministic order) into one + # compilation unit. `resolve_pkg` maps a pkg name to its registered + # source path(s) (String, comma-list String, or Array) so sibling + # `pkg:` requires can be recognized; unknown names resolve to nil and + # are treated as external. + sig do + params( + member_paths: T::Array[String], + resolve_pkg: T.proc.params(name: String).returns(T.nilable(T.any(String, T::Array[String]))), + ).returns(MergedPackage) + end + def self.merge(member_paths, resolve_pkg:) + members = member_paths.map { |p| File.expand_path(p) } + member_set = members.to_set + + external_requires = T.let([], T::Array[String]) + seen_requires = T.let(Set.new, T::Set[[String, T.nilable(String)]]) + bodies = T.let([], T::Array[String]) + + members.each do |path| + dir = File.dirname(path) + body_lines = T.let([], T::Array[String]) + # Materialize the lines so ruby-to-clear lowers this as a FOR loop. + # `String#each_line` is callback-based and cannot carry the mutable + # body_lines accumulator through a CLEAR closure capture. + File.read(path).split("\n").each do |raw_line| + line = raw_line + m = REQUIRE_LINE.match(line) + unless m + body_lines << "#{line}\n" + next + end + + target = T.cast(m[1], String).dup + alias_name = T.let(nil, T.nilable(String)) + alias_match = REQUIRE_ALIAS.match(line) + alias_name = T.cast(alias_match[1], String).dup if alias_match + resolved = PackageSource.resolve_require_targets(target, dir, resolve_pkg) + if resolved.any? { |t| member_set.include?(t) } + # Sibling require: the declaration is part of this unit already. + next + end + + emitted_target = PackageSource.portable_require_target(target, dir) + key_alias = T.let(nil, T.nilable(String)) + key_alias = alias_name.dup if alias_name + key = T.let([emitted_target.dup, key_alias], [String, T.nilable(String)]) + next if seen_requires.include?(key) + + seen_requires << key + if alias_name + external_requires << "REQUIRE \"#{emitted_target}\" AS #{alias_name}\n" + else + external_requires << "REQUIRE \"#{emitted_target}\"\n" + end + end + bodies << "# FILE: #{path}\n" + bodies << body_lines.join + end + + MergedPackage.new( + source: PackageSource.dedupe_generated_functions((external_requires + ["\n"] + bodies).join), + member_paths: members, + ) + end + + # Translated members each emit their own generated support helpers + # (castXToY, ruby_array_concat_T, ...). Merged into one unit, identical + # copies collide as duplicate declarations. Drop repeat definitions of a + # top-level FN whose full text matches one already kept; differing bodies + # under one name are left alone so the compiler surfaces the conflict. + sig { params(source: String).returns(String) } + def self.dedupe_generated_functions(source) + seen = T.let({}, T::Hash[String, String]) + out = T.let([], T::Array[String]) + lines = source.lines + i = 0 + seen_extern = T.let(Set.new, T::Set[String]) + while i < lines.length + line = T.must(lines[i]) + # Single-line EXTERN declarations (FN/STRUCT) repeat verbatim across + # members that share an FFI module — keep the first. + stripped_line = line.strip + if stripped_line.start_with?("EXTERN ") + key = stripped_line + if seen_extern.include?(key) + i += 1 + else + seen_extern << key + out << line.dup + i += 1 + end + next + end + m = /\A(?:PUB |PRIVATE )?FN ([A-Za-z_][A-Za-z0-9_]*)\(/.match(line) + unless m + out << line.dup + i += 1 + next + end + + block = T.let([line.dup], T::Array[String]) + j = i + 1 + while j < lines.length + block_line = T.must(lines[j]) + break if block_line == "END\n" || block_line == "END" + + block << block_line.dup + j += 1 + end + block << T.must(lines[j]).dup if j < lines.length + text = block.join + name = T.must(m[1]) + if seen[name] == text + # exact duplicate — skip + else + out << text + seen[name] = text unless seen.key?(name) + end + i = j + 1 + end + out.join + end + + # All absolute file paths a require target can resolve to. + sig do + params( + target: String, + member_dir: String, + resolve_pkg: T.proc.params(name: String).returns(T.nilable(T.any(String, T::Array[String]))), + ).returns(T::Array[String]) + end + def self.resolve_require_targets(target, member_dir, resolve_pkg) + if target.start_with?("pkg:") + resolved = resolve_pkg.call(target.delete_prefix("pkg:")) + return [] unless resolved + + list = if resolved.is_a?(String) + resolved.split(",") + else + resolved + end + expanded = T.let([], T::Array[String]) + list.each { |path| expanded << File.expand_path(path.strip) } + return expanded + end + + [File.expand_path(target, member_dir)] + end + + # Path-relative externals are re-anchored to the member's own directory so + # they survive the merged unit's single source_dir. + sig { params(target: String, member_dir: String).returns(String) } + def self.portable_require_target(target, member_dir) + return target if target.start_with?("pkg:") + + File.expand_path(target, member_dir) + end +end diff --git a/compiler/ruby/ffi/c_header_importer.rb b/compiler/ruby/ffi/c_header_importer.rb index 8ced7e243..454f6789a 100644 --- a/compiler/ruby/ffi/c_header_importer.rb +++ b/compiler/ruby/ffi/c_header_importer.rb @@ -47,19 +47,30 @@ def self.import(header, library, source_dir:) raise Error, "C header not found: #{header_path}" unless File.file?(header_path) zig = zig_executable - stdout, stderr, status = Open3.capture3(zig, "translate-c", "-I#{source_dir}", header_path) - unless status.success? - detail = stderr.lines.first(8).join.strip - raise Error, "Zig could not import C header #{header.inspect}: #{detail}" - end + stdout = compiler_zig_translate_c(zig, source_dir, header_path) header_source = File.read(header_path) Translator.new(stdout, header: header, library: library, allowed_names: declaration_names(header_source)).declarations - rescue Errno::ENOENT => e - raise Error, "Zig is required to import C headers: #{e.message}" + rescue Errno::ENOENT + raise Error, "Zig is required to import C headers" end + # Ruby shells out through Open3. The self-hosted compiler maps this adapter + # onto compilerZigTranslateC in the compiler native support module. + # ruby-to-clear: skip + sig { params(zig: String, source_dir: String, header_path: String).returns(String) } + def self.compiler_zig_translate_c(zig, source_dir, header_path) + stdout, stderr, status = Open3.capture3(zig, "translate-c", "-I#{source_dir}", header_path) + unless status.success? + detail = stderr.split("\n").take(8).join("\n").strip + raise Error, "Zig could not import C header #{header_path.inspect}: #{detail}" + end + + stdout + end + + # ruby-to-clear: skip sig { returns(String) } def self.zig_executable candidates = [ @@ -101,7 +112,7 @@ def declarations structs = translate_structs functions = translate_functions declarations = structs + functions - if declarations.empty? + if declarations.length == 0 raise Error, "C header #{@header.inspect} contains no ABI declarations CLEAR can import" end (declarations + [""]).join("\n") @@ -111,8 +122,9 @@ def declarations sig { void } def collect_aliases! - @zig_source.each_line do |line| - next unless (match = line.match(/^pub const ([A-Za-z_]\w*) = (.+);$/)) + @zig_source.split("\n").each do |line| + match = T.let(line.match(/^pub const ([A-Za-z_]\w*) = (.+);$/), T.nilable(MatchData)) + next unless match name = T.must(match[1]) rhs = T.must(match[2]).strip @aliases[name] = rhs @@ -126,50 +138,74 @@ def translate_structs c_name = T.cast(name, String) next unless @allowed_names.include?(c_name) clear_name = clear_type_name(c_name) - fields = T.cast(body, String).lines.filter_map do |line| + fields = T.let([], T::Array[String]) + T.cast(body, String).split("\n").each do |line| match = line.match(/^\s*([A-Za-z_]\w*):\s*(.+?)(?:\s*=\s*.+)?,$/) next unless match field_type = map_type(T.must(match[2]), position: :field) next unless field_type - "#{T.must(match[1])}: #{field_type}" + fields << "#{T.must(match[1])}: #{field_type}" end next if fields.empty? && !T.cast(body, String).strip.empty? output << %(EXTERN STRUCT #{clear_name} { #{fields.join(', ')} } AS "#{c_name}" FROM "#{@library}" ABI C HEADER "#{@header}";) end - @aliases.each do |name, rhs| - next unless @allowed_names.include?(name) - next unless rhs.start_with?("struct_") - next unless @zig_source.match?(/^pub const #{Regexp.escape(rhs)} = opaque \{/) - output << %(EXTERN STRUCT #{clear_type_name(name)} {} AS "#{name}" FROM "#{@library}" ABI C HEADER "#{@header}";) + alias_names = @aliases.keys + alias_index = T.let(0, Integer) + while alias_index < alias_names.length + name = alias_names.fetch(alias_index) + rhs = @aliases.fetch(name) + if @allowed_names.include?(name) && + rhs.start_with?("struct_") && + @zig_source.match?(/^pub const #{Regexp.escape(rhs)} = opaque \{/) + output << %(EXTERN STRUCT #{clear_type_name(name)} {} AS "#{name}" FROM "#{@library}" ABI C HEADER "#{@header}";) + end + alias_index += 1 end output.uniq end sig { returns(T::Array[String]) } def translate_functions - @zig_source.each_line.filter_map do |line| + output = T.let([], T::Array[String]) + @zig_source.split("\n").each do |line| parsed = parse_function_line(line) next unless parsed name, raw_params, raw_return = parsed next unless @allowed_names.include?(name) return_type = map_type(raw_return, position: :return) next unless return_type - params = split_top_level(raw_params).filter_map.with_index do |raw, index| - next if raw.strip == "..." - param_match = raw.match(/^([A-Za-z_]\w*):\s*(.+)$/) - param_name = param_match ? T.must(param_match[1]) : "arg#{index}" - raw_type = param_match ? T.must(param_match[2]) : raw - mapped = map_param(param_name, raw_type) - mapped + raw_param_parts = split_top_level(raw_params) + params = T.let([], T::Array[Param]) + valid_params = T.let(true, T::Boolean) + raw_param_parts.each_with_index do |raw, index| + if raw.strip == "..." + valid_params = false + break + else + param_match = raw.match(/^([A-Za-z_]\w*):\s*(.+)$/) + param_name = T.let("arg#{index}", String) + raw_type = T.let(raw, String) + if param_match + param_name = T.must(param_match[1]) + raw_type = T.must(param_match[2]) + end + param = map_param(param_name, raw_type) + unless param + valid_params = false + break + end + params << param + end end - next unless params.length == split_top_level(raw_params).length + next unless valid_params rendered_params = params.map do |param| prefix = param.mutable ? "MUTABLE " : "" "#{prefix}#{param.name}: #{param.type}" end - %(EXTERN FN #{name}(#{rendered_params.join(', ')}) RETURNS #{return_type} AS "#{name}" FROM "#{@library}" ABI C HEADER "#{@header}";) + output << %(EXTERN FN #{name}(#{rendered_params.join(', ')}) RETURNS #{return_type} AS "#{name}" FROM "#{@library}" ABI C HEADER "#{@header}";) end + output end sig { params(line: String).returns(T.nilable([String, String, String])) } @@ -264,11 +300,16 @@ def opaque_alias?(raw) def map_callback(raw) match = raw.match(/^\?\*const fn \((.*)\) callconv\(\.c\) (.+)$/) return nil unless match - params = split_top_level(T.must(match[1])).filter_map do |item| + raw_params = split_top_level(T.must(match[1])) + # `filter_map` lowered this as `?String[]`, so `join` could not resolve + # an overload even after its length check established success. + params = T.let([], T::Array[String]) + raw_params.each do |item| type = item.sub(/^[A-Za-z_]\w*:\s*/, "") - map_type(type, position: :param) + mapped = map_type(type, position: :param) + params << mapped if mapped end - return nil unless params.length == split_top_level(T.must(match[1])).length + return nil unless params.length == raw_params.length result = map_type(T.must(match[2]), position: :return) result ? "FN(#{params.join(', ')}) -> #{result} CALLCONV C" : nil end @@ -304,7 +345,23 @@ def c_named_type?(raw) sig { params(c_name: String).returns(String) } def clear_type_name(c_name) - @type_names[c_name] ||= c_name.sub(/^struct_/, "").split("_").reject(&:empty?).map(&:capitalize).join + existing = T.let(@type_names[c_name], T.nilable(String)) + return existing if existing + + words = T.let([], T::Array[String]) + c_name.sub(/^struct_/, "").split("_").each do |part| + next if part.empty? + + # CLEAR has `upcase`/`downcase`, but no Ruby `capitalize` intrinsic. + # Normalize both halves explicitly to retain Ruby's capitalization + # semantics for mixed-case C spelling. + first = T.must(part[0, 1]).upcase + rest = T.must(part[1..]).downcase + words << (first + rest) + end + clear_name = words.join + @type_names[c_name] = clear_name + clear_name end sig { params(text: String).returns(T::Array[String]) } diff --git a/compiler/ruby/incremental/dependency_snapshot.rb b/compiler/ruby/incremental/dependency_snapshot.rb index 669a8abd3..d1d3a4248 100644 --- a/compiler/ruby/incremental/dependency_snapshot.rb +++ b/compiler/ruby/incremental/dependency_snapshot.rb @@ -26,17 +26,22 @@ def initialize(entries) sig { params(paths: T::Enumerable[String]).returns(DependencySnapshot) } def self.capture(paths) - entries = paths.map { |path| File.expand_path(path) }.uniq.sort.map do |path| - DependencyFingerprint.new(path: path, digest: file_digest(path)) + expanded = T.let([], T::Array[String]) + paths.each { |path| expanded << File.expand_path(path) } + entries = T.let([], T::Array[DependencyFingerprint]) + expanded.uniq.sort.each do |path| + entries << DependencyFingerprint.new(path: path, digest: file_digest(path)) end new(entries) end sig { returns(T::Array[String]) } def changed_paths - @entries.filter_map do |entry| - entry.path unless self.class.file_digest(entry.path) == entry.digest + changed = T.let([], T::Array[String]) + @entries.each do |entry| + changed << entry.path unless self.class.file_digest(entry.path) == entry.digest end + changed end sig { returns(T::Boolean) } @@ -46,9 +51,13 @@ def current? sig { params(path: String).returns(String) } def self.file_digest(path) - return "missing" unless File.file?(path) + return "missing" unless File.exist?(path) - Digest::SHA256.file(path).hexdigest + # Keep the self-hosted dependency check exact and deterministic without + # depending on Ruby's Digest implementation. The retained snapshot owns + # one source copy per dependency; equality then detects edits and + # edit/revert cycles without mtime or hash-collision ambiguity. + File.read(path) end end end diff --git a/compiler/ruby/incremental/portable_cache.rb b/compiler/ruby/incremental/portable_cache.rb index 2e40ff9b1..81a7db796 100644 --- a/compiler/ruby/incremental/portable_cache.rb +++ b/compiler/ruby/incremental/portable_cache.rb @@ -78,7 +78,7 @@ def write(source:, artifact:, function_counter_snapshots:, dependencies:) raise ArgumentError, "incremental cache exceeds #{MAX_BYTES} bytes" if bytes.bytesize > MAX_BYTES FileUtils.mkdir_p(File.dirname(@path)) - temporary = "#{@path}.tmp.#{$$}" + temporary = "#{@path}.tmp.#{Process.pid}" begin File.binwrite(temporary, bytes) File.rename(temporary, @path) @@ -167,7 +167,7 @@ def decode_emission_state(value) sig { params(snapshots: T::Hash[String, MIRLoweringCounterSnapshot]).returns(T::Hash[String, T.untyped]) } def encode_counter_snapshots(snapshots) snapshots.transform_values do |snapshot| - snapshot.values.to_h { |kind, value| [kind.serialize, value] } + snapshot.values.dup end end @@ -175,9 +175,9 @@ def encode_counter_snapshots(snapshots) def decode_counter_snapshots(value) result = T.let({}, T::Hash[String, MIRLoweringCounterSnapshot]) hash!(value).each do |name, raw_snapshot| - values = T.let({}, T::Hash[MIRLoweringCounterKind, Integer]) + values = T.let({}, T::Hash[String, Integer]) hash!(raw_snapshot).each do |kind, counter| - values[MIRLoweringCounterKind.deserialize(String(kind))] = Integer(counter) + values[String(kind)] = Integer(counter) end result[String(name)] = MIRLoweringCounterSnapshot.new(values: values) end diff --git a/compiler/ruby/incremental/source_catalog.rb b/compiler/ruby/incremental/source_catalog.rb index 7e7b92833..46495c143 100644 --- a/compiler/ruby/incremental/source_catalog.rb +++ b/compiler/ruby/incremental/source_catalog.rb @@ -83,7 +83,14 @@ def self.build(source, module_path:) start_line: range.start_line, end_line: range.end_line, exact_fingerprint: digest(exact_source), - interface_fingerprint: token_fingerprint(interface_source), + # Retained identity v4: keep-analysis infers a param's handle ABI + # from body shapes (struct-literal stores and call-argument + # positions of param identifiers), so those shapes are interface, + # not implementation - a body-only edit that changes them must + # invalidate callers. + interface_fingerprint: token_fingerprint( + interface_source + retention_shape_source(node) + ), called_functions: calls, ) end @@ -100,7 +107,7 @@ def self.build(source, module_path:) source: source, module_path: module_path, functions: functions, - non_function_fingerprint: digest(without_functions(source, functions)), + non_function_fingerprint: digest(non_function_tokens(tokens, functions)), non_function_calls: non_function_calls, ) end @@ -162,21 +169,31 @@ def mask_functions(source, functions) bytes[offset] = 0x20 unless byte == 0x0A || byte == 0x0D end end - bytes.pack("C*").force_encoding(source.encoding) + # Compiler sources are UTF-8. CLEAR strings do not carry a mutable + # encoding tag, so keep the Ruby host result aligned with that + # contract instead of deriving a dynamic encoding from the input. + bytes.pack("C*").force_encoding(Encoding::UTF_8) end private - sig { params(source: String, functions: T::Array[FunctionItem]).returns(String) } - def without_functions(source, functions) - cursor = 0 - pieces = T.let([], T::Array[String]) - functions.sort_by(&:start_offset).each do |item| - pieces << (source.byteslice(cursor...item.start_offset) || "") - cursor = item.end_offset + # Fingerprint the TOKENS outside function bodies, not the raw bytes. + # Digesting the source made a comment or a reflowed line read as a + # semantic change, which dropped the whole file off the incremental fast + # path -- the common edit during development. The lexer has already run + # by this point, so the token stream costs nothing extra, and it carries + # exactly what the compiler acts on. + sig { params(tokens: T::Array[Lexer::Token], functions: T::Array[FunctionItem]).returns(String) } + def non_function_tokens(tokens, functions) + ranges = functions.map { |item| [item.start_offset, item.end_offset] } + parts = T.let([], T::Array[String]) + tokens.each do |token| + offset = token.start_offset + next if offset && ranges.any? { |from, to| offset >= from && offset < to } + + parts << "#{token.type}\u0000#{token.value}" end - pieces << (source.byteslice(cursor..-1) || "") - pieces.join + parts.join("\u0001") end sig { params(module_path: String, name: String).returns(String) } @@ -189,6 +206,51 @@ def digest(value) Digest::SHA256.hexdigest(value) end + # Retained identity v4: the body shapes keep-analysis reads participate + # in the interface fingerprint - see the FunctionItem construction + # above. Each arm mirrors its semantic deriver EXACTLY (same unwraps, + # same bare-identifier requirement); a looser capture here would let a + # body edit flip the derived ABI without changing the fingerprint: + # - struct-literal fields and field assignments feed + # Lifetimes#keep_param_identity!, which unwraps OR_ELSE (keeping the + # provided identity) but treats COPY/GIVE wrappers as not-kept; + # - call args feed KeepAnalysis transitive propagation, which fires on + # bare identifiers only. + sig { params(node: AST::FunctionDef).returns(String) } + def retention_shape_source(node) + param_names = node.params.map { |p| p.name.to_s }.to_set + shape = T.let([], T::Array[String]) + AST.each_locatable(node.body, descend_functions: false) do |child| + case child + when AST::StructLit + child.fields.each do |field_name, value| + inner = unwrap_keep_value(value) + next unless inner.is_a?(AST::Identifier) && param_names.include?(inner.name) + shape << "#{child.name}.#{field_name}<-#{inner.name}" + end + when AST::Assignment + target = child.name + next unless target.is_a?(AST::GetField) + inner = unwrap_keep_value(child.value) + next unless inner.is_a?(AST::Identifier) && param_names.include?(inner.name) + shape << "#{target.field}=<-#{inner.name}" + when AST::FuncCall + child.args.each_with_index do |arg, idx| + next unless arg.is_a?(AST::Identifier) && param_names.include?(arg.name) + shape << "#{child.name}(#{idx})<-#{arg.name}" + end + end + end + shape.sort.join(";") + end + + sig { params(value: AST::Node).returns(AST::Node) } + def unwrap_keep_value(value) + return unwrap_keep_value(value.left) if value.is_a?(AST::BinaryOp) && value.op == :OR_ELSE + + value + end + sig { params(source: String).returns(String) } def token_fingerprint(source) payload = Lexer.new(source).tokenize.reject { |token| token.type == :EOF }.map do |token| diff --git a/compiler/ruby/lsp/diagnostics.rb b/compiler/ruby/lsp/diagnostics.rb index 4a0f81d2b..b350d95af 100644 --- a/compiler/ruby/lsp/diagnostics.rb +++ b/compiler/ruby/lsp/diagnostics.rb @@ -143,7 +143,7 @@ def self.code_for(finding) DiagnosticRegistry::DIAGNOSTICS.each do |code, entry| template = entry[:template] - next unless template + next unless template.is_a?(String) # Skip umbrella templates whose body is a single placeholder # (e.g. "%{message}") — they'd match anything. next if template.start_with?('%{') && template.end_with?('}') && template.count('%') == 1 diff --git a/compiler/ruby/lsp/logger.rb b/compiler/ruby/lsp/logger.rb index b5b343596..be281ef1b 100644 --- a/compiler/ruby/lsp/logger.rb +++ b/compiler/ruby/lsp/logger.rb @@ -10,7 +10,7 @@ class Logger LEVELS = T.let({ debug: 0, info: 1, warn: 2, error: 3 }.freeze, T::Hash[Symbol, Integer]) sig { params(level: Symbol, io: IO).void } - def initialize(level: :info, io: $stderr) + def initialize(level: :info, io: STDERR) @level = T.let(LEVELS.fetch(level), Integer) @io = T.let(io, IO) end diff --git a/compiler/ruby/lsp/server.rb b/compiler/ruby/lsp/server.rb index 0d9e4f4c4..e0ea7d67b 100644 --- a/compiler/ruby/lsp/server.rb +++ b/compiler/ruby/lsp/server.rb @@ -25,7 +25,7 @@ class Server # path without sleeping for half a second; production runs at the # default 500. sig { params(stdin: T.any(IO, StringIO), stdout: T.any(IO, StringIO), log_level: Symbol, debounce_ms: Integer).void } - def initialize(stdin: $stdin, stdout: $stdout, log_level: :info, debounce_ms: 500) + def initialize(stdin: STDIN, stdout: STDOUT, log_level: :info, debounce_ms: 500) @stdin = stdin @stdout = stdout @stdout.sync = true diff --git a/compiler/ruby/mir/alloc.rb b/compiler/ruby/mir/alloc.rb index 9789b18e7..43b5760dc 100644 --- a/compiler/ruby/mir/alloc.rb +++ b/compiler/ruby/mir/alloc.rb @@ -22,7 +22,8 @@ def downgrade_frame_to_stack(node, storage) node.full_type!.mark_stack_value! node.storage = :stack - node.value.storage = :stack + value_node = node.value + value_node.storage = :stack :stack end @@ -72,7 +73,7 @@ def expression_allocates?(node) end when AST::Cast expression_allocates?(node.value) - when AST::MoveNode, AST::CopyNode, AST::CloneNode, AST::ShareNode, AST::LinkNode, AST::ResolveNode, AST::CapabilityWrap + when AST::MoveNode, AST::CopyNode, AST::KeepNode, AST::ShareNode, AST::LinkNode, AST::ResolveNode, AST::CapabilityWrap expression_allocates?(node.value) else true diff --git a/compiler/ruby/mir/cleanup_classifier.rb b/compiler/ruby/mir/cleanup_classifier.rb index 0a6080f25..ac5170c76 100644 --- a/compiler/ruby/mir/cleanup_classifier.rb +++ b/compiler/ruby/mir/cleanup_classifier.rb @@ -343,6 +343,8 @@ def self.place_for_binding_node(name, node) return if lifecycle && !lifecycle.needs_drop? ti = Type.from_node!(value, context: "cleanup lifetime promotion") + # Interned/borrowed slices never own storage, with or without a plan. + return if ti.symbol? || (ti.string? && ti.rodata?) entry_obj.promote_to_cleanup!(kind: ti.string? ? :heap_string : :uniform, alloc: :heap, has_moved_guard: true) end @@ -469,6 +471,15 @@ def self.stamp_field_pre_cleanups!(body, facts, schema_lookup: nil, lifecycle_re next end + # A value BlockExpr (`{ stmts...; result }`, e.g. a desugared pipeline + # fold) carries real bindings in its body that need cleanup classification + # like any other statement sequence. + if child.is_a?(AST::BlockExpr) + classify_cleanup_binding_body(child.body, schema_lookup, lifecycle_registry, bindings, entries_by_place) if child.body + classify_inline_bg_binding_body(child.result, schema_lookup, lifecycle_registry, bindings, entries_by_place) if child.result + next + end + classify_inline_bg_binding_body(child, schema_lookup, lifecycle_registry, bindings, entries_by_place) end end @@ -602,6 +613,49 @@ def self.stamp_field_pre_cleanups!(body, facts, schema_lookup: nil, lifecycle_re base[:via_pointer] = true if ti.respond_to?(:needs_pointer_passing?) && ti.needs_pointer_passing? bindings[name] = base end + + # A MONOMORPHIC TAKES param is emitted anytype and threads the caller's + # actual carrier. Its declared payload type may need no drop, but the carrier + # that arrives may be an Rc/Arc handle that must be released. The carrier's + # release lifecycle is registered at param introduction (LifecycleRegistry + # keyed by LifecyclePlanner.monomorphic_carrier_key); fetch it here rather + # than reconstructing memory logic in this consumer. DROP lowering then emits + # the carrier-agnostic CheatLib.cleanup(@TypeOf(u), ...) (no-op for a plain + # carrier, releaseOne for Rc/Arc). + fn_node.params.each do |p| + next unless p.takes && p.carrier_contract == :monomorphic + next if bindings.key?(p.name.to_s) + + base = entry(:rc, rc_alloc: :heap) + base.mark_moved_guard! + base.set_alloc!(:heap) + base[:source_kind] ||= :takes_param + plan = lifecycle_registry ? lifecycle_registry.fetch_monomorphic(p.type) : Semantic::LifecyclePlanner.monomorphic_carrier_plan(p.type) + base.set_lifecycle_plan!(plan) + bindings[p.name.to_s] = base + end + + # Kept-identity params (retained identity v4) own the Rc handle their + # call edge normalized; release at scope exit unless a consumer moved it. + fn_node.params.each do |p| + next if p.takes + next unless p.symbol&.kept_identity + + base = entry(:rc, rc_alloc: :heap) + base.mark_moved_guard! + base.set_alloc!(:heap) + base[:source_kind] ||= :kept_param + bindings[p.name.to_s] = base + end + end + + # INV-14: consuming sites inherit the source's cleanup recipe via the same + # entry builders locals use. Pipeline consumer loops that dequeue OWNED + # stream items reuse the TAKES-param recipe for the item's type — an owned + # channel payload has exactly the ownership shape of a consumed TAKES arg. + sig { params(ti: Type, schema_lookup: Proc).returns(T.nilable(CleanupEntry)) } + def self.owned_value_entry(ti, schema_lookup) + takes_param_base_entry(ti, schema_lookup) end # Build the base cleanup entry for a TAKES param of type ti. Defers to the @@ -1010,7 +1064,7 @@ def self.stamp_field_pre_cleanups!(body, facts, schema_lookup: nil, lifecycle_re symbol = sym.is_a?(SymbolEntry) ? (AST.declaration_symbol(sym) || sym) : sym storage = symbol&.storage reflected = T.unsafe(node) - storage || (reflected.respond_to?(:storage) ? T.cast(reflected.public_send(:storage), T.nilable(Symbol)) : nil) + storage || (reflected.respond_to?(:storage) ? T.cast(reflected.storage, T.nilable(Symbol)) : nil) end # ── Individual classifiers ─────────────────────────────────────── @@ -1028,7 +1082,7 @@ def self.stamp_field_pre_cleanups!(body, facts, schema_lookup: nil, lifecycle_re sig { params(ti: Type, node: AST::Node, schema_lookup: Proc).returns(T::Boolean) } private_class_method def self.mutable_owning_slot?(ti, node, schema_lookup) - return false unless node.respond_to?(:var_mutated) && node.var_mutated == true + return false unless node.var_mutated == true ownership_bearing_type?(ti, schema_lookup) end diff --git a/compiler/ruby/mir/cleanup_entry.rb b/compiler/ruby/mir/cleanup_entry.rb index 8a423e1df..f56197b23 100644 --- a/compiler/ruby/mir/cleanup_entry.rb +++ b/compiler/ruby/mir/cleanup_entry.rb @@ -2,7 +2,7 @@ # frozen_string_literal: true require "sorbet-runtime" -require_relative "../ast/schemas" +require_relative "../ast/type" require_relative "../semantic/lifecycle_plan" require_relative "placement" diff --git a/compiler/ruby/mir/control_flow.rb b/compiler/ruby/mir/control_flow.rb index f9bee74b9..28400955c 100644 --- a/compiler/ruby/mir/control_flow.rb +++ b/compiler/ruby/mir/control_flow.rb @@ -1135,7 +1135,7 @@ def collect_ownership_transfers(node, step) when AST::ShareNode collect_share_transfer(node, step) - when AST::CopyNode, AST::CloneNode, AST::FreezeNode + when AST::CopyNode, AST::KeepNode, AST::FreezeNode # COPY / FREEZE do NOT move the source. when AST::CapabilityWrap @@ -1428,7 +1428,7 @@ def check_call_reads(call_node, state) check_reads_in_expr(arg, state) unless arg.value.is_a?(AST::Identifier) elsif arg.is_a?(AST::ShareNode) check_share_reads(arg, state) - elsif arg.is_a?(AST::CopyNode) || arg.is_a?(AST::CloneNode) || arg.is_a?(AST::FreezeNode) + elsif arg.is_a?(AST::CopyNode) || arg.is_a?(AST::KeepNode) || arg.is_a?(AST::FreezeNode) # COPY/FREEZE: the source IS read (must be live to copy/freeze from). check_reads_in_expr(arg.value, state) else @@ -1446,7 +1446,7 @@ def check_reads_in_expr(node, state) when AST::Identifier check_identifier_node_read(node, state) - when AST::CopyNode, AST::CloneNode, AST::FreezeNode + when AST::CopyNode, AST::KeepNode, AST::FreezeNode # COPY/FREEZE x: x IS read (must be live to copy/freeze from). check_reads_in_expr(node.value, state) diff --git a/compiler/ruby/mir/fsm_lowering.rb b/compiler/ruby/mir/fsm_lowering.rb index 2ffa20fa0..15087e523 100644 --- a/compiler/ruby/mir/fsm_lowering.rb +++ b/compiler/ruby/mir/fsm_lowering.rb @@ -124,17 +124,23 @@ def lower_step_stmts(stmts, no_result:, ctx_id: nil, async_result_shape: nil) end expr_t = retained_error ? Type.new(retained_error) : (expr_type.is_a?(Type) ? expr_type : Type.new(expr_type)) - result_alloc = escaping_value_alloc(expr_t) + # The tail expression can be more narrowly inferred than the promise + # slot (notably a string literal inside Tuple becomes `[N]Byte`). Its + # allocator must follow the declared async payload, because the value + # escapes the worker even when the source inference itself looks + # non-owning. + declared_result_type = async_result_shape&.payload_type || expr_t + result_alloc = escaping_value_alloc(declared_result_type) raw_last_mir = with_decl_alloc(result_alloc) { lower(last_step.expr) } last_mir = T.let(raw_last_mir.is_a?(MIR::Emittable) ? raw_last_mir : nil, T.nilable(MIR::Node)) # The tail expression's annotated type describes the source. A rodata # String is borrowed there, but the promise result slot is an owning # cross-fiber destination. Give placement the destination contract so # it emits the ordinary lifecycle-approved heap copy before transfer. - result_destination_type = if expr_t.string? && expr_t.rodata? + result_destination_type = if declared_result_type.string? && expr_t.rodata? Type.new(:String, location: result_alloc) else - expr_t + declared_result_type end last_mir = place_value_for_destination( last_mir, @@ -320,6 +326,27 @@ def fsm_result_transfer_facts(result_mir, ast_node, destination_type = nil) move_guarded: guarded, ) end + + # Aggregate results can own compiler-generated temporaries without + # consuming an AST binding. For example, `Tuple{COPY "x", 1}` lowers the + # copied string to a guarded `__tmp_N`, then stores a tuple containing that + # temp into the promise. The AST walk above intentionally skips COPY's + # source, but the generated copy still crosses the FSM boundary and must + # disarm its segment-local cleanup. + MIR.nodes(result_mir).grep(MIR::Ident).each do |ident| + name = ident.name.to_s + safe = rename_map.fetch(name, name) + entry = bindings[name] || bindings[safe] || CleanupEntry::NONE + guarded = guarded_cleanup_names[name] == true || guarded_cleanup_names[safe] == true + next unless guarded || entry.present? + next if facts.any? { |fact| fact.name == safe } + + facts << MIR::FsmResultTransferFact.new( + name: safe, + target_alloc: entry.present? ? entry.alloc : :heap, + move_guarded: guarded || entry.has_moved_guard?, + ) + end facts end @@ -512,7 +539,7 @@ def fsm_cap_metadata(cap, with_node, ctx_id, captured) lock_field_ref = if any_rc "#{base_field}.ctrl.data.*" elsif polymorphic_locked - "(if (comptime @hasField(@TypeOf(#{base_field}), \"ctrl\")) #{base_field}.ctrl.data.* else #{base_field})" + "(if (comptime @hasDecl(@TypeOf(#{base_field}), \"__clear_ref_carrier\")) #{base_field}.ctrl.data.* else #{base_field})" else base_field end diff --git a/compiler/ruby/mir/fsm_ops.rb b/compiler/ruby/mir/fsm_ops.rb index 465a4f471..8e6751c5a 100644 --- a/compiler/ruby/mir/fsm_ops.rb +++ b/compiler/ruby/mir/fsm_ops.rb @@ -514,8 +514,28 @@ def self.walk(node, &block) return end yield node - T.unsafe(node).each_pair do |_, v| - walk(v, &block) - end if node.respond_to?(:each_pair) + case node + when AssignField, LetConst + walk(node.value, &block) + when ErrDeferCall, StmtCall, CallExpr + walk(node.args, &block) + when IoSubmit + walk(node.waiter, &block) + walk(node.extra_args, &block) + when IfFieldSubLtZeroReturnCall + walk(node.return_args, &block) + when SubField + walk(node.base, &block) + when AddrOf, IntCast + walk(node.expr, &block) + when AllocExpr + walk(node.count, &block) + when SliceUntilIntCast + walk(node.base, &block) + walk(node.end_expr, &block) + when BinOp + walk(node.left, &block) + walk(node.right, &block) + end end end diff --git a/compiler/ruby/mir/fsm_transform/emit.rb b/compiler/ruby/mir/fsm_transform/emit.rb index 78b35692a..16d4ccaea 100644 --- a/compiler/ruby/mir/fsm_transform/emit.rb +++ b/compiler/ruby/mir/fsm_transform/emit.rb @@ -1242,11 +1242,11 @@ def self.expand_lock_segment(spec, ctx, capture_map, lowering, base_idx) # which locks are still held. try_success_idx = held_set_idx - prior_meta = prior.map { |c| + prior_meta = T.let(prior.map { |c| m = lowering_api.fsm_cap_metadata(c, with_node, id, captured) return nil if m.nil? - m - } + T.cast(m, FsmLowering::FsmCapMetadata) + }, T::Array[FsmLowering::FsmCapMetadata]) pointer_captures = ctx.pointer_captures err = diff --git a/compiler/ruby/mir/fsm_transform/liveness.rb b/compiler/ruby/mir/fsm_transform/liveness.rb index 39138d457..7dca5f91f 100644 --- a/compiler/ruby/mir/fsm_transform/liveness.rb +++ b/compiler/ruby/mir/fsm_transform/liveness.rb @@ -29,6 +29,12 @@ module FsmTransform module Liveness + # Structural walk domain of collect_defs/collect_tail_uses: AST and + # MIR nodes mixed on one stack. Named (not an inline T.any) so the + # CLEAR translation can dispatch the each_pair reflection walk over a + # closed union. + CollectValue = T.type_alias { T.any(AST::Node, MIR::Node) } + class CrossSegmentVarFact < T::Struct extend T::Sig @@ -196,13 +202,14 @@ def self.collect_tail_uses(seg, uses_by_seg) case tail when Segments::IoSuspend next_idx = seg.index + 1 - bucket = (uses_by_seg[next_idx] ||= Set.new) + uses_by_seg[next_idx] ||= Set.new + bucket = uses_by_seg[next_idx] call_node = tail.call_node if call_node.is_a?(AST::MethodCall) - walk_idents(call_node.object) { |name| bucket << name } + walk_idents(call_node.object) { |name| T.must(bucket) << name } end call_node.args.each do |a| - walk_idents(a) { |name| bucket << name } + walk_idents(a) { |name| T.must(bucket) << name } end end end @@ -269,7 +276,7 @@ def self.collect_uses(stmt, into) def self.walk_idents(node, &block) return if node.nil? - stack = T.let([], T::Array[T.any(AST::Node, MIR::Node)]) + stack = T.let([], T::Array[CollectValue]) case node when Array node.reverse_each do |child| diff --git a/compiler/ruby/mir/fsm_transform/recursive_splitter.rb b/compiler/ruby/mir/fsm_transform/recursive_splitter.rb index cd6705da1..fda3fcf2c 100644 --- a/compiler/ruby/mir/fsm_transform/recursive_splitter.rb +++ b/compiler/ruby/mir/fsm_transform/recursive_splitter.rb @@ -204,11 +204,8 @@ def self.split(body, lowering, ctx: nil) builder = Builder.new done_idx = builder.reserve_index - begin - entry = emit_stmts(body, done_idx, builder, lowering, ctx || {}) - rescue UnsupportedShape - return nil - end + entry = emit_stmts_or_nil(body, done_idx, builder, lowering, ctx || {}) + return nil unless entry # The Done segment has no body; it's the final exit. builder.fill(done_idx, [], Segments::Done.new) @@ -230,6 +227,21 @@ def self.split(body, lowering, ctx: nil) ) end + sig do + params( + body: T::Array[AST::Node], + after_idx: Integer, + builder: Builder, + lowering: FsmTransform::LoweringApi, + ctx: SplitContext, + ).returns(T.nilable(Integer)) + end + def self.emit_stmts_or_nil(body, after_idx, builder, lowering, ctx) + emit_stmts(body, after_idx, builder, lowering, ctx) + rescue UnsupportedShape + nil + end + class UnsupportedShape < StandardError; end # Emit segments for `stmts` such that control flow exits to @@ -317,17 +329,18 @@ def self.stmt_introduces_split?(stmt) sig { params(stmts: T.nilable(T.any(SegmentStmt, T::Array[SegmentStmt]))).returns(T::Boolean) } def self.contains_suspend_anywhere?(stmts) T.bind(self, T.untyped) rescue nil - Array(stmts).any? do |stmt| - next true if stmt.is_a?(AST::Locatable) && Segments.classify_suspend(stmt) + Array(stmts).each do |stmt| + return true if stmt.is_a?(AST::Locatable) && Segments.classify_suspend(stmt) case stmt when AST::WithBlock - with_lock_suspend?(stmt) || contains_suspend_anywhere?(stmt.body) + return true if with_lock_suspend?(stmt) || contains_suspend_anywhere?(stmt.body) else - next false unless stmt.is_a?(Struct) + next unless stmt.is_a?(Struct) - AST.child_bodies(stmt).any? { |body| contains_suspend_anywhere?(body) } + return true if AST.child_bodies(stmt).any? { |body| contains_suspend_anywhere?(body) } end end + false end # A WITH "lock-suspends" if any of its capabilities require the @@ -632,6 +645,7 @@ def self.remap_tail(tail, mapping) private_class_method :emit_for_range_fragment private_class_method :emit_if_fragment private_class_method :emit_stmts + private_class_method :emit_stmts_or_nil private_class_method :emit_suspend private_class_method :emit_suspend_with_pre private_class_method :emit_while_fragment diff --git a/compiler/ruby/mir/fsm_transform/segments.rb b/compiler/ruby/mir/fsm_transform/segments.rb index d61cc469b..6a20d0a3d 100644 --- a/compiler/ruby/mir/fsm_transform/segments.rb +++ b/compiler/ruby/mir/fsm_transform/segments.rb @@ -38,8 +38,8 @@ module FsmTransform module Segments extend T::Sig - SegmentBodyInput = T.type_alias { T.nilable(T.any(AST::Node, AST::RawBody)) } - LockWithNode = T.type_alias { T.untyped } + SegmentBody = T.type_alias { T::Array[AST::Locatable] } + LockWithNode = T.type_alias { AST::WithBlock } LockCap = T.type_alias { T.any(CapabilityPlan::CapabilityTransition, Symbol) } Done = Struct.new(:_) do @@ -62,7 +62,14 @@ def with_next_index(index) sig { returns(T.nilable(Type)) } def result_type - call_node ? Type.from_node!(call_node, context: "FSM IO suspend result") : nil + return nil unless call_node + + node = call_node + type_object = node.type_object + raise "FSM IO suspend result: missing type info" unless type_object + concrete_type = type_object + raise "FSM IO suspend result: unresolved type info" if concrete_type.untyped? + concrete_type end end NextSuspend = Struct.new(:promise_ast, :result_var, :next_index) do @@ -80,10 +87,14 @@ def with_next_index(index) sig { returns(T.nilable(Type)) } def result_type - promise_ft = promise_ast ? Type.from_node!(promise_ast, context: "FSM NEXT suspend promise") : nil - return nil unless promise_ft - - pt = Type.new(promise_ft) + return nil unless promise_ast + + node = promise_ast + type_object = node.type_object + raise "FSM NEXT suspend result: missing type info" unless type_object + concrete_type = type_object + raise "FSM NEXT suspend result: unresolved type info" if concrete_type.untyped? + pt = Type.new(concrete_type) pt.tense_type end end @@ -177,7 +188,7 @@ def self.suspend_tail?(tail) # # Adding new shapes (IF with suspend, WhileLoop+IO, etc.) extends # this method's case dispatch + adds a new tail variant if needed. - sig { params(body: AST::RawBody, lowering: T.untyped).returns(T.nilable(SplitResult)) } + sig { params(body: SegmentBody, lowering: T.untyped).returns(T.nilable(SplitResult)) } def self.split(body, lowering) # Rewrite pipeline+IO shapes (`readFile(p) |> stage`) into # linear stmts so the standard splitter sees the suspending @@ -192,22 +203,18 @@ def self.split(body, lowering) return nil if contains_unsupported_shape?(body) segments = T.let([], T::Array[Segment]) - current_stmts = T.let([], AST::RawBody) - - flush = lambda do |tail| - segments << Segment.new(segments.length, current_stmts, tail) - current_stmts = [] - end + current_stmts = T.let([], SegmentBody) body.each do |stmt| suspend = classify_suspend(stmt) if suspend - flush.call(suspend) + segments << Segment.new(segments.length, current_stmts, suspend) + current_stmts = [] else current_stmts << stmt end end - flush.call(Done.new(nil)) + segments << Segment.new(segments.length, current_stmts, Done.new(nil)) SplitResult.new(segments: segments) end @@ -222,7 +229,7 @@ def self.split(body, lowering) # 2 loop_pre -- NextSuspend / IoSuspend -> 3 # 3 loop_post -- LoopBack(1) # 4 post -- Done - sig { params(body: AST::RawBody).returns(T.nilable(SplitResult)) } + sig { params(body: SegmentBody).returns(T.nilable(SplitResult)) } def self.split_while_loop_next(body) T.bind(self, T.untyped) rescue nil return nil unless body.is_a?(Array) @@ -241,14 +248,14 @@ def self.split_while_loop_next(body) end return nil if loop_idx.nil? - pre = body[0...loop_idx] || [] - post = body[(loop_idx + 1)..] || [] + pre = segment_slice(body, 0, loop_idx) + post = segment_slice(body, loop_idx + 1, body.length) pre.each { |s| return nil if contains_suspend_anywhere?([s]) } post.each { |s| return nil if contains_suspend_anywhere?([s]) } - loop_node = T.cast(body[loop_idx], T.any(AST::WhileLoop, AST::WhileBindLoop)) - loop_body = loop_node.do_branch.is_a?(Array) ? - loop_node.do_branch : [loop_node.do_branch] + loop_node = T.must(body[loop_idx]) + loop_body = loop_body_for(loop_node) + return nil unless loop_body # Find the single suspend inside the loop body. Accept either a # top-level NEXT (B2-LOOP+NEXT shape) or a top-level IO call @@ -257,27 +264,30 @@ def self.split_while_loop_next(body) # nested suspends. sus_idx = T.let(nil, T.nilable(Integer)) sus_tail = T.let(nil, T.nilable(SegmentTail)) - loop_body.each_with_index do |s, j| + index = 0 + while index < loop_body.length + s = loop_body.fetch(index) sus = classify_suspend(s) - if suspend_tail?(sus) - return nil if sus_idx # multiple suspends in loop body - sus_idx = j - sus_tail = sus - elsif sus.nil? + if sus.nil? # Nested suspend inside an expression -- bail. return nil if contains_suspend_anywhere?([s]) + else + return nil unless sus_idx.nil? # multiple suspends in loop body + sus_idx = index + sus_tail = sus end + index += 1 end return nil if sus_idx.nil? - loop_pre = loop_body[0...sus_idx] - loop_post = loop_body[(sus_idx + 1)..] || [] + loop_pre = segment_slice(loop_body, 0, sus_idx) + loop_post = segment_slice(loop_body, sus_idx + 1, loop_body.length) # Reject if loop_pre/loop_post contain further suspends (Stage 3). loop_pre.each { |s| return nil if contains_suspend_anywhere?([s]) } loop_post.each { |s| return nil if contains_suspend_anywhere?([s]) } - cond_node = loop_node.respond_to?(:condition) ? T.unsafe(loop_node).condition : nil + cond_node = loop_condition_for(loop_node) return nil if cond_node.nil? segments = [ @@ -290,9 +300,47 @@ def self.split_while_loop_next(body) SplitResult.new(segments: segments) end + sig { params(body: SegmentBody, start_index: Integer, end_index: Integer).returns(SegmentBody) } + def self.segment_slice(body, start_index, end_index) + result = T.let([], SegmentBody) + index = start_index + while index < end_index + result << T.must(body[index]) + index += 1 + end + result + end + private_class_method :segment_slice + + sig { params(node: AST::Locatable).returns(T.nilable(SegmentBody)) } + def self.loop_body_for(node) + case node + when AST::WhileLoop + node.do_branch + when AST::WhileBindLoop + node.do_branch + else + nil + end + end + private_class_method :loop_body_for + + sig { params(node: AST::Locatable).returns(T.nilable(AST::Locatable)) } + def self.loop_condition_for(node) + case node + when AST::WhileLoop + T.cast(node.condition, AST::Locatable) + when AST::WhileBindLoop + T.cast(node.condition, AST::Locatable) + else + nil + end + end + private_class_method :loop_condition_for + # Stage 1 punt: anything outside top-level linear stmts + # top-level suspends is not yet handled. - sig { params(body: AST::RawBody).returns(T::Boolean) } + sig { params(body: SegmentBody).returns(T::Boolean) } def self.contains_unsupported_shape?(body) T.bind(self, T.untyped) rescue nil body.any? { |stmt| stmt_unsupported?(stmt) } @@ -309,8 +357,10 @@ def self.stmt_unsupported?(stmt) when AST::WithBlock, AST::CatchBlock true # Stage 3/4 territory. when AST::IfStatement - branches = [stmt.then_branch, stmt.else_branch].compact - branches.any? { |b| contains_suspend_anywhere?(b) } + return true if contains_suspend_anywhere?(stmt.then_branch) + + else_branch = stmt.else_branch + else_branch ? contains_suspend_anywhere?(else_branch) : false else # Top-level linear stmt (assign, var decl, bare expr). # Top-level suspends are handled by the splitter; nested @@ -325,25 +375,44 @@ def self.stmt_unsupported?(stmt) # Recursive scan for any suspend anywhere in a subtree -- # used to reject control-flow constructs that contain # suspends (Stage 1 punts those to the legacy emitters). - sig { params(stmts: SegmentBodyInput).returns(T::Boolean) } - def self.contains_suspend_anywhere?(stmts) - T.bind(self, T.untyped) rescue nil - Array(stmts).any? do |stmt| - case stmt - when AST::WhileLoop, AST::WhileBindLoop - contains_suspend_anywhere?(stmt.do_branch) - when AST::ForRange, AST::ForEach - contains_suspend_anywhere?(stmt.body) - when AST::WithBlock, AST::CatchBlock - true - when AST::IfStatement - contains_suspend_anywhere?(stmt.then_branch) || - contains_suspend_anywhere?(stmt.else_branch || []) - else - !classify_suspend(stmt).nil? + sig { params(stmts: T.nilable(SegmentBody)).returns(T::Boolean) } + def self.contains_suspend_anywhere?(stmts) + T.bind(self, T.untyped) rescue nil + return false if stmts.nil? + + items = stmts + index = 0 + while index < items.length + stmt = items.fetch(index) + case stmt + when AST::WhileLoop + loop_stmt = stmt + return true if contains_suspend_anywhere?(loop_stmt.do_branch) + when AST::WhileBindLoop + loop_stmt = stmt + return true if contains_suspend_anywhere?(loop_stmt.do_branch) + when AST::ForRange + range_stmt = stmt + return true if contains_suspend_anywhere?(range_stmt.body) + when AST::ForEach + each_stmt = stmt + return true if contains_suspend_anywhere?(each_stmt.body) + when AST::WithBlock, AST::CatchBlock + return true + when AST::IfStatement + if_stmt = stmt + return true if contains_suspend_anywhere?(if_stmt.then_branch) + else_branch = if_stmt.else_branch + unless else_branch.nil? + return true if contains_suspend_anywhere?(else_branch) end + else + return true unless classify_suspend(stmt).nil? end + index += 1 end + false + end # Identify the suspend tail (if any) that this top-level stmt # represents. Returns one of IoSuspend / NextSuspend / nil. @@ -369,11 +438,14 @@ def self.classify_suspend(stmt) sig { params(v: T.nilable(AST::Node), name: T.nilable(String)).returns(T.nilable(SegmentTail)) } def self.suspend_for(v, name) T.bind(self, T.untyped) rescue nil - case v + return nil if v.nil? + + value = v + case value when AST::FuncCall, AST::MethodCall - IoSuspend.new(v, v.matched_stdlib_def, name) if io_suspending_call?(v) + IoSuspend.new(value, value.matched_stdlib_def, name) if io_suspending_call?(value) when AST::NextExpr - NextSuspend.new(v.expr, name) + NextSuspend.new(value.expr, name) end end @@ -411,7 +483,7 @@ def self.suspending_call?(expr) # (suspending pipeline as the value of a bind) # - Multi-stage chains where the suspend isn't at the LHS-most # position - sig { params(body: AST::RawBody).returns(AST::RawBody) } + sig { params(body: SegmentBody).returns(SegmentBody) } def self.rewrite_pipeline_io(body) T.bind(self, T.untyped) rescue nil return body unless body.is_a?(Array) diff --git a/compiler/ruby/mir/hoist.rb b/compiler/ruby/mir/hoist.rb index c5ffefe48..64983a628 100644 --- a/compiler/ruby/mir/hoist.rb +++ b/compiler/ruby/mir/hoist.rb @@ -136,7 +136,12 @@ def self.collect_stmt_hoists!(stmt, hoists, counter, schema_lookup, return_type: call.args.each_with_index do |arg, idx| next if arg.is_a?(AST::MoveNode) && arg.value.is_a?(AST::Identifier) next unless allocating?(arg, schema_lookup) - call.args[idx] = make_temp!(arg, hoists, counter.next_name, moved: moved_arg?(arg), schema_lookup: schema_lookup) + replacement = make_temp!(arg, hoists, counter.next_name, moved: moved_arg?(arg), schema_lookup: schema_lookup) + if call.is_a?(AST::FuncCall) + call.args[idx] = replacement + elsif call.is_a?(AST::MethodCall) + call.args[idx] = replacement + end end end @@ -358,6 +363,19 @@ def self.make_temp!(concat, hoists, name, moved: true, expected_type: nil, schem :heap elsif ast_borrow_expr?(concat, moved) || (concat.respond_to?(:container_borrow) && concat.container_borrow) :borrow + elsif moved && ti.needs_explicit_cleanup?(:heap, T.unsafe(schema_lookup)) + # An owned, heap-owning value (a collection/struct that needs cleanup) + # ESCAPES only when it is moved out of this frame: return / yield / element + # or field store, or a consuming (TAKES/GIVE) call argument. There a frame + # allocation would be rewound out from under the escaped value, so it must + # live on the heap. `moved` is the escape signal -- every escaping caller + # of make_temp! passes moved: true (the default). An ordinary BORROWED call + # argument (moved: false, e.g. a concat passed to a print/borrow parameter) + # is consumed within the current frame and stays frame-local; heap- + # promoting it needlessly moved dozens of temporaries off the frame + # allocator. A borrowed access-path source is handled above and keeps + # :borrow; this branch is only for freshly-allocated owned temps. + :heap else (concat.respond_to?(:storage) && concat.storage) || :frame end @@ -461,7 +479,7 @@ module MIRHoistLowering ALLOC_MIR_CLASSES = [ MIR::DupeSlice, MIR::AllocSlice, MIR::MakeList, MIR::CapWrap, MIR::SharePromote, MIR::RcRetain, MIR::RcDowngrade, MIR::WeakUpgrade, - MIR::DeepCopy, MIR::ConcatStr, MIR::ContainerInit, + MIR::DeepCopy, MIR::ConcatStr, MIR::ContainerInit, MIR::MonomorphicKeep, ].freeze sig { returns(T::Array[MIR::Stmt]) } @@ -528,7 +546,11 @@ def with_pending(pending, node) def descend(parent, field) T.bind(self, MIRLowering) rescue nil - child = parent.public_send(field) + child = case field + when :left then parent.left + when :right then parent.right + else raise "unknown BinaryOp field #{field}" + end if parent.respond_to?(:lazy_fields) && parent.lazy_fields.include?(field) lower_scoped do lower(child) @@ -768,6 +790,10 @@ def mir_alloc_mark_type_info(mir, ast_node = nil, context: "MIR allocation") Type.new(mir.zig_base.to_s, ownership: :shared, location: :heap) when MIR::RcRetain, MIR::RcDowngrade, MIR::FreezeExpr Type.new(mir.zig_base.to_s, ownership: :multiowned, location: :heap) + when MIR::MonomorphicKeep + # Carrier resolved per monomorphization; the retained arm is the owning + # shape the AllocMark tracks (the plain arm needs no drop). + Type.new(mir.zig_base.to_s, ownership: :multiowned, location: :heap) when MIR::Cast, MIR::TryExpr, MIR::TryOptional, MIR::OptionalUnwrap mir_alloc_mark_type_info(mir.expr, nil, context: context) when MIR::Call, MIR::MethodCall, MIR::TailCall @@ -1011,15 +1037,82 @@ def normalize_stmt_child_exprs!(stmt) sig { params(stmt: MIR::Node, attr: Symbol, transfer_on_success: T::Boolean).returns(T::Array[MIR::Node]) } def normalize_used_expr_attr!(stmt, attr, transfer_on_success: false) - value = stmt.public_send(attr) + value = normalized_expr_attr(stmt, attr) return [] unless value prefix, normalized = normalize_allocating_used_expr(value, transfer_on_success: transfer_on_success) - setter = :"#{attr}=" - stmt.public_send(setter, normalized) + set_normalized_expr_attr!(stmt, attr, normalized) prefix end + sig { params(stmt: MIR::Node, attr: Symbol).returns(T.nilable(MIR::Node)) } + def normalized_expr_attr(stmt, attr) + case attr + when :target + T.cast(stmt, MIR::Set).target + when :value + case stmt + when MIR::ReassignWithCleanup then stmt.value + when MIR::ReturnStmt then stmt.value + when MIR::BreakStmt then stmt.value + end + when :expr + T.cast(stmt, MIR::ExprStmt).expr + when :cond + stmt.is_a?(MIR::IfStmt) ? stmt.cond : T.cast(stmt, MIR::WhileStmt).cond + when :update + T.cast(stmt, MIR::WhileStmt).update + when :iter + T.cast(stmt, MIR::ForStmt).iter + when :subject + stmt.is_a?(MIR::SwitchStmt) ? stmt.subject : T.cast(stmt, MIR::UnionMatchStmt).subject + when :item_expr + T.cast(stmt, MIR::BatchWindowPush).item_expr + when :value_expr + stmt.is_a?(MIR::BatchWindowPush) ? stmt.value_expr : T.cast(stmt, MIR::BatchWindowFlush).value_expr + end + end + + sig { params(stmt: MIR::Node, attr: Symbol, value: MIR::Node).void } + def set_normalized_expr_attr!(stmt, attr, value) + case attr + when :target + T.cast(stmt, MIR::Set).target = value + when :value + case stmt + when MIR::ReassignWithCleanup then stmt.value = value + when MIR::ReturnStmt then stmt.value = value + when MIR::BreakStmt then stmt.value = value + end + when :expr + T.cast(stmt, MIR::ExprStmt).expr = value + when :cond + if stmt.is_a?(MIR::IfStmt) + stmt.cond = value + else + T.cast(stmt, MIR::WhileStmt).cond = value + end + when :update + T.cast(stmt, MIR::WhileStmt).update = value + when :iter + T.cast(stmt, MIR::ForStmt).iter = value + when :subject + if stmt.is_a?(MIR::SwitchStmt) + stmt.subject = value + else + T.cast(stmt, MIR::UnionMatchStmt).subject = value + end + when :item_expr + T.cast(stmt, MIR::BatchWindowPush).item_expr = value + when :value_expr + if stmt.is_a?(MIR::BatchWindowPush) + stmt.value_expr = value + else + T.cast(stmt, MIR::BatchWindowFlush).value_expr = value + end + end + end + sig { params(stmt: MIR::IfBindStmt, name: String).returns(T::Boolean) } def if_bind_transfer_present?(stmt, name) [stmt.then_body, stmt.else_body].compact.any? do |body| @@ -1195,26 +1288,38 @@ def replace_mir_expr_child!(parent, old_child, new_child) def replace_mir_expr_in_value!(value, old_child, new_child) case value when Array + replaced = T.let(false, T::Boolean) value.each_with_index do |item, idx| if item.equal?(old_child) value[idx] = new_child - return true + replaced = true + break end if item.is_a?(Array) || item.is_a?(Hash) - return true if replace_mir_expr_in_value!(item, old_child, new_child) + if replace_mir_expr_in_value!(item, old_child, new_child) + replaced = true + break + end end end + return replaced when Hash + replaced = T.let(false, T::Boolean) value.each_key do |key| item = value[key] if item.equal?(old_child) value[key] = new_child - return true + replaced = true + break end if item.is_a?(Array) || item.is_a?(Hash) - return true if replace_mir_expr_in_value!(item, old_child, new_child) + if replace_mir_expr_in_value!(item, old_child, new_child) + replaced = true + break + end end end + return replaced end false end @@ -1369,7 +1474,7 @@ def hoist_cleanup_entry(mir, ast_node) end when MIR::SharePromote rc_cleanup_entry(ast_node, source: "MIR::SharePromote", mir: mir) - when MIR::RcRetain, MIR::RcDowngrade, MIR::WeakUpgrade + when MIR::RcRetain, MIR::RcDowngrade, MIR::WeakUpgrade, MIR::MonomorphicKeep cleanup_entry_for_owned_result(ast_node, alloc: alloc) || CleanupEntry.build(:rc, alloc: alloc, has_moved_guard: false) when MIR::FreezeExpr CleanupEntry.build(:frozen, alloc: :heap, has_moved_guard: false, fixed_alloc: true) diff --git a/compiler/ruby/mir/lower/pipeline/pipeline_batch_window_lowerer.rb b/compiler/ruby/mir/lower/pipeline/pipeline_batch_window_lowerer.rb index a08725a51..3e813eda3 100644 --- a/compiler/ruby/mir/lower/pipeline/pipeline_batch_window_lowerer.rb +++ b/compiler/ruby/mir/lower/pipeline/pipeline_batch_window_lowerer.rb @@ -6,6 +6,7 @@ require_relative "../../../ast/ast" require_relative "../../../ast/type" require_relative "../../mir" +require_relative "./pipeline_records" PipelineBatchWindowTypeInput = T.type_alias { T.any(Type, Symbol, String) } @@ -26,7 +27,13 @@ class PipelineBatchWindowPlan < T::Struct const :element_zig, String const :result_zig, String const :size_mir, MIR::Node - const :expr_mir, MIR::Node + # Fresh per-use lowering of the per-window expression: its pending hoists + # (an owned call, a nested pipeline) must be emitted INSIDE each batch + # scope that declares the placeholder — a shared pre-lowered tree flushed + # them to the enclosing statement (undeclared __bw_batch Zig). Each call + # produces fresh temp ids so the linear ownership checker (name-keyed) + # sees distinct allocations at the push and flush sites. + const :make_expr, T.proc.returns(MIR::Node) const :alloc, Symbol const :placeholder_var, String const :timeout_ns, String @@ -38,6 +45,7 @@ class PipelineBatchWindowLowerer < T::Struct const :bc_target, T.proc.returns(T::Boolean) const :visit_mir, T.proc.params(node: AST::Node).returns(MIR::Node) const :visit_mir_with_placeholder, T.proc.params(node: AST::Node, placeholder: String).returns(MIR::Node) + const :visit_expr_head, T.proc.params(expr_node: AST::Node, placeholder: String).returns(PipelineElementHead) const :pipeline_block, T.proc.params(list_node: AST::Node, blk: T.proc.params(items: String, label: String).returns(T::Array[MIR::Emittable])).returns(MIR::BlockExpr) const :next_label, T.proc.returns(String) const :set_current_label, T.proc.params(label: String).void @@ -70,7 +78,7 @@ def batch_window_plan(list_node, smooth_node, bw_node) element_zig: self.transpile_type.call(batch_element_type(lhs_type).to_s), result_zig: self.transpile_type.call(bw_node.expression.full_type!.to_s), size_mir: batch_size_mir(bw_node), - expr_mir: self.visit_mir_with_placeholder.call(bw_node.expression, placeholder_var), + make_expr: batch_expr_builder(bw_node, placeholder_var), alloc: self.pipeline_alloc.call(smooth_node), placeholder_var: placeholder_var, timeout_ns: batch_window_timeout_ns(bw_node), @@ -132,6 +140,23 @@ def batch_element_type(lhs_type) lhs_type.runtime_stream_storage_element_type || T.must(lhs_type.element_type) end + sig { params(bw_node: AST::BatchWindowOp, placeholder_var: String).returns(T.proc.returns(MIR::Node)) } + def batch_expr_builder(bw_node, placeholder_var) + lambda do + head = self.visit_expr_head.call(bw_node.expression, placeholder_var) + next head.value if head.pending.empty? && !head.owned + + # The batch placeholder only exists inside the emitter's batch block — + # fence the expression as a lazy-boundary value block so the post-pass + # allocation normalizer materializes owned work INSIDE it instead of + # lifting a statement above the scope (undeclared __bw_batch Zig). + label = self.next_label.call + block = MIR::BlockExpr.new(label, [*head.pending, MIR::BreakStmt.new(label, head.value)]) + block.lazy_boundary = true + block + end + end + sig { params(bw_node: AST::BatchWindowOp).returns(MIR::Node) } def batch_size_mir(bw_node) size = bw_node.options["size"] @@ -314,7 +339,7 @@ def bc_materialized_loop_body(plan, source, append_uses_allocator:) false, nil, nil), - MIR::Let.new("__bw_val", plan.expr_mir, false, nil, nil), + MIR::Let.new("__bw_val", plan.make_expr.call, false, nil, nil), bc_append_value_stmt(plan.alloc, append_uses_allocator: append_uses_allocator), MIR::Set.new(MIR::Ident.new("__bw_offset"), MIR::Ident.new("__bw_end")), ] @@ -374,7 +399,7 @@ def batch_window_push_stmt(plan, item_var) plan.placeholder_var, plan.element_zig, "res_list", - plan.expr_mir, + plan.make_expr.call, plan.alloc, ) end @@ -386,7 +411,7 @@ def batch_window_flush_stmt(plan) plan.placeholder_var, plan.element_zig, "res_list", - plan.expr_mir, + plan.make_expr.call, plan.alloc, ) end diff --git a/compiler/ruby/mir/lower/pipeline/pipeline_binding_chain_lowerer.rb b/compiler/ruby/mir/lower/pipeline/pipeline_binding_chain_lowerer.rb index 4f0c304a3..44cdfa78b 100644 --- a/compiler/ruby/mir/lower/pipeline/pipeline_binding_chain_lowerer.rb +++ b/compiler/ruby/mir/lower/pipeline/pipeline_binding_chain_lowerer.rb @@ -66,7 +66,17 @@ def unwrap_chain(node) return nil unless node.smooth? fold = node.right - return nil unless AST.pipeline_range_fold?(fold) + unless AST.pipeline_range_fold?(fold) + # A chain that binds names (AS $v) but ends in a materializing op has + # no supported lowering. Say so — falling through leaves the $v idents + # unresolved and produces a misleading "Undefined pipeline binding". + if binding_chain_shape?(node) + raise "Materializing terminals are not supported in AS $v binding chains. " \ + "End the chain with a fold terminal (COUNT/SUM/MIN/MAX/AVERAGE/ANY/ALL/FIND), " \ + "or restructure as nested pipelines (e.g. UNNEST _.field |> SELECT ...)." + end + return nil + end cursor = T.let(node.left, AST::Node) stages = T.let([], T::Array[AST::Node]) @@ -298,6 +308,31 @@ def fold_plan(init, body, post_inner, result) ) end + # The UNNEST binding-chain shape unwrap_chain handles, minus the fold + # terminal: a BIND_VAR source AND an UNNEST stage in the left spine. A bare + # `xs AS $u |> SELECT:concurrent ...` (named CONCURRENT binding) is a + # different, SUPPORTED shape and must fall through to its own lowerer. + sig { params(node: AST::BinaryOp).returns(T::Boolean) } + def binding_chain_shape?(node) + has_bind_source = T.let(false, T::Boolean) + has_unnest_stage = T.let(false, T::Boolean) + cursor = T.let(node.left, AST::Node) + while cursor.is_a?(AST::BinaryOp) + has_bind_source = true if cursor.op == :BIND_VAR + + rhs = cursor.right + if rhs.is_a?(AST::UnnestOp) + has_unnest_stage = true + unnest_expr = rhs.expression + has_bind_source = true if unnest_expr.is_a?(AST::BinaryOp) && unnest_expr.op == :BIND_VAR + end + break unless cursor.smooth? + + cursor = cursor.left + end + has_bind_source && has_unnest_stage + end + sig { params(stages: T::Array[AST::Node], placeholder: String, accum_stmts: T::Array[MIR::Emittable]).returns(T::Array[MIR::Emittable]) } def wrap_stages(stages, placeholder, accum_stmts) body = T.let(accum_stmts, T::Array[MIR::Emittable]) diff --git a/compiler/ruby/mir/lower/pipeline/pipeline_concurrent_lowerer.rb b/compiler/ruby/mir/lower/pipeline/pipeline_concurrent_lowerer.rb index ffc228287..864211b08 100644 --- a/compiler/ruby/mir/lower/pipeline/pipeline_concurrent_lowerer.rb +++ b/compiler/ruby/mir/lower/pipeline/pipeline_concurrent_lowerer.rb @@ -1236,9 +1236,7 @@ def shard_context(conc_op) sig { params(map_type: Type).returns(Type) } def shard_key_type(map_type) - return map_type.key_type if map_type.numeric_map? - - Type.new(:String) + map_type.key_type end sig { params(map_var_name: String, caps: FiberCtxBuilder::Result).returns(T::Hash[String, String]) } diff --git a/compiler/ruby/mir/lower/pipeline/pipeline_context.rb b/compiler/ruby/mir/lower/pipeline/pipeline_context.rb index e78c42552..e725c00be 100644 --- a/compiler/ruby/mir/lower/pipeline/pipeline_context.rb +++ b/compiler/ruby/mir/lower/pipeline/pipeline_context.rb @@ -89,8 +89,13 @@ def with_soa_rewrite(each_mode, fields) sig { returns(T::Boolean) } def active? - !!(placeholder_name || acc_placeholder || join_param_map || - soa_each_mode || soa_rewrite_active || !named_bindings.empty?) + return true unless placeholder_name.nil? + return true unless acc_placeholder.nil? + return true unless join_param_map.nil? + return true if soa_each_mode + return true if soa_rewrite_active + + !named_bindings.empty? end sig { params(name: String).returns(T.nilable(String)) } @@ -98,22 +103,42 @@ def replacement_for_identifier(name) return placeholder_name if name == "_" && placeholder_name return acc_placeholder if name == "acc" && acc_placeholder - joined = join_param_map&.[](name) - return joined if joined + map = join_param_map + if map + joined = map[name] + return joined if joined + end named_bindings[name] end sig { params(node: AST::Node).returns(T::Boolean) } def soa_each_field_node?(node) - soa_each_mode && node.is_a?(AST::GetField) && - node.target.is_a?(AST::Identifier) && node.target.name == "_" + return false unless soa_each_mode + + case node + when AST::GetField + placeholder_identifier?(node.target) + else + false + end end sig { params(node: AST::GetField).returns(T::Boolean) } def soa_rewrite_field_node?(node) - soa_rewrite_active && node.target.is_a?(AST::Identifier) && node.target.name == "_" + soa_rewrite_active && placeholder_identifier?(node.target) + end + + sig { params(node: AST::Locatable).returns(T::Boolean) } + def placeholder_identifier?(node) + case node + when AST::Identifier + node.name == "_" + else + false + end end + private :placeholder_identifier? sig { params(field: T.any(Symbol, String)).void } def record_soa_field(field) @@ -144,9 +169,12 @@ def substitute(node) when AST::BindExpr then substitute_bind_expr(node) when AST::Assignment then substitute_assignment(node) when AST::UnaryOp then substitute_unary_op(node) + when AST::CopyNode, AST::MoveNode, AST::KeepNode, AST::ShareNode + substitute_value_wrapper(node) when AST::WithBlock then substitute_with_block(node) when AST::StructLit then substitute_struct_lit(node) when AST::HashLit then substitute_hash_lit(node) + when AST::ListLit then substitute_list_lit(node) when AST::BlockExpr then substitute_block_expr(node) when AST::Assert then substitute_assert(node) when AST::IfStatement then substitute_if_statement(node) @@ -210,7 +238,7 @@ def substitute_binary_op(node) new_bin = AST::BinaryOp.new(node.token, new_left, node.op, new_right) copy_type_info(node, new_bin) - new_bin.string_concat = node.string_concat if node.respond_to?(:string_concat) && node.string_concat + new_bin.string_concat = true if node.string_concat == true new_bin end @@ -219,9 +247,9 @@ def substitute_get_field(node) if @context.soa_rewrite_field_node?(node) @context.record_soa_field(node.field) soa_field = AST::Identifier.new(node.token, "__soa_#{node.field}") - AST.stamp_synthetic_type!(soa_field, soa_field_slice_type(node), context: "synthetic AST type") + soa_field.type_object = soa_field_slice_type(node) soa_idx = AST::Identifier.new(node.token, "__soa_i") - AST.stamp_synthetic_type!(soa_idx, :Int64, context: "synthetic AST type") + soa_idx.type_object = Type.new(:Int64) new_gi = AST::GetIndex.new(node.token, soa_field, soa_idx) copy_type_info(node, new_gi) return new_gi @@ -276,9 +304,52 @@ def substitute_assignment(node) new_assign end - sig { params(node: T.any(AST::Node, String)).returns(T.any(AST::Node, String)) } + sig { params(node: AST::AssignmentName).returns(AST::AssignmentName) } def substitute_assignment_target(node) - node.is_a?(AST::GetField) || node.is_a?(AST::GetIndex) ? substitute(node) : node + if node.is_a?(AST::GetField) + rewritten = substitute(node) + return T.cast(rewritten, AST::AssignmentName) + end + if node.is_a?(AST::GetIndex) + rewritten = substitute(node) + return T.cast(rewritten, AST::AssignmentName) + end + + node + end + + # COPY/GIVE/CLONE/SHARE wrappers: substitute inside the wrapped value. + sig { params(node: T.any(AST::CopyNode, AST::MoveNode, AST::KeepNode, AST::ShareNode)).returns(AST::Node) } + def substitute_value_wrapper(node) + value = T.let(nil, T.nilable(AST::Locatable)) + case node + when AST::CopyNode + value = node.value + when AST::MoveNode + value = node.value + when AST::KeepNode + value = node.value + when AST::ShareNode + value = node.value + end + value = T.must(value) + new_value = substitute(value) + return node if new_value == value + + new_node = T.let(nil, T.nilable(AST::PipelineRewriteNode)) + case node + when AST::CopyNode + new_node = AST::CopyNode.new(node.token, new_value) + when AST::MoveNode + new_node = AST::MoveNode.new(node.token, new_value) + when AST::KeepNode + new_node = AST::KeepNode.new(node.token, new_value) + when AST::ShareNode + new_node = AST::ShareNode.new(node.token, new_value) + end + new_node = new_node + copy_type_info(node, new_node) + new_node end sig { params(node: AST::UnaryOp).returns(AST::Node) } @@ -293,19 +364,31 @@ def substitute_unary_op(node) sig { params(node: AST::WithBlock).returns(AST::Node) } def substitute_with_block(node) - new_body = node.body.map { |stmt| substitute(stmt) } - new_arms = node.arms&.map do |arm| - new_arm_body = arm.body.map { |stmt| substitute(stmt) } - if new_arm_body == arm.body - arm - else - AST::WithMatchArm.new( - family: arm.family, - body: new_arm_body, - lock_error_clauses: arm.lock_error_clauses, - token: arm.token, - ) + new_body = substitute_body(node.body) + arms = node.arms + new_arms = T.let(nil, T.nilable(T::Array[AST::WithMatchArm])) + if arms + rewritten_arms = T.let([], T::Array[AST::WithMatchArm]) + arm_index = 0 + while arm_index < arms.length + arm = arms.fetch(arm_index) + arm_body = T.let(arm.body_nodes, T::Array[AST::Node]) + new_arm_body = substitute_body(arm_body) + rewritten_arm = + if new_arm_body == arm_body + arm + else + AST::WithMatchArm.new( + family: arm.family_value, + body: new_arm_body, + lock_error_clauses: arm.lock_error_clauses_value, + token: arm.token_value, + ) + end + rewritten_arms << rewritten_arm + arm_index += 1 end + new_arms = rewritten_arms end return node if new_body == node.body && new_arms == node.arms @@ -322,20 +405,57 @@ def substitute_with_block(node) new_with end + sig { params(body: T::Array[AST::Node]).returns(T::Array[AST::Node]) } + def substitute_body(body) + result = T.let([], T::Array[AST::Node]) + index = 0 + while index < body.length + result << substitute(body.fetch(index)) + index += 1 + end + result + end + sig { params(node: AST::StructLit).returns(AST::Node) } def substitute_struct_lit(node) - new_fields = node.fields.transform_values { |value| substitute(value) } - return node if new_fields == node.fields + fields = T.let(node.fields, T::Hash[String, AST::Node]) + new_fields = T.let({}, T::Hash[String, AST::Node]) + keys = fields.keys + index = 0 + while index < keys.length + key = keys.fetch(index) + new_fields[key] = substitute(fields.fetch(key)) + index += 1 + end + return node if new_fields == fields new_sl = AST::StructLit.new(node.token, node.name, new_fields, node.storage, node.type_args) copy_type_info(node, new_sl) new_sl end + sig { params(node: AST::ListLit).returns(AST::Node) } + def substitute_list_lit(node) + new_items = node.items.map { |item| substitute(item) } + return node if new_items == node.items + + new_ll = AST::ListLit.new(node.token, new_items, node.storage, node.constructor_options) + copy_type_info(node, new_ll) + new_ll + end + sig { params(node: AST::HashLit).returns(AST::Node) } def substitute_hash_lit(node) - new_pairs = node.pairs.transform_values { |value| substitute(value) } - return node if new_pairs == node.pairs + pairs = T.let(node.pairs, T::Hash[AST::Node, AST::Node]) + new_pairs = T.let({}, T::Hash[AST::Node, AST::Node]) + keys = pairs.keys + index = 0 + while index < keys.length + key = keys.fetch(index) + new_pairs[key] = substitute(pairs.fetch(key)) + index += 1 + end + return node if new_pairs == pairs new_hl = AST::HashLit.new(node.token, new_pairs, node.storage) copy_type_info(node, new_hl) @@ -344,7 +464,13 @@ def substitute_hash_lit(node) sig { params(node: AST::BlockExpr).returns(AST::Node) } def substitute_block_expr(node) - new_body = node.body.map { |stmt| substitute(stmt) } + body = T.let(node.body, T::Array[AST::Node]) + new_body = T.let([], T::Array[AST::Node]) + index = 0 + while index < body.length + new_body << substitute(body.fetch(index)) + index += 1 + end result = T.let(node.result, T.nilable(AST::Node)) new_result = result ? substitute(result) : nil return node if new_body == node.body && new_result == node.result @@ -379,19 +505,22 @@ def substitute_if_statement(node) new_if end - sig { params(src: AST::Locatable, dst: AST::Locatable).void } + sig { params(src: AST::PipelineRewriteNode, dst: AST::PipelineRewriteNode).void } def copy_type_info(src, dst) - AST.copy_pipeline_rewrite_metadata!(src, dst) + AST.copy_pipeline_rewrite_metadata!(dst, src) end - sig { params(src: AST::Locatable, dst: AST::Locatable).void } + sig { params(src: AST::PipelineRewriteNode, dst: AST::PipelineRewriteNode).void } def copy_call_metadata(src, dst) - AST.copy_pipeline_rewrite_metadata!(src, dst, include_call_metadata: true) + AST.copy_pipeline_rewrite_metadata!(dst, src, include_call_metadata: true) end sig { params(field_node: AST::GetField).returns(Type) } def soa_field_slice_type(field_node) - field_type = field_node.full_type!(context: "SOA field slice") - Type.new(:"#{field_type.resolved}[]") + field_type = field_node.type_object + raise "SOA field slice: missing annotated type" unless field_type + concrete_type = field_type + raise "SOA field slice: unresolved annotated type" if concrete_type.untyped? + Type.new(:"#{concrete_type.resolved}[]") end end diff --git a/compiler/ruby/mir/lower/pipeline/pipeline_each_lowerer.rb b/compiler/ruby/mir/lower/pipeline/pipeline_each_lowerer.rb index f56fb2073..d6056e590 100644 --- a/compiler/ruby/mir/lower/pipeline/pipeline_each_lowerer.rb +++ b/compiler/ruby/mir/lower/pipeline/pipeline_each_lowerer.rb @@ -48,6 +48,7 @@ class PipelineEachLowerer < T::Struct const :lower_sharded_each, T.proc.params(list_node: AST::Node, each_op: AST::EachOp).returns(MIR::ScopeBlock) const :ast_stmts_use_placeholder, T.proc.params(body_stmts: T::Array[AST::Node]).returns(T::Boolean) const :next_index_name, T.proc.returns(String) + const :source_alloc_fact, T.proc.params(value: MIR::Node, name: String, type_info: Type).returns(T.nilable([MIR::AllocMark, CleanupEntry])) sig { params(list_node: AST::Node, each_op: AST::EachOp).returns(PipelineEachResult) } def lower(list_node, each_op) @@ -109,6 +110,11 @@ def each_source_kind(list_node, lhs_type, range_chain, bc_target) return PipelineEachSourceKind::List if lhs_type.list_collection? || lhs_type.fixed_soa? return PipelineEachSourceKind::Set if lhs_type.set_collection? return PipelineEachSourceKind::RangeLiteral if list_node.is_a?(AST::RangeLit) + # Bare arrays LAST: the rewriter fuses most array EACHes, but a source it + # must bypass (an ORDER_BY chain) arrives here typed as a plain array — + # materialize it and iterate like any list. After the capability checks so + # set/pool/sharded shapes keep their dedicated routes. + return PipelineEachSourceKind::List if lhs_type.array? PipelineEachSourceKind::Unsupported end @@ -223,12 +229,18 @@ def lower_list_each(list_node, each_op, bc_target:) return MIR::ScopeBlock.new([lower_bc_indexed_each(list_node, list_body_mir, skip_nil: false)]) end - MIR::ScopeBlock.new([ - MIR::Let.new("__each_src", source_mir, false, nil, nil), - MIR::Let.new("__each_items", - MIR::ItemsAccess.new(MIR::Ident.new("__each_src"), true), false, nil, nil), - MIR::ForStmt.new(MIR::Ident.new("__each_items"), "__each_item", list_body_mir, nil), - ]) + # An OWNED materialized source (an ORDER_BY chain the rewriter must + # bypass) is a first-class synthetic binding: AllocMark + Cleanup, so the + # checker can verify it and the list is freed after the loop. + stmts = T.let([], T::Array[MIR::Emittable]) + fact = self.source_alloc_fact.call(source_mir, "__each_src", list_node.full_type!) + stmts << fact[0] if fact + stmts << MIR::Let.new("__each_src", source_mir, false, nil, nil) + stmts << MIR::Cleanup.new("__each_src", fact[1]) if fact + stmts << MIR::Let.new("__each_items", + MIR::ItemsAccess.new(MIR::Ident.new("__each_src"), true), false, nil, nil) + stmts << MIR::ForStmt.new(MIR::Ident.new("__each_items"), "__each_item", list_body_mir, nil) + MIR::ScopeBlock.new(stmts) end sig { params(list_node: AST::Node, each_op: AST::EachOp).returns(MIR::ScopeBlock) } diff --git a/compiler/ruby/mir/lower/pipeline/pipeline_host.rb b/compiler/ruby/mir/lower/pipeline/pipeline_host.rb index 4fc442351..b45c84f58 100644 --- a/compiler/ruby/mir/lower/pipeline/pipeline_host.rb +++ b/compiler/ruby/mir/lower/pipeline/pipeline_host.rb @@ -93,6 +93,14 @@ def build_list_lowerer visit_expr: ->(_list_node, expr_node, placeholder) { with_pipeline_context(placeholder: placeholder) { visit_mir(expr_node) } }, + visit_expr_head: ->(expr_node, placeholder, alloc) { + # Owned field copies inside the element (COPY dup(_) in a composite) + # must land on the RESULT list's allocator: a frame copy moved into a + # heap-listed row dangles once the per-iteration rewind fires. + with_pipeline_context(placeholder: placeholder) do + @lowering_bridge.with_pipeline_decl_alloc(alloc) { visit_mir_head(expr_node) } + end + }, visit_reduce_expr: ->(expr_node, item_placeholder, acc_placeholder) { with_pipeline_context(placeholder: item_placeholder, acc: acc_placeholder) { visit_mir(expr_node) } }, @@ -114,6 +122,9 @@ def build_list_lowerer append_owned_value_stmt: ->(receiver, alloc, value_expr) { append_owned_value_stmt(receiver, alloc, value_expr) }, + append_fresh_owned_value_stmt: ->(receiver, alloc, value_expr, owned_type) { + append_owned_value_stmt(receiver, alloc, value_expr, known_owned: true, owned_type: owned_type) + }, borrowed_pipeline_value: ->(value, type_info, alloc) { borrowed_pipeline_value(value, type_info, alloc) }, @@ -121,6 +132,7 @@ def build_list_lowerer owning_pipeline_temp_stmts: ->(name, source, type_info, zig_type, alloc) { owning_pipeline_temp_stmts(name, source, type_info, zig_type, alloc) }, + loop_mark_stmts: -> { @lowering_bridge.pipeline_iteration_loop_marks }, ) end @@ -179,6 +191,14 @@ def build_each_lowerer lower_each_range: ->(source_node, stages, each_op) { lower_each_range(source_node, stages, each_op) }, lower_sharded_each: ->(list_node, each_op) { lower_sharded_each(list_node, each_op) }, ast_stmts_use_placeholder: ->(body_stmts) { ast_stmts_use_placeholder?(body_stmts) }, + source_alloc_fact: ->(value, name, type_info) { + fact = @lowering_bridge.pipeline_alloc_mark_fact( + value, name, fallback_alloc: :heap, type_info: type_info, + context: "each source materialization") + next nil unless fact + entry = CleanupEntry.build(:uniform, alloc: fact.alloc, has_moved_guard: false, zig_type: type_info.zig_type) + [fact.mark, entry] + }, next_index_name: -> { @each_idx_counter += 1 "__each_i_#{@each_idx_counter}" @@ -204,8 +224,8 @@ def build_set_index_lowerer typed_block_expr(label, T.cast(body, T::Array[MIR::Node]), result_type) }, range_chain: ->(node) { unwrap_range_chain(node) }, - lazy_range_prefix: ->(source_node, stages, on_skip) { - build_lazy_range_prefix(source_node, stages, on_skip: on_skip) + lazy_range_prefix: ->(source_node, stages, on_skip, track_owned_items) { + build_lazy_range_prefix(source_node, stages, on_skip: on_skip, track_owned_items: track_owned_items) }, range_fold_observable_distinct: ->(prefix, distinct_op, smooth_node, label, source_node) { lower_range_fold_observable_distinct(prefix, distinct_op, smooth_node, label, source_node) @@ -241,6 +261,9 @@ def build_batch_window_lowerer visit_mir_with_placeholder: ->(node, placeholder) { with_pipeline_context(placeholder: placeholder) { visit_mir(node) } }, + visit_expr_head: ->(expr_node, placeholder) { + with_pipeline_context(placeholder: placeholder) { visit_mir_head(expr_node) } + }, pipeline_block: ->(list_node, blk) { pipeline_block(list_node) { |items, label| blk.call(items, label) } }, @@ -262,6 +285,14 @@ def build_range_lowerer_host visit_mir_with_context: ->(node, placeholder, acc) { with_pipeline_context(placeholder: placeholder, acc: acc) { visit_mir(node) } }, + lower_head_with_context: ->(node, placeholder) { + @lowering_bridge.lower_head do + with_pipeline_context(placeholder: placeholder) { visit_mir(node) } + end + }, + with_fiber_rt: ->(rt_name, blk) { + with_fiber_capture_map({}, rt_override: rt_name) { blk.call } + }, visit_pipeline_body_mir: ->(body_stmts, placeholder) { visit_pipeline_body_mir(body_stmts, placeholder: placeholder) }, @@ -595,6 +626,25 @@ def visit_mir(node) @lowering_bridge.lower_node(substituted) end + # Lower a per-element expression, capturing the hoisted temps it produces + # (owned struct fields, nested owned calls, ...) so the caller can emit them + # INSIDE the pipeline loop body. Without this the hoists flush to the + # enclosing statement -- outside the loop -- and a per-iteration allocation + # looks to the checker like a once-allocated binding consumed each iteration. + sig { params(node: AST::Node).returns(PipelineElementHead) } + def visit_mir_head(node) + substituted = substitute_placeholders(node) + head = @lowering_bridge.lower_head { @lowering_bridge.lower_node(substituted) } + # Capture ownership from MIR facts NOW, while both the value's effect and its + # captured hoists are visible. After hoisting, the value may be a plain Ident + # whose owned parts moved into `pending` (an owned struct field, a nested + # owned call), so the value's effect alone under-reports ownership. An owned + # allocation among the hoists means the element constructed something owned. + owned = MIR::OwnershipEffect.of(head.value).produces_owned || + head.pending.any? { |stmt| stmt.is_a?(MIR::AllocMark) } + PipelineElementHead.new(value: head.value, pending: head.pending, owned: owned) + end + # Lower an array of AST body statements to MIR nodes, with pipeline # placeholder substitution. Used by side-effect operators (Tap, Each, Join) # whose loop bodies contain multiple statements. @@ -630,6 +680,13 @@ def substitute_placeholders(node) public + # Public placement read for lower_complex_smooth's sink decision: the + # pipeline domain owns the annotator's storage stamp (INV-16). + sig { params(smooth_node: AST::BinaryOp).returns(T::Boolean) } + def pipeline_result_heap?(smooth_node) + pipeline_alloc(smooth_node) == :heap + end + # MIR entry point: returns MIR node tree for migrated pipeline operators. # Returns nil for non-migrated operators (caller falls back to string path). sig { params(node: AST::BinaryOp).returns(PipelineLoweringResult) } @@ -770,12 +827,20 @@ def lower_stream_select(site, op) caps.specs.map { |spec| MIR::StructInitField.new(name: spec.name, value: spec.init_value_mir) } selector_prefix = T.let([], T::Array[MIR::Emittable]) - selector = with_pipeline_context(placeholder: "__select_item") do - with_fiber_capture_map(caps.capture_map, - capture_symbols: caps.capture_symbols, rt_override: "__rt") do - visit_mir(op.expression) + # Lower the selector with its own pending-statement scope: allocating + # sub-expressions (COPY temps, TAKES dupes) must materialize inside the + # consumer loop, where the per-item capture is in scope, not at the + # enclosing statement. + selector_head = @lowering_bridge.lower_head do + with_pipeline_context(placeholder: "__select_item") do + with_fiber_capture_map(caps.capture_map, + capture_symbols: caps.capture_symbols, rt_override: "__rt") do + visit_mir(op.expression) + end end end + selector = T.let(selector_head.value, MIR::Node) + selector_prefix.concat(selector_head.pending) tense_plan = select_tense_plan(op) if tense_plan.asynchronous? promise_name = "__select_promise#{id}" @@ -826,45 +891,106 @@ def lower_stream_select(site, op) end end - source_ref = MIR::FieldGet.new(MIR::Ident.new("ctx"), "source") - loop = if source_type.array? - # List sources are borrowed; elements stay owned by the list. - push = MIR::ExprStmt.new(MIR::MethodCall.new( + # `push` TRANSFERS the value into the channel: the consumer owns every + # dequeued item and frees it (the WHILE-NEXT path's guarded defer). The + # producer must therefore push an OWNED value and never free it itself — + # a `no_ownership` push contract here once let the hoisted owned selector + # temp keep its borrowed-arg (unguarded) cleanup, and both sides freed the + # same memory: an accepted program that segfaulted (invalid free). + # - fresh-owned selector (a call like dup(_), concat, composite): move + # it into the push under a moved-guard; + # - identity selector: the dequeued item's ownership goes into the push + # (existing behavior — the item guard below is skipped); + # - borrowing projection of a heap-owning type: push a deep copy, so + # the consumer never frees memory the producer still owns. + identity_selector = PipelinePlaceholderUsage.identity_placeholder?(op.expression) + # A fresh composite construction (Box{ name: GIVE _ } / tuple / array) is + # owned BY CONSTRUCTION even though nothing in it allocates: its fields + # took ownership of their values (a GIVE-moved item, a COPY dupe). Deep- + # copying it would strand the moved-in originals (leak); it must move. + fresh_composite = (selector.is_a?(MIR::StructInit) || + selector.is_a?(MIR::TupleLiteral) || selector.is_a?(MIR::ArrayInit)) && + item_type.recursive_cleanup_shape?(T.unsafe(pipeline_schema_lookup)) + selector_owned = MIR::OwnershipEffect.of(selector).produces_owned || + selector_prefix.any? { |stmt| stmt.is_a?(MIR::AllocMark) } || + fresh_composite + if !selector_owned && !identity_selector && + item_type.recursive_cleanup_shape?(T.unsafe(pipeline_schema_lookup)) + selector = MIR::DeepCopy.new(selector, item_type.zig_type, nil, :full_value, :heap) + selector_owned = true + end + + push_stmts = T.let([], T::Array[MIR::Emittable]) + if selector_owned + push_contract = T.let(nil, T.nilable(MIR::CallableContract)) + transfer_name = T.let(nil, T.nilable(String)) + pushed = T.let(nil, T.nilable(MIR::Node)) + if selector.is_a?(MIR::Ident) + # The owned value was hoisted into the prefix with its own cleanup; + # consume THAT binding: guard its cleanup and mark the move. + transfer_name = selector.name.to_s + selector_prefix.each do |stmt| + next unless (stmt.is_a?(MIR::Cleanup) || stmt.is_a?(MIR::ErrCleanup)) && + stmt.name.to_s == transfer_name + + stmt.cleanup_entry.mark_moved_guard! + end + pushed = selector + else + transfer_name = "__select_push#{id}" + entry = CleanupClassifier.owned_value_entry(item_type, T.unsafe(pipeline_schema_lookup)) || + CleanupEntry.build(:uniform, alloc: :heap, has_moved_guard: true, zig_type: item_type.zig_type) + entry = entry.with_alloc(:heap) + entry.mark_moved_guard! + push_stmts << MIR::AllocMark.new(transfer_name, :heap, item_type, :heap) + push_stmts << MIR::Let.new(transfer_name, selector, false, nil, nil) + push_stmts << MIR::ErrCleanup.new(transfer_name, entry) + pushed = MIR::Ident.new(transfer_name) + end + push_contract = MIR::CallableContract.new( + MIR::CallableContract.no_ownership(1).signature, + MIR::OwnershipContract.consume_operands([ + MIR::OwnershipOperandFact.owned_binding( + transfer_name, item_type, "stream SELECT push transfers the item to the consumer", :heap), + ]), + 1, + ) + push_stmts << MIR::ExprStmt.new(MIR::MethodCall.new( + MIR::Ident.new(local_stream), "push", [pushed], true, push_contract), false) + push_stmts.concat(MIR::OwnershipTransferPlan.new( + name: transfer_name, + target: :owned_sink, + target_alloc: :heap, + move_guarded: true, + ).marks) + else + push_stmts << MIR::ExprStmt.new(MIR::MethodCall.new( MIR::Ident.new(local_stream), "push", [selector], true, MIR::CallableContract.no_ownership(1)), false) - MIR::ForStmt.new(MIR::ItemsAccess.new(source_ref, true), "__select_item", [*selector_prefix, push], nil) + end + item_body = T.let([*selector_prefix, *push_stmts], T::Array[MIR::Emittable]) + # A stream source dequeues OWNED items (the producer's YIELD moved them + # into the channel; the same runtime :heap allocator frees them — the + # allocator identity the WHILE-NEXT consumer path already uses). Unless + # the selector passes the item through by identity (ownership goes into + # the push) the item gets a guarded per-iteration defer; a moving selector + # (GIVE / TAKES) suppresses it via the moved flag. A borrowing projection + # ALSO frees the item: the pushed value is an independent deep copy (see + # the push-transfer above), so the dequeued item would otherwise leak. + # Array sources iterate borrows of the source list — never released here. + source_item_type = T.let(source_type.runtime_stream_storage_element_type, T.nilable(Type)) + if !source_type.array? && source_item_type && + !PipelinePlaceholderUsage.identity_placeholder?(op.expression) + owned_stmts = @range_lowerer.owned_stream_item_stmts("__select_item", source_item_type) + item_body = owned_stmts + item_body unless owned_stmts.empty? + end + source_ref = MIR::FieldGet.new(MIR::Ident.new("ctx"), "source") + loop = if source_type.array? + MIR::ForStmt.new(MIR::ItemsAccess.new(source_ref, true), "__select_item", item_body, nil) else - # Stream sources hand each popped item over OWNED. The defer-based - # AllocMark/Cleanup pair (same contract as FOR-over-stream) frees it; - # an identity selector transfers the item into the result stream, so - # the push carries a consume-operands contract and the ownership-fact - # finalizer move-marks it. - source_elem_t = source_type.runtime_stream_storage_element_type || - source_type.tense_type&.element_type - item_ownership = source_elem_t ? - @range_lowerer.stream_item_ownership_prelude(source_type, source_elem_t, "__select_item") : [] - identity_transfer = !item_ownership.empty? && - selector.is_a?(MIR::Ident) && selector.name == "__select_item" - push_contract = if identity_transfer - MIR::CallableContract.new( - FunctionSignature.new(params: [ - AST::Param.new(name: "item", type: T.must(source_elem_t), takes: true), - ], return_type: Type.new(:Void)), - MIR::OwnershipContract.consume_operands([ - MIR::OwnershipOperandFact.owned_binding( - "__select_item", T.must(source_elem_t), "stream SELECT identity transfer", :heap), - ]), - 1, - ) - else - MIR::CallableContract.no_ownership(1) - end - push = MIR::ExprStmt.new(MIR::MethodCall.new( - MIR::Ident.new(local_stream), "push", [selector], true, push_contract), false) next_method = source_type.inf_stream? || source_type.bounded_stream? ? "nextOrNull" : "next" MIR::WhileStmt.new(MIR::MethodCall.new(source_ref, next_method, [], true, - MIR::CallableContract.no_ownership(0)), - [*item_ownership, *selector_prefix, push], "__select_item", nil, nil, nil) + MIR::CallableContract.no_ownership(0)), item_body, "__select_item", nil, nil, nil) end stream_zig = if stream_type.inf_stream? @@ -930,9 +1056,9 @@ def pipeline_schema_lookup @materializer.schema_lookup end - sig { params(receiver: String, alloc: Symbol, value_expr: MIR::Node).returns(MIR::Emittable) } - def append_owned_value_stmt(receiver, alloc, value_expr) - @materializer.append_owned_value_stmt(receiver, alloc, value_expr) + sig { params(receiver: String, alloc: Symbol, value_expr: MIR::Node, known_owned: T::Boolean, owned_type: T.nilable(Type)).returns(MIR::Emittable) } + def append_owned_value_stmt(receiver, alloc, value_expr, known_owned: false, owned_type: nil) + @materializer.append_owned_value_stmt(receiver, alloc, value_expr, known_owned: known_owned, owned_type: owned_type) end sig { params(value: MIR::Node, type_info: Type, alloc: Symbol).returns(MIR::Node) } @@ -1146,6 +1272,12 @@ def lower_find(site, find_node) sig { params(smooth_node: AST::BinaryOp).returns(Symbol) } def pipeline_alloc(smooth_node) + # The storage stamp is the annotator/escape-analysis placement decision + # (INV-16: use sites READ, they do not re-classify). A :heap stamp + # (promise lists, rewind-forcing elements) overrides the declaration + # context; otherwise the destination's allocator decides. + return :heap if smooth_node.storage == :heap + pipeline_result_alloc end @@ -1287,14 +1419,16 @@ def numeric_fold_expr_typed(expr_ast, item_var, acc_zig) stages: T::Array[AST::Node], on_skip: T.nilable(PipelineRangeSkipHook), source_alloc: T.nilable(Symbol), + track_owned_items: T::Boolean, ).returns(PipelineHost::LazyRangePrefix) end - def build_lazy_range_prefix(source_node, stages, on_skip: nil, source_alloc: nil) + def build_lazy_range_prefix(source_node, stages, on_skip: nil, source_alloc: nil, track_owned_items: false) @range_lowerer.build_lazy_range_prefix( source_node, stages, on_skip: on_skip, source_alloc: source_alloc, + track_owned_items: track_owned_items, ) end @@ -1360,9 +1494,9 @@ def lower_range_fold_observable(p, smooth_node, label, source_node, # Single shared lowering for SUM/COUNT/MAX/MIN/AVG/ANY/ALL/FIND. # REDUCE and DISTINCT need seeded inits or inline CAS, so they keep # dedicated helpers below. - sig { params(p: PipelineHost::LazyRangePrefix, fold_op: DefaultObservableFoldOp, smooth_node: AST::BinaryOp, label: String, source_node: AST::Node, terminal: Symbol).returns(MIR::BlockExpr) } - def lower_range_fold_observable_default(p, fold_op, smooth_node, label, source_node, terminal:) - @range_lowerer.lower_range_fold_observable_default(p, fold_op, smooth_node, label, source_node, terminal: terminal) + sig { params(p: PipelineHost::LazyRangePrefix, fold_op: DefaultObservableFoldOp, smooth_node: AST::BinaryOp, label: String, source_node: AST::Node, terminal: Symbol, observable_id: T.nilable(Integer)).returns(MIR::BlockExpr) } + def lower_range_fold_observable_default(p, fold_op, smooth_node, label, source_node, terminal:, observable_id: nil) + @range_lowerer.lower_range_fold_observable_default(p, fold_op, smooth_node, label, source_node, terminal: terminal, observable_id: observable_id) end sig { params(type_info: Type).returns(T::Boolean) } @@ -1370,9 +1504,9 @@ def pipeline_element_owns_heap?(type_info) @range_lowerer.pipeline_element_owns_heap?(type_info) end - sig { params(item_var: String, source_node: AST::Node).returns(T::Array[MIR::Emittable]) } - def consumed_stream_item_cleanup(item_var, source_node) - @range_lowerer.consumed_stream_item_cleanup(item_var, source_node) + sig { params(item_var: String, elem_t: T.nilable(Type)).returns(T::Array[MIR::Emittable]) } + def owned_stream_item_stmts(item_var, elem_t) + @range_lowerer.owned_stream_item_stmts(item_var, elem_t) end # REDUCE-scalar: per-item publish is a CAS loop that applies the diff --git a/compiler/ruby/mir/lower/pipeline/pipeline_list_lowerer.rb b/compiler/ruby/mir/lower/pipeline/pipeline_list_lowerer.rb index aefd92d9d..f8ec8de7d 100644 --- a/compiler/ruby/mir/lower/pipeline/pipeline_list_lowerer.rb +++ b/compiler/ruby/mir/lower/pipeline/pipeline_list_lowerer.rb @@ -29,6 +29,7 @@ class PipelineListLowerer < T::Struct extend T::Sig const :visit_mir, T.proc.params(node: AST::Node).returns(MIR::Node) const :visit_expr, T.proc.params(list_node: AST::Node, expr_node: AST::Node, placeholder: String).returns(MIR::Node) + const :visit_expr_head, T.proc.params(expr_node: AST::Node, placeholder: String, alloc: Symbol).returns(PipelineElementHead) const :visit_reduce_expr, T.proc.params(expr_node: AST::Node, item_placeholder: String, acc_placeholder: String).returns(MIR::Node) const :visit_body, T.proc.params(body_stmts: T::Array[AST::Node], placeholder: String).returns(T::Array[MIR::Emittable]) const :visit_join_lambda, T.proc.params(body: AST::Node, join_params: T::Hash[String, String]).returns(MIR::Node) @@ -40,9 +41,11 @@ class PipelineListLowerer < T::Struct const :next_label, T.proc.returns(String) const :set_current_label, T.proc.params(label: String).void const :append_owned_value_stmt, T.proc.params(receiver: String, alloc: Symbol, value_expr: MIR::Node).returns(MIR::Emittable) + const :append_fresh_owned_value_stmt, T.proc.params(receiver: String, alloc: Symbol, value_expr: MIR::Node, owned_type: Type).returns(MIR::Emittable) const :borrowed_pipeline_value, T.proc.params(value: MIR::Node, type_info: Type, alloc: Symbol).returns(MIR::Node) const :cleanup_bearing_type, T.proc.params(type_info: Type).returns(T::Boolean) const :owning_pipeline_temp_stmts, T.proc.params(name: String, source: MIR::Node, type_info: Type, zig_type: String, alloc: Symbol).returns(T::Array[MIR::Emittable]) + const :loop_mark_stmts, T.proc.returns(T::Array[MIR::Emittable]) sig { params(site: PipelineSite, op: PipelineListTerminalOp).returns(MIR::BlockExpr) } def lower(site, op) @@ -98,17 +101,18 @@ def lower_where(site, expr_node) alloc = self.pipeline_alloc.call(smooth_node) pred_mir = visit_pipeline_expr_mir(list_node, expr_node) self.pipeline_block.call(list_node, lambda do |items, label| + res = result_binding_name(label) [ - MIR::Let.new("res_list", + MIR::Let.new(res, MIR::MakeList.new(elem_zig, [], alloc), true, nil, nil), MIR::ForStmt.new(MIR::Ident.new(items), "it", [ MIR::Let.new("matches", pred_mir, false, nil, nil), MIR::IfStmt.new(MIR::Ident.new("matches"), [ - self.append_owned_value_stmt.call("res_list", alloc, + self.append_owned_value_stmt.call(res, alloc, self.borrowed_pipeline_value.call(MIR::Ident.new("it"), Type.new(elem_type), alloc)), ], nil), ], nil), - MIR::BreakStmt.new(label, MIR::Ident.new("res_list")), + MIR::BreakStmt.new(label, MIR::Ident.new(res)), ] end) end @@ -120,21 +124,89 @@ def lower_select(site, expr_node) res_type = select_result_type(expr_node) res_zig = self.transpile_type.call(res_type) alloc = self.pipeline_alloc.call(smooth_node) - expr_mir = visit_pipeline_expr_mir(list_node, expr_node) self.pipeline_block.call(list_node, lambda do |items, label| + res = result_binding_name(label) + # Per-label capture: a nested pipeline in the element expression emits + # its own loop, and a shared literal `it` shadows in Zig. + raw_it = label.to_s + raw_it = T.must(raw_it[1..]) while raw_it.start_with?("_") + it_name = "it_#{raw_it}" + # Lower the element INSIDE the loop body so its hoisted temps stay + # per-iteration instead of flushing to the enclosing statement. + head = self.visit_expr_head.call(expr_node, it_name, alloc) + # An element that allocates frame TRANSIENTS each iteration (a nested + # pipeline's apparatus, an owned composite field hoist) needs a + # per-iteration arena rewind (FRAME_OVERFLOW otherwise). Rewind frees + # everything frame-allocated during the iteration, so it is only sound + # when the escaping result is HEAP (the annotator stamps such pipelines + # heap; the append's transfer copies elements out of the doomed region + # before the rewind). If the stamp missed a shape, DON'T rewind — the + # checker's FRAME_NO_REWIND rejection stays fail-closed instead of + # emitting a rewind that frees moved frame elements (UAF). + rewind_per_iter = alloc == :heap && element_head_frame_transients?(head) + # A SELECT element that yields a FRESH owned value is MOVED into res_list; + # a borrowed element (a plain field/`it` projection) is COPIED so res_list + # owns an independent value. Ownership is carried on the head, established + # from MIR facts at lowering time (value effect + captured owned hoists) -- + # not re-derived from AST syntax after hoisting has flattened the value. + element_owned = head.owned + body = T.let(rewind_per_iter ? self.loop_mark_stmts.call.dup : [], T::Array[MIR::Emittable]) + body.concat(head.pending) + if element_owned + body << self.append_fresh_owned_value_stmt.call(res, alloc, head.value, res_type) + else + body << MIR::Let.new("val", head.value, false, nil, nil) + body << self.append_owned_value_stmt.call(res, alloc, + self.borrowed_pipeline_value.call(MIR::Ident.new("val"), res_type, alloc)) + end [ - MIR::Let.new("res_list", + # Explicit allocation + block-result transfer facts (mirrors + # lower_order_by): statement-level finalization stamps these when the + # pipeline is a statement, but NOT when the block is nested in an + # expression position — the hoist's cleanup allocator is read from + # these marks (a heap result with no mark got a frame defer: leak). + MIR::AllocMark.new(res, alloc, Type.new(smooth_node.full_type!), + MIR::Placement.alloc_scope(alloc)), + MIR::Let.new(res, MIR::MakeList.new(res_zig, [], alloc), true, nil, nil), - MIR::ForStmt.new(MIR::Ident.new(items), "it", [ - MIR::Let.new("val", expr_mir, false, nil, nil), - self.append_owned_value_stmt.call("res_list", alloc, - self.borrowed_pipeline_value.call(MIR::Ident.new("val"), res_type, alloc)), - ], nil), - MIR::BreakStmt.new(label, MIR::Ident.new("res_list")), + MIR::ForStmt.new(MIR::Ident.new(items), it_name, body, nil), + *MIR::OwnershipTransferPlan.new( + name: res, + target: :block_result, + target_alloc: alloc, + move_guarded: false, + ).marks, + MIR::BreakStmt.new(label, MIR::Ident.new(res)), ] end) end + # True when the lowered element carries frame AllocMarks — per-iteration + # frame transients that require the loop rewind above. Fibers/lambdas are + # boundaries (their frames are their own); nested loops are NOT (their + # allocations still land in this function's arena every outer iteration). + sig { params(head: PipelineElementHead).returns(T::Boolean) } + def element_head_frame_transients?(head) + found = T.let(false, T::Boolean) + boundary = ->(node) { node.is_a?(MIR::BgBlock) || node.is_a?(MIR::LambdaExpr) } + nodes = T.let([*head.pending, head.value], T::Array[MIR::Emittable]) + MIR.each_node_until(nodes, boundary) do |node| + found = true if node.is_a?(MIR::AllocMark) && MIR::Placement.frame?(node.alloc) + end + found + end + + # Per-label result binding (mirrors pipe_src_list_