Skip to content

New resolution - #564

Open
LPTK wants to merge 235 commits into
hkust-taco:hkmc2from
LPTK:new-resolution
Open

LPTK wants to merge 235 commits into
hkust-taco:hkmc2from
LPTK:new-resolution

Conversation

@LPTK

@LPTK LPTK commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

This PR implements a new resolution system independent of type inference and type checking, broadly following the ideas I presented at WITS 2026 in the talk Decoupling Resolution from Type Inference.

The approach is actually a lot more subtle and interesting than what's suggested in the talk.

First, I want to allow unrestricted forward and indirect references, including to other and nested module/class members. This requires moving to a "listener"-based architecture, which is the easiest way of implementing something like that. It bears some resemblance to Tate's so-called type outference, although in our case it's used to resolve everything in the program, and not just for local type inference with method resolution.

Second, I want for resolution to work with unannotated definitions and to have some amount of context sensitivity, so that something like this works:

module A { val u = 1 }
module B { val u = 2 }

fun foo(x) = x

foo(A).u // resolves precisely to A's u

foo(B).u // resolves precisely to B's u

Making this works as part of the resolution of everything else is quite hard, as it means we can't just generalize and instantiate constraint schemes, since some things are not yet resolved by the time a definition is completed (this problem is similar to that of "omnidirectional" type inference for ML, but our solution is very different). This means we need to maintain a graph of related listeners along with contextual marks that delineate the various scopes, so that we can separate different uses of a given definition (what the slop machine calls "activations") by filtering scope entries that meet unrelated scope exits. There are interesting parallels between this and optimal reduction in the lambda calculus.

Doing this correctly is tricky and requires keeping track of captured identifiers, where captures act like unfiltered scope exits. Ensuring that this terminates requires bounding the depth of mark stacks by preventing recursive edges from entering scopes repeatedly. We do this by treating recursive references as scope exits. As for indirect recursion, since such recursion always goes through scope exits, the approach works out of the box.

This PR contains only the first implementation of context sensitivity. As it stands, it's roughly equivalen to a higher-order 1-CFA. But I have plans to relax it by computing call-graph SCCs, with the effect of maintaining much more context sensitivity (more similar to that of ML type inference) while preserving termination and reasonable performance in the common case.

Third, I want for this whole resolution machinery to be integrated with the type system and leverage type information, including partial type annotations, which should still preserve context sensitivity of the unannotated parts. Type annotations essentially provide abstraction boundaries, so that the appraoch remains local and scalable.

An important design decision is that resolution does not let unknowns flow into places that would conceptually affect resolution. This means that, for example, in a file Test.mls, this is allowed:

fun foo(xs) = xs.length
module Test {
  fun bar() = foo([1, 2, 3])
}

but this is disallowed

module Test {
  fun foo(xs) = xs.length
  fun bar() = foo([1, 2, 3])
}

Indeed, even though we can resolve xs.length as the Array method from bar's foo call, in the latter code snippet, foo is also exported to the outside world. To preserve separate resolution and compilation, we couldn't let outside users affect the resolution of Test.mls, and we also don't eant to make it look like foo could handle any any length method on its input. So, the latter code snippets yields an error:

//│ ╔══[COMPILATION ERROR] Resolution error in selection
//│ ║  l.232: 	  fun foo(xs) = xs.length
//│ ║         	                ^^^^^^^^^
//│ ╟── Cannot resolve member 'length' of a value with unknown shape.
//│ ╟── Parameter 'xs' admits values of unknown shape.
//│ ║  l.232: 	  fun foo(xs) = xs.length
//│ ║         	          ^^
//│ ╟── Member 'foo' is accessible here.
//│ ║  l.232: 	  fun foo(xs) = xs.length
//│ ║         	  ^^^^^^^^^^^^^^^^^^^^^^^
//│ ╟── 'Test' is exposed by this compilation unit.
//│ ║  l.231: 	module Test {
//│ ║         	^^^^^^^^^^^^^
//│ ║  l.232: 	  fun foo(xs) = xs.length
//│ ║         	^^^^^^^^^^^^^^^^^^^^^^^^^
//│ ║  l.233: 	  fun bar() = foo([1, 2, 3])
//│ ║         	^^^^^^^^^^^^^^^^^^^^^^^^^^^^
//│ ║  l.234: 	}
//│ ╙──       	^

This can be solved by adding a partial type annotation:

module Test {
  fun foo(xs: Array) = xs.length
  fun bar() = foo([1, 2, 3])
}

(The type argument is left unspecified, and thus inferred at call sites.)

Fourth, I want separate resolution even when members that are not fully annotated are exported. To make this work, we need to be very careful about where we register listeners and what state resolution is allowed to modified, to avoid logic and concurrency issues. Basically, we perform on-the-fly lazy copies of a frozen view of the resolution state of each imported file into each downstream user.


I implemented the initial prototype and use GPT-6 Astra (and a bit of Claude Fable 5.1) extensively to fill in the many, many blanks, especially for things related to the handling of type annotations, and handle much of the migration. Not everything is migrated as part of this PR yet, as there are areas left for future development.

A few things that are intentionally not included and left for future PRs are:

  • more advanced context-sensitive reasoning
  • implicits (ie, implicit parameters, implicit namespacing, etc.)
  • using the resolution info for producing more precise erased types during lowering

LPTK and others added 30 commits September 18, 2026 16:23
…o new-resolution

# Conflicts:
#	hkmc2/shared/src/test/mlscript/newres/ValCtxSens.mls
LPTK and others added 27 commits October 3, 2026 13:07
…o new-resolution

# Conflicts:
#	hkmc2/shared/src/test/mlscript/newres/Opens.mls
#	hkmc2/shared/src/test/mlscript/newres/Projections.mls
# Conflicts:
#	hkmc2/shared/src/test/mlscript/codegen/ConfigDirective.mls
#	hkmc2/shared/src/test/mlscript/decls/Prelude.mls
#	hkmc2/shared/src/test/mlscript/deforest/fusibility.mls
#	hkmc2/shared/src/test/mlscript/deforest/recursive.mls
@LPTK
LPTK requested a balanced review from Copilot October 3, 2026 16:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The watcher ignores directory-configuration edits, and an existing callable-return regression is being converted into an expected error.

Review effort: Balanced
Findings: 1 High severity · 1 Medium severity · 1 Low severity

Open (3)
What changed in this PR

Introduces the new resolution system and migrates compiler fixtures and diff tests to exercise it.

Changes:

  • Adds resolution support for super, dynamic constructs, forward references, imports, and erased types.
  • Migrates test suites to inherited directory configuration and updates golden outputs.
  • Updates test runners, watcher behavior, and supporting compiler passes.
File(s) Description
.codex/​config.toml Configures Metals MCP.
hkmc2DiffTests/​src/​test/​scala/​hkmc2/​{Watcher,InvalmlDiffMaker}.scala Updates watcher filtering and InvalML erasure.
hkmc2/​jvm/​src/​test/​scala/​hkmc2/​{TestFolders,CompileTestRunnerBase}.scala Excludes directory configuration files from test discovery.
hkmc2/​shared/​src/​main/​scala/​hkmc2/​{package,utils/​utils}.scala Adds collection and debug-printing support.
hkmc2/​shared/​src/​main/​scala/​hkmc2/​syntax/​ParseRule.scala Parses dyn and super.
hkmc2/​shared/​src/​main/​scala/​hkmc2/​semantics/​{ups/​Instantiator,ups/​FixedPointCompiler,ucs/​TermSynthesizer}.scala Propagates erasure and flow symbols.
hkmc2/​shared/​src/​main/​scala/​hkmc2/​codegen/​{ReflectionInstrumenter,Printer,BlockTraverser,BlockTransformer,BlockSimplifier}.scala Handles super selections throughout code generation.
hkmc2/​shared/​src/​test/​mlscript-compile/​cpp/​Makefile Removes Homebrew-specific paths.
hkmc2/​shared/​src/​test/​mlscript-compile/​NoFreeze.mjs Updates generated default export.
hkmc2/​shared/​src/​test/​mlscript-compile/​{AsyncLibrary,GenericLibrary,LegacyGenericLibrary,LenientSelections,NamedFieldLibrary,OverloadedClassLibrary,OverloadedDefault,RecordProducers,SealedClasses}.mls Adds resolution/import fixtures.
hkmc2/​shared/​src/​test/​mlscript-compile/​{Benchmark,CachedHash,Char,Example,FilePrivateFunctions,NestedFunctionsNoLift,NoFreeze,QuoteExample,QuoteExample2,Record,Stack,StrOps,TreeTracer,XML}.mls Migrates compile fixtures and annotations.
hkmc2/​shared/​src/​test/​mlscript-compile/​quotes/​{CSPBar,CSPBaz,CSPFoo,CSPNest,Cubic,Gib12,Opened,QuoteFoo,QuoteInc,SafeDiv}.mls Migrates quotation fixtures.
hkmc2/​shared/​src/​test/​mlscript-compile/​ups/​{DnfCnf,EvenOddTree,TruthyFalsy}.mls Migrates UPS fixtures.
hkmc2/​shared/​src/​test/​mlscript-compile/​nofib/​{atom,banner,boyer,boyer2,calendar,cichelli,clausify,cryptarithm1,eliza,gcd,lcss,life,mandel,mandel2,puzzle,rsa,scc,secretary,sphere}.mls Migrates nofib modules and annotations.
hkmc2/​shared/​src/​test/​mlscript-compile/​apps/​{CSV}.mls Adds language mode and input shape.
hkmc2/​shared/​src/​test/​mlscript-compile/​apps/​parsing/​{BasicExpr,Expr,Lexer,PrattParsing,RecursiveDescent,Test,TokenHelpers}.mls Migrates parsing libraries.
hkmc2/​shared/​src/​test/​mlscript-compile/​apps/​parsing-web-demo/​Examples.mls Adds typed example metadata.
hkmc2/​shared/​src/​test/​mlscript/​.mls Sets inherited language mode.
hkmc2/​shared/​src/​test/​mlscript/​newres/​{.mls,DynamicInstantiation,HostInterfaces,ImportedRecords,IRClassHeaders,LexerMigration,MemberAssignments,TypeApplications}.mls Adds new-resolution regression coverage.
hkmc2/​shared/​src/​test/​mlscript/​newres/​loose/​.mls Configures lenient resolution.
hkmc2/​shared/​src/​test/​mlscript/​newres/​wasm/​{.mls,BackendFlags,WasmBasics}.mls Adds new-resolution WASM coverage.
hkmc2/​shared/​src/​test/​mlscript/​wasm/​{.mls,Binaryen,ClassInheritance,ControlFlow,DeadConstructorElim,DeadParamElim,Exceptions,MainFunctions,Matching,ReplImports,ScopedLocals,Singletons,SingletonUnit,Strings,Tuples,TypeResolution}.mls Migrates WASM tests and snapshots.
hkmc2/​shared/​src/​test/​mlscript/​basics/​{.mls,AppOp,BadModules,ByNameTailPosition,CallSyntaxes,CompanionModules_Functions,CompanionModules_Patterns,CompanionModules_Types,CyclicValues,DisruptiveComments,Drop,DynamicFields,ExplicitSelections,FunnyRecordKeys,Getters,IfThenNewline,LabeledIfWhileBodies,LetBindings,LocalVal,MultiParamLists,MutArr,MutCls,MutVal,NamedArgs,NewlineOperatorEnd,OpDivPrecedence,Overloading,ShortcircuitingOps,Super,SuspensionComments,ThenRecord}.mls Migrates basic language tests and adds super coverage.
hkmc2/​shared/​src/​test/​mlscript/​codegen/​{.mls,BadFunctions,BadImport,BadInit,BadValInit,CaseOfCase,CheckedCasts,Classes,ClassInClass,ClassMatching,CodegenScratch,Comma,CurriedFunctions,EarlyReturn,ElseLess,Formatting,Functions,FunctionsThis,FunInClass,GlobalThis,i382,IfThenElse,ImportedOps,ImportJSClass,ImportJSModule,InlineLambdas,InlineMultiArgLists,Inliner,InnerNameHygiene,InterleavedRecords,Lazy,MergeMatchArms,ModuleMatching,NestedLoops,NestedTypes,NoFreeze,NoInline,ObjectInit,OpenJS,OptMatch,PredefUsage,PrivateConstructors,PrivateMemberExtrusion,Pwd,QQImport,ReboundLet,RecordSpreads,Repl,Scoping,SetIn,ShortCircuitReturn,SimplePatMat,SingletonInit,SpecialJSNames,SwitchSpecialization,TailRecFormerFailure,This,TraceLog,WhileDefaults}.mls Migrates code-generation tests and snapshots.
hkmc2/​shared/​src/​test/​mlscript/​lifter/​{ClassWithCompanion,CompanionsInFun,CurriedClassInFun,DefnsInClass,FunInFun,FunInMethod,Imports,Labels,Loops,Mutation,PatternInFun}.mls Migrates lifting tests and outputs.
hkmc2/​shared/​src/​test/​mlscript/​handlers/​{.mls,Debugging,EffectInHandler,GeneratorStack,HandlerReset,HandlersScratch,Loops,MultiResumption,NestedHandlers,NonLocalLabelControlFlow,UserThreadsSafe,UserThreadsUnsafe,ZCombinator}.mls Migrates effect-handler fixtures.
hkmc2/​shared/​src/​test/​mlscript/​deforest/​{.mls,append,basic,clashes,cyclic,determinism,dropLast,erasedTypes,fusibility,imperative,listComprehension,module,multiArgLists,nestedMatch,zipunzip}.mls Migrates deforestation tests and snapshots.
hkmc2/​shared/​src/​test/​mlscript/​deforest/​eta-expansion/​{.mls,basic,recursive}.mls Migrates eta-expansion tests.
hkmc2/​shared/​src/​test/​mlscript/​dead-param-elim/​{.mls,clash,class-in-fun,config-flags,dead-ref,lambda,module,multiArgLists,recursive,todos,ups}.mls Migrates dead-parameter tests.
hkmc2/​shared/​src/​test/​mlscript/​ups/​{.mls,EmptyJunctions,JoinPatterns,SimpleConjunction,TransformFree,UpsBugsBacklog}.mls Migrates UPS tests.
hkmc2/​shared/​src/​test/​mlscript/​ups/​{examples,fixpoint,nondeterminism,parametric,recursion,regex,specialization,syntax,transformation}/​.mls Adds inherited UPS directory configuration.
hkmc2/​shared/​src/​test/​mlscript/​ups/​examples/​{BadRecStackParse,DnfCnf,DoubleOrSum,DoubleTripleList,Extraction,Flatten,Negation,TupleSpread}.mls Migrates UPS examples.
hkmc2/​shared/​src/​test/​mlscript/​ups/​fixpoint/​{MoreAlternatives,SimpleExample}.mls Migrates fixed-point tests.
hkmc2/​shared/​src/​test/​mlscript/​ups/​nondeterminism/​{BitArithmetic,EvenOddTree,LaRbTree}.mls Migrates nondeterminism tests.
hkmc2/​shared/​src/​test/​mlscript/​ups/​parametric/​ListLike.mls Migrates parametric tests.
hkmc2/​shared/​src/​test/​mlscript/​ups/​recursion/​{BitSeq,BitTree,LeafEvenOddTree,NatBox,NullTree,SignBox}.mls Migrates recursion tests.
hkmc2/​shared/​src/​test/​mlscript/​ups/​regex/​{EmailAddress,EmptyString,Identifier,Number,Simplification}.mls Migrates regex tests.
hkmc2/​shared/​src/​test/​mlscript/​ups/​specialization/​SimpleLiterals.mls Migrates specialization snapshots.
hkmc2/​shared/​src/​test/​mlscript/​ups/​syntax/​CrossCompilation.mls Migrates syntax tests.
hkmc2/​shared/​src/​test/​mlscript/​ups/​transformation/​{BasicTransformations,BindingLess}.mls Migrates transformation tests.
hkmc2/​shared/​src/​test/​mlscript/​ucs/​{.mls,AndNewline}.mls Adds UCS inherited configuration.
hkmc2/​shared/​src/​test/​mlscript/​ucs/​{examples,future,general,hygiene,normalization,papers,patterns,syntax}/​.mls Adds UCS subdirectory configuration.
hkmc2/​shared/​src/​test/​mlscript/​ucs/​examples/​{BinarySearchTree,LeftistTree,ListFold,Permutations,SimpleTree,ULC}.mls Migrates UCS examples.
hkmc2/​shared/​src/​test/​mlscript/​ucs/​future/​Or.mls Migrates future-syntax snapshots.
hkmc2/​shared/​src/​test/​mlscript/​ucs/​general/​{BooleanPatterns,CardSuits,CrossModules,DualOptions,InterleavedLet,JoinPoints,List,Seqs,Simple}.mls Migrates general UCS tests.
hkmc2/​shared/​src/​test/​mlscript/​ucs/​hygiene/​{Hygiene,PatVars}.mls Migrates hygiene tests.
hkmc2/​shared/​src/​test/​mlscript/​ucs/​normalization/​{DeduplicationWhile,ExcessiveDeduplication,InheritanceNormalization,RecordImpliedByClass,SimplePairMatches,UnifySubScrutinees,UnifyTupleElements}.mls Migrates normalization tests.
hkmc2/​shared/​src/​test/​mlscript/​ucs/​papers/​OperatorSplit.mls Migrates paper example.
hkmc2/​shared/​src/​test/​mlscript/​ucs/​patterns/​{AliasPattern,CompiledQualifiedConstructors,ConjunctionPattern,Literals,NamePattern,RestTuple,SpreadTrailingIndex,String}.mls Migrates pattern tests.
hkmc2/​shared/​src/​test/​mlscript/​ucs/​syntax/​{And,ConjunctMatches,Empty,Is,Of,Split,TupleRest,WithBraces}.mls Migrates syntax tests.
hkmc2/​shared/​src/​test/​mlscript/​opt/​{.mls,AbortivePrefix,AlmostTailCall,BasicVarPropag,CapturedVariableMutation,ConstantFolding,DeadAssignmentRemoval,DeadBranchRemoval,DeadObjRemoval,DeadSelRemoval,DeadStatRemoval,DeadVarRemoval,IllFormedAccesses,InfiniteInlining,InlineAcrossFiles,InlineModuleMethods,InlineNestedFunctionsAcrossFiles,LoopOpt,MatchSimplification,NestedLoopOpt,NoCaseOfCase,PureCallPropagation,TailRecOpt,UnspecifiedBehavior,ValPropag,VarPropagStress,WorkerWrapper}.mls Migrates optimization tests.
hkmc2/​shared/​src/​test/​mlscript/​nofib/​{.mls,ansi,atom,awards,banner,boyer,boyer2,calendar,cichelli,circsim,clausify,constraints,cryptarithm1,cryptarithm2,cse,eliza,fish,gcd,integer,knights,lambda,last-piece,lcss,life,mandel,mandel2,mate,minimax,para,power,pretty,primetest,puzzle,rsa,scc,secretary,sorting,sphere,treejoin}.mls Migrates nofib runtime tests.
hkmc2/​shared/​src/​test/​mlscript/​apps/​{.mls,AccountingTest,CSVTest}.mls Migrates application tests.
hkmc2/​shared/​src/​test/​mlscript/​apps/​parsing/​{.mls,CamlLightTest,DirectiveTest,LeftRecursion,LoopExpressions,ParseRuleVisualizerTest,ParserErrorTest,ParserTest,PrattParsingTest,RecursiveDescentTest,RulesTest,TypeTest}.mls Migrates parser tests and adds regressions.
hkmc2/​shared/​src/​test/​mlscript/​apps/​parsing-web-demo/​{.mls,ExamplesTest}.mls Migrates web-demo tests.
hkmc2/​shared/​src/​test/​mlscript/​block-staging/​{.mls,Casts,Nested,PrintCode,Syntax}.mls Migrates block-staging tests.
hkmc2/​shared/​src/​test/​mlscript/​interop/​{.mls,CtorBypass,Functions,JSCollections,JSDerp,Null,Number,Symbols}.mls Migrates JavaScript interop tests.
hkmc2/​shared/​src/​test/​mlscript/​invalml/​{InvalMLCodeGen,InvalMLPrelude}.mls Updates InvalML fixtures.
hkmc2/​shared/​src/​test/​mlscript/​meta/​{.mls,BlockDiffTesting,ImportedTest,ImporterTest,LocDebugging}.mls Migrates metadata tests.
hkmc2/​shared/​src/​test/​mlscript/​std/​{.mls,LazyArrayTest,RecordTest,StackTests}.mls Migrates standard-library tests.
hkmc2/​shared/​src/​test/​mlscript/​ctx/​{.mls,MissingDefinitions2}.mls Migrates context tests.
hkmc2/​shared/​src/​test/​mlscript/​backlog/​{.mls,Lifter,NonReturningStatements}.mls Migrates backlog tests.
hkmc2/​shared/​src/​test/​mlscript/​objbuf/​{.mls,ObjectBufferAllocator}.mls Migrates object-buffer tests.
hkmc2/​shared/​src/​test/​mlscript/​syntax/​{.mls,BackslashApps,BasicIfs}.mls Migrates syntax tests.
hkmc2/​shared/​src/​test/​mlscript/​syntax/​annotations/​.mls Adds inherited syntax configuration.
hkmc2/​shared/​src/​test/​mlscript/​tailrec/​.mls Adds inherited tail-recursion configuration.
hkmc2/​shared/​src/​test/​mlscript/​flows/​LeadingDotAccesses.mls Updates flow snapshot.
hkmc2/​shared/​src/​test/​mlscript/​OverloadedModulesInSignatures.mls Updates companion-module diagnostic.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@@ -80,7 +81,13 @@ fun foo() =
//│ —————————————————| Output |—————————————————————————————————————————————————————————————————————————

:expect 2
:e
Comment thread hkmc2DiffTests/src/test/scala/hkmc2/Watcher.scala
Str then JSON.stringify(value)
StyleAttributeValue then value.toValue()

// * TOOD: Allow just `StyleAttributeValue then value.toValue()`

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants