What is the API story for this new codebase? #455
Replies: 24 comments 54 replies
|
What are your IPC plans for WebAssembly in the browser? |
|
I don't think we have a concrete plan (just prototypes), but we expect to be able to at least do the same IPC via mechanisms like WASI (what I do in https://jakebailey.dev/esbuild-playground, not to pump this side project). There's also the possibility to use plain messaging and only offer async in the browser, or some sort of async worker with SharedArrayBuffer based communication; the latter is already pretty well thought out for vscode.dev/github.dev, so there's theoretically some reusability there. |
|
Have you considered WASM as a solution to providing a cross-language API? I haven't used it for this purpose, but it seems like there could be a good match in requirements. WASM modules can pass typed data to one another, call one another's functions, etc. |
|
Actually, this has its own discussion here: |
|
People really need to provide their test code. I'm pretty sure the results are incorrect. Notably, threading needs to be disabled in WASM but we didn't do that in the repo yet. |
|
Wouldn't https://extism.org be ideal here? |
|
Why is exitism ideal? Which problem are you specifically trying to solve? Plugins? To be clear, I do not think it's the direction we want to take to allow embedding languages into tsc. It could be technically feasible, but we'd now be a "runtime" and have to go through a lot of compliance work..... |
|
Jake Bailey (@jakebailey) it could allow us to hook into the compiler similarly to how its currently being done with the transformer API. |
|
How's the story for TypeScript plugin developers and people integrating with the language service? More specifically, will method patching still work? The official way to create a plugin is to patch methods. And across such use cases there are probably dozens of cases where people reach into TypeScript semi-private internals to patch other methods (so that things can be intercepted for example) - will that also still work? If not, how do you envisage plugin authors to migrate away from doing that? |
|
We love ts-morph; it is explicitly an anti-goal to prevent ts-morph from working altogether. Type information is effectively the reason why we need to have an API at all 😄 |
|
Okay whew! |
|
WE'RE SO BACK |
|
Jake Bailey (@jakebailey) Any news about the plugin API? |
|
No. We've been pretty clear in the blogs and so on that this is a 7.1 item. Follow https://github.com/microsoft/typescript-go/issues/2824, and there's already a prototype shipped that covers some aspects like refactors |
|
I've used the tsc api for various codegen purposes, will you be exposing an API to walk/visit the AST tree as was available in the js tsc api? |
|
Will error reporting change? For a long time, TS errors have been a heated topic and there have been efforts by the community (https://github.com/yoavbls/pretty-ts-errors) to make it a little better. Will solutions like Pretty TS still work in the new world? |
|
+1 |
|
Are there plans about instead of IPC there would be something like FFI? |
|
Loading multiple runtimes into the same process means they are all competing to register signal handlers, do preemption, manage certain process-specific limits, etc. It might work if all are built with the same exact version of Go, since things like signal handlers would still grab data at the right offsets into things like TLS, but it's definitely not a guaranteed. See also: golang/go#65050 (comment) |
|
What I mean is, I'm surprised it's not possible to only load a single runtime on the first FFI call, and have subsequent FFI calls instantiate structs into that same runtime... Which from skimming that thread, sounds like it is possible using plugins instead of c-shared libraries...but I guess there's still a problem if different versions of |
|
The Go runtime is entirely different between Go releases, and potentially Go patches. I don't even think it's guaranteed that it'll work if they happen to match. Plugins are not an option; that's for loading Go code into Go code, not Go code into other things. That and Go plugins do not support Windows. |
|
Go's lack of a stable ABI seems to make it a terrible choice. Why was Go chosen instead of something with a stable ABI? This almost seems like a choice made explicitly to create conflict and incompatibility more than an effort to improve performance. |
|
We wouldn't want the only API to be tied to any specific language anyway. An IPC-based API was always going to be needed, and at that point it's probably not worth the effort to make a native API even if it was written in a language that would allow for that. |
|
Not a question, but an observation: Right now, lots of tools which depend on types (TypeScript Eslint, Typia, etc) run their own Lexer, parser and type checker. It is not uncommon to have separate type checkers running in the ESLint process, in a development server and in the TypeScript language service. This leads to doing a lot of work twice or more and can also lead to inconsistencies if the tooling does some weird stuff. With an IPC layer it could become very feasible to have all of these hook into one tsgo process and have one source of truth for type information, meaning that the development experience when using these kinds of tools could get a lot smoother. Maybe this is something that can be considered when designing the IPC layer - i.e. to allow having one tsgo "server" and multiple clients consuming it's API. |
|
Is now maybe the time to also re-consider adopting semver from the API perspective? I could usually care less about breaking changes from the language and type checking perspective, but I would love it if from the tooling perspective we could get some sort of guarantees of what versions of the typescript api are supported and compatible. I will acknowledge that in the past this was a harder sell because of the nature of javascript and how the entire AST and compiler internals are essentially all public api. As a result, any change or internal refactoring could therefore be considered a breaking change, and thus nothing was. If the team is being more intentional about what is exposed across the API boundary, it can be more reasonable. My ideal would be something like: Note: changes to the language that add new features may change the resulting AST that is returned from the API. This should not be considered a breaking change to the API, but maybe things like removing documented properties from existing nodes, might be? That way, tooling providers can set their expectations of supported typescript releases based on the |
|
Absolutely; this is going to be mandatory going forward since it won't be apparent to API consumers what version they're talking to. A better versioning story is one of the reasons we want to make a more intentional design here. That may or may not correspond to the "language version", but we're going to have some concept of an "API version" that probably revs on a different axis. |
|
Hey there, I'm developing an experimental Node.js binding here: https://github.com/Brooooooklyn/typescript-go-napi/tree/main/rust The basic idea is to compile typescript-go into a static link library, and then wrap it into a JavaScript API in Rust using NAPI-RS. I have now completed two basic APIs, among which transform is like this: import { transform } from '@typescript-go/api'
transform('const a: number = 1;', '/absolute/path/a.ts')
// 'const a = 1;\n' |
|
Do note #455 (reply in thread) This works but is likely not going to be safe for a number of important users. |
|
I do actually have the code to do NAPI directly with Go, it'd be awesome if we could use it 😞 |
Perhaps my implementation can be provided to users who don't need to load multiple different ts-go instances 👀 |
How future-proof is this module? Webdriverio badly wanted to provide a sync API for writing tests, and relied on |
|
Andrew Branch (@andrewbranch) Jake Bailey (@jakebailey) thoughts on how futureproof I guess the fact that it's based upon NAPI is better than |
|
I don't really understand the question. This surely isn't worse than anyone else using |
|
Okay yeah I hadn't realized that if native code simply makes blocking calls that will block the source JS function call (and event loop). |
|
Adding my two cents here, if API is not possible for major languages (mainly JS and probably Rust for SWC/OXC), a static type info snapshot can also be helpful. (though it is not possible to do transformation) Related issues https://github.com/microsoft/typescript-go/issues/504 |
|
Would it be possible to use tsc-go as a library from Go and hook into it that way? As opposed to plugins and IPC etc? |
|
The TypeDoc project uses TypeScript compiler APIs (including watch-related APIs) to generate documentation from source projects (via JSDoc as well as TypeScript type info) and to respond to changes in the code base and non-code documentation files. Will the same or substantially similar watch APIs be available from JS? (i.e. including the ability to add arbitrary watches as well as incremental-compile-on-change) |
|
How would we use something typedoc with ts plugins? like, supporting .vue, .svelte, .gjs, or .gts files? |
|
I think it’s very unlikely that any watch APIs will exist. |
|
The beginnings of the IPC API are now in a PR at #711. |
|
No, you cannot make use of any of this right now. The APIs are going to be different, so that API example from the old repo is unlikely to work. What is in the current API package is a prototype, a proof of concept. We're certainly aware of the annoyances in the old API, and want to make it better, but we have a challenge ahead of us as to how plugins are supposed to work since we can't actually load random code into a Go binary like we did a JS process, nor would there be any way to access anything internal (only what we explicitly expose). |
I can invoke Go from within JS using the bridge library you implemented (AFAIK that's what powers |
|
I was able to create a rust client for the IPC-based API in a few hours, the hardest part was the encoding format. Excited to see where this goes! https://github.com/abhiaagarwal/tsgo-rust-ipc/ |
Sure, but only if we actually launch the server via that bridge. |
|
I don't get how the callbacks design doesn't already explicitly support this? Go is essentially invoking JS by asking it to return the results of a desired callback. |
|
However the API is implemented, it would be great if the TS Compiler, Language Service, etc APIs were better documented and provided more detailed and complex examples than the currently limited wiki pages. Sample integrations and such could also be very useful. Per microsoft/TypeScript#49790, microsoft/TypeScript#49050, and links therein, I've had to dive into the source code on numerous occasions to understand how to use pieces of the API and fix implementation issues, and sometimes haven't even been able to find an answer there either. This makes maintaining an integration really difficult in certain areas as there's little-to-no information to be found on proper usage, especially when it comes to various edge-cases like |
|
One limitation I've encountered as a (very early!) consumer of the API is that concurrency is not doable, in that due to the nature of pipes, I have to wait to consume a response from stdout before sending another request via stdin. I'd ideally want them to be able to operate independently. Is there any plan to address this? Could use an uds socket over stdin or maybe even correlation ids? I was originally designing my API around async processing, but I'm switching to a mpsc approach with tsgo subprocess running in a dedicated thread. If the goal is for tsgo to become the canonical source of truth with multiple processes hooking into its API with it serving as more of a server, it'd be great to support a more traditional client-server architecture. UDS would be a good middlepoint between the overhead of tcp sockets |
|
Concurrent requests in a single session will definitely be supported eventually. I’m not sure whether “one server many clients” will be a generally supported pattern or not. |
|
One thing that would be nice (and I apologize, I don't mean to turn this into a feature request thread) off the top of my head is the ability to pass an AST directly to the api server. The big blocker for rust-based plugins like oxlint, biome, etc is the inconsistent mapping between their CST/AST and TS's AST (see typescript-eslint for the magic they do to convert back and forth). I could imagine an universe where a plugin creates their own internal AST, converts it to a TS AST (with an mapping from their nodes -> TS nodes), and then passes it directly to the API. Then, the linter can query the nodes they're interested in type information about directly without having to interpret typescript's AST back to theirs. The big blocker in the ecosystem from what I see is how ts interprets trivia in terms of leading spaces etc, which shouldn't matter for type-checking. Basically, give tsgo the ability to deserialize an encoded |
|
Well, I'm planning to create a similar system. I need something where, for example, an OXC or SWC transformer can query the TypeScript TypeChecker for type information about a symbol. I'll probably fork tsgo and build an API layer on top of it, unfortunately, since tsgo keeps everything internal, I have to create a custom build. But I want to keep it compatible so I can still merge changes from the original tsgo repository into my project. It would be nice if tsgo exposed a proper Go API so I could use it as a package instead of having to fork it. The motivation is to provide extended type inference capabilities for RTTIST - runtime type information system (reflection). Rewriting everything (RTTIST's typegen) in Go might make sense if tsgo became the standard for build tools, replacing SWC, esbuild, and OXC - but I don't think that's likely to happen. |
|
For what it's worth, I did look into this yesterday because I'm interested in doing the same thing (for oxc to solve oxc-project/oxc#3105) and I'm convinced it's possible: the API layer would have to expose the This is a cursory look and is probably incorrect in places, but the API is designed well-enough that this is feasible. |
|
Ohhh… I have an OXC-based transformer where I need to infer types, and that’s why I’m doing this. I didn’t know about that issue, but I’m basically solving it. 😆 |
|
Since EDIT: got simple diagnostics working |
|
Just wondering: Is there still no way to use the new TypeScript Go Compiler to print an AST e.g. as JSON? I don't need to modify nodes or transform anything. I just want the parsed AST from the compiler on stdout. Possible/Not possible? |
|
You can do something like that with the existing API client prototype |
|
Had a poke at the new API (directly, not via the node package) and while checking to see if I could do a full VFS found it currently barfs if you give it a CWD that doesn't exist or return Hoping to eventually wire up a UX to this so you can poke around in a build and see exactly what is taking time a lot easier: it's no good being 10x faster, if that just means you make 10x more expensive types :P |
|
Different angle on this. I'm working on The shim approach is borrowed from Plugins ship as Go source. puchm The shim is an overlay I maintain by hand, but the AST and checker surfaces have been stable enough in practice that tsgo bumps haven't been a serious cost. Unlike the |
|
I'm making a compiler in Go that needs to parse ASTs of type declaration files. Is there a way to do that from Go? |
|
I tested the current project-backed native AST against the published After mechanical compatibility adapters, 7/7 valid fixtures convert strictly to ESTree, including JS, JSX, comments, generics, This narrows the syntax-only tooling request considerably. We do not appear to need the full legacy compiler API or matching numeric enum values. The remaining blocker is one isolated parsing operation: api.parseSourceFile({
fileName,
text,
scriptKind,
languageVersion,
setParentNodes: true,
})
// -> { sourceFile, diagnostics }The operation would:
An optional batch form could reduce IPC overhead: api.parseSourceFiles(requests)The project-backed model in https://github.com/microsoft/typescript-go/issues/2824 is appropriate for typed linting and semantic tools. It does not replace the syntax-only Executable evidence and the detailed minimal proposal are here:
Would an isolated |
|
I have a stronger downstream integration result to add to the earlier isolated-parser proposal. The previous experiment proved that the project-backed native parse(sourceText, options);
parseAndGenerateServices(sourceText, options);I loaded Results:
Evidence:
This narrows the required integration surface further: the syntax-only public parser APIs do not need redesign. They currently reach one replaceable runtime dependency, The temporary-project facade is not proposed as the solution—it still creates an API process, project, and |
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
How will the API work? Will developers have a canonical JavaScript-based API for integrating with this new version of TypeScript?
While we are porting most of the existing TypeScript compiler and language service, that does not necessarily mean that all APIs will be ported over. Because of the challenges between language runtime interoperability, API consumers will typically not communicate within the same process. Instead, we expect our API to leverage a message-passing scheme, typically over an IPC layer. Because this sort of bridging is not "free", exposing all functionality will be impractical. We expect to have a more curated API that is informed by critical use-cases (e.g. linting, transforms, resolution behavior, language service embedding, etc.).
We knew that providing a solid API would be a big challenge, so as soon as we started exploring the TypeScript port, we investigated the possibilities here. Beyond how capable the API would be, we asked ourselves whether the sorts of use-cases would be constrained by the performance of IPC solutions. More specifically, even if we could come up with a set of APIs that did what users wanted, would the throughput of requests be a limiting factor? Would the new TypeScript be fast enough to offset the cost of serialization and data transfer? Would the chattiness of API usage overwhelm the wins of having a much faster compiler?
We've become increasingly confident and optimistic in answers around the IPC performance. Kat Marchán (@zkat) has built a Node native module to use synchronous communication over standard I/O between external processes. Building on that, Andrew Branch (@andrewbranch) has experimented with exposing an API server entrypoint to our compiler, along with a JavaScript client that "speaks" to the server over that communication layer. What we've found is fairly promising - while IPC overhead is not entirely negligible, it is small enough. We also can imagine opportunities to optimize, use other underlying IPC strategies, and provide batch-style APIs to minimize call overhead.
As our experiments solidify, we will post more concrete details on our plans, and what the API will look like.
All reactions