support for top level await

[buttercubz]

I am currently working with deno which has top level support, is there any way to use esbuild with top level await?

[evanw]

Sorry, top-level await is not supported. It messes with a lot of things and adding support for it is quite complicated. It likely won't be supported for a long time.

[guybedford]

Out of interest / for the issue history, is it possible to give some idea of the sort of issues that arise here?

…

On Sun, Jul 12, 2020 at 14:36 Erick Sosa Garcia ***@***.***> wrote: Closed #253 <#253>. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#253 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAESFSS2NEUIRROFVJVY7GLR3IUGHANCNFSM4OX6XTIQ> .

[evanw]

Right now there is an equivalence in esbuild between CommonJS modules and ES6 modules. A module can be written in either syntax and can even mix both syntaxes. That no longer works with CommonJS because the implementation of require() can't suspend execution in JavaScript. So I'll have to figure out something for that. It's probably some form of conditionally disabling features along the module dependency graph when a module contains a top-level await.

I'd like to punt this feature until later because there are already a lot of edge cases to work through with all of the features added so far (tree shaking, scope hoisting, code splitting, star re-exports, etc.) and this feature adds another dimension to the test case matrix. This feature is also lower priority for me than other features such as watch mode, plugins, finishing code splitting, and adding additional CSS and HTML file types, especially given that top-level await is a new feature and not yet widely used.

Let's keep this issue open because I'm sure it'll come up again. At the very least esbuild should be able to parse top-level await and pass it through in non-bundling mode, which will be useful for people using esbuild as a transformer and not a bundler.

I haven't added parsing support for top-level await yet because it's not clear exactly how to parse it. The way it works the JavaScript spec is that you're supposed to know beforehand whether the parsing goal is "script" or "module" and then only parse the await operator with the "module" goal. However, esbuild doesn't work that way because it makes no distinction between the two goals.

TypeScript is another tool that does this so I've been waiting to see how they handle things. When I looked into this last they had a pretty severe bug so I'm going to wait a while longer for their implementation to settle before I consider how to approach this in esbuild. It looks like they still have open bugs on this such as this one, which interestingly appears to require a symbol table to parse correctly kind of like C. Not sure how I feel about that approach.

Also, while I'm on the topic, apparently this feature also requires supporting top-level for await loops. Just adding that as a note to myself for when I get to implementing this.

[guybedford]

That no longer works with CommonJS because the implementation of require() can't suspend execution in JavaScript

This was one of the major issues that the Node.js integration of ES modules had to deal with. The result was we did not permit CJS modules to load ES modules although bundlers like Webpack do. I believe previous dicussions here for eg Webpack have been around allowing require of ES modules so long as they do not use top-level await. Because of the timeline for adoption of top-level await, we may actually be ok here in terms of real world usage.

I haven't added parsing support for top-level await yet because it's not clear exactly how to parse it. The way it works the JavaScript spec is that you're supposed to know beforehand whether the parsing goal is "script" or "module" and then only parse the await operator with the "module" goal. However, esbuild doesn't work that way because it makes no distinction between the two goals.

This was something else the Node.js modules integration had to carefully define with "type": "module" package.json boundaries. Mainly for the strict-mode-by-default handling for modules without module syntax, but also for other reasons like this too.

I would avoid trying to follow anything TypeScript does personally as they have made it clear they will not lead the ecosystem but instead follow the winning conventions when they emerge. As such TypeScript is a typical friction point in modules workflows these days.

One of the other implementation difficulties is that when bundling parallel dependencies, they are supposed to behave like a Promise.all, but in theory that is just a matter of wrapping.

Definitely agreed this is a low priority, but thanks for the detailed explanation and for keeping this open for future reference, however long that might be.

[evanw]

I'm planning to add pass-through support for this now that node is enabling it by default. It won't work while bundling but it will still be useful for certain use cases such as using esbuild as a library for converting TypeScript to JavaScript.

TypeScript is another tool that does this so I've been waiting to see how they handle things.

It looks like TypeScript's implementation ended up being to do an initial parse where top-level await is an identifier, then to reparse all expressions containing those identifiers as await expressions instead if the AST has an import or export statement. This approach isn't a good fit for esbuild's parser for many reasons including lack of a persistent token stream and the fact that parse errors are streamed in real-time and cannot be un-emitted. It's also a very complex approach because it may involve rewriting subsequent expressions (e.g. await \n foo; becomes await foo; instead of await; foo;). And it seems too weird to me to not be able to use await import() in a file without having to add export {} like you have to in TypeScript.

I've decided to just try unconditionally parsing the await operator at the top-level. I'm guessing that the number of modules in the wild that actually expect await to be an available top-level identifier is very small and the benefit of supporting top-level await syntax far outweighs making those modules work, especially for a modern tool like esbuild.

[MylesBorins]

@evanw an added benefit is that TLA is only supported in the module goal and await is a reserved keyword in the module goal 🎉

[Soremwar]

Note that while transforming code containing top-level await is supported, bundling code containing top-level await is not yet supported.

:(

Is the situation for this still the same?

[evanw]

Is the situation for this still the same?

Yes, top-level await still doesn't work with bundling.

[evanw]

TLA is so complicated. I'm still trying to figure out how to support it but it has been really challenging. I'm trying to match all of the real evaluation order rules including parallel evaluation and microtask ordering while still doing scope hoisting and my attempts keep failing. I might need to undo some of esbuild's existing optimizations to get this in.

One of the other implementation difficulties is that when bundling parallel dependencies, they are supposed to behave like a Promise.all, but in theory that is just a matter of wrapping.

It would be amazing if that actually worked but joining together modules using promises causes things to evaluate in an incorrect order. I have written a TLA correctness fuzzer to verify that a TLA bundling strategy is correct by comparing it against V8, which I assume is correct. The only strategy that I've gotten to work is manually creating a module registry and using it to track which dependencies are remaining for each module. Presumably this is how the runtime works as well. But that's too much generated code size and run-time performance overhead for what I'm going for.

I haven't quite gotten to the bottom of exactly what's so hard about this, but I have a hunch that the problem might be a fundamental difference between how the spec works and how promises work. I think it might be something like "the spec adds dependencies as modules are evaluated but the Promise API adds dependencies at the time .then() is called." I'm still investigating this.

FWIW I couldn't get Webpack or SystemJS to match V8's behavior either. Does anyone know if there is a bundler-like implementation of TLA that is able to match V8's behavior? Also before I get even deeper into this, does anyone know if V8's behavior is incorrect somehow?

[evanw]

Small update: It turns out that the top-level await specification and/or V8 have some subtle bugs that cause undesirable behavior. More details are here: evanw/tla-fuzzer#1. Many thanks to @sokra on the Webpack team for getting to the bottom of these issues.