mirror of
https://github.com/hanzoai/swc.git
synced 2026-10-30 19:42:53 +00:00
**Background:** In https://github.com/swc-project/swc/pull/10377 and https://github.com/swc-project/swc/pull/10399, we create another crate `swc_ecma_lexer` from `swc_ecma_parser`. 1. The main goal is that we want to split `TokenKind` and `TokenValue` to make lexer and parser run faster. This has been done in `swc_ecma_parser`, whose `Token` is only 1 byte. This also means we should refactor the lexer and the parser. 2. SWC always made `Lexer` and `Token` public, so the change of `Token` will introduce a large breaking for rust users. So for compatibility, we have to keep the legacy set of lexer and parser to produce compatible `Token`s. 3. However, I made a wrong decision in https://github.com/swc-project/swc/pull/10399. That is, we separate the common lexer/parser functions as much as possible in the `swc_ecma_lexer/common` by introducing a set of complexy `ParseTrait`, `LexerTrait`, etc, which makes the project chaotic and less comprehensive. You can see that `swc_ecma_parser` depends on `swc_ecma_lexer` and calls the common functions everywhere. **Motivation:** 1. Apparently it makes DX better. You don't need to jump to empty trait functions before jump to their implementations. You don't need to jump cross two crates forward and back. 2. It also improve the space for performance optimization because you won't be restricted by the trait or the legacy lexer and parser. **Description:** Now it's time to correct the decision. This pr makes `swc_ecma_parser` self-contained and doesn't depends `swc_ecma_lexer` any more. On the contrary, this pr makes `swc_ecma_lexer` depends on some `swc_ecma_parser` instead such as some common simple data structure like `Syntax`. For compatibility, I also move and import legacy `Token` in `swc_ecma_parser`. **After this pr the `swc_ecma_lexer` is nearly marked as no longer maintained. All the bug fixes and performance optimization should only be applied in `swc_ecma_parser`.** Specifically, what I do in this pr is only copy all common functions from `swc_ecma_lexer/common` to `swc_ecma_parser` and eliminate the trait-based generics. For example: ```rs // Before // crates/swc_ecma_lexer/common/... pub trait Parser { fn xxx(); fn yyy() { ... } } pub fn parse_xx<P: Parser>(p: &mut P) { ... } // Impl for both legacy Parser and new performant Parser impl common::Parser for Parser { ... } ``` ```rs // After // crates/swc_ecma_parser/... impl Parser { pub fn parse_xx(&mut self) { // Copy and paste the code } } ``` Note that I nearly doesn't change anything in `swc_ecma_lexer` so the lexer and parser in that crate are still based on trait and common function. **Breaking Changes:** If you don't use `Token` API, then there's no breaking changes, which means for most rust api users, there's no breaking change. Otherwise you may need to remove the dependencies of `swc_ecma_lexer` and related imports of traits. Test in community crates: - [rspack](https://github.com/web-infra-dev/rspack): Remove `swc_ecma_lexer` and related imports of traits. - [deno_ast](https://github.com/denoland/deno_ast): No breaking changes - [dprint-swc-ext](https://github.com/dprint/dprint-swc-ext): No breaking changes **Future Works:** Actually when I finished copying all the code, the performance got regression. It takes me lots of time to figure it out but I finally keep the regression around -1%. So I have to do some complex optimization ahead of time such as refactor of `parse_subscripts`.