Files
CPunisherandGitHub a3d2ddd369 refactor(es/parser): Detach swc_ecma_parser from swc_ecma_lexer (#11148)
**Background:**

In https://github.com/swc-project/swc/pull/10377 and
https://github.com/swc-project/swc/pull/10399, we create another crate
`swc_ecma_lexer` from `swc_ecma_parser`.
1. The main goal is that we want to split `TokenKind` and `TokenValue`
to make lexer and parser run faster. This has been done in
`swc_ecma_parser`, whose `Token` is only 1 byte. This also means we
should refactor the lexer and the parser.
2. SWC always made `Lexer` and `Token` public, so the change of `Token`
will introduce a large breaking for rust users. So for compatibility, we
have to keep the legacy set of lexer and parser to produce compatible
`Token`s.
3. However, I made a wrong decision in
https://github.com/swc-project/swc/pull/10399. That is, we separate the
common lexer/parser functions as much as possible in the
`swc_ecma_lexer/common` by introducing a set of complexy `ParseTrait`,
`LexerTrait`, etc, which makes the project chaotic and less
comprehensive. You can see that `swc_ecma_parser` depends on
`swc_ecma_lexer` and calls the common functions everywhere.

**Motivation:**
1. Apparently it makes DX better. You don't need to jump to empty trait
functions before jump to their implementations. You don't need to jump
cross two crates forward and back.
2. It also improve the space for performance optimization because you
won't be restricted by the trait or the legacy lexer and parser.

**Description:**

Now it's time to correct the decision. This pr makes `swc_ecma_parser`
self-contained and doesn't depends `swc_ecma_lexer` any more. On the
contrary, this pr makes `swc_ecma_lexer` depends on some
`swc_ecma_parser` instead such as some common simple data structure like
`Syntax`. For compatibility, I also move and import legacy `Token` in
`swc_ecma_parser`.

**After this pr the `swc_ecma_lexer` is nearly marked as no longer
maintained. All the bug fixes and performance optimization should only
be applied in `swc_ecma_parser`.**

Specifically, what I do in this pr is only copy all common functions
from `swc_ecma_lexer/common` to `swc_ecma_parser` and eliminate the
trait-based generics. For example:

```rs
// Before
// crates/swc_ecma_lexer/common/...
pub trait Parser {
   fn xxx();
   fn yyy() { ... }
}

pub fn parse_xx<P: Parser>(p: &mut P) { ... }

// Impl for both legacy Parser and new performant Parser
impl common::Parser for Parser { ... }
```

```rs
// After
// crates/swc_ecma_parser/...
impl Parser {
   pub fn parse_xx(&mut self) { // Copy and paste the code }
}
```

Note that I nearly doesn't change anything in `swc_ecma_lexer` so the
lexer and parser in that crate are still based on trait and common
function.

**Breaking Changes:**

If you don't use `Token` API, then there's no breaking changes, which
means for most rust api users, there's no breaking change. Otherwise you
may need to remove the dependencies of `swc_ecma_lexer` and related
imports of traits.

Test in community crates:
- [rspack](https://github.com/web-infra-dev/rspack): Remove
`swc_ecma_lexer` and related imports of traits.
- [deno_ast](https://github.com/denoland/deno_ast): No breaking changes 
- [dprint-swc-ext](https://github.com/dprint/dprint-swc-ext): No
breaking changes

**Future Works:**

Actually when I finished copying all the code, the performance got
regression. It takes me lots of time to figure it out but I finally keep
the regression around -1%. So I have to do some complex optimization
ahead of time such as refactor of `parse_subscripts`.
2025-10-27 19:31:36 +09:00

93 lines
2.6 KiB
Rust

use std::{ffi::OsStr, path::Path};
use swc_common::{
errors::{ColorConfig, Handler},
sync::Lrc,
SourceMap,
};
use swc_ecma_ast::Program;
use swc_ecma_parser::{lexer::Lexer, EsSyntax, Parser, StringInput, Syntax, TsSyntax};
fn main() {
let cm: Lrc<SourceMap> = Default::default();
let handler = Handler::with_tty_emitter(ColorConfig::Auto, true, false, Some(cm.clone()));
// read file path from command line argument or use a default
let file_path = std::env::args()
.nth(1)
.unwrap_or_else(|| "test.js".to_string());
let file_path = Path::new(&file_path);
let ext = file_path.extension().map(OsStr::to_ascii_lowercase);
let is_ts = ext.as_ref().is_some_and(|e| {
e == "ts" || e == "tsx" || e == "mts" || e == "cts" || e == "mtsx" || e == "ctsx"
});
let is_d_ts = is_ts
&& file_path
.file_name()
.and_then(OsStr::to_str)
.is_some_and(|s| {
let mut iter = s.rsplit('.');
if iter.next() != Some("ts") {
return false;
}
iter.next() == Some("d") || iter.next() == Some("d")
});
let is_jsx = ext
.as_ref()
.is_some_and(|e| e == "jsx" || e == "tsx" || e == "mjsx" || e == "cjsx");
let is_esm = ext
.as_ref()
.is_some_and(|e| e == "mjs" || e == "mts" || e == "mjsx" || e == "mtsx");
let is_commonjs = ext
.as_ref()
.is_some_and(|e| e == "cjs" || e == "cts" || e == "cjsx" || e == "ctsx");
let fm = cm.load_file(file_path).expect("failed to load test.ts");
let syntax = if is_ts {
Syntax::Typescript(TsSyntax {
tsx: is_jsx,
dts: is_d_ts,
..Default::default()
})
} else {
Syntax::Es(EsSyntax {
jsx: is_jsx,
..Default::default()
})
};
let lexer = Lexer::new(syntax, Default::default(), StringInput::from(&*fm), None);
let mut parser = Parser::new_from(lexer);
let program = if is_esm {
parser
.parse_module()
.map(Program::Module)
.map_err(|e| e.into_diagnostic(&handler).emit())
} else if is_commonjs {
parser
.parse_commonjs()
.map(Program::Script)
.map_err(|e| e.into_diagnostic(&handler).emit())
} else {
parser
.parse_program()
.map_err(|e| e.into_diagnostic(&handler).emit())
};
for e in parser.take_errors() {
e.into_diagnostic(&handler).emit();
}
let program = program.expect("Failed to parse program.");
eprintln!("Parsed program:\n {program:#?}");
}