mirror of https://github.com/hanzo-ds/sqlparser.git synced 2026-10-30 20:53:50 +00:00

Files

T

zeekay 3583e59e56 debrand: rename the CLI, docs and fixture content the server does not define

This repo was previously a module-path-only rebrand; the CLI, build and docs
still described the upstream project.

The binary is now sqlparser. Makefile's PKG_FILES sed still stripped the
UPSTREAM module prefix, so it matched nothing and left full import paths in the
coverage list. main.go dropped its vendor import alias, and the package no
longer shadows itself.

README described the upstream project end to end: its CI, coverage and report
card badges, its install path, and a brew formula we do not publish. Rewritten
for this fork, and its Go examples were compiled and run rather than hand
written.

Fixture content that is merely arbitrary — a greeting string literal and a
table name — was renamed, and the golden files REGENERATED with -update rather
than edited, because they encode character offsets that shift with the text.

Two classes of fixture content are deliberately left alone, and CI records why:
ReplicatedMergeTree '/clickhouse/tables/...' ZooKeeper paths are persisted
state (our own production server stores replicas under exactly that prefix),
and SOURCE(CLICKHOUSE(...)) is a keyword in the server's dictionary grammar
that this parser must accept. Renaming either would make the parser reject
real SQL.

2026-07-26 11:55:25 -07:00

9.8 KiB

Raw Permalink Blame History

Datastore SQL Parser

The goal of this project is to build a Datastore SQL parser in Go with the following key features:

Parse Datastore SQL into AST
Beautify Datastore SQL format

This project is inspired by memefish which is a SQL parser for Spanner in Go.

How to use

You can use it as your Go library or CLI tool, see the following examples:

Use sqlparser as a Go library

package main

import (
    "github.com/hanzo-ds/sqlparser/parser"
)

query := "SELECT * FROM events"
p := parser.NewParser(query)
// Parse query into AST
statements, err := p.ParseStmts()
if err != nil {
    return nil, err
}

Install sqlparser as a CLI tool

On Linux:

$ go install github.com/hanzo-ds/sqlparser@latest

On macOS:

Parse Datastore SQL into AST or beautify Datastore SQL format:

## Parse query into AST
$ sqlparser "SELECT * FROM events WHERE a=100"

## Beautify query
$ sqlparser -format "SELECT * FROM events WHERE a=100"

## Parse query from file
$ sqlparser -file ./test.sql

Parsed tree(AST) back into a SQL statement

p := parser.NewParser("SELECT * FROM events")
// Parse query into AST
statements, err := p.ParseStmts()
if err != nil {
    return nil, err
}

// Call the String method to unparsed AST into a SQL string
for _, stmt := range statements {
  fmt.Println(stmt.String())
}

AST Traversal

Walk Pattern (Recommended)

The Walk pattern provides a simple and efficient way to traverse AST nodes. Use the Walk function to visit all nodes in the AST:

import (
    "github.com/hanzo-ds/sqlparser/parser"
)

p := parser.NewParser("SELECT * FROM table WHERE id = 1")
statements, err := p.ParseStmts()
if err != nil {
    return err
}

// Walk through all nodes in the AST
parser.Walk(statements[0], func(node parser.Expr) bool {
    fmt.Printf("Node type: %T\n", node)
    return true // return false to stop traversal for this subtree
})

Walk Pattern Functions

Walk(node Expr, fn WalkFunc) - Traverses all nodes in depth-first order
WalkWithBreak(node Expr, fn WalkFunc) - Allows early termination of traversal
Find(root Expr, predicate func(Expr) bool) - Finds the first node matching a condition
FindAll(root Expr, predicate func(Expr) bool) - Finds all nodes matching a condition
Transform(root Expr, transformer func(Expr) Expr) - Applies transformations to nodes

Examples

Find all table identifiers:

tables := parser.FindAll(stmt, func(node parser.Expr) bool {
    _, ok := node.(*parser.TableIdentifier)
    return ok
})

Find the first WHERE clause:

whereClause, found := parser.Find(stmt, func(node parser.Expr) bool {
    _, ok := node.(*parser.WhereClause)
    return ok
})

Update test assets

For the files inside output and format dir are generated by the test cases,

if you want to update them, you can run the following command:

$ make update_test

Benchmarks

go test -bench=. -benchmem ./parser

Results

$ go test -bench=. -benchmem ./parser
goos: linux
goarch: amd64
pkg: github.com/hanzo-ds/sqlparser/parser
cpu: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
BenchmarkParseSQLFiles/access_tuple_with_dot.sql-28                23294             58467 ns/op           13448 B/op        293 allocs/op
BenchmarkParseSQLFiles/query_with_expr_compare.sql-28              43560             25704 ns/op            6240 B/op        132 allocs/op
BenchmarkParseSQLFiles/select_cast.sql-28                          75055             16518 ns/op            4648 B/op         92 allocs/op
BenchmarkParseSQLFiles/select_column_alias_string.sql-28          499798              2785 ns/op             704 B/op         13 allocs/op
BenchmarkParseSQLFiles/select_expr.sql-28                         488187              2448 ns/op             696 B/op         12 allocs/op
BenchmarkParseSQLFiles/select_item_with_modifiers.sql-28           54124             23305 ns/op            6232 B/op        136 allocs/op
BenchmarkParseSQLFiles/select_order_by_timestamp.sql-28           232302              5809 ns/op            1368 B/op         28 allocs/op
BenchmarkParseSQLFiles/select_simple.sql-28                        30602             50022 ns/op            9920 B/op        216 allocs/op
BenchmarkParseSQLFiles/select_simple_field_alias.sql-28           178126              6316 ns/op            1712 B/op         39 allocs/op
BenchmarkParseSQLFiles/select_simple_with_bracket.sql-28           71902             16172 ns/op            3824 B/op         85 allocs/op
BenchmarkParseSQLFiles/select_simple_with_cte_with_column_aliases.sql-28                   67050             19916 ns/op            4648 B/op        101 allocs/op
BenchmarkParseSQLFiles/select_simple_with_group_by_with_cube_totals.sql-28                107047             10348 ns/op            2768 B/op         58 allocs/op
BenchmarkParseSQLFiles/select_simple_with_is_not_null.sql-28                               55285             21957 ns/op            5224 B/op        111 allocs/op
BenchmarkParseSQLFiles/select_simple_with_is_null.sql-28                                   66648             22412 ns/op            4728 B/op        102 allocs/op
BenchmarkParseSQLFiles/select_simple_with_top_clause.sql-28                               269910              4166 ns/op            1088 B/op         22 allocs/op
BenchmarkParseSQLFiles/select_simple_with_with_clause.sql-28                               58494             18417 ns/op            5144 B/op        109 allocs/op
BenchmarkParseSQLFiles/select_table_alias_without_keyword.sql-28                          121261             10011 ns/op            2896 B/op         65 allocs/op
BenchmarkParseSQLFiles/select_table_function_with_query.sql-28                             98017             14929 ns/op            4168 B/op         81 allocs/op
BenchmarkParseSQLFiles/select_when_condition.sql-28                                       220394              5457 ns/op            1304 B/op         28 allocs/op
BenchmarkParseSQLFiles/select_with_distinct.sql-28                                        172948              6531 ns/op            1560 B/op         33 allocs/op
BenchmarkParseSQLFiles/select_with_join_only.sql-28                                       286346              5594 ns/op            1520 B/op         32 allocs/op
BenchmarkParseSQLFiles/select_with_left_join.sql-28                                        88200             13627 ns/op            3880 B/op         75 allocs/op
BenchmarkParseSQLFiles/select_with_literal_table_name.sql-28                              241094              5099 ns/op            1304 B/op         27 allocs/op
BenchmarkParseSQLFiles/select_with_multi_join.sql-28                                       44700             31964 ns/op            8240 B/op        188 allocs/op
BenchmarkParseSQLFiles/select_with_multi_line_comment.sql-28                              363499              4460 ns/op             824 B/op         18 allocs/op
BenchmarkParseSQLFiles/select_with_multi_union.sql-28                                     146233              7827 ns/op            2176 B/op         36 allocs/op
BenchmarkParseSQLFiles/select_with_number_field.sql-28                                    129945              8746 ns/op            2352 B/op         51 allocs/op
BenchmarkParseSQLFiles/select_with_query_parameter.sql-28                                  33850             36346 ns/op            9936 B/op        209 allocs/op
BenchmarkParseSQLFiles/select_with_string_expr.sql-28                                     142882              7530 ns/op            1880 B/op         34 allocs/op
BenchmarkParseSQLFiles/select_with_union_distinct.sql-28                                  147031              9601 ns/op            2352 B/op         47 allocs/op
BenchmarkParseSQLFiles/select_with_variable.sql-28                                        179158              6669 ns/op            1880 B/op         36 allocs/op
BenchmarkParseSQLFiles/select_with_window_function.sql-28                                  54925             31320 ns/op            6720 B/op        136 allocs/op
BenchmarkParseSQLFiles/select_with_placeholder.sql-28                                    196771              5145 ns/op            1272 B/op         26 allocs/op
BenchmarkParseSQLFiles/set_simple.sql-28                                                  172419              7062 ns/op            2480 B/op         49 allocs/op
BenchmarkParseComplexQueries/testdata/query/select_with_multi_join.sql-28                  39056             36897 ns/op            8240 B/op        188 allocs/op
BenchmarkParseComplexQueries/testdata/query/select_with_window_function.sql-28             47629             29916 ns/op            6720 B/op        136 allocs/op
BenchmarkParseComplexQueries/testdata/query/select_simple_with_with_clause.sql-28                  69210             19731 ns/op            5144 B/op        109 allocs/op
BenchmarkParseComplexQueries/testdata/query/select_with_left_join.sql-28                           74576             15338 ns/op            3880 B/op         75 allocs/op
BenchmarkParseComplexQueries/testdata/benchdata/posthog_huge_0.sql-28                                235           6231253 ns/op         1236189 B/op      26696 allocs/op
BenchmarkParseComplexQueries/testdata/benchdata/posthog_huge_1.sql-28                                279           4438280 ns/op         1043374 B/op      22717 allocs/op
PASS
ok      github.com/hanzo-ds/sqlparser/parser       66.547s

Contact us

Feel free to open an issue or discussion if you have any issues or questions.

9.8 KiB Raw Permalink Blame History