diff --git a/AGENTS.md b/AGENTS.md index 372447e..cc415a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,3 +40,4 @@ keep it fast: `make bench` (and pprof for anything non-trivial). Do not add allocations, map lookups or function calls to per-byte loops. Never trade hot-path performance for minor readability gains. +- In prose docs (Markdown, comments), do not hard-wrap lines mid-sentence. Put each sentence (up to its period) on its own line; do not break a line just because it reached some character count. diff --git a/readme.md b/readme.md index f51354e..db39943 100644 --- a/readme.md +++ b/readme.md @@ -1,40 +1,63 @@ -# Tokenizer +# Tokenizer [![Build Status](https://github.com/bzick/tokenizer/actions/workflows/tokenizer.yml/badge.svg)](https://github.com/bzick/tokenizer/actions/workflows/tokenizer.yml) [![codecov](https://codecov.io/gh/bzick/tokenizer/branch/master/graph/badge.svg?token=MFY5NWATGC)](https://codecov.io/gh/bzick/tokenizer) [![Go Report Card](https://goreportcard.com/badge/github.com/bzick/tokenizer?rnd=2)](https://goreportcard.com/report/github.com/bzick/tokenizer) [![GoDoc](https://godoc.org/github.com/bzick/tokenizer?status.svg)](https://godoc.org/github.com/bzick/tokenizer) -Tokenizer — parse any string, slice or infinite buffer to any tokens. +High-performance, generic tokenizer (lexer) for Go. +It parses any string, byte slice or infinite `io.Reader` stream into a stream of typed tokens. +Use it as the foundation for higher-level parsers and DSLs. -Main features: +## Features -* High performance. -* No regexp. -* Provides [simple API](https://pkg.go.dev/github.com/bzick/tokenizer). -* Supports [integer](#integer-number) and [float](#float-number) numbers. -* Supports [quoted string or other "framed"](#framed-string) strings. -* Supports [injection](#injection-in-framed-string) in quoted or "framed" strings. -* Supports unicode. -* [Customization of tokens](#user-defined-tokens). -* Autodetect white space symbols. -* Parse any data syntax (xml, [json](https://github.com/bzick/tokenizer/blob/master/example_test.go), yaml), any programming language. -* Single pass through the data. -* Parses infinite incoming data and don't panic. +- **Fast.** Single pass over the data, zero allocations per token on the hot path (token values point directly into the source buffer — no copies), lookup tables instead of regexp. +- **Simple [API](https://pkg.go.dev/github.com/bzick/tokenizer).** +- Recognizes [integer](#integer-number) and [float](#float-number) numbers. +- Recognizes [quoted (framed) strings](#framed-string) with [escaping](#framed-string) and [embedded injections](#injections-in-a-framed-string). +- [User-defined tokens](#user-defined-tokens) (operators, punctuation, keywords). +- Unicode-aware keywords. +- Configurable whitespace symbols. +- Streams infinite input without loading it fully into memory and without panicking. -Use cases: -- Parsing html, xml, [json](./example_test.go), yaml and other text formats. -- Parsing huge or infinite texts. -- Parsing any programming languages. -- Parsing templates. -- Parsing formulas. +## Use cases -For example, parsing SQL `WHERE` condition `user_id = 119 and modified > "2020-01-01 00:00:00" or amount >= 122.34`: +- Parsing text formats: XML, HTML, [JSON](./example_test.go), YAML, etc. +- Parsing huge or infinite inputs. +- Parsing programming languages, DSLs, templates and formulas. + +## Installation + +```bash +go get github.com/bzick/tokenizer +``` + +## Table of contents + +- [Quick start](#quick-start) +- [Getting started](#getting-started) + - [Create and parse](#create-and-parse) + - [Parse an infinite stream](#parse-an-infinite-stream) +- [Built-in tokens](#built-in-tokens) + - [Unknown token](#unknown-token) + - [Keywords](#keywords) + - [Integer number](#integer-number) + - [Float number](#float-number) + - [Framed string](#framed-string) + - [Injections in a framed string](#injections-in-a-framed-string) +- [User-defined tokens](#user-defined-tokens) +- [Stream API](#stream-api) +- [Known issues](#known-issues) +- [Benchmark](#benchmark) + +## Quick start + +For example, parsing the SQL `WHERE` condition `user_id = 119 and modified > "2020-01-01 00:00:00" or amount >= 122.34`: ```go import "github.com/bzick/tokenizer" -// define custom tokens keys +// define keys for the custom tokens const ( TEquality = iota + 1 TDot @@ -42,7 +65,7 @@ const ( TDoubleQuoted ) -// configure tokenizer +// configure the tokenizer parser := tokenizer.New() parser.DefineTokens(TEquality, []string{"<", "<=", "==", ">=", ">", "!="}) parser.DefineTokens(TDot, []string{"."}) @@ -50,66 +73,77 @@ parser.DefineTokens(TMath, []string{"+", "-", "/", "*", "%"}) parser.DefineStringToken(TDoubleQuoted, `"`, `"`).SetEscapeSymbol(tokenizer.BackSlash) parser.AllowKeywordSymbols(tokenizer.Underscore, tokenizer.Numbers) -// create tokens' stream +// create the token stream stream := parser.ParseString(`user_id = 119 and modified > "2020-01-01 00:00:00" or amount >= 122.34`) defer stream.Close() -// iterate over each token +// iterate over the tokens for stream.IsValid() { if stream.CurrentToken().Is(tokenizer.TokenKeyword) { field := stream.CurrentToken().ValueString() // ... + _ = field } stream.GoNext() } ``` -stream tokens: +The stream of tokens: + ``` string: user_id = 119 and modified > "2020-01-01 00:00:00" or amount >= 122.34 tokens: |user_id| =| 119| and| modified| >| "2020-01-01 00:00:00"| or| amount| >=| 122.34| - | 0 | 1| 2 | 3 | 4 | 5| 6 | 7 | 8 | 9 | 10 | + | 0 | 1| 2 | 3 | 4 | 5| 6 | 7| 8 | 9 | 10 | -0: {key: TokenKeyword, value: "user_id"} token.Value() == "user_id" -1: {key: TEquality, value: "="} token.Value() == "=" -2: {key: TokenInteger, value: "119"} token.ValueInt64() == 119 -3: {key: TokenKeyword, value: "and"} token.Value() == "and" -4: {key: TokenKeyword, value: "modified"} token.Value() == "modified" -5: {key: TEquality, value: ">"} token.Value() == ">" -6: {key: TokenString, value: "\"2020-01-01 00:00:00\""} token.ValueUnescaped() == "2020-01-01 00:00:00" -7: {key: TokenKeyword, value: "or"} token.Value() == "and" -8: {key: TokenKeyword, value: "amount"} token.Value() == "amount" -9: {key: TEquality, value: ">="} token.Value() == ">=" -10: {key: TokenFloat, value: "122.34"} token.ValueFloat64() == 122.34 + 0: {key: TokenKeyword, value: "user_id"} token.ValueString() == "user_id" + 1: {key: TEquality, value: "="} token.ValueString() == "=" + 2: {key: TokenInteger, value: "119"} token.ValueInt64() == 119 + 3: {key: TokenKeyword, value: "and"} token.ValueString() == "and" + 4: {key: TokenKeyword, value: "modified"} token.ValueString() == "modified" + 5: {key: TEquality, value: ">"} token.ValueString() == ">" + 6: {key: TokenString, value: "\"2020-01-01 00:00:00\""} token.ValueUnescapedString() == "2020-01-01 00:00:00" + 7: {key: TokenKeyword, value: "or"} token.ValueString() == "or" + 8: {key: TokenKeyword, value: "amount"} token.ValueString() == "amount" + 9: {key: TEquality, value: ">="} token.ValueString() == ">=" +10: {key: TokenFloat, value: "122.34"} token.ValueFloat64() == 122.34 ``` More examples: + - [JSON parser](./example_test.go) -## Begin +## Getting started ### Create and parse ```go import "github.com/bzick/tokenizer" -var parser := tokenizer.New() -parser.AllowKeywordSymbols(tokenizer.Underscore, []rune{}) -// ... and other configuration code - +parser := tokenizer.New() +parser.AllowKeywordSymbols(tokenizer.Underscore, tokenizer.Numbers) +// ... and any other configuration ``` -There are two ways to **parse string or slice**: +There are two ways to **parse a string or a byte slice**. Both return a `*Stream`: - `parser.ParseString(str)` - `parser.ParseBytes(slice)` -The package allows to **parse an endless stream** of data into tokens. -For parsing, you need to pass `io.Reader`, from which data will be read (chunk-by-chunk): +Always `Close()` the stream when you are done — it releases token objects back to the internal pool: + +```go +stream := parser.ParseString(`user_id = 119`) +defer stream.Close() +``` + +### Parse an infinite stream + +The tokenizer can also parse an **endless stream** of data. +Pass an `io.Reader` and a buffer size (in bytes); data is read and parsed chunk by chunk, so the whole input never needs to fit in memory: ```go fp, err := os.Open("data.json") // huge JSON file -// check fs, configure tokenizer ... +// handle err, configure the tokenizer ... stream := parser.ParseStream(fp, 4096).SetHistorySize(10) defer stream.Close() @@ -119,18 +153,24 @@ for stream.IsValid() { } ``` -## Embedded tokens +In stream mode only a window of tokens is kept in memory. +`SetHistorySize(n)` controls how many already-visited tokens remain available behind the current one (for `GoPrev`, `GoTo`, `GetSnippet`, etc.). + +## Built-in tokens -- `tokenizer.TokenUnknown` — unspecified token key. -- `tokenizer.TokenKeyword` — keyword, any combination of letters, including unicode letters. -- `tokenizer.TokenInteger` — integer value -- `tokenizer.TokenFloat` — float/double value -- `tokenizer.TokenString` — quoted string -- `tokenizer.TokenStringFragment` — fragment framed (quoted) string +Every token carries one of these built-in keys unless it matches a user-defined token: + +- `tokenizer.TokenUnknown` — a symbol that does not match any known token. +- `tokenizer.TokenKeyword` — a word: any combination of letters, including unicode letters. +- `tokenizer.TokenInteger` — an integer number. +- `tokenizer.TokenFloat` — a float/double number. +- `tokenizer.TokenString` — a quoted (framed) string. +- `tokenizer.TokenStringFragment` — a fragment of a framed string that contains injections. +- `tokenizer.TokenUndef` — returned when the stream pointer is out of range (see [Stream API](#stream-api)). ### Unknown token -A token marks as `tokenizer.TokenUnknown` if the parser detects an unknown token: +A token is marked as `tokenizer.TokenUnknown` when the parser encounters a symbol that does not match any known token: ```go stream := parser.ParseString(`one!`) @@ -138,42 +178,32 @@ stream := parser.ParseString(`one!`) ``` stream: [ - { - Key: tokenizer.TokenKeyword - Value: "One" - }, - { - Key: tokenizer.TokenUnknown - Value: "!" - } + {Key: tokenizer.TokenKeyword, Value: "one"}, + {Key: tokenizer.TokenUnknown, Value: "!"}, ] ``` -By default, `TokenUnknown` tokens are added to the stream. -Setting `tokenizer.StopOnUndefinedToken()` stops parser when `tokenizer.TokenUnknown` appears in the stream. +By default, `TokenUnknown` tokens are added to the stream. +Calling `parser.StopOnUndefinedToken()` makes the parser stop as soon as an unknown token appears: ```go +parser.StopOnUndefinedToken() stream := parser.ParseString(`one!`) ``` ``` stream: [ - { - Key: tokenizer.TokenKeyword - Value: "one" - } + {Key: tokenizer.TokenKeyword, Value: "one"}, ] ``` -Please note that if the `StopOnUndefinedToken()` setting is enabled, then the string may not be fully parsed. -To find out that the string was not fully parsed, check the length of the parsed string `stream.GetParsedLength()` -and the length of the original string. +Note that with `StopOnUndefinedToken()` enabled the input may not be fully parsed. +To detect that, compare `stream.GetParsedLength()` with the length of the original input. ### Keywords -Any word that is not a custom token is stored in a single token as `tokenizer.TokenKeyword`. - -The word can contain unicode characters, and it can be configured to contain other characters, like numbers and underscores (see `tokenizer.AllowKeywordSymbols()`). +Any word that is not a custom token is stored as a single `tokenizer.TokenKeyword`. +A keyword can contain unicode characters, and can be configured to contain additional symbols such as numbers and underscores (see `AllowKeywordSymbols`). ```go stream := parser.ParseString(`one 二 три`) @@ -181,25 +211,16 @@ stream := parser.ParseString(`one 二 три`) ``` stream: [ - { - Key: tokenizer.TokenKeyword - Value: "one" - }, - { - Key: tokenizer.TokenKeyword - Value: "二" - }, - { - Key: tokenizer.TokenKeyword - Value: "три" - } + {Key: tokenizer.TokenKeyword, Value: "one"}, + {Key: tokenizer.TokenKeyword, Value: "二"}, + {Key: tokenizer.TokenKeyword, Value: "три"}, ] ``` -Keyword may be modified with `tokenizer.AllowKeywordSymbols(majorSymbols, minorSymbols)` +Keywords can be extended with `parser.AllowKeywordSymbols(majorSymbols, minorSymbols)`: -- Major symbols (any quantity in the keyword) can be in the beginning, in the middle and at the end of the keyword. -- Minor symbols (any quantity in the keyword) can be in the middle and at the end of the keyword. +- **Major** symbols may appear anywhere in the keyword — at the beginning, in the middle and at the end. +- **Minor** symbols may appear only in the middle and at the end (not at the beginning). ```go parser.AllowKeywordSymbols(tokenizer.Underscore, tokenizer.Numbers) @@ -211,7 +232,7 @@ parser.AllowKeywordSymbols([]rune{'_', '@'}, tokenizer.Numbers) ### Integer number -Any integer is stored as one token with key `tokenizer.TokenInteger`. +Any integer is stored as one token with the key `tokenizer.TokenInteger`: ```go stream := parser.ParseString(`223 999`) @@ -219,158 +240,136 @@ stream := parser.ParseString(`223 999`) ``` stream: [ - { - Key: tokenizer.TokenInteger - Value: "223" - }, - { - Key: tokenizer.TokenInteger - Value: "999" - }, + {Key: tokenizer.TokenInteger, Value: "223"}, + {Key: tokenizer.TokenInteger, Value: "999"}, ] ``` -To get int64 from the token value use `stream.GetInt64()`: +Use `token.ValueInt64()` to get the value as `int64`: ```go -stream := tokenizer.ParseString("123") -fmt.Print("Token is %d", stream.CurrentToken().GetInt64()) // Token is 123 +stream := parser.ParseString("123") +fmt.Printf("Token is %d", stream.CurrentToken().ValueInt64()) // Token is 123 ``` +Underscores between digits (e.g. `1_000`) can be allowed with `parser.AllowNumberUnderscore()`. + ### Float number -Any float number is stored as one token with key `tokenizer.TokenFloat`. Float number may -- have point, for example `1.2` -- have exponent, for example `1e6` -- have lower `e` or upper `E` letter in the exponent, for example `1E6`, `1e6` -- have sign in the exponent, for example `1e-6`, `1e6`, `1e+6` +Any float is stored as one token with the key `tokenizer.TokenFloat`. A float may + +- have a fractional point, e.g. `1.2` +- have an exponent, e.g. `1e6` +- use lower- or upper-case `e`/`E` in the exponent, e.g. `1E6`, `1e6` +- have a sign in the exponent, e.g. `1e-6`, `1e+6` ```go -stream := parser.ParseString(`1.3e-8`): +stream := parser.ParseString(`1.3e-8`) ``` ``` stream: [ - { - Key: tokenizer.TokenFloat - Value: "1.3e-8" - }, + {Key: tokenizer.TokenFloat, Value: "1.3e-8"}, ] ``` -To get float64 from the token value use `token.GetFloat64()`: +Use `token.ValueFloat64()` to get the value as `float64`: ```go stream := parser.ParseString("1.3e2") -fmt.Print("Token is %d", stream.CurrentToken().GetFloat64()) // Token is 130 +fmt.Printf("Token is %g", stream.CurrentToken().ValueFloat64()) // Token is 130 ``` ### Framed string -Strings that are framed with tokens are called framed strings. An obvious example is quoted a string like `"one two"`. -There are quotes — edge tokens. +A string that is enclosed between two tokens is called a *framed string*. +The most common example is a quoted string such as `"one two"`, where the quotes are the frame (edge) tokens. -You can create and customize framed string through `tokenizer.DefineStringToken()`: +Define a framed string with `parser.DefineStringToken(key, startToken, endToken)`. +`SetEscapeSymbol` makes the escape character (usually a backslash) ignore the closing frame, and `AddSpecialStrings` lists the sequences that the escape character can unescape: ```go -const TokenDoubleQuotedString = 10 -// ... -parser.DefineStringToken(TokenDoubleQuotedString, `"`, `"`).SetEscapeSymbol('\\') +const TokenDoubleQuoted = 10 // ... +parser.DefineStringToken(TokenDoubleQuoted, `"`, `"`). + SetEscapeSymbol(tokenizer.BackSlash). + AddSpecialStrings([]string{`"`}) + stream := parser.ParseString(`"two \"three"`) ``` ``` stream: [ - { - Key: tokenizer.TokenString - Value: "\"two \\"three\"" - }, + {Key: tokenizer.TokenString, Value: `"two \"three"`}, ] ``` -To get a framed string without edge tokens and special characters, use the `stream.ValueUnescaped()` method: +The raw value (`token.Value()` / `token.ValueString()`) includes the frame tokens and escape symbols. +Use `token.ValueUnescapedString()` (or `token.ValueUnescaped()` for `[]byte`) to get the string without the frame tokens and with escape sequences resolved: ```go -value := stream.CurrentToken().ValueUnescaped() // result: two "three +value := stream.CurrentToken().ValueUnescapedString() // two "three ``` -The method `token.StringKey()` will be return token string key defined in the `DefineStringToken`: +`token.StringKey()` returns the key that was passed to `DefineStringToken`: ```go -if stream.CurrentToken().StringKey() == TokenDoubleQuotedString { - // true +if stream.CurrentToken().StringKey() == TokenDoubleQuoted { + // ... } ``` -### Injection in framed string +### Injections in a framed string -Strings can contain expression substitutions that can be parsed into tokens. For example `"one {{two}} three"`. -Fragments of strings before, between and after substitutions will be stored in tokens as `tokenizer.TokenStringFragment`. +A framed string can contain embedded expressions (injections) that are parsed as regular tokens, for example `"one {{ two }} three"`. +The pieces of the string before, between and after the injections are emitted as `tokenizer.TokenStringFragment` tokens (they keep the frame quote and the surrounding whitespace); the injected part is parsed with the full set of tokens. ```go const ( - TokenOpenInjection = 1 - TokenCloseInjection = 2 - TokenQuotedString = 3 + TokenOpenInjection = 1 + TokenCloseInjection = 2 + TokenQuotedString = 3 ) parser := tokenizer.New() parser.DefineTokens(TokenOpenInjection, []string{"{{"}) parser.DefineTokens(TokenCloseInjection, []string{"}}"}) -parser.DefineStringToken(TokenQuotedString, `"`, `"`).AddInjection(TokenOpenInjection, TokenCloseInjection) - -parser.ParseString(`"one {{ two }} three"`) -``` - -Tokens: - -``` -{ - { - Key: tokenizer.TokenStringFragment, - Value: "one" - }, - { - Key: TokenOpenInjection, - Value: "{{" - }, - { - Key: tokenizer.TokenKeyword, - Value: "two" - }, - { - Key: TokenCloseInjection, - Value: "}}" - }, - { - Key: tokenizer.TokenStringFragment, - Value: "three" - }, -} +parser.DefineStringToken(TokenQuotedString, `"`, `"`). + AddInjection(TokenOpenInjection, TokenCloseInjection) + +stream := parser.ParseString(`"one {{ two }} three"`) ``` -Use cases: -- parse templates -- parse placeholders +``` +stream: [ + {Key: tokenizer.TokenStringFragment, Value: `"one `}, + {Key: TokenOpenInjection, Value: "{{"}, + {Key: tokenizer.TokenKeyword, Value: "two"}, + {Key: TokenCloseInjection, Value: "}}"}, + {Key: tokenizer.TokenStringFragment, Value: ` three"`}, +] +``` -## User defined tokens +Use cases: parsing templates and placeholders. -The new token can be defined via the `DefineTokens()` method: +## User-defined tokens -```go +Custom tokens (operators, punctuation, brackets, ...) are registered with `parser.DefineTokens(key, tokens)`. +The `key` must be a positive integer; a single key can map to multiple token strings. +When several tokens can match at the same position, the longest one wins (e.g. `>=` is preferred over `>`). +```go const ( - TokenCurlyOpen = 1 - TokenCurlyClose = 2 - TokenSquareOpen = 3 - TokenSquareClose = 4 - TokenColon = 5 - TokenComma = 6 - TokenDoubleQuoted = 7 + TokenCurlyOpen TokenKey = iota + 1 + TokenCurlyClose + TokenSquareOpen + TokenSquareClose + TokenColon + TokenComma + TokenDoubleQuoted ) -// json parser +// a minimal JSON tokenizer parser := tokenizer.New() parser. DefineTokens(TokenCurlyOpen, []string{"{"}). @@ -379,19 +378,57 @@ parser. DefineTokens(TokenSquareClose, []string{"]"}). DefineTokens(TokenColon, []string{":"}). DefineTokens(TokenComma, []string{","}). - DefineStringToken(TokenDoubleQuoted, `"`, `"`).SetSpecialSymbols(tokenizer.DefaultStringEscapes) + DefineStringToken(TokenDoubleQuoted, `"`, `"`). + SetEscapeSymbol(tokenizer.BackSlash). + AddSpecialStrings(tokenizer.DefaultSpecialString) stream := parser.ParseString(`{"key": [1]}`) ``` +See [`example_test.go`](./example_test.go) for a complete JSON parser built on top of this configuration. + +## Stream API + +The `*Stream` is a bidirectional iterator over the parsed tokens. Common methods: + +| Method | Description | +| --- | --- | +| `IsValid() bool` | `true` while the pointer is on a real token. | +| `GoNext() *Stream` | Move to the next token (loads the next chunk in stream mode). | +| `GoPrev() *Stream` | Move to the previous token (bounded by `SetHistorySize`). | +| `GoTo(id int) *Stream` | Move to a token by its id. | +| `CurrentToken() *Token` | The current token (a `TokenUndef` token when out of range). | +| `NextToken() / PrevToken() *Token` | Peek without moving the pointer. | +| `GoNextIfNextIs(key, ...) bool` | Advance only if the next token matches. | +| `IsNextSequence(keys...) bool` | Check that the following tokens match a sequence. | +| `GetSnippet(before, after int) []Token` | Tokens around the current one. | +| `GetParsedLength() int` | Number of bytes parsed so far. | +| `Close()` | Release tokens back to the pool. | + +Token accessors: + +| Method | Description | +| --- | --- | +| `Key() TokenKey` | The token key. | +| `Is(key, ...keys) bool` | Whether the token matches any of the keys. | +| `Value() []byte` / `ValueString() string` | The raw value from the source. | +| `ValueInt64() int64` / `ValueFloat64() float64` | The numeric value. | +| `ValueUnescaped() []byte` / `ValueUnescapedString() string` | A framed string without frame tokens and escapes. | +| `StringKey() TokenKey` | The key defined in `DefineStringToken`. | +| `Line() int` / `Offset() int` | Position in the input (line starts at 1). | +| `Indent() []byte` | Whitespace before the token. | + +> Do not store a `*Token` returned by the stream — the underlying object may be reused as the pointer moves. +> Copy the value if you need to keep it. ## Known issues -* zero-byte `\x00` (`\0`) stops parsing. +- A zero byte (`\x00`) in the input stops parsing. ## Benchmark -Parse string/bytes +Parse a string / byte slice: + ``` pkg: tokenizer cpu: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz @@ -403,7 +440,8 @@ BenchmarkParseBytes BenchmarkParseBytes-8 158481 7358 ns/op ``` -Parse infinite stream +Parse an infinite stream: + ``` pkg: tokenizer cpu: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz