Skip to main content

A simple generic string tokenizer written in TypeScript

This package works with Bun, Cloudflare Workers, Node.js, Deno, Browsers
JSR Score
100%
Published
a month ago (1.0.3)
v
default
No documentation available
v
Matchers

A collection of common token matchers. These can be used as examples on how to use RegEx in token matching.

T
Token

The token data structure. The line and column always start from 1.

f
token_iterator

Cretaes a generator for iterating over tokens.

f
tokenize

Parses the text and collects the found tokens into a list and returns it.

T
TokenMatcher

Specifies how a token is matched. The token is always matched from the beginning of the string (+ current index).

T
TokenMatcherFunction

A function that returns the matched string that the text parameter starts with. Should return null if there is no match.

T
TokenSpecification

Specifies all tokenizer rules to be used on an input string. I.e. an object where the keys are token types, and values token matchers.

c
TokenStream

Creates a stream of tokens that can be read with the next method.