The Book

Class: Scanner

Base~ Scanner

Wraps a function which tokenizes a string. Token patterns should not partially overlap literals, e.g., /[a-z]+/ would conceal 'formula1'.

new Scanner(factory, skip [, terminals])

Creates the pattern used to tokenize a string; see factory method grammar.scanner().

Parameters:
Name Type Argument Description
factory module:Base~Factory

supplies literals and tokens; unused terminals and $eof and $error, if any, are ignored.
skip RegExp

a pattern to define ignorable character sequences, must not accept empty input, must not use d, g, or y flag, should not be anchored, should use (:? ) rather than ( ) for grouping.
terminals Array.<T> <optional>

ordered list corresponding to .pattern; if omitted, tokens by ascending name and then literals by decreasing length.
Properties:
Name Type Description
assert function

bound to factory.assert().
tuple function

bound to factory.tuple().
terminals Array.<module:Base~T>

ordered for pattern; first tokens ordered by ascending name then literals ordered by decreasing length.
skip RegExp

a pattern to define ignorable character sequences, should not accept empty input, should not use flags, should not be anchored, should use (:? ) rather than ( ) for grouping.
pattern RegExp

read-only, concatenates capture groups with skip and terminals, used to disect input.
Source:

Methods

escape(s)

Escapes most characters by \. or \x.. or \u..... Similar to Tuple.escape().

Parameters:
Name Type Description
s string

string to escape.
Source:
Returns:

escaped string. escape(null) // will crash escape('a') // [alphanumerics] returns string containing a escape('\b') // [controls] returns string containing \b escape('s') // [specials] returns string containing \s escape('x') // [other] returns string containing \x## or \u####

Type
string

scan(input)

Tokenizes a string.

Parameters:
Name Type Description
input string

to be divided into literals and tokens.
Source:
Returns:

a list of literals and tokens. The list contains one Tuple with a null terminal for each character sequence which is neither ignorable nor a literal or a token.

Type
Array.<module:Base~Tuple>