The available implementations of Token
are primarily guided by the implementation of the tokenization
and the needs of the parser and included visitors. It does not distinguish all types of tokens. For example there is
QuotedIdentifierToken
because the tokenization needs handling for
quoted identifiers, while a normal identifier is a GenericToken
, because that is handled by the fallback
tokenization after checking for all other types. On the other hand, open and close curly braces, square brackets and
parentheses each have their own type, as the parser may need this to find nested contexts.
- Since:
- 5
- Author:
- Mark Rotteveel
-
Method Summary
Modifier and TypeMethodDescriptiondefault void
Appends the current token to the supplied String builder.default boolean
equalsIgnoreCase
(String tokenText) Case-insensitive equality of this tokens text using an equivalent ofString.equalsIgnoreCase(String)
.default boolean
Detects if the token is valid as an identifier (ignoring length constraints).default boolean
int
length()
Token text length.int
position()
Token position.text()
Token text.default CharSequence
Token text asCharSequence
.
-
Method Details
-
text
String text()Token text.- Returns:
- the text of the token; this is the original text from the source
- See Also:
-
textAsCharSequence
Token text asCharSequence
.Default implementation returns
text()
. As an optimization, implementations may return their containedCharSequence
to avoid unnecessary conversion to string.- Returns:
- the text of the token; this is the original text from the source
- See Also:
-
appendTo
Appends the current token to the supplied String builder.- Parameters:
sb
- String builder to append to
-
position
int position()Token position.- Returns:
- 0-based position of the occurrence of this token in the source (the first character)
-
length
int length()Token text length.- Returns:
- Length of the token text
-
isWhitespaceOrComment
default boolean isWhitespaceOrComment()- Returns:
true
if this token is whitespace or a comment,false
for all other tokens
-
equalsIgnoreCase
Case-insensitive equality of this tokens text using an equivalent ofString.equalsIgnoreCase(String)
.- Parameters:
tokenText
- Token text to compare- Returns:
true
iftokenText
is equal - ignoring case - to the text of this token,false
otherwise
-
isValidIdentifier
default boolean isValidIdentifier()Detects if the token is valid as an identifier (ignoring length constraints).This will always return
false
forReservedToken
or other specialised tokens (e.g.OperatorToken
withIS
orLIKE
) that can't occur as an identifier.- Returns:
true
if the token is valid as an identifier,false
otherwise
-