Tokenizes a document that may use Triplate templating, opt-in. Default host lexing
is unaffected: a non-template document is tokenized exactly as lexer.tokenize(text)
would, and Triplate token types never appear in any host token list.
For a template, the host parser should consume TriplateLexingResult.parseTokens
(placeholders) while every other consumer uses TriplateLexingResult.tokens
(the faithful stream). The frontmatter's prefixed-name/IRI/literal values are surfaced as real RDF
tokens (via triplate's positioned symbols) so tooltips and prefix rename work inside it; if
triplate reports nothing usable it falls back to a single opaque TRIPLATE_FRONTMATTER block.
Tokenizes a document that may use Triplate templating, opt-in. Default host lexing is unaffected: a non-template document is tokenized exactly as
lexer.tokenize(text)would, and Triplate token types never appear in any host token list.For a template, the host parser should consume TriplateLexingResult.parseTokens (placeholders) while every other consumer uses TriplateLexingResult.tokens (the faithful stream). The frontmatter's prefixed-name/IRI/literal values are surfaced as real RDF tokens (via
triplate's positioned symbols) so tooltips and prefix rename work inside it; iftriplatereports nothing usable it falls back to a single opaqueTRIPLATE_FRONTMATTERblock.