‪TYPO3CMS  ‪main
TYPO3\CMS\Core\TypoScript\Tokenizer\TokenizerInterface Interface Reference
Inheritance diagram for TYPO3\CMS\Core\TypoScript\Tokenizer\TokenizerInterface:
TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer

Public Member Functions

 tokenize (string $source)

Detailed Description

A lossless tokenizer for TypoScript syntax.

tokenize() creates a stream of LineInterface objects from a TypoScript string, each line contains the important streams or tokens of a single line.

There are two tokenizer implementations:

  • LossyTokenizer: This one skip all invalid lines and comments and everything that is not needed for AST building.
  • LosslessTokenizer: This one creates a stream of lines useful for backend template module to elaborate on details and failures in TypoScript.

The tokenizer does not parse conditions or includes itself (no file / db lookups), this is part of the IncludeTree parser.

: Internal tokenizer structure.

Definition at line 39 of file TokenizerInterface.php.

Member Function Documentation

◆ tokenize()