TYPO3\CMS\Core\TypoScript\Tokenizer\TokenizerInterface Interface Reference

Inheritance diagram for TYPO3\CMS\Core\TypoScript\Tokenizer\TokenizerInterface:

Public Member Functions
	tokenize (string $source)

Detailed Description

A lossless tokenizer for TypoScript syntax.

tokenize() creates a stream of LineInterface objects from a TypoScript string, each line contains the important streams or tokens of a single line.

There are two tokenizer implementations:

• ‪LossyTokenizer: This one skip all invalid lines and comments and everything that is not needed for AST building.
• ‪LosslessTokenizer: This one creates a stream of lines useful for backend template module to elaborate on details and failures in TypoScript.

The tokenizer does not parse conditions or includes itself (no file / db lookups), this is part of the IncludeTree parser.

: Internal tokenizer structure.

Definition at line 39 of file TokenizerInterface.php.

Member Function Documentation

tokenize()

TYPO3\CMS\Core\TypoScript\Tokenizer\TokenizerInterface::tokenize

(

string

$source

)

Implemented in TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer, and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer.

Referenced by TYPO3\CMS\Core\TypoScript\IncludeTree\StringTreeBuilder\getTreeFromString(), and TYPO3\CMS\Core\TypoScript\IncludeTree\TsConfigTreeBuilder\getTreeFromString().