LosslessTokenizer

final class \TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer

Implements
TokenizerInterface

A lossless tokenizer for TypoScript syntax.

tokenize() creates a flat stream of tokens from a TypoScript string. It is lossless and never "looses" characters to allow syntax linting and creating linter-fixed source strings: tokenize() to create a TokenStream and using string cast (__toString()) on that stream creates the same source string again.

The tokenizer does not parse conditions or includes itself (no file / db lookups), this is part of the IncludeTree parser.

This class is unit test covered by TokenizerInterfaceTest and paired with LossyTokenizer. Never change anything in this class without additional test coverage!

Tags

internal:

Internal tokenizer structure.

Table of Contents

Interfaces

TokenizerInterface
A lossless tokenizer for TypoScript syntax.

Methods

tokenize()  : LineStream

Methods

On this page