LossyTokenizer implements TokenizerInterface

FinalYes

A lossy tokenizer implementation: Do not create invalid lines, do not create empty lines, do not create token line and column positions.

This tokenizer creates a much smaller streams of only relevant lines. All information not essential for the AstBuilder is skipped. This tokenizer is used in frontend rendering for quicker AST building.

An instance of this tokenizer is injected by DI when injecting TokenizerInterface.

This class is unit test covered by TokenizerInterfaceTest and paired with LossyTokenizer. Never change anything in this class without additional test coverage!

Tags
internal:

Internal tokenizer structure.

Attributes
#[AsAlias]
\TYPO3\CMS\Core\TypoScript\Tokenizer\TokenizerInterface::class

Table of Contents

Interfaces

TokenizerInterface
A lossless tokenizer for TypoScript syntax.

Methods

tokenize()  : LineStream

Methods


        
On this page

Search results