LossyTokenizer implements TokenizerInterface
FinalYes
A lossy tokenizer implementation: Do not create invalid lines, do not create empty lines, do not create token line and column positions.
This tokenizer creates a much smaller streams of only relevant lines. All information not essential for the AstBuilder is skipped. This tokenizer is used in frontend rendering for quicker AST building.
An instance of this tokenizer is injected by DI when injecting TokenizerInterface.
This class is unit test covered by TokenizerInterfaceTest and paired with LossyTokenizer. Never change anything in this class without additional test coverage!
Tags
Table of Contents
Interfaces
- TokenizerInterface
- A lossless tokenizer for TypoScript syntax.
Methods
Methods
tokenize()
public
tokenize(string $source) : LineStream
Parameters
- $source : string