LosslessTokenizer implements TokenizerInterface
FinalYes
A lossless tokenizer for TypoScript syntax.
tokenize() creates a flat stream of tokens from a TypoScript string. It is lossless and never "looses" characters to allow syntax linting and creating linter-fixed source strings: tokenize() to create a TokenStream and using string cast (__toString()) on that stream creates the same source string again.
The tokenizer does not parse conditions or includes itself (no file / db lookups), this is part of the IncludeTree parser.
This class is unit test covered by TokenizerInterfaceTest and paired with LossyTokenizer. Never change anything in this class without additional test coverage!
Tags
Table of Contents
Interfaces
- TokenizerInterface
- A lossless tokenizer for TypoScript syntax.
Methods
Methods
tokenize()
public
tokenize(string $source) : LineStream
Parameters
- $source : string