‪TYPO3CMS  ‪main
TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer Class Reference
Inheritance diagram for TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer:
TYPO3\CMS\Core\TypoScript\Tokenizer\TokenizerInterface

Public Member Functions

 tokenize (string $source)
 

Private Member Functions

 splitLines ($source)
 
 createEmptyLine ()
 
 parseTabsAndWhitespaces ()
 
 makeComment ()
 
 createHashCommentLine ()
 
 parseHashComment ()
 
 createDoubleSlashCommentLine ()
 
 parseDoubleSlashComment ()
 
 createMultilineCommentLine ()
 
 parseMultilineComment ()
 
 createConditionLine ()
 
 createBlockStopLine ()
 
 parseBlockStart ()
 
 parseImportLine ()
 
 parseImportOld ()
 
 parseIdentifier ()
 
 parseOperatorAssignment ()
 
 parseOperatorMultilineAssignment ()
 
 parseOperatorUnset ()
 
 parseOperatorCopy ()
 
 parseOperatorReference ()
 
 parseIdentifierAtEndOfLine ()
 
 parseIdentifierUntilStopChar (array $splitLine, bool $isRelative=false)
 
 parseOperatorFunction ()
 
 parseValueForConstants ()
 

Private Attributes

LineStream $lineStream
 
TokenStreamInterface $tokenStream
 
IdentifierTokenStream $identifierStream
 
TokenStreamInterface $valueStream
 
array $lines
 
int $currentLineNumber
 
string $currentLineString
 
closure $currentLinebreakCallback
 
int $currentColumnInLine = 0
 

Detailed Description

A lossless tokenizer for TypoScript syntax.

tokenize() creates a flat stream of tokens from a TypoScript string. It is lossless and never "looses" characters to allow syntax linting and creating linter-fixed source strings: tokenize() to create a TokenStream and using string cast (__toString()) on that stream creates the same source string again.

The tokenizer does not parse conditions or includes itself (no file / db lookups), this is part of the IncludeTree parser.

This class is unit test covered by TokenizerInterfaceTest and paired with LossyTokenizer. Never change anything in this class without additional test coverage!

: Internal tokenizer structure.

Definition at line 60 of file LosslessTokenizer.php.

Member Function Documentation

◆ createBlockStopLine()

TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::createBlockStopLine ( )
private

◆ createConditionLine()

TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::createConditionLine ( )
private

◆ createDoubleSlashCommentLine()

TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::createDoubleSlashCommentLine ( )
private

◆ createEmptyLine()

TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::createEmptyLine ( )
private

◆ createHashCommentLine()

TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::createHashCommentLine ( )
private

◆ createMultilineCommentLine()

TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::createMultilineCommentLine ( )
private

◆ makeComment()

◆ parseBlockStart()

◆ parseDoubleSlashComment()

◆ parseHashComment()

◆ parseIdentifier()

◆ parseIdentifierAtEndOfLine()

◆ parseIdentifierUntilStopChar()

◆ parseImportLine()

◆ parseImportOld()

◆ parseMultilineComment()

◆ parseOperatorAssignment()

◆ parseOperatorCopy()

◆ parseOperatorFunction()

◆ parseOperatorMultilineAssignment()

◆ parseOperatorReference()

◆ parseOperatorUnset()

◆ parseTabsAndWhitespaces()

◆ parseValueForConstants()

TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::parseValueForConstants ( )
private

◆ splitLines()

TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::splitLines (   $source)
private

◆ tokenize()

Member Data Documentation

◆ $currentColumnInLine

◆ $currentLinebreakCallback

◆ $currentLineNumber

◆ $currentLineString

string TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::$currentLineString
private

Definition at line 70 of file LosslessTokenizer.php.

◆ $identifierStream

IdentifierTokenStream TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::$identifierStream
private

◆ $lines

array TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::$lines
private

Definition at line 68 of file LosslessTokenizer.php.

◆ $lineStream

LineStream TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::$lineStream
private

◆ $tokenStream

TokenStreamInterface TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::$tokenStream
private

◆ $valueStream

TokenStreamInterface TYPO3\CMS\Core\TypoScript\Tokenizer\LosslessTokenizer::$valueStream
private