‪TYPO3CMS  ‪main
TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer Class Reference
Inheritance diagram for TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer:
TYPO3\CMS\Core\TypoScript\Tokenizer\TokenizerInterface

Public Member Functions

 tokenize (string $source)
 

Private Member Functions

 splitLines ($source)
 
 ignoreUntilEndOfMultilineComment ()
 
 createConditionLine ()
 
 parseBlockStart ()
 
 parseImportLine ()
 
 parseImportOld ()
 
 parseIdentifier ()
 
 parseOperatorUnset ()
 
 parseOperatorAssignment ()
 
 parseOperatorMultilineAssignment ()
 
 parseOperatorCopy ()
 
 parseOperatorReference ()
 
 parseIdentifierAtEndOfLine ()
 
 parseIdentifierUntilStopChar (array $splitLine, bool $isRelative=false)
 
 parseOperatorFunction ()
 
 parseValueForConstants (TokenStreamInterface $valueStream, string $value)
 

Private Attributes

LineStream $lineStream
 
IdentifierTokenStream $identifierStream
 
TokenStreamInterface $valueStream
 
array $lines
 
int $currentLineNumber
 
string $currentLineString
 

Detailed Description

A lossy tokenizer implementation: Do not create invalid lines, do not create empty lines, do not create token line and column positions.

This tokenizer creates a much smaller streams of only relevant lines. All information not essential for the AstBuilder is skipped. This tokenizer is used in frontend rendering for quicker AST building.

An instance of this tokenizer is injected by DI when injecting TokenizerInterface.

This class is unit test covered by TokenizerInterfaceTest and paired with LossyTokenizer. Never change anything in this class without additional test coverage!

: Internal tokenizer structure.

Definition at line 56 of file LossyTokenizer.php.

Member Function Documentation

◆ createConditionLine()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::createConditionLine ( )
private

Create a condition line from token stream of this line.

Definition at line 148 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\tokenize().

◆ ignoreUntilEndOfMultilineComment()

◆ parseBlockStart()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseBlockStart ( )
private

◆ parseIdentifier()

◆ parseIdentifierAtEndOfLine()

◆ parseIdentifierUntilStopChar()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseIdentifierUntilStopChar ( array  $splitLine,
bool  $isRelative = false 
)
private

◆ parseImportLine()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseImportLine ( )
private

◆ parseImportOld()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseImportOld ( )
private

Parse everything behind <INCLUDE_TYPOSCRIPT: at least until end of line or more if there is a multiline comment at end.

Definition at line 273 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\tokenize().

◆ parseOperatorAssignment()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseOperatorAssignment ( )
private

◆ parseOperatorCopy()

◆ parseOperatorFunction()

◆ parseOperatorMultilineAssignment()

◆ parseOperatorReference()

◆ parseOperatorUnset()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseOperatorUnset ( )
private

◆ parseValueForConstants()

◆ splitLines()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::splitLines (   $source)
private

◆ tokenize()

Member Data Documentation

◆ $currentLineNumber

int TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$currentLineNumber
private

◆ $currentLineString

string TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$currentLineString
private

Definition at line 65 of file LossyTokenizer.php.

◆ $identifierStream

IdentifierTokenStream TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$identifierStream
private

◆ $lines

array TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$lines
private

Definition at line 63 of file LossyTokenizer.php.

◆ $lineStream

LineStream TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$lineStream
private

◆ $valueStream

TokenStreamInterface TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$valueStream
private