‪TYPO3CMS  ‪main
Public Member Functions | Private Member Functions | Private Attributes | List of all members
TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer Class Reference
Inheritance diagram for TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer:
TYPO3\CMS\Core\TypoScript\Tokenizer\TokenizerInterface

Public Member Functions

 tokenize (string $source)
 

Private Member Functions

 splitLines ($source)
 
 ignoreUntilEndOfMultilineComment ()
 
 createConditionLine ()
 
 parseBlockStart ()
 
 parseImportLine ()
 
 parseImportOld ()
 
 parseIdentifier ()
 
 parseOperatorUnset ()
 
 parseOperatorAssignment ()
 
 parseOperatorMultilineAssignment ()
 
 parseOperatorCopy ()
 
 parseOperatorReference ()
 
 parseIdentifierAtEndOfLine ()
 
 parseIdentifierUntilStopChar (array $splitLine, bool $isRelative=false)
 
 parseOperatorFunction ()
 
 parseValueForConstants ()
 

Private Attributes

LineStream $lineStream
 
IdentifierTokenStream $identifierStream
 
TokenStreamInterface $valueStream
 
array $lines
 
int $currentLineNumber
 
string $currentLineString
 

Detailed Description

A lossy tokenizer implementation: Do not create invalid lines, do not create empty lines, do not create token line and column positions.

This tokenizer creates a much smaller streams of only relevant lines. All information not essential for the AstBuilder is skipped. This tokenizer is used in frontend rendering for quicker AST building.

An instance of this tokenizer is injected by DI when injecting TokenizerInterface.

This class is unit test covered by TokenizerInterfaceTest and paired with LossyTokenizer. Never change anything in this class without additional test coverage!

: Internal tokenizer structure.

Definition at line 56 of file LossyTokenizer.php.

Member Function Documentation

◆ createConditionLine()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::createConditionLine ( )
private

Create a condition line from token stream of this line.

Definition at line 149 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\tokenize().

◆ ignoreUntilEndOfMultilineComment()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::ignoreUntilEndOfMultilineComment ( )
private

Definition at line 132 of file LossyTokenizer.php.

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\createConditionLine(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifier(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifierAtEndOfLine(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseImportLine(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseImportOld(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorCopy(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorFunction(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorMultilineAssignment(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorReference(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorUnset(), and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\tokenize().

◆ parseBlockStart()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseBlockStart ( )
private

Definition at line 217 of file LossyTokenizer.php.

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifier().

◆ parseIdentifier()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseIdentifier ( )
private

Definition at line 308 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseBlockStart(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifierUntilStopChar(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorAssignment(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorCopy(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorFunction(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorMultilineAssignment(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorReference(), and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorUnset().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\tokenize().

◆ parseIdentifierAtEndOfLine()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseIdentifierAtEndOfLine ( )
private

Definition at line 464 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment(), and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifierUntilStopChar().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorCopy(), and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorReference().

◆ parseIdentifierUntilStopChar()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseIdentifierUntilStopChar ( array  $splitLine,
bool  $isRelative = false 
)
private

Definition at line 495 of file LossyTokenizer.php.

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifier(), and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifierAtEndOfLine().

◆ parseImportLine()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseImportLine ( )
private

Definition at line 229 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\tokenize().

◆ parseImportOld()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseImportOld ( )
private

Parse everything behind <INCLUDE_TYPOSCRIPT: at least until end of line or more if there is a multiline comment at end.

Definition at line 274 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\tokenize().

◆ parseOperatorAssignment()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseOperatorAssignment ( )
private

Definition at line 359 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseValueForConstants().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifier().

◆ parseOperatorCopy()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseOperatorCopy ( )
private

Definition at line 424 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\$identifierStream, TYPO3\CMS\Core\TypoScript\Tokenizer\Token\IdentifierTokenStream\append(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment(), and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifierAtEndOfLine().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifier().

◆ parseOperatorFunction()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseOperatorFunction ( )
private

Definition at line 545 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifier().

◆ parseOperatorMultilineAssignment()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseOperatorMultilineAssignment ( )
private

Definition at line 367 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\$currentLineNumber, TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment(), and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseValueForConstants().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifier().

◆ parseOperatorReference()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseOperatorReference ( )
private

Definition at line 444 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\$identifierStream, TYPO3\CMS\Core\TypoScript\Tokenizer\Token\IdentifierTokenStream\append(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment(), and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifierAtEndOfLine().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifier().

◆ parseOperatorUnset()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseOperatorUnset ( )
private

Definition at line 350 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment().

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifier().

◆ parseValueForConstants()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::parseValueForConstants ( )
private

Definition at line 604 of file LossyTokenizer.php.

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorAssignment(), and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorMultilineAssignment().

◆ splitLines()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::splitLines (   $source)
private

Definition at line 120 of file LossyTokenizer.php.

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\tokenize().

◆ tokenize()

TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::tokenize ( string  $source)

Implements TYPO3\CMS\Core\TypoScript\Tokenizer\TokenizerInterface.

Definition at line 67 of file LossyTokenizer.php.

References TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\$lineStream, TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\createConditionLine(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\ignoreUntilEndOfMultilineComment(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseIdentifier(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseImportLine(), TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseImportOld(), and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\splitLines().

Member Data Documentation

◆ $currentLineNumber

int TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$currentLineNumber
private

Definition at line 64 of file LossyTokenizer.php.

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorMultilineAssignment().

◆ $currentLineString

string TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$currentLineString
private

Definition at line 65 of file LossyTokenizer.php.

◆ $identifierStream

IdentifierTokenStream TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$identifierStream
private

Definition at line 60 of file LossyTokenizer.php.

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorCopy(), and TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\parseOperatorReference().

◆ $lines

array TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$lines
private

Definition at line 63 of file LossyTokenizer.php.

◆ $lineStream

LineStream TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$lineStream
private

Definition at line 58 of file LossyTokenizer.php.

Referenced by TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer\tokenize().

◆ $valueStream

TokenStreamInterface TYPO3\CMS\Core\TypoScript\Tokenizer\LossyTokenizer::$valueStream
private

Definition at line 61 of file LossyTokenizer.php.