Indexer

class \TYPO3\CMS\IndexedSearch\Indexer

Indexing class for TYPO3 frontend

Table of Contents

Properties

$conf  : array
$content_md5h  : string
Content of TYPO3 page
$defaultIndexingDataPayload  : array
$excludeSections  : string
HTML code blocks to exclude from indexing
$externalFileCounter  : int
$file_phash_arr  : array
Hash array, contains phash and phash_grouping
$flagBitMask  : int
$forceIndexing  : bool
Max number of external files to index.
$freqMax  : float
$freqRange  : int
$hash  : array
Indexer configuration, coming from TYPO3's system configuration for EXT:indexed_search
$indexerConfig  : array
Configuration set internally (see init functions for required keys and their meaning)
$indexingDataStringDto  : IndexingDataAsString
$maxExternalFiles  : int
If set, this tells a minimum limit before a document can be indexed again. This is regardless of mtime.
$tstamp_minAge  : int
If set, this tells a number of seconds that is the maximum age of an indexed document.
$wordcount  : int

Methods

__construct()  : mixed
analyzeBody()  : void
Calculates relevant information for body content
analyzeHeaderinfo()  : void
Calculates relevant information for headercontent
bodyDescription()  : string
Extracts the sample description text from the content array.
charsetEntity2utf8()  : void
Convert character set and HTML entities in the value of input content array keys
checkContentHash()  : array|true
Check content hash in phash table
checkExternalDocContentHash()  : bool
Check content hash for external documents Returns TRUE if the document needs to be indexed (that is, there was no result)
checkWordList()  : void
Adds new words to db
convertHTMLToUtf8()  : string
Converts a HTML document to utf-8
embracingTags()  : bool
Finds first occurrence of embracing tags and returns the embraced content and the original string with the tag removed in the two passed variables. Returns FALSE if no match found. i.e. useful for finding <title> of document or removing <script>-sections
extractBaseHref()  : string
Extracts the "base href" from content string.
extractHyperLinks()  : array<int, array{tag: string, href: string}>
Extracts all links from the HTML content string.
extractLinks()  : void
Extract links (hrefs) from HTML content and if indexable media is found, it is indexed.
freqMap()  : int
maps frequency from a real number in [0;1] to an integer in [0;$this->freqRange] with anything above $this->freqMax as 1 and back.
getHTMLcharset()  : string
Extract the charset value from HTML meta tag.
getIndexStatus()  : IndexStatus
Check the mtime / tstamp of the currently indexed page/file (based on phash)
getRootLineFields()  : void
Adding values for root-line fields.
indexAnalyze()  : array
Analyzes content to use for indexing,
indexFileDocument()  : void
Indexes a file of a FAL storage with the text extractor which is responsible for it.
indexTypo3PageContent()  : void
Start indexing of the TYPO3 page
init()  : void
is_grlist_set()  : bool
Checks if a grlist record has been set for the phash value input (looking at the "real" phash of the current content, not the linked-to phash of the common search result page)
log_setTSlogMessage()  : void
processWordsInArrays()  : IndexingDataAsArray
Processing words in the array from split*Content -functions. Values are ensured to be unique.
removeOldIndexedFiles()  : void
Removes records for the indexed page, $phash
removeOldIndexedPages()  : void
Removes records for the indexed page, $phash
setExtHashes()  : array{phash_grouping: string, phash: string}
Get search hash, external files
setT3Hashes()  : void
Get search hash, T3 pages
splitHTMLContent()  : IndexingDataAsString
Splits HTML content and returns an associative array, with title, a list of meta tags, and a list of words in the body.
splitRegularContent()  : IndexingDataAsString
Splits non-HTML content (from external files for instance)
submit_grlist()  : void
Stores gr_list in the database.
submit_section()  : void
Stores section $hash and $hash_t3 are the same for TYPO3 pages, but different when it is external files.
submitFile_section()  : void
Stores file section for a file IF it does not exist
submitFilePage()  : void
Updates db with information about the file
submitPage()  : void
Updates db with information about the page (TYPO3 page, not external media)
submitWords()  : void
Submits RELATIONS between words and phash
typoSearchTags()  : bool
Removes content that shouldn't be indexed according to TYPO3SEARCH-tags.
update_grlist()  : void
Check if a grlist-entry for this hash exists and if not so, write one.
updateParsetime()  : void
Update parse time for phash row.
updateRootline()  : void
Update section rootline for the page
updateSetId()  : void
Update SetID of the index_phash record.
updateTstamp()  : void
Update tstamp for a phash row.
addSpacesToKeywordList()  : string
Makes sure that keywords are space-separated. This is important for their proper displaying as a part of fulltext index.
buildFileDumpParameters()  : array
Rebuilds the parameters the file dump token has been calculated with.
createFileFromFileDumpUrl()  : FileInterface|null
Files of storages which are not publicly accessible are not linked directly, but delivered by the file dump eID script. Such a URL is resolved back into the FAL object it points to.
createFileFromLocalPath()  : FileInterface|null
Finds the file in the public storage which contains the file in the public web path.
createFileObject()  : FileInterface|null
Resolves the target of a link to a file of a FAL storage. Links to files outside of any storage and links to other websites are not resolved. A public file is only resolved if its extension can be indexed, so FAL does not add other files to its index.
createIndexingData()  : IndexingDataAsString
If there is no title, the file name is set as title. This will raise the hits considerably if the search matches the document name.
createLocalPathFromAbsoluteURL()  : string
Attempts to create a local file path from the absolute URL without schema.
createLocalPathFromRelativeURL()  : string
Attempts to create a local file path from the relative URL.
createLocalPathUsingAbsRefPrefix()  : string
Attempts to create a local file path by matching absRefPrefix.
createLocalPathUsingDomainURL()  : string
Attempts to create a local file path by matching a current request URL.
isAllowedLocalFile()  : bool
Checks if the path points to the file inside the website
isRelativeURL()  : bool
Checks if URL is relative.
milliseconds()  : int
Gets the unixtime as milliseconds.

Properties

$content_md5h

Content of TYPO3 page

public string $content_md5h = ''

$defaultIndexingDataPayload

public array $defaultIndexingDataPayload = ['title' => '', 'description' => '', 'keywords' => '', 'body' => '']

$excludeSections

HTML code blocks to exclude from indexing

public string $excludeSections = 'script,style'

$externalFileCounter

public int $externalFileCounter = 0

$file_phash_arr

Hash array, contains phash and phash_grouping

public array $file_phash_arr = []

$forceIndexing

Max number of external files to index.

public bool $forceIndexing = false

$hash

Indexer configuration, coming from TYPO3's system configuration for EXT:indexed_search

public array $hash = []

$indexerConfig

Configuration set internally (see init functions for required keys and their meaning)

public array $indexerConfig = []

$maxExternalFiles

If set, this tells a minimum limit before a document can be indexed again. This is regardless of mtime.

public int $maxExternalFiles = 0

$tstamp_minAge

If set, this tells a number of seconds that is the maximum age of an indexed document.

public int $tstamp_minAge = 0

Regardless of mtime the document will be re-indexed if this limit is exceeded.

Methods

__construct()

public __construct(TimeTracker $timeTracker, Lexer $lexer, ConnectionPool $connectionPool, ResourceFactory $resourceFactory, FileDumpService $fileDumpService, TextExtractorRegistry $textExtractorRegistry, ExternalToolConfiguration $externalToolConfiguration, ExternalMediaTypes $externalMediaTypes, ExtensionConfiguration $extensionConfiguration, LoggerInterface $logger) : mixed

Parameters

$timeTracker : TimeTracker
$lexer : Lexer
$connectionPool : ConnectionPool
$resourceFactory : ResourceFactory
$fileDumpService : FileDumpService
$textExtractorRegistry : TextExtractorRegistry
$externalToolConfiguration : ExternalToolConfiguration
$externalMediaTypes : ExternalMediaTypes
$extensionConfiguration : ExtensionConfiguration
$logger : LoggerInterface

analyzeBody()

Calculates relevant information for body content

public analyzeBody(array &$retArr, IndexingDataAsArray $indexingDataDto) : void

Parameters

$retArr : array

Index array, passed by reference

$indexingDataDto : IndexingDataAsArray

analyzeHeaderinfo()

Calculates relevant information for headercontent

public analyzeHeaderinfo(array &$retArr, array $content, int $offset) : void

Parameters

$retArr : array

Index array, passed by reference

$content : array

Standard content array

$offset : int

Bit-wise priority to type

checkContentHash()

Check content hash in phash table

public checkContentHash() : array|true

Return values

array|true —

Returns TRUE if the page needs to be indexed (that is, there was no result), otherwise the phash value (in an array) of the phash record to which the grlist_record should be related!

checkExternalDocContentHash()

Check content hash for external documents Returns TRUE if the document needs to be indexed (that is, there was no result)

public checkExternalDocContentHash(string $hashGr, string $content_md5h) : bool

Parameters

$hashGr : string

phash value to check (phash_grouping)

$content_md5h : string

Content hash to check

Return values

bool

checkWordList()

Adds new words to db

public checkWordList(array $wordListArray) : void

Parameters

$wordListArray : array

Word List array (where each word has information about position, etc.).

convertHTMLToUtf8()

Converts a HTML document to utf-8

public convertHTMLToUtf8(string $content[, string $charset = '' ]) : string

Parameters

$content : string
$charset : string = ''

Return values

string

embracingTags()

Finds first occurrence of embracing tags and returns the embraced content and the original string with the tag removed in the two passed variables. Returns FALSE if no match found. i.e. useful for finding <title> of document or removing <script>-sections

public embracingTags(string $string, string $tagName, string|null &$tagContent, string|null &$stringAfter, string|null &$paramList) : bool

Parameters

$string : string

String to search in

$tagName : string

Tag name, eg. "script

$tagContent : string|null

Passed by reference: Content inside found tag

$stringAfter : string|null

Passed by reference: Content after found tag

$paramList : string|null

Passed by reference: Attributes of the found tag.

Return values

bool

extractBaseHref()

Extracts the "base href" from content string.

public extractBaseHref(string $html) : string

Parameters

$html : string

Return values

string

Extracts all links from the HTML content string.

public extractHyperLinks(string $html) : array<int, array{tag: string, href: string}>

Parameters

$html : string

Return values

array<int, array{tag: string, href: string}>

Extract links (hrefs) from HTML content and if indexable media is found, it is indexed.

public extractLinks(string $content) : void

Parameters

$content : string

freqMap()

maps frequency from a real number in [0;1] to an integer in [0;$this->freqRange] with anything above $this->freqMax as 1 and back.

public freqMap(float $freq) : int

Parameters

$freq : float

Frequency

Return values

int —

Frequency in range.

getHTMLcharset()

Extract the charset value from HTML meta tag.

public getHTMLcharset(string $content) : string

Parameters

$content : string

Return values

string

getIndexStatus()

Check the mtime / tstamp of the currently indexed page/file (based on phash)

public getIndexStatus(int $mtime, string $phash) : IndexStatus

Parameters

$mtime : int

mtime value to test against limits and indexed page (usually this is the mtime of the cached document)

$phash : string

"phash" used to select any already indexed page to see what its mtime is.

Return values

IndexStatus

getRootLineFields()

Adding values for root-line fields.

public getRootLineFields(array &$fieldArray) : void

rl0, rl1 and rl2 are standard. A hook might add more.

Parameters

$fieldArray : array

Field array, passed by reference

indexFileDocument()

Indexes a file of a FAL storage with the text extractor which is responsible for it.

public indexFileDocument(FileInterface $file, string $fileUrl[, bool $force = false ]) : void

Parameters

$file : FileInterface

The file to index

$fileUrl : string

The URL the file was linked with on the page, which identifies the file in the index

$force : bool = false

If set, indexing is forced (despite content hashes, mtime etc).

indexTypo3PageContent()

Start indexing of the TYPO3 page

public indexTypo3PageContent() : void

init()

public init([array|null $configuration = null ]) : void

Parameters

$configuration : array|null = null

will be used to set $this->conf, otherwise $this->conf MUST be set with proper values prior to this call

is_grlist_set()

Checks if a grlist record has been set for the phash value input (looking at the "real" phash of the current content, not the linked-to phash of the common search result page)

public is_grlist_set(string $phash_x) : bool

Parameters

$phash_x : string

Return values

bool

log_setTSlogMessage()

public log_setTSlogMessage(string $msg[, string $logLevel = LogLevel::INFO ]) : void

Parameters

$msg : string
$logLevel : string = LogLevel::INFO

removeOldIndexedFiles()

Removes records for the indexed page, $phash

public removeOldIndexedFiles(string $phash) : void

Parameters

$phash : string

phash value to flush

removeOldIndexedPages()

Removes records for the indexed page, $phash

public removeOldIndexedPages(string $phash) : void

Parameters

$phash : string

phash value to flush

setExtHashes()

Get search hash, external files

public setExtHashes(string $file[, array $subinfo = [] ]) : array{phash_grouping: string, phash: string}

Parameters

$file : string

File name / path which identifies it on the server

$subinfo : array = []

Additional content identifying the (subpart of) content. For instance; PDF files are divided into groups of pages for indexing.

Return values

array{phash_grouping: string, phash: string}

setT3Hashes()

Get search hash, T3 pages

public setT3Hashes() : void

splitHTMLContent()

Splits HTML content and returns an associative array, with title, a list of meta tags, and a list of words in the body.

public splitHTMLContent(string $content) : IndexingDataAsString

Parameters

$content : string

HTML content to index. To some degree expected to be made by TYPO3 (i.e. splitting the header by ":")

Return values

IndexingDataAsString

submit_grlist()

Stores gr_list in the database.

public submit_grlist(string $hash, string $phash_x) : void

Parameters

$hash : string

Search result record phash

$phash_x : string

Actual phash of current content

submit_section()

Stores section $hash and $hash_t3 are the same for TYPO3 pages, but different when it is external files.

public submit_section(string $hash, string $hash_t3) : void

Parameters

$hash : string

phash of TYPO3 parent search result record

$hash_t3 : string

phash of the file indexation search record

submitFile_section()

Stores file section for a file IF it does not exist

public submitFile_section(string $hash) : void

Parameters

$hash : string

phash value of file

submitFilePage()

Updates db with information about the file

public submitFilePage(array $hash, string $file, array $subinfo, string $ext, int $mtime, int $ctime, int $size, string $content_md5h, IndexingDataAsString $indexingDataDto) : void

Parameters

$hash : array

Array with phash and phash_grouping keys for file

$file : string

File name

$subinfo : array

Array of "static_page_arguments" for files: This is for instance the page index for a PDF file (other document types it will be a zero)

$ext : string

File extension determining the type of media.

$mtime : int

Modification time of file.

$ctime : int

Creation time of file.

$size : int

Size of file in bytes

$content_md5h : string

Content HASH value.

$indexingDataDto : IndexingDataAsString

submitPage()

Updates db with information about the page (TYPO3 page, not external media)

public submitPage() : void

submitWords()

Submits RELATIONS between words and phash

public submitWords(array $wordList, string $phash) : void

Parameters

$wordList : array
$phash : string

typoSearchTags()

Removes content that shouldn't be indexed according to TYPO3SEARCH-tags.

public typoSearchTags(string &$body) : bool

Parameters

$body : string

HTML Content, passed by reference

Return values

bool —

Returns TRUE if a TYPOSEARCH_ tag was found, otherwise FALSE.

update_grlist()

Check if a grlist-entry for this hash exists and if not so, write one.

public update_grlist(string $phash, string $phash_x) : void

Parameters

$phash : string

phash of the search result that should be found

$phash_x : string

The real phash of the current content. The two values are different when a page with userlogin turns out to contain the exact same content as another already indexed version of the page; This is the whole reason for the grlist table in fact...

updateParsetime()

Update parse time for phash row.

public updateParsetime(string $phash, int $parsetime) : void

Parameters

$phash : string
$parsetime : int

updateRootline()

Update section rootline for the page

public updateRootline() : void

updateSetId()

Update SetID of the index_phash record.

public updateSetId(string $phash) : void

Parameters

$phash : string

updateTstamp()

Update tstamp for a phash row.

public updateTstamp(string $phash[, int $mtime = 0 ]) : void

Parameters

$phash : string
$mtime : int = 0

addSpacesToKeywordList()

Makes sure that keywords are space-separated. This is important for their proper displaying as a part of fulltext index.

protected addSpacesToKeywordList(string $keywordList) : string

Parameters

$keywordList : string

Tags

see
https://forge.typo3.org/issues/14959

Return values

string

buildFileDumpParameters()

Rebuilds the parameters the file dump token has been calculated with.

protected buildFileDumpParameters(array $queryParameters) : array

Parameters

$queryParameters : array

Tags

see
FileDumpController

Return values

array

createFileFromFileDumpUrl()

Files of storages which are not publicly accessible are not linked directly, but delivered by the file dump eID script. Such a URL is resolved back into the FAL object it points to.

protected createFileFromFileDumpUrl(string $sourcePath) : FileInterface|null

Parameters

$sourcePath : string

Return values

FileInterface|null

createFileFromLocalPath()

Finds the file in the public storage which contains the file in the public web path.

protected createFileFromLocalPath(string $localPath) : FileInterface|null

Parameters

$localPath : string

Return values

FileInterface|null

createFileObject()

Resolves the target of a link to a file of a FAL storage. Links to files outside of any storage and links to other websites are not resolved. A public file is only resolved if its extension can be indexed, so FAL does not add other files to its index.

protected createFileObject(string $sourcePath, array<int, string> $supportedExtensions) : FileInterface|null

Parameters

$sourcePath : string
$supportedExtensions : array<int, string>

Return values

FileInterface|null

createLocalPathFromAbsoluteURL()

Attempts to create a local file path from the absolute URL without schema.

protected createLocalPathFromAbsoluteURL(string $sourcePath) : string

Parameters

$sourcePath : string

Return values

string

createLocalPathFromRelativeURL()

Attempts to create a local file path from the relative URL.

protected createLocalPathFromRelativeURL(string $sourcePath) : string

Parameters

$sourcePath : string

Return values

string

createLocalPathUsingAbsRefPrefix()

Attempts to create a local file path by matching absRefPrefix.

protected createLocalPathUsingAbsRefPrefix(string $sourcePath) : string

Parameters

$sourcePath : string

Return values

string

createLocalPathUsingDomainURL()

Attempts to create a local file path by matching a current request URL.

protected createLocalPathUsingDomainURL(string $sourcePath) : string

Parameters

$sourcePath : string

Return values

string

isAllowedLocalFile()

Checks if the path points to the file inside the website

protected static isAllowedLocalFile(string $filePath) : bool

Parameters

$filePath : string

Return values

bool

isRelativeURL()

Checks if URL is relative.

protected static isRelativeURL(string $url) : bool

Parameters

$url : string

Return values

bool

milliseconds()

Gets the unixtime as milliseconds.

protected milliseconds() : int

Return values

int
On this page