Public Member Functions
	__construct ()

	init (array $configuration=null)

	initializeExternalParsers ()

	indexTypo3PageContent ()

	splitHTMLContent (string $content)

string	getHTMLcharset (string $content)

	convertHTMLToUtf8 (string $content, string $charset='')

bool	embracingTags (string $string, string $tagName, ?string &$tagContent, ?string &$stringAfter, ?string &$paramList)

bool	typoSearchTags (string &$body)

	extractLinks (string $content)

array< int, extractHyperLinks(string $html):array { $htmlParser=GeneralUtility::makeInstance(HtmlParser::class);$htmlParts=$htmlParser->	splitTags ('a', $html)

	extractBaseHref (string $html)

	indexExternalUrl (string $externalUrl)

array false	getUrlHeaders (string $url)

	indexRegularDocument (string $file, bool $force=false, string $contentTmpFile='', string $altExtension='')

	readFileContent (string $fileExtension, string $absoluteFileName, string\|int $sectionPointer)

array	fileContentParts (string $ext, string $absFile)

	splitRegularContent (string $content)

	charsetEntity2utf8 (IndexingDataAsString $indexingDataDto)

	processWordsInArrays (IndexingDataAsString $input)

	bodyDescription (IndexingDataAsString $indexingDataDto)

array	indexAnalyze (IndexingDataAsArray $indexingDataDto)

	analyzeHeaderinfo (array &$retArr, array $content, int $offset)

	analyzeBody (array &$retArr, IndexingDataAsArray $indexingDataDto)

	submitPage ()

	submit_grlist (int $hash, int $phash_x)

	submit_section (int $hash, int $hash_t3)

	removeOldIndexedPages (int $phash)

	submitFilePage (array $hash, string $file, array $subinfo, string $ext, int $mtime, int $ctime, int $size, int $content_md5h, IndexingDataAsString $indexingDataDto)

	submitFile_section (int $hash)

	removeOldIndexedFiles (int $phash)

int	checkMtimeTstamp (int $mtime, int $phash)

array true	checkContentHash ()

bool	checkExternalDocContentHash (int $hashGr, int $content_md5h)

bool	is_grlist_set (int $phash_x)

	update_grlist (int $phash, int $phash_x)

	updateTstamp (int $phash, int $mtime=0)

	updateSetId (int $phash)

	updateParsetime (int $phash, int $parsetime)

	updateRootline ()

	getRootLineFields (array &$fieldArray)

	checkWordList (array $wordListArray)

	submitWords (array $wordList, int $phash)

int	freqMap (float $freq)

	setT3Hashes ()

array	setExtHashes (string $file, array $subinfo=[])

	log_setTSlogMessage (string $msg, string $logLevel=LogLevel::INFO)

Public Attributes
array	$reasons

string	$excludeSections = 'script,style'

array	$external_parsers = []

int	$tstamp_maxAge = 0

int	$tstamp_minAge = 0

int	$maxExternalFiles = 0

bool	$forceIndexing = false

array	$defaultIndexingDataPayload

int	$wordcount = 0

int	$externalFileCounter = 0

array	$conf = []

array	$indexerConfig = []

array	$hash = []

array	$file_phash_arr = []

IndexingDataAsString	$indexingDataStringDto

int	$content_md5h

array	$internal_log = []

string	$indexExternalUrl_content = ''

int	$freqRange = 32000

float	$freqMax = 0.1

Lexer	$lexerObj

int	$flagBitMask

	$hyperLinksData = []

foreach($htmlParts as $index=> $tagData) return	$hyperLinksData

Protected Member Functions
string	createLocalPath (string $sourcePath)

	createLocalPathUsingDomainURL (string $sourcePath)

	createLocalPathUsingAbsRefPrefix (string $sourcePath)

	createLocalPathFromAbsoluteURL (string $sourcePath)

	createLocalPathFromRelativeURL (string $sourcePath)

string	addSpacesToKeywordList (string $keywordList)

Static Protected Member Functions
static	isRelativeURL (string $url)

static	isAllowedLocalFile (string $filePath)

Protected Attributes
TimeTracker	$timeTracker

Private Member Functions
array	removePhashCollisions (array $wordList)

Detailed Description

Indexing class for TYPO3 frontend

Definition at line 41 of file Indexer.php.

Constructor & Destructor Documentation

◆ __construct()

TYPO3\CMS\IndexedSearch\Indexer::__construct ( )

Definition at line 127 of file Indexer.php.

References TYPO3\CMS\Core\Utility\MathUtility\forceIntegerInRange().

Member Function Documentation

◆ addSpacesToKeywordList()

string TYPO3\CMS\IndexedSearch\Indexer::addSpacesToKeywordList ( string $keywordList )

protected

Makes sure that keywords are space-separated. This is important for their proper displaying as a part of fulltext index.

Parameters

string $keywordList

Returns: ‪string

See also: ‪https://forge.typo3.org/issues/14959

Definition at line 1786 of file Indexer.php.

References TYPO3\CMS\Core\Utility\GeneralUtility\trimExplode().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\splitHTMLContent().

◆ analyzeBody()

TYPO3\CMS\IndexedSearch\Indexer::analyzeBody	(	array &	$retArr,
		IndexingDataAsArray	$indexingDataDto
	)

Calculates relevant information for bodycontent

Parameters

array $retArr ‪Index array, passed by reference

Definition at line 971 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\md5inthash().

◆ analyzeHeaderinfo()

TYPO3\CMS\IndexedSearch\Indexer::analyzeHeaderinfo	(	array &	$retArr,
		array	$content,
		int	$offset
	)

Calculates relevant information for headercontent

Parameters

array	$retArr	‪Index array, passed by reference
array	$content	‪Standard content array
int	$offset	‪Bit-wise priority to type

Definition at line 945 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\md5inthash().

◆ bodyDescription()

TYPO3\CMS\IndexedSearch\Indexer::bodyDescription ( IndexingDataAsString $indexingDataDto )

Extracts the sample description text from the content array.

Definition at line 909 of file Indexer.php.

References TYPO3\CMS\Core\Utility\MathUtility\forceIntegerInRange().

◆ charsetEntity2utf8()

TYPO3\CMS\IndexedSearch\Indexer::charsetEntity2utf8 ( IndexingDataAsString $indexingDataDto )

Convert character set and HTML entities in the value of input content array keys

Definition at line 873 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Dto\IndexingDataAsString\toArray().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ checkContentHash()

array true TYPO3\CMS\IndexedSearch\Indexer::checkContentHash ( )

Check content hash in phash table

Returns: ‪array|true Returns TRUE if the page needs to be indexed (that is, there was no result), otherwise the phash value (in an array) of the phash record to which the grlist_record should be related!

Definition at line 1362 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ checkExternalDocContentHash()

bool TYPO3\CMS\IndexedSearch\Indexer::checkExternalDocContentHash	(	int	$hashGr,
		int	$content_md5h
	)

Check content hash for external documents Returns TRUE if the document needs to be indexed (that is, there was no result)

Parameters

int	$hashGr	‪phash value to check (phash_grouping)
int	$content_md5h	‪Content hash to check

Returns: ‪bool Returns TRUE if the document needs to be indexed (that is, there was no result)

Definition at line 1396 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument().

◆ checkMtimeTstamp()

int TYPO3\CMS\IndexedSearch\Indexer::checkMtimeTstamp	(	int	$mtime,
		int	$phash
	)

Check the mtime / tstamp of the currently indexed page/file (based on phash) Return positive integer if the page needs to be indexed

Parameters

int	$mtime	‪mtime value to test against limits and indexed page (usually this is the mtime of the cached document)
int	$phash	‪"phash" used to select any already indexed page to see what its mtime is.

Returns: ‪int Result integer: Generally: <0 = No indexing, >0 = Do indexing (see $this->reasons): -2) Min age was NOT exceeded and so indexing cannot occur. -1) mtime matched so no need to reindex page. 0) N/A 1) Max age exceeded, page must be indexed again. 2) mtime of indexed page doesn't match mtime given for current content and we must index page. 3) No mtime was set, so we will index... 4) No indexed page found, so of course we will index.

Definition at line 1299 of file Indexer.php.

References $GLOBALS, and TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument(), and TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ checkWordList()

TYPO3\CMS\IndexedSearch\Indexer::checkWordList ( array $wordListArray )

Adds new words to db

Parameters

array $wordListArray ‪Word List array (where each word has information about position, etc.).

Definition at line 1595 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed(), and TYPO3\CMS\Core\Database\Connection\PARAM_INT_ARRAY.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument(), and TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ convertHTMLToUtf8()

TYPO3\CMS\IndexedSearch\Indexer::convertHTMLToUtf8	(	string	$content,
		string	$charset = `''`
	)

Converts a HTML document to utf-8

Definition at line 343 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Indexer\getHTMLcharset().

◆ createLocalPath()

string TYPO3\CMS\IndexedSearch\Indexer::createLocalPath ( string $sourcePath )

protected

Checks if the file is local

Returns: ‪string Absolute path to file if file is local, else empty string

Definition at line 577 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Indexer\createLocalPathFromAbsoluteURL(), TYPO3\CMS\IndexedSearch\Indexer\createLocalPathFromRelativeURL(), TYPO3\CMS\IndexedSearch\Indexer\createLocalPathUsingAbsRefPrefix(), and TYPO3\CMS\IndexedSearch\Indexer\createLocalPathUsingDomainURL().

◆ createLocalPathFromAbsoluteURL()

TYPO3\CMS\IndexedSearch\Indexer::createLocalPathFromAbsoluteURL ( string $sourcePath )

protected

Attempts to create a local file path from the absolute URL without schema.

Definition at line 636 of file Indexer.php.

References TYPO3\CMS\Core\Core\Environment\getPublicPath().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\createLocalPath().

◆ createLocalPathFromRelativeURL()

TYPO3\CMS\IndexedSearch\Indexer::createLocalPathFromRelativeURL ( string $sourcePath )

protected

Attempts to create a local file path from the relative URL.

Definition at line 652 of file Indexer.php.

References TYPO3\CMS\Core\Core\Environment\getPublicPath().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\createLocalPath().

◆ createLocalPathUsingAbsRefPrefix()

TYPO3\CMS\IndexedSearch\Indexer::createLocalPathUsingAbsRefPrefix ( string $sourcePath )

protected

Attempts to create a local file path by matching absRefPrefix. This requires TSFE. If TSFE is missing, this function does nothing.

Definition at line 616 of file Indexer.php.

References $GLOBALS, and TYPO3\CMS\Core\Core\Environment\getPublicPath().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\createLocalPath().

◆ createLocalPathUsingDomainURL()

TYPO3\CMS\IndexedSearch\Indexer::createLocalPathUsingDomainURL ( string $sourcePath )

protected

Attempts to create a local file path by matching a current request URL.

Definition at line 597 of file Indexer.php.

References $GLOBALS, and TYPO3\CMS\Core\Core\Environment\getPublicPath().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\createLocalPath().

◆ embracingTags()

bool TYPO3\CMS\IndexedSearch\Indexer::embracingTags	(	string	$string,
		string	$tagName,
		?string &	$tagContent,
		?string &	$stringAfter,
		?string &	$paramList
	)

Finds first occurrence of embracing tags and returns the embraced content and the original string with the tag removed in the two passed variables. Returns FALSE if no match found. ie. useful for finding <title> of document or removing <script>-sections

Parameters

string	$string	‪String to search in
string	$tagName	‪Tag name, eg. "script
string \| null	$tagContent	‪Passed by reference: Content inside found tag
string \| null	$stringAfter	‪Passed by reference: Content after found tag
string \| null	$paramList	‪Passed by reference: Attributes of the found tag.

Returns: ‪bool Returns FALSE if tag was not found, otherwise TRUE.

Definition at line 368 of file Indexer.php.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\splitHTMLContent().

◆ extractBaseHref()

TYPO3\CMS\IndexedSearch\Indexer::extractBaseHref ( string $html )

Extracts the "base href" from content string.

Definition at line 502 of file Indexer.php.

◆ extractLinks()

TYPO3\CMS\IndexedSearch\Indexer::extractLinks ( string $content )

Extract links (hrefs) from HTML content and if indexable media is found, it is indexed.

Definition at line 423 of file Indexer.php.

References TYPO3\CMS\Core\Core\Environment\getPublicPath(), TYPO3\CMS\IndexedSearch\Indexer\indexExternalUrl(), and TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ fileContentParts()

array TYPO3\CMS\IndexedSearch\Indexer::fileContentParts	(	string	$ext,
		string	$absFile
	)

Creates an array with pointers to divisions of document.

Parameters

string	$ext	‪File extension
string	$absFile	‪Absolute filename (must exist and be validated OK before calling function)

Returns: ‪array Array of pointers to sections that the document should be divided into

Definition at line 844 of file Indexer.php.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument().

◆ freqMap()

int TYPO3\CMS\IndexedSearch\Indexer::freqMap ( float $freq )

maps frequency from a real number in [0;1] to an integer in [0;$this->freqRange] with anything above $this->freqMax as 1 and back.

Parameters

float $freq ‪Frequency

Returns: ‪int Frequency in range.

Definition at line 1710 of file Indexer.php.

◆ getHTMLcharset()

string TYPO3\CMS\IndexedSearch\Indexer::getHTMLcharset ( string $content )

Extract the charset value from HTML meta tag.

Parameters

string $content ‪HTML content

Returns: ‪string The charset value if found.

Definition at line 329 of file Indexer.php.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\convertHTMLToUtf8().

◆ getRootLineFields()

TYPO3\CMS\IndexedSearch\Indexer::getRootLineFields ( array & $fieldArray )

Adding values for root-line fields. rl0, rl1 and rl2 are standard. A hook might add more.

Parameters

array $fieldArray ‪Field array, passed by reference

Definition at line 1575 of file Indexer.php.

References $GLOBALS.

◆ getUrlHeaders()

array false TYPO3\CMS\IndexedSearch\Indexer::getUrlHeaders ( string $url )

Getting HTTP request headers of URL

Parameters

string $url ‪The URL

Returns: ‪array|false If no answer, returns FALSE. Otherwise, an array where HTTP headers are keys

Definition at line 556 of file Indexer.php.

References TYPO3\CMS\Webhooks\Message\$url.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexExternalUrl().

◆ indexAnalyze()

array TYPO3\CMS\IndexedSearch\Indexer::indexAnalyze ( IndexingDataAsArray $indexingDataDto )

Analyzes content to use for indexing,

Returns: ‪array Index Array (whatever that is...)

Definition at line 928 of file Indexer.php.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument(), and TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ indexExternalUrl()

TYPO3\CMS\IndexedSearch\Indexer::indexExternalUrl ( string $externalUrl )

Index External URLs HTML content

Parameters

string $externalUrl ‪URL, eg. "https://typo3.org/

Definition at line 532 of file Indexer.php.

References TYPO3\CMS\Core\Utility\GeneralUtility\getUrl(), TYPO3\CMS\IndexedSearch\Indexer\getUrlHeaders(), TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument(), and TYPO3\CMS\Core\Utility\GeneralUtility\writeFile().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\extractLinks().

◆ indexRegularDocument()

TYPO3\CMS\IndexedSearch\Indexer::indexRegularDocument	(	string	$file,
		bool	$force = `false`,
		string	$contentTmpFile = `''`,
		string	$altExtension = `''`
	)

Indexing a regular document given as $file (relative to public web path, local file)

Parameters

string	$file	‪Relative Filename, relative to public web path. It can also be an absolute path as long as it is inside the lockRootPath. Finally, if $contentTmpFile is set, this value can be anything, most likely a URL
bool	$force	‪If set, indexing is forced (despite content hashes, mtime etc).
string	$contentTmpFile	‪Temporary file with the content to read it from (instead of $file). Used when the $file is a URL.
string	$altExtension	‪File extension for temporary file.

Definition at line 697 of file Indexer.php.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\extractLinks(), and TYPO3\CMS\IndexedSearch\Indexer\indexExternalUrl().

◆ indexTypo3PageContent()

TYPO3\CMS\IndexedSearch\Indexer::indexTypo3PageContent ( )

Start indexing of the TYPO3 page

Definition at line 193 of file Indexer.php.

◆ init()

TYPO3\CMS\IndexedSearch\Indexer::init ( array $configuration = null )

Initializes the object.

Parameters

array | null $configuration ‪will be used to set $this->conf, otherwise $this->conf MUST be set with proper values prior to this call

Definition at line 148 of file Indexer.php.

References $GLOBALS, TYPO3\CMS\IndexedSearch\Indexer\initializeExternalParsers(), and TYPO3\CMS\IndexedSearch\Indexer\setT3Hashes().

◆ initializeExternalParsers()

TYPO3\CMS\IndexedSearch\Indexer::initializeExternalParsers ( )

Initialize external parsers

Definition at line 173 of file Indexer.php.

References $GLOBALS.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\init().

◆ is_grlist_set()

bool TYPO3\CMS\IndexedSearch\Indexer::is_grlist_set ( int $phash_x )

Checks if a grlist record has been set for the phash value input (looking at the "real" phash of the current content, not the linked-to phash of the common search result page)

Parameters

int $phash_x ‪Phash integer to test.

Returns: ‪bool

Definition at line 1422 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ isAllowedLocalFile()

static TYPO3\CMS\IndexedSearch\Indexer::isAllowedLocalFile ( string $filePath )

staticprotected

Checks if the path points to the file inside the website

Definition at line 676 of file Indexer.php.

References TYPO3\CMS\Core\Core\Environment\getPublicPath().

◆ isRelativeURL()

static TYPO3\CMS\IndexedSearch\Indexer::isRelativeURL ( string $url )

staticprotected

Checks if URL is relative.

Definition at line 667 of file Indexer.php.

References TYPO3\CMS\Webhooks\Message\$url.

◆ log_setTSlogMessage()

TYPO3\CMS\IndexedSearch\Indexer::log_setTSlogMessage	(	string	$msg,
		string	$logLevel = `LogLevel::INFO`
	)

Definition at line 1769 of file Indexer.php.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument(), and TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ processWordsInArrays()

TYPO3\CMS\IndexedSearch\Indexer::processWordsInArrays ( IndexingDataAsString $input )

Processing words in the array from split*Content -functions. Values are ensured to be unique.

Definition at line 887 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Dto\IndexingDataAsArray\fromArray(), and TYPO3\CMS\IndexedSearch\Dto\IndexingDataAsString\toArray().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument(), and TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ readFileContent()

TYPO3\CMS\IndexedSearch\Indexer::readFileContent	(	string	$fileExtension,
		string	$absoluteFileName,
		string\|int	$sectionPointer
	)

Reads the content of an external file being indexed. The content from the external parser MUST be returned in utf-8!

Parameters

string	$fileExtension	‪File extension, eg. "pdf", "doc" etc.
string	$absoluteFileName	‪Absolute filename of file (must exist and be validated OK before calling function)
string \| int	$sectionPointer	‪Pointer to section (zero for all other than PDF which will have an indication of pages into which the document should be splitted.)

Definition at line 810 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Dto\IndexingDataAsString\fromArray().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument().

◆ removeOldIndexedFiles()

TYPO3\CMS\IndexedSearch\Indexer::removeOldIndexedFiles ( int $phash )

Removes records for the indexed page, $phash

Parameters

int $phash ‪phash value to flush

Definition at line 1273 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

◆ removeOldIndexedPages()

TYPO3\CMS\IndexedSearch\Indexer::removeOldIndexedPages ( int $phash )

Removes records for the indexed page, $phash

Parameters

int $phash ‪phash value to flush

Definition at line 1125 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

◆ removePhashCollisions()

array TYPO3\CMS\IndexedSearch\Indexer::removePhashCollisions ( array $wordList )

private

Make sure that the word list only contains words with unique phash values. All words with phash collisions are filtered from the list.

Parameters

array $wordList ‪the input word list

Returns: ‪array the filtered word list

Definition at line 1799 of file Indexer.php.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument(), and TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ setExtHashes()

array TYPO3\CMS\IndexedSearch\Indexer::setExtHashes	(	string	$file,
		array	$subinfo = `[]`
	)

Get search hash, external files

Parameters

string	$file	‪File name / path which identifies it on the server
array	$subinfo	‪Additional content identifying the (subpart of) content. For instance; PDF files are divided into groups of pages for indexing.

Returns: ‪array Array with "phash_grouping" and "phash" inside.

Definition at line 1754 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\md5inthash().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument().

◆ setT3Hashes()

TYPO3\CMS\IndexedSearch\Indexer::setT3Hashes ( )

Get search hash, T3 pages

Definition at line 1730 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\md5inthash().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\init().

◆ splitHTMLContent()

TYPO3\CMS\IndexedSearch\Indexer::splitHTMLContent ( string $content )

Splits HTML content and returns an associative array, with title, a list of metatags, and a list of words in the body.

Parameters

string $content ‪HTML content to index. To some degree expected to be made by TYPO3 (i.e. splitting the header by ":")

Definition at line 278 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Indexer\addSpacesToKeywordList(), TYPO3\CMS\IndexedSearch\Indexer\embracingTags(), TYPO3\CMS\IndexedSearch\Dto\IndexingDataAsString\fromArray(), and TYPO3\CMS\IndexedSearch\Indexer\typoSearchTags().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ splitRegularContent()

TYPO3\CMS\IndexedSearch\Indexer::splitRegularContent ( string $content )

Splits non-HTML content (from external files for instance)

Definition at line 857 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Dto\IndexingDataAsString\fromArray().

◆ splitTags()

array<int, extractHyperLinks(string $html): array { $htmlParser = GeneralUtility::makeInstance(HtmlParser::class); $htmlParts = $htmlParser-> TYPO3\CMS\IndexedSearch\Indexer::splitTags	(	'a'	,
			$html
	)

Extracts all links to external documents from the HTML content string

Parameters

string $html

Returns: ‪array<int, array{tag: string, href: string, localPath: string}>

◆ submit_grlist()

TYPO3\CMS\IndexedSearch\Indexer::submit_grlist	(	int	$hash,
		int	$phash_x
	)

Stores gr_list in the database.

Parameters

int	$hash	‪Search result record phash
int	$phash_x	‪Actual phash of current content

Definition at line 1082 of file Indexer.php.

References $fields, TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed(), and TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\md5inthash().

◆ submit_section()

TYPO3\CMS\IndexedSearch\Indexer::submit_section	(	int	$hash,
		int	$hash_t3
	)

Stores section $hash and $hash_t3 are the same for TYPO3 pages, but different when it is external files.

Parameters

int	$hash	‪phash of TYPO3 parent search result record
int	$hash_t3	‪phash of the file indexation search record

Definition at line 1105 of file Indexer.php.

References $fields, and TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

◆ submitFile_section()

TYPO3\CMS\IndexedSearch\Indexer::submitFile_section ( int $hash )

Stores file section for a file IF it does not exist

Parameters

int $hash ‪phash value of file

Definition at line 1239 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed(), and TYPO3\CMS\Core\Database\Connection\PARAM_INT.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument().

◆ submitFilePage()

TYPO3\CMS\IndexedSearch\Indexer::submitFilePage	(	array	$hash,
		string	$file,
		array	$subinfo,
		string	$ext,
		int	$mtime,
		int	$ctime,
		int	$size,
		int	$content_md5h,
		IndexingDataAsString	$indexingDataDto
	)

Updates db with information about the file

Parameters

array	$hash	‪Array with phash and phash_grouping keys for file
string	$file	‪File name
array	$subinfo	‪Array of "static_page_arguments" for files: This is for instance the page index for a PDF file (other document types it will be a zero)
string	$ext	‪File extension determining the type of media.
int	$mtime	‪Modification time of file.
int	$ctime	‪Creation time of file.
int	$size	‪Size of file in bytes
int	$content_md5h	‪Content HASH value.

Definition at line 1163 of file Indexer.php.

References $fields, $GLOBALS, TYPO3\CMS\Core\Utility\PathUtility\basename(), TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed(), and TYPO3\CMS\IndexedSearch\Dto\IndexingDataAsString\toArray().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument().

◆ submitPage()

TYPO3\CMS\IndexedSearch\Indexer::submitPage ( )

Updates db with information about the page (TYPO3 page, not external media)

Definition at line 1000 of file Indexer.php.

References $fields, $GLOBALS, and TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ submitWords()

TYPO3\CMS\IndexedSearch\Indexer::submitWords	(	array	$wordList,
		int	$phash
	)

Submits RELATIONS between words and phash

Parameters

array	$wordList	‪Word list array
int	$phash	‪phash value

Definition at line 1660 of file Indexer.php.

References $fields, TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed(), and TYPO3\CMS\Core\Database\Connection\PARAM_INT.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument(), and TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ typoSearchTags()

bool TYPO3\CMS\IndexedSearch\Indexer::typoSearchTags ( string & $body )

Removes content that shouldn't be indexed according to TYPO3SEARCH-tags.

Parameters

string $body ‪HTML Content, passed by reference

Returns: ‪bool Returns TRUE if a TYPOSEARCH_ tag was found, otherwise FALSE.

Definition at line 397 of file Indexer.php.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\splitHTMLContent().

◆ update_grlist()

TYPO3\CMS\IndexedSearch\Indexer::update_grlist	(	int	$phash,
		int	$phash_x
	)

Check if a grlist-entry for this hash exists and if not so, write one.

Parameters

int	$phash	‪phash of the search result that should be found
int	$phash_x	‪The real phash of the current content. The two values are different when a page with userlogin turns out to contain the exact same content as another already indexed version of the page; This is the whole reason for the grlist table in fact...

Definition at line 1445 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed(), and TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\md5inthash().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ updateParsetime()

TYPO3\CMS\IndexedSearch\Indexer::updateParsetime	(	int	$phash,
		int	$parsetime
	)

Update parsetime for phash row.

Parameters

int	$phash	‪phash value.
int	$parsetime	‪Parsetime value to set.

Definition at line 1527 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument(), and TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ updateRootline()

TYPO3\CMS\IndexedSearch\Indexer::updateRootline ( )

Update section rootline for the page

Definition at line 1549 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ updateSetId()

TYPO3\CMS\IndexedSearch\Indexer::updateSetId ( int $phash )

Update SetID of the index_phash record.

Parameters

int $phash ‪phash value

Definition at line 1502 of file Indexer.php.

References TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

◆ updateTstamp()

TYPO3\CMS\IndexedSearch\Indexer::updateTstamp	(	int	$phash,
		int	$mtime = `0`
	)

Update tstamp for a phash row.

Parameters

int	$phash	‪phash value
int	$mtime	‪If set, update the mtime field to this value.

Definition at line 1472 of file Indexer.php.

References $GLOBALS, and TYPO3\CMS\IndexedSearch\Utility\IndexedSearchUtility\isTableUsed().

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument(), and TYPO3\CMS\IndexedSearch\Indexer\indexTypo3PageContent().

Member Data Documentation

◆ $conf

array TYPO3\CMS\IndexedSearch\Indexer::$conf = []

Definition at line 95 of file Indexer.php.

◆ $content_md5h

int TYPO3\CMS\IndexedSearch\Indexer::$content_md5h

Content of TYPO3 page

Definition at line 117 of file Indexer.php.

Referenced by TYPO3\CMS\IndexedSearch\Indexer\indexRegularDocument().

◆ $defaultIndexingDataPayload

array TYPO3\CMS\IndexedSearch\Indexer::$defaultIndexingDataPayload

Initial value:

= [
        'title' => '',
        'description' => '',
        'keywords' => '',
        'body' => '',
    ]

Definition at line 86 of file Indexer.php.

◆ $excludeSections

string TYPO3\CMS\IndexedSearch\Indexer::$excludeSections = 'script,style'

HTML code blocks to exclude from indexing

Definition at line 58 of file Indexer.php.

◆ $external_parsers

array TYPO3\CMS\IndexedSearch\Indexer::$external_parsers = []

Supported Extensions for external files

Definition at line 63 of file Indexer.php.

◆ $externalFileCounter

int TYPO3\CMS\IndexedSearch\Indexer::$externalFileCounter = 0

Definition at line 94 of file Indexer.php.

◆ $file_phash_arr

array TYPO3\CMS\IndexedSearch\Indexer::$file_phash_arr = []

Hash array, contains phash and phash_grouping

Definition at line 110 of file Indexer.php.

◆ $flagBitMask

int TYPO3\CMS\IndexedSearch\Indexer::$flagBitMask

Definition at line 124 of file Indexer.php.

◆ $forceIndexing

bool TYPO3\CMS\IndexedSearch\Indexer::$forceIndexing = false

Max number of external files to index.

Definition at line 84 of file Indexer.php.

◆ $freqMax

float TYPO3\CMS\IndexedSearch\Indexer::$freqMax = 0.1

Definition at line 122 of file Indexer.php.

◆ $freqRange

int TYPO3\CMS\IndexedSearch\Indexer::$freqRange = 32000

Definition at line 121 of file Indexer.php.

◆ $hash

array TYPO3\CMS\IndexedSearch\Indexer::$hash = []

Indexer configuration, coming from TYPO3's system configuration for EXT:indexed_search

Definition at line 105 of file Indexer.php.

◆ $hyperLinksData [1/2]

TYPO3\CMS\IndexedSearch\Indexer::$hyperLinksData = []

Definition at line 480 of file Indexer.php.

◆ $hyperLinksData [2/2]

foreach ( $htmlParts as $index=> $tagData) return TYPO3\CMS\IndexedSearch\Indexer::$hyperLinksData

Definition at line 496 of file Indexer.php.

◆ $indexerConfig

array TYPO3\CMS\IndexedSearch\Indexer::$indexerConfig = []

Configuration set internally (see init functions for required keys and their meaning)

Definition at line 100 of file Indexer.php.

◆ $indexExternalUrl_content

string TYPO3\CMS\IndexedSearch\Indexer::$indexExternalUrl_content = ''

Definition at line 120 of file Indexer.php.

◆ $indexingDataStringDto

IndexingDataAsString TYPO3\CMS\IndexedSearch\Indexer::$indexingDataStringDto

Definition at line 112 of file Indexer.php.

◆ $internal_log

array TYPO3\CMS\IndexedSearch\Indexer::$internal_log = []

Definition at line 119 of file Indexer.php.

◆ $lexerObj

Lexer TYPO3\CMS\IndexedSearch\Indexer::$lexerObj

Definition at line 123 of file Indexer.php.

◆ $maxExternalFiles

int TYPO3\CMS\IndexedSearch\Indexer::$maxExternalFiles = 0

If set, this tells a minimum limit before a document can be indexed again. This is regardless of mtime.

Definition at line 79 of file Indexer.php.

◆ $reasons

array TYPO3\CMS\IndexedSearch\Indexer::$reasons

Initial value:

= [
        -1 => 'mtime matched the document, so no changes detected and no content updated',
        -2 => 'The minimum age was not exceeded',
        1 => 'The configured max-age was exceeded for the document and thus it\'s indexed.',
        2 => 'The minimum age was exceed and mtime was set and the mtime was different, so the page was indexed.',
        3 => 'The minimum age was exceed, but mtime was not set, so the page was indexed.',
        4 => 'Page has never been indexed (is not represented in the index_phash table).',
    ]

Definition at line 46 of file Indexer.php.

◆ $timeTracker

TimeTracker TYPO3\CMS\IndexedSearch\Indexer::$timeTracker

protected

Definition at line 125 of file Indexer.php.

◆ $tstamp_maxAge

int TYPO3\CMS\IndexedSearch\Indexer::$tstamp_maxAge = 0

Min/Max times

Definition at line 68 of file Indexer.php.

◆ $tstamp_minAge

int TYPO3\CMS\IndexedSearch\Indexer::$tstamp_minAge = 0

If set, this tells a number of seconds that is the maximum age of an indexed document. Regardless of mtime the document will be re-indexed if this limit is exceeded.

Definition at line 74 of file Indexer.php.

◆ $wordcount

int TYPO3\CMS\IndexedSearch\Indexer::$wordcount = 0

Definition at line 93 of file Indexer.php.

Public Member Functions

Public Attributes

Protected Member Functions

Static Protected Member Functions

Protected Attributes

Private Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ __construct()

Member Function Documentation

◆ addSpacesToKeywordList()

◆ analyzeBody()

◆ analyzeHeaderinfo()

◆ bodyDescription()

◆ charsetEntity2utf8()

◆ checkContentHash()

◆ checkExternalDocContentHash()

◆ checkMtimeTstamp()

◆ checkWordList()

◆ convertHTMLToUtf8()

◆ createLocalPath()

◆ createLocalPathFromAbsoluteURL()

◆ createLocalPathFromRelativeURL()

◆ createLocalPathUsingAbsRefPrefix()

◆ createLocalPathUsingDomainURL()

◆ embracingTags()

◆ extractBaseHref()

◆ extractLinks()

◆ fileContentParts()

◆ freqMap()

◆ getHTMLcharset()

◆ getRootLineFields()

◆ getUrlHeaders()

◆ indexAnalyze()

◆ indexExternalUrl()

◆ indexRegularDocument()

◆ indexTypo3PageContent()

◆ init()

◆ initializeExternalParsers()

◆ is_grlist_set()

◆ isAllowedLocalFile()

◆ isRelativeURL()

◆ log_setTSlogMessage()

◆ processWordsInArrays()

◆ readFileContent()

◆ removeOldIndexedFiles()

◆ removeOldIndexedPages()

◆ removePhashCollisions()

◆ setExtHashes()

◆ setT3Hashes()

◆ splitHTMLContent()

◆ splitRegularContent()

◆ splitTags()

◆ submit_grlist()

◆ submit_section()

◆ submitFile_section()

◆ submitFilePage()

◆ submitPage()

◆ submitWords()

◆ typoSearchTags()

◆ update_grlist()

◆ updateParsetime()

◆ updateRootline()

◆ updateSetId()

◆ updateTstamp()

Member Data Documentation

◆ $conf

◆ $content_md5h

◆ $defaultIndexingDataPayload

◆ $excludeSections

◆ $external_parsers

◆ $externalFileCounter

◆ $file_phash_arr

◆ $flagBitMask

◆ $forceIndexing

◆ $freqMax

◆ $freqRange

◆ $hash

◆ $hyperLinksData [1/2]

◆ $hyperLinksData [2/2]