SearchIndexer..::..Dispatcher_Action Method
Assembly: AdvantageCSP.Keyoti.SiteIndexer (in AdvantageCSP.Keyoti.SiteIndexer.dll)
Syntax
Parameters
- sender
- Type: Object
The source of the event.
- e
- Type: ActionEventArgs
The ActionEventArgs instance containing the event data.
Remarks
Every crawl decision this indexer makes happens here. The dispatcher raises the actions in crawl order, and each case below owns one step:
| Action | What this indexer does |
|---|---|
| ImportStarted | Clears the link sets, loads the previous run's page map into PreviousScrapeResults, deletes the old 404.txt, and calls WarmUpSite so the first real request does not pay the application's cold-start cost. |
| IsDocumentToBeCrawledPrescreen | Sets WillCrawl = false for URLs rejected by IsInvalidUrl or already present in ProcessedLinks/NotFoundLinks. This is the cheapest place to stop work — the request is never issued. |
| UseDocumentEncoding | Left as a no-op; the document's own encoding is accepted. |
| ResponseFromServerReceived | Allocates the current PageInformation and AIInfo from the response and records whether the URL should be processed at all. |
| ResponseErrorFromServerReceived | Files the URL under NotFoundLinks (404) or ErrorLinks (anything else, also written to the log table). |
| RawBodyReceived | Captures the response body length and, when the domain has the AI payload enabled, stashes the raw text for later cleaning. |
| ReadingText | Runs VerifyDomain against the parsed document and turns processing off when the page belongs to another domain or language; harvests title, meta data and categories. |
| ReadLinks | De-duplicates the harvested links, records which page each link was found on in LinkDocumentMapList, then prunes the list through GetUniqueListUri. |
| IsDocumentToBeIndexed | The final gate: detects redirects, honours the processing flag, and skips pages whose timestamp and length match the previous run. |
| DocumentIndexed | Records the URL as processed and writes its payload. |
| AutoAssignContent | Delegates to AssignDefaultContentCategories(Object, ActionEventArgs). |
| ImportFinished | Writes the 404 report, clears the sets, and sets ScrapeComplete so BuildIndex()()()() can safely prune stale documents. |
RequestingUri()()()() is deliberately absent. The base subscribes its own handler ahead of this one and uses that action to attach the domain's cookies, the configured request headers, the search key and the platform user agent to every outgoing request. A custom indexer that handles RequestingUri runs second and may adjust the request the base prepared, but must not rebuild it from scratch — doing so drops the search-key header and the domain/language cookies, and the crawler starts indexing the wrong language's pages.
Ordering is a real constraint, not a style choice. PageInformation is only allocated on ResponseFromServerReceived()()()(), so any action that fires before it — or for a URL whose response never arrived — reads the previous page's object. The error case here mutates that stale instance instead of allocating a fresh one; a custom indexer is better off allocating.
Examples
public override void Dispatcher_Action(object sender, ActionEventArgs e) { switch (e.ActionData.Name) { case ActionName.ImportStarted: ProcessedLinks = new HashSet<string>(); NotFoundLinks = new HashSet<string>(); ErrorLinks = new HashSet<string>(); PreviousScrapeResults = SearchHelper.GetPageLinksFromFile(Configuration); WarmUpSite(e); break; case ActionName.ReadLinks: GetUniqueListUri((ArrayList)e.ActionData.Data); break; case ActionName.ImportFinished: Generate404List(); NotFoundLinks.Clear(); ErrorLinks.Clear(); break; } }