Extending the Indexer

The Search Index DLL setting names the assembly BWA loads to perform indexing. It defaults to the indexer shipped with the platform:

AdvantageCSP.Keyoti.SiteIndexer.dll

Because the indexer is resolved by name at build time rather than compiled in, a custom indexer can be deployed alongside the platform and selected per domain. One domain can use a custom indexer while others continue on the standard one.

A custom indexer derives from SearchIndexPluginBase (in AdvantageCSP.Keyoti), which supplies the crawl bookkeeping — processed, error and not-found link tracking, redirect mapping, and the page and metadata models — and leaves three members to implement:

The base class exposes the current AdvantageSearchDomain, giving an implementation access to the domain and language being indexed together with every setting configured on the Search screen — crawl depth, exclude paths, headers, cookies, categories and weights. A custom indexer should honour those settings so that the configuration screen continues to mean what it says.

The class implements IDisposable; override the protected Dispose to release anything your implementation holds.

The Build with Advantage menu in Visual Studio scaffolds the project for you — BWA → Search Index Plugin — and the generated class library already references the assemblies the contract needs. The skeleton it produces looks like this:

using Keyoti.SearchEngine;
using Keyoti.SearchEngine.Events;
using AdvantageCSP.Keyoti;

public class SearchIndexer : SearchIndexPluginBase
{
    public SearchIndexer(Configuration configuration, AdvantageSearchDomain searchDomain,
                         string cookieLanguage)
        : base(configuration, searchDomain, cookieLanguage)
    {
    }

    public override SearchIndexPluginIndexResult BuildIndex()
    {
        SearchIndexPluginIndexResult result = new SearchIndexPluginIndexResult();
        result.Success = false;

        DocumentIndex documentIndex = new DocumentIndex(Configuration);
        try
        {
            if (!string.IsNullOrEmpty(SearchDomain.PrimaryUrlPath))
            {
                documentIndex.Import(new WebsiteBasedIndexableSourceRecord(
                    SearchDomain.PrimaryUrlPath, PathsToExclude, PathsToInclude));
            }

            documentIndex.Optimize();
            result.Success = true;
            result.Message = "Success";
        }
        catch (Exception ex)
        {
            result.Message = "Failed";
            result.Error = ex;
        }
        finally
        {
            documentIndex.Close();
        }

        return result;
    }

    public override void Dispatcher_NeedObject(object sender, NeedObjectEventArgs e)
    {
        if (e.RequiredObject is ParserProvider)
            e.RequiredObject = new ExtendedParserProvider(e.Configuration, SearchDomain);
    }

    public override void Dispatcher_Action(object sender, ActionEventArgs e)
    {
        switch (e.ActionData.Name)
        {
            case ActionName.AutoAssignContent:
                // Classify the document being written to the index.
                break;
        }
    }
}

Three things in that skeleton are worth drawing out.

The engine reaches your code through events, not method calls, and the base class has already wired them. The SearchIndexPluginBase constructor subscribes both Dispatcher_Action and Dispatcher_NeedObject to the configuration's central dispatcher on your behalf. Do not subscribe them again in BuildIndex — a second subscription does not replace the first, so every crawl event is then handled twice.

Caution
The base constructor catches and logs every exception it raises rather than letting it escape. A binding that fails leaves a live object with no subscriptions and a crawl that indexes nothing without reporting an error, so check the search error log before treating an empty index as a configuration problem.

Returning a result is how the build reports itself. Populate Success, Message and, on failure, Error. That result is what the Search screen and the scheduled indexing log display, so a failure swallowed here becomes a build that silently produces nothing.

Replacing the parser provider is optional. Returning ExtendedParserProvider gives your crawl the platform's own parsing behaviour, including the boost markers driven by the keyword and description weights configured on the Search screen. Leave the override empty and the engine uses its default parser instead.

The base class exposes the settings the Search screen writes, and a custom indexer is expected to respect them — otherwise the configuration screen stops describing what actually happens. The members most implementations need:

Note
These members are protected internal. They are available to a class deriving from SearchIndexPluginBase, which is how a custom indexer reaches them, but they are not a public API for other code to call.

The shipped SearchIndexer is a working implementation of this contract and the best starting point for a custom one. It crawls the site through a headless browser so that pages relying on client-side rendering are indexed as a visitor sees them, and assigns default content categories as pages are processed.

Its Dispatcher_Action is the part most worth reading. The engine raises a named action at each decision point in the crawl — whether a document will be crawled, whether it will be indexed, what categories it receives — and the standard indexer answers them from the configured settings. A custom indexer overrides the same switch and answers them from whatever its own rules require.

Replacing the indexer is a significant commitment — the custom assembly becomes your responsibility across platform upgrades. Most requirements that appear to need one are met by configuration:

A custom indexer is justified where content must be indexed that has no crawlable web presence at all, or where classification depends on rules only your systems can supply.