How Search Works

BWA Search is built on Keyoti SearchUnit, a commercial .NET search engine from Keyoti that has been deeply embedded into the platform.

It is included as part of BWA. There is nothing separate to buy, install, license or maintain, and no separate administration tool — BWA wraps SearchUnit's configuration, crawling and index management behind the administration screens described in these topics. You do not interact with SearchUnit directly.

Advantage also extends it. Most significantly, pages are crawled through a headless browser rather than fetched as raw HTML, so content assembled by JavaScript is indexed correctly. That is a BWA addition, not stock SearchUnit behaviour, and it is the reason the indexing cycle below renders each page rather than simply downloading it.

A few terms on the configuration screen come from the search engine rather than from BWA, notably Lemma and Stop Words. Those are explained in plain language in the topics where they appear.

Note
Keyoti's own documentation describes the engine's capabilities in general terms and can be useful background. Treat the BWA topics here as authoritative for anything to do with configuration: what SearchUnit exposes and what BWA exposes are not the same set, and some SearchUnit features are driven by BWA on your behalf or are not surfaced at all.

When an index build runs, BWA works through the following steps for the selected domain and language:

1. Start at the site root. The crawler begins at the domain's primary URL.
2. Render the page. Each page is opened in a real browser engine rather than simply downloaded. This matters: pages that build part of their content with JavaScript are indexed as a visitor would see them, not as empty shells.
3. Extract and weight the text. Visible text is collected. Meta keywords and meta description are collected separately so they can be given extra weight. Regions marked as ignored in the page markup are skipped.
4. Apply categories. The page is tagged with any content, location, and security categories that apply, so results can later be filtered.
5. Follow links. Links found on the page are queued, subject to the maximum link depth, the exclude-path list, and the external domain list.
6. Record the outcome. Every URL is logged as indexed, redirected, or not found, and those lists are what you review on the Results tab.

The cycle repeats until every reachable page within the depth limit has been visited.

Because search reads from an index rather than from live content, a newly published page is invisible to search until the next index build includes it. If a page is missing from search results, the first question to ask is whether the index has been rebuilt since the page went live.

Nothing rebuilds the index on its own. A build happens when someone clicks Build Index, or when a scheduled job runs the indexing tool. There is no built-in interval setting, and publishing a page does not queue a build.

On any site whose content changes regularly, set up the schedule. BWA ships a console application, AdvantageCSP.BuildIndex.exe, in the site's bin folder; point Windows Task Scheduler at it and choose a frequency that suits how often your content changes. It runs the same build as the button, so the result is identical. See Scheduled and Command-Line Indexing.

BWA keeps a record of what it indexed on the previous run so that subsequent builds do not have to start from nothing. Clearing that record from the Results tab forces a complete rebuild on the next run. See Reviewing Index Results and Logs.

Each domain and language combination gets its own index directory on disk, organised as a domain folder containing one subfolder per language. Alongside the index, the same location holds the run reports — the processed-link list, the redirect list, and the not-found list — which is how the Results tab and log viewer get their data.

Note
Because the index is a set of files on disk rather than rows in the database, an index is not carried across by a database restore. A newly restored or newly deployed environment needs its index built.

When a visitor submits a search, the query runs against the index for the current domain and language only. Supported query forms include:

• Phrases — words surrounded by double quotes are matched as a unit
• Wildcards — an asterisk indicates a partial match
• Boolean expressions — AND, OR and NOT, which may be grouped and nested
• Word variants — controlled by the Lemma setting, so related forms of a word match one another

The visitor reaches this through a search box on the site, which sends them to a results page carrying the search results module. BWA supplies both, and the default theme unpacks them when a site is created. See Adding Search to Your Site.