Scheduled and Command-Line Indexing

Clicking Build Index in the admin screen is fine for a first build or a one-off refresh, but it depends on someone remembering to do it. For a site whose content changes regularly, the index should be rebuilt on a schedule instead.

BWA ships a console application for exactly this. AdvantageCSP.BuildIndex.exe sits in the site's bin folder alongside the rest of the application, and can be driven by Windows Task Scheduler or any other job scheduler. It runs the same indexing code as the admin screen, so the resulting index is identical — the only difference is what starts it.

What It Does

On each run the tool:

Works out which domain and language combinations to index.
Reads each one's search configuration from the database — the same settings you edit on the Search configuration screen.
Crawls and indexes each one in turn, writing to the same index directory the site reads from.
Writes the same run reports the Results and Logs tabs display.

Because the reports are the same, a scheduled build is reviewed exactly like a manual one. See Reviewing Index Results and Logs.

Note
With no arguments at all, the tool indexes every domain and language that has Enable Indexing ticked, using the database the site itself uses. For most installations that is the whole job, and no parameters are needed.

Command-Line Parameters

Parameters are written as /Name value and are matched without regard to case. Every one of them is optional.

Parameter	Purpose
/Domains	Which domain and language combinations to index. Either the word all, or a JSON array of domain/language ID pairs in the form [{"Key":1,"Value":2}], where Key is the domain ID and Value is the language ID. Omitted, or empty, means all.
/ConnectionString	The database to read configuration from. Rarely needed — see Finding the Database below.
/BaseDir	The root folder of the website. Defaults to the parent of the working directory when that directory is named bin.
/WorkingDirectory	The folder holding the executable and its dependencies. Defaults to the folder the executable is running from, and is set as the process's current directory before indexing starts.
/BrowserDirectory	Where the headless browser used for crawling is stored. Defaults to app_Data\IndexDirectory\ beneath the base directory. The browser is downloaded here on first use, so the account running the job needs write access to it.
/DictionaryDirectory	Where the language dictionary files live. Defaults to a Dictionaries\ folder beneath the browser directory.
/IsSecureConnection	true or 1 to crawl the site over HTTPS; anything else crawls over HTTP. Set this to match how the site is actually served, or the crawler will follow a redirect on every page and waste link depth.

Caution
A parameter's value is everything between it and the next recognised parameter. That means values do not need quoting even when they contain spaces — but it also means a misspelled or unsupported switch is not rejected. It is silently absorbed into the preceding parameter's value, which usually makes that parameter invalid. If a scheduled run behaves as though a parameter were ignored, check the spelling of the switch that follows it.

Caution

A parameter's value is everything between it and the next recognised parameter. That means values do not need quoting even when they contain spaces — but it also means a misspelled or unsupported switch is not rejected. It is silently absorbed into the preceding parameter's value, which usually makes that parameter invalid. If a scheduled run behaves as though a parameter were ignored, check the spelling of the switch that follows it.

Settings in the .config File

Anything that stays the same from run to run is better placed in AdvantageCSP.BuildIndex.exe.config, which ships beside the executable. Command-line parameters override the config file, so the file holds the defaults and the scheduled task only supplies what varies.

appSettings key	Purpose
DefaultConnection	The name of the entry in the connectionStrings section to use. Shipped set to AdvantageCSPConnectionString.
SkipFileExtensions	A comma-separated list of file extensions the crawler will not fetch. Shipped with a broad list covering images, audio, video, fonts, archives, scripts, stylesheets and data files. Add to it if your site links to file types that are wasting crawl time; remove an entry only if you genuinely want that file type indexed.
BaseDirectory	Same as /BaseDir. Ignored unless the folder actually exists.
WorkingDirectory	Same as /WorkingDirectory.
BrowserDirectory	Same as /BrowserDirectory.
DictionaryDirectory	Same as /DictionaryDirectory.
IsSecureConnection	Same as /IsSecureConnection.
Domains	Same as /Domains.

The connectionStrings section holds the connection string itself. It ships empty, which is deliberate — see below.

Finding the Database

The tool locates the database in this order, stopping at the first one that works:

A connection string supplied on the command line.
The connection string named by DefaultConnection in the tool's own .config file — but only if a test connection to it actually succeeds.
The site's own web.config in the base directory, again using its DefaultConnection setting, and again only if a test connection succeeds.

Because of the last step, leaving the connection string empty in the tool's .config file is normally the right choice: the tool picks up whatever the site is using, and there is no second copy of the credentials to keep in step when the database moves.

Note
If no database can be reached, the tool reports that a connection string is required and stops without indexing anything.

Setting Up a Scheduled Task

Create a Windows Task Scheduler task on the web server.
Set the action to start AdvantageCSP.BuildIndex.exe from the site's bin folder.
Set Start in to that same bin folder. The tool resolves the site root from it, so getting this wrong is the most common cause of a task that runs and indexes nothing.
Add arguments only if you need to narrow the scope or override a path.
Run the task under an account with read access to the site folder, write access to the index and browser directories, and access to the database.
Choose a schedule that suits how often content changes — nightly, outside business hours, suits most sites.
Set the task to stop if it runs longer than expected, so a stalled crawl cannot overlap the next run.

Caution
Do not schedule runs so close together that one build can still be running when the next starts. Two builds writing to the same index directory will interfere with each other. Pick an interval comfortably longer than a full build takes, and use the scheduler's own setting to prevent a second instance starting.

Checking That It Worked

The tool writes nothing to the console, so a task that reports as having run tells you very little on its own, and its exit code is not a reliable success signal either. Confirm the outcome from the admin screen instead: open the Results tab and check that Index Results reflects the run, then read the build log on the Logs tab. Errors encountered during a run are recorded through the platform's normal error logging.

Remember that a scheduled build is incremental, exactly as a manual one is: pages whose content has not changed since the previous run are left alone. If a scheduled run needs to reindex everything — after a stop list change, for instance — use Clear on the Index Results tab beforehand. See Reviewing Index Results and Logs.