Custom Words and Stop Lists

Stop words are words so common that indexing them adds noise rather than value — typically articles, conjunctions and prepositions such as "the", "and", "of". Words on the stop list are left out of the index entirely, which keeps the index smaller and stops near-universal terms from returning most of the site.

Caution
A stop-listed word cannot be found, even inside a quoted phrase. Before adding a word, consider whether it is genuinely meaningless on your site. Short words that are ordinary English but significant in your subject matter — a product called "The One", an organisation abbreviated "IT" — will disappear from search if stop-listed.

Note

Stop list changes only take effect after the index is rebuilt, because the list is applied while pages are being indexed. Saving holds the edited list aside as a pending list; the next index build adopts it before it starts crawling. You do not need to use Clear on the Results tab first. Adopting a pending stop list discards the record of previously indexed pages automatically, so the build that picks up your change is always a full rebuild.

The Custom Words list adds vocabulary the engine does not know from ordinary language — product names, brand names, acronyms, technical terms, and proper nouns specific to your organisation.

Its purpose is to stop the engine second-guessing those terms. Word matching is driven by the Lemma language set on the General tab, which recognises a word and its variants by comparing it against that language's vocabulary. A term the lemma language has never heard of — an acronym, a made-up product name — looks to it like a misspelling of something it does know, and the visitor gets offered a different word instead of the results they asked for. Listing the term under Custom Words makes it known vocabulary, so it is accepted as typed and no alternative is suggested.

Acronyms are the clearest case. An unfamiliar three- or four-letter term is close enough to several ordinary words that the engine will readily suggest one of them. Any acronym your organisation uses in content and expects visitors to search for belongs on this list.

Note
Unlike the stop list, Custom Words takes effect as soon as you save. The list is not used while pages are being indexed — it is consulted when a visitor types a query — so there is nothing to rebuild.

Caution

Custom Words only has an effect on a site whose search page includes a suggestion control. The results module supplied with the default theme does not, so on a site using it as delivered the list is saved but never consulted. Making it live is a development task, not a configuration one: the search page needs a Keyoti SearchSuggestions control, and the platform helper SearchHelper.SetCustomDictionary called against it to point it at the list saved for the current domain and language. If suggestions are already working on your site, Custom Words is already being applied.

Both lists use the standard text editor. Enter one word per line, then save the configuration. The format is deliberately plain, and both lists are tidied up when you save:

• One word per line. Blank lines are ignored, so you can leave them in while editing.
• Case does not matter. Entries are converted to lower case when saved and matched without regard to case. An acronym typed in capitals will read back in lower case — that is expected, and it still matches the capitalised term on your pages.
• Duplicates are removed, and the saved order is not the order you typed. Expect the list to come back rearranged; there is no point sorting or grouping it by hand.
• Comments are not supported. There is no comment character. A line beginning with a hash or a slash is stored as an ordinary entry, so anything you type as a note becomes a word on the list.
• Do not indent. Leading and trailing spaces are kept as part of the entry, so an indented line will not match the word you meant.
• Single words only. A line containing spaces is stored exactly as typed, but neither list is built for phrases — the stop list is applied word by word as pages are indexed, and Custom Words is a vocabulary of terms. Enter each word on its own line instead.

A Custom Words list holding the acronyms and product names an organisation uses in its own content:

aquaflow
hydroline
polybraid
astm
nsf

A stop list of ordinary English words that add nothing to the index:

the
and
of
a
for
with

Note
Both lists are held as plain text files beside the index, under the domain's search index directory in a folder named for the language — customDictionary.txt and stoplist.txt, with an edited stop list waiting in stoplist_pending.txt until the next build adopts it. Because they live with the index rather than in the database, they do not travel with a database restore.