Controlling Which Pages Are Indexed
By default the crawler follows every link it finds within the site, up to the configured link depth. The Link Paths tab lets you narrow that — keeping areas out of the index — or widen it, by permitting the crawler to follow links onto named external domains.
Paths to Exclude
Each entry in this list is matched against every URL the crawler is about to visit. If the entry's text appears anywhere in the URL, that URL is skipped and its links are not followed.
The match is a plain substring match, not a pattern. An entry of /dir1/ causes http://mydomain.com/dir1/file.aspx to be skipped.
|
Column |
Purpose |
|---|---|
|
Description |
A label for your own reference. Not used in matching — use it to record why the exclusion exists. |
|
Path |
The text matched against the URL. |
Caution
|
|---|
|
Because the match is a substring anywhere in the URL, short entries exclude far more than intended. An entry of news would exclude /news/, but also /newsletter/ and any page whose title slug happens to contain the word. Include the surrounding slashes — /news/ — to keep an exclusion tight. |
Typical exclusions include printer-friendly duplicates of existing pages, paginated archives that add no new text, and administrative or utility paths that should never appear in visitor results.
Entries can be toggled active or inactive, which is a safer way to test the effect of an exclusion than deleting and re-adding it.
External Domains
Normally the crawler stays within the current site. Adding a domain to the External Domains list permits it to follow links onto that domain and index what it finds there, which is how content spread across several hosts is brought into one search index.
|
Column |
Purpose |
|---|---|
|
Domain Name |
A label for your own reference. |
|
Host Entry |
The host the crawler is permitted to follow links onto. |
Note
|
|---|
|
External Domain linking requires the Enterprise edition. Where it is not licensed, the tab shows a notice in place of the list. |
|
Caution
|
|---|
|
Add external domains deliberately and one at a time. A permitted domain that links onward to further sites can pull a large volume of unrelated content into your index and greatly extend build times. Check the Index Results after the first build following any addition. |
Choosing Between the Available Tools
There is more than one way to keep something out of search results. Pick by scope:
|
To exclude |
Use |
|---|---|
|
A whole area of the site, by URL |
Paths to Exclude, on this tab |
|
Part of a page — navigation, footers, repeated furniture |
Ignore blocks in the template — see Controlling Indexing from Page Markup |
|
Individual common words |
The stop list — see Custom Words and Stop Lists |
|
Everything beyond a certain depth |
Maximum Link Depth — see General Settings |