HTTP Headers and Cookies
The crawler visits your site as an anonymous visitor. Any content that depends on being signed in, on a particular consent choice, or on a header your infrastructure requires, is therefore invisible to it — the crawler sees the login page, the consent wall, or an error, and indexes that instead of your content.
The Headers and Cookies tabs let you attach values to every request the crawler makes, so it presents itself the way your site needs.
Note
|
|---|
|
Both tabs require the Professional or Enterprise edition. Where not licensed, a notice appears in place of the list. |
When You Need These
- Members-only content that should be searchable by the members entitled to see it
- A staging or pre-release environment sitting behind a gateway that requires a header
- A site that hides content until a cookie consent choice has been recorded
- Infrastructure — a firewall, CDN, or load balancer — that requires an identifying header before it will serve pages
If your site has none of these, leave both lists empty.
Adding an Entry
Both tabs work identically. Each entry has:
- Name — the header name or cookie name
- Value — the value sent with every request
- An Active / Inactive toggle, and a Delete action
Values apply to every request in the build, including requests to any permitted external domains.
Indexing Restricted Content Safely
Caution
|
|---|
|
Giving the crawler credentials means restricted content enters the search index. The index does not enforce permissions by itself. Unless results are filtered for the current visitor, indexing restricted content can expose it — through result titles and summaries — to people who cannot open the pages themselves. Before configuring the crawler to sign in, confirm how results will be restricted at query time. Security Categories exist for this purpose; see Content, Location and Security Categories. |
|
Important
|
|---|
|
TODO — Confirm and document the supported way to give the crawler an authenticated session. A session cookie captured from a browser will expire and silently stop working between builds, which would make it poor advice. If there is a recommended approach — a long-lived service account, a dedicated header the site recognises — describe it here. If there is not, say so plainly and note the expiry caveat. |
Practical Notes
- Values are stored with the search configuration. Treat any credential placed here as you would any other stored secret.
- If a build that previously worked starts returning login pages in the Index Results, an expired cookie or rotated header value is the first thing to check.
- Use the Active toggle rather than deleting entries when testing whether a header is still required.