YottaBot-SearchBot is the web crawler for Yotta Search, the index behind the web search tool in YottaBot products. It fetches public pages so that a search can return them: each result shows the page's address, its title and short passages from it, and links back to the page.
It crawls for search only. YottaBot does not use what YottaBot-SearchBot fetches to train AI models. If we ever collect web content for training, it will be under a separate crawler name that you can block on its own, and we will list that crawler on this page before it starts.
Who sends requests under this name
| Crawler | What it's for | What happens when you block it |
|---|---|---|
| Yotta Search | Builds the shared index that YottaBot's web search reads. Today it crawls a reviewed list of public documentation sites, such as programming language references and standards. | Your pages leave the index and stop appearing in YottaBot's web search results. |
| A customer's installation | A YottaBot customer's own installation indexes sites its administrators choose, for that customer's use only. Its requests come from the customer's network. | Your pages leave that customer's index and stop appearing in its search results. |
Both use the YottaBot-SearchBot product token, so one robots.txt group controls both. They identify themselves like this (the version number may change):
Mozilla/5.0 (compatible; YottaBot-SearchBot/1.0; +https://yotta.bot/searchbot)Mozilla/5.0 (compatible; YottaBot-SearchBot/1.0; customer; +https://yotta.bot/searchbot)When either fetches robots.txt, it adds robots.txt; before the link, so you can tell those requests apart even if your logs leave out paths:
Mozilla/5.0 (compatible; YottaBot-SearchBot/1.0; robots.txt; +https://yotta.bot/searchbot)Control it with robots.txt
YottaBot-SearchBot follows robots.txt. It obeys a group for YottaBot-SearchBot when your file has one, and the User-agent: * group otherwise. The product token is not case-sensitive. To keep it off your whole site:
User-agent: YottaBot-SearchBot
Disallow: /To keep it out of part of your site:
User-agent: YottaBot-SearchBot
Disallow: /private/robots.txt applies to one host, so add the group to every subdomain you want to exclude. To appear in results, allow YottaBot-SearchBot and requests from our addresses.
- Changes take effect within 24 hours. It rereads robots.txt at least once a day. When it reads a file that disallows pages already in the index, it removes those pages at once.
- Crawl-delay is honoured, up to 30 seconds between requests. Whatever robots.txt says, it never makes more than one request a second to a host.
- If robots.txt can't be read (an error other than 404 or 410, or no answer), it doesn't crawl the site, and tries again later. A 404 or 410 means the whole site may be crawled.
- A 429 or 503 response slows it down. It waits for the time your
Retry-Afterheader asks, from one minute to a day, and five minutes when there is none. - An earlier name,
YottaSearch, still works in robots.txt and in the tags below.
Control single pages
To keep a page out of the index while still letting it be fetched, mark it noindex in either place:
<meta name="robots" content="noindex">
<meta name="YottaBot-SearchBot" content="noindex">
X-Robots-Tag: noindex
X-Robots-Tag: YottaBot-SearchBot: noindexA page marked noindex is removed when it is next fetched. nofollow, in the same places or on a link's rel, stops it following links. none means both.
How it crawls
- Openly. Every request names the crawler and links to this page.
- Gently. One request a second to a host at most, slower when robots.txt or your server asks. When it has a page from before, it asks with
If-None-MatchorIf-Modified-Since, so an unchanged page costs you a 304. - Only what's public. It fetches HTTPS pages served as HTML or plain text. It never logs in, sends cookies, submits forms or runs JavaScript, and it doesn't try to get past CAPTCHAs or other access controls.
- Within your site. It follows redirects only within the same site, at most five in a row.
Check that a request is ours
The Yotta Search crawler, the first user agent above, crawls only from these IP addresses. They are also published as JSON at https://yotta.bot/searchbot.json, in the same format other search crawlers use.
34.232.248.25354.145.60.11554.225.229.90
A request that sends that user agent from any other address is not from YottaBot. A customer's installation crawls from the customer's own network, which we don't publish, and always says customer in its user agent.
Blocking these addresses isn't a reliable way to opt out. It stops the crawler, but it also stops it reading your robots.txt, so it can't learn which pages to remove: pages already in the index stay searchable for up to 30 days. Use robots.txt instead.
Ask for pages to be removed
robots.txt is the fastest way, and needs nothing from us. If you can't change your robots.txt, or want pages removed for another reason, email support@yotta.bot with the subject YottaBot-SearchBot removal. Include the addresses or the address prefix to remove. Write from an address at the site's domain if you can: it lets us verify the request.
We review every request. Once accepted, the pages are deleted from Yotta Search's index and are not indexed again. We email you when that's done. If you think the crawler is misbehaving, for example making too many requests, write to the same address.
A customer's installation keeps its own index inside the customer's network, so a removal request doesn't reach it. robots.txt and noindex do: it removes pages they exclude just as Yotta Search does.