About our crawler
TheirMoves is a weekly competitor-monitoring service. To tell a customer what changed on a competitor's website, we read that website's public pages. This page explains exactly what that means, so you can decide whether to allow it.
Why you might see us in your logs
Someone added your site as a competitor to watch. When we check it, the request comes from our servers with the user agent below. We do not pretend to be a browser, and we do not hide behind a different identity.
- User-Agent:
TheirMovesBot/1.0 (+https://theirmoves.com/bot)
What we fetch
For each site we watch, we read only public, unauthenticated URLs:
robots.txt, to read the site's rules before we ask for anything.- The public
sitemap.xml(or the sitemap listed inrobots.txt), to see which pages exist and when they were last updated. - The small number of pages that look new or changed since the last check.
We do not submit forms, we do not log in, and we do not fetch anything behind a paywall, a login, or a bot check. If a page needs a session or a cookie to render, we leave it alone.
How often and how fast
- About once a week per site, when the customer's briefing runs.
- At most one request per second to any one site. In practice we are far slower: a normal site is a handful of requests, then we stop.
- If a site returns errors or rate limits, we back off instead of retrying harder.
robots.txt, and how to block us
We respect robots.txt. To keep us out of one path, or out of the whole site, add a rule for our user agent:
User-agent: TheirMovesBot
Disallow: /
# or just the private parts
User-agent: TheirMovesBot
Disallow: /account/
We used to be called CompetitorRadarBot. Rules you wrote for that name still apply.
A Disallow for the whole site means we cannot include your site in any briefing, so a customer watching you will simply see no changes. If you would rather be read but not followed into a section, disallow just that section.
What we keep
For the pages we read, we keep the URL, the page's last-modified date, and short excerpts of the text we compare week to week. We do not keep your visitors' data, we do not run scripts on your pages, and we do not republish your pages. What we do publish is covered by our privacy policy.
Questions or a problem
If our crawler caused trouble, ignored your robots.txt, or hit you harder than one request per second, tell us and we will fix it. Email [email protected] with the user agent line from your logs and the time (UTC) of the request.
Want to see what we can tell a customer about any site? Try the free website change checker — it lists the pages any website added or updated in the last 30 days, with no signup.