TheirMoves
TheirMoves

About our crawler

TheirMoves is a weekly competitor-monitoring service. To tell a customer what changed on a competitor's website, we read that website's public pages. This page explains exactly what that means, so you can decide whether to allow it.

Why you might see us in your logs

Someone added your site as a competitor to watch. When we check it, the request comes from our servers with the user agent below. We do not pretend to be a browser, and we do not hide behind a different identity.

What we fetch

For each site we watch, we read only public, unauthenticated URLs:

We do not submit forms, we do not log in, and we do not fetch anything behind a paywall, a login, or a bot check. If a page needs a session or a cookie to render, we leave it alone.

How often and how fast

robots.txt, and how to block us

We respect robots.txt. To keep us out of one path, or out of the whole site, add a rule for our user agent:

User-agent: TheirMovesBot
Disallow: /

# or just the private parts
User-agent: TheirMovesBot
Disallow: /account/

We used to be called CompetitorRadarBot. Rules you wrote for that name still apply.

A Disallow for the whole site means we cannot include your site in any briefing, so a customer watching you will simply see no changes. If you would rather be read but not followed into a section, disallow just that section.

What we keep

For the pages we read, we keep the URL, the page's last-modified date, and short excerpts of the text we compare week to week. We do not keep your visitors' data, we do not run scripts on your pages, and we do not republish your pages. What we do publish is covered by our privacy policy.

Questions or a problem

If our crawler caused trouble, ignored your robots.txt, or hit you harder than one request per second, tell us and we will fix it. Email [email protected] with the user agent line from your logs and the time (UTC) of the request.

Want to see what we can tell a customer about any site? Try the free website change checker — it lists the pages any website added or updated in the last 30 days, with no signup.