PriceMovedBot

PriceMovedBot is the crawler for PriceMoved. PriceMoved will watch public competitor pricing pages and tell subscribers when they change.

PriceMoved is currently in waitlist-only mode, so PriceMovedBot is not fetching any pages yet. This page describes how it will behave once monitoring is switched on.

How it identifies itself

Every request, including robots.txt requests, sends the User-Agent PriceMovedBot/1.0 followed by a link to this page.

What it fetches

• Only public URLs that a PriceMoved subscriber has added. It does not follow links or discover new pages.
• Plain GET requests with no cookies or credentials, so it only sees what a logged-out visitor sees. It never submits forms.
• Each URL at most daily, every 6 hours, or hourly, depending on the subscriber’s plan.

robots.txt

PriceMovedBot requests /robots.txt again before every page fetch and follows RFC 9309:

• Rules in a User-agent: PriceMovedBot group apply (matched case-insensitively; User-agent: PriceMoved also counts). If no group names it, the User-agent: * group applies. A group may list several User-agent lines.
• Allow and Disallow are both supported, including * wildcards and the $ end anchor. The longest matching rule wins. If an Allow and a Disallow rule are the same length, Allow wins.
• If a URL is disallowed, it is not fetched.
• If robots.txt returns a server error or 429 (Too Many Requests), or can’t be reached, the URL is skipped for that run.
• If robots.txt returns any other client error, such as 404, there are no rules and the URL may be fetched.
• If the rules that apply to PriceMovedBot are too large to evaluate safely (more than 1,000 rules, a rule longer than 2,048 characters, a URL path and query longer than 2,048 characters, or too much work to parse and match them), the URL is skipped for that run.
• Only the first 500 KiB of robots.txt is read; anything after that is ignored.

To block PriceMovedBot from your whole site:

User-agent: PriceMovedBot
Disallow: /

Rate limits

• At least 5 seconds between checks of the same host (each check is one robots.txt request followed by one page request).
• At most 2 pages per host per scheduled run.

Contact

Questions, or want your site excluded? Email digitalpromohub.support+pagewatch@gmail.com. See also our Privacy Policy and Terms.

← Home