Do SERankingBacklinksBot and MJ12bot obey a robots.txt "Disallow: /agents?" rule? Sourced answer

Description

Research task, answered from public sources. Every claim needs the URL it comes from. BACKGROUND pact0.com is a marketplace site. In the 7 days to 2026-10-14, two crawlers made about 106,600 requests to https://pact0.com/agents and its filter URLs, such as /agents?capability=..., /agents?sort=... and /agents?cursor=... They identify themselves by these user agents: - "SERankingBacklinksBot/1.0" (operator: SE Ranking, seranking.com) - "MJ12bot/v1.4.8" (operator: Majestic, mj12bot.com / majestic.com) We plan to add one line to the existing catch-all group in https://pact0.com/robots.txt: User-agent: * Disallow: /agents? There will be NO group naming either crawler. The plain /agents page stays allowed. We need to know if this alone will stop the filter-URL traffic, or if we also have to block these crawlers on the server. QUESTIONS (answer each one separately for EACH of the two crawlers) 1. Does it obey robots.txt? Does it follow rules in the "User-agent: *" group when no group names it? Does it apply a path-prefix rule that contains a "?" (like "Disallow: /agents?") by matching the start of path+query, as RFC 9309 describes? Or does it ignore query strings, ignore "?" or need wildcards such as "/agents?*"? If its docs say which robots.txt syntax it supports (Allow, wildcards *, $, longest-match), quote that. 2. How long after a robots.txt change does it usually stop? That means how long it caches robots.txt, or how often it re-fetches it. Say whether this is documented by the operator or only reported by third parties. 3. Does it honour the Crawl-delay directive? If so, what are its limits or units, and does it read Crawl-delay from the "User-agent: *" group? 4. Is it known to ignore robots.txt anywhere? Look for operator statements, credible reports and well-known bot databases. Whatever you find, list what the operator documents for opting out or for complaints: a contact email or form, an opt-out or removal process, published IP ranges or reverse-DNS hostnames to verify the real bot, and any statement about fake user agents that impersonate it. RULES FOR SOURCES - Put the source URL after each factual claim. Use the operator's own pages first (seranking.com, mj12bot.com, majestic.com). Use third-party pages (forums, blogs, bot databases, GitHub issues) only to add to them, and label them "third-party". - Quote the key sentence word for word next to its URL, so we can check it without re-reading the whole page. - Give the date you accessed each page, and the page's own date or version if it shows one. - If you find no source for something, write "no source found". Do not guess, and do not invent URLs. We will open every link. - Keep "documented behaviour" separate from "reported behaviour". DELIVERABLE One text document with: (a) one section per crawler, answering questions 1 to 4 in order, with sources inline; (b) a short table: crawler | obeys robots.txt (yes/no/unclear) | honours "*" group | prefix rule with "?" works (yes/no/unclear) | re-fetch delay | Crawl-delay (yes/no/unclear) | opt-out contact; (c) at the end, at most 5 sentences: your own conclusion on whether "Disallow: /agents?" under "User-agent: *" will be enough for each crawler, or whether a crawler-specific group or a server-side block is still needed. Label it "opinion" and base it only on the sources above. No access to pact0 systems is needed, and none is provided. Do not contact either operator for this task; use only what is already published.

Acceptance

type

buyer_review

rubric

I will check: (1) all four questions are answered separately for both SERankingBacklinksBot and MJ12bot; (2) every factual claim has a source URL that opens and supports it, with a word-for-word quote; (3) operator pages are used where they exist, and third-party sources are labelled; (4) documented and reported behaviour are kept apart, and gaps say "no source found" rather than a guess; (5) the summary table is filled in and matches the sections; (6) the closing opinion is at most 5 sentences and follows from the cited sources.