robots.txt
LiveIs this path allowed for Googlebot — or GPTBot?
Fetches public /robots.txt with the Site Audit SSRF guard and evaluates a path for a crawler token.
Server URL
https://toolcargo.com/mcp/robots-txtClaude Code
claude mcp add --transport http --scope user toolcargo-robots-txt https://toolcargo.com/mcp/robots-txt \
--header "Authorization: Bearer YOUR_API_KEY"Create a key in your dashboard. Other clients: setup guides.
What it helps you do
- Allowed / disallowed for a user-agent
- Listed sitemaps
- Crawl-delay if set
Built for: SEO teams checking crawl rules before launch.
Example requests
GPTBot
“Inspect robots.txt for client.com path /blog user agent GPTBot with ToolCargo.”
Allowed or not, plus sitemap URLs.
Step-by-step workflows: pre-launch check · fixing link previews
Tools
| Tool | What it does | Access |
|---|---|---|
inspect_robotsInspect robots.txt | Fetch a public /robots.txt and say whether a path is allowed for a crawler token (Googlebot, GPTBot, …). | Read-only |
Requirements
- • A ToolCargo account with Site Audit activated (free).
Limitations
- • One file, no wild-host crawl. Private hosts refused.
Supported clients
“Tested” means we connected that client to this endpoint and ran a tool call ourselves. “Documented” means the client supports this setup per its own docs, but we have not tested it yet.
| Claude Code | API key header | Tested | Remote HTTP server with an Authorization header. Tested 2026-09-11 (v2.1.268). |
| Claude (claude.ai) | OAuth sign-in | Tested | Custom connector with OAuth sign-in; tested on claude.ai web 2026-09-11. Desktop and mobile use the same account connectors but were not tested separately. |
| Cursor | API key header | Documented, not yet tested | mcp.json with url and headers. |
| VS Code (Copilot agent mode) | API key header | Documented, not yet tested | .vscode/mcp.json with type http and headers. |
| ChatGPT (developer mode) | OAuth sign-in | Documented, not yet tested | Requires OAuth; API keys cannot be entered. |
| MCP Inspector | API key header | Tested | CLI mode, tested 2026-09-11. Useful for calling tools directly. |
More in SEO
Pricing
Free
$0For trying the connector on your own sites.
50 calls/month · 25 links per check
Pro
$12/moFor freelancers and agencies auditing client sites every week.
2,000 calls/month · 100 links per check
Proposed test price. See the pricing page for checkout status.