Robots.txt + Sitemap + Meta Analyzer
Inspect robots.txt, sitemap discovery and head meta of a page.
- slug
- robots-meta-analyzer
- visibility
- Public
- risk
- safe
- level
- 0
What this tool does.
Fetches the root robots.txt of the target site, discovers declared sitemaps, and inspects the canonical, title and robots meta of the page itself.
Built for fast SEO triage of a single URL.
Run this tool.
Submit an input and see the runner output here. Execution is server-side and governed by the policy shown in the sidebar.
Everything runs server-side. Inputs and outputs are subject to the tool's policy shown in the sidebar.
Dev-docs · 2 documents.
Related articles.
Why Robots.txt, Sitemaps and Metadata Still Matter
A practical guide to reading robots.txt, XML sitemaps, canonical links and indexing metadata as separate evidence layers — and using contradictions between them to generate better OSINT questions.
Getting Started with Public Surface Analysis
A practical, passive-first workflow for reading DNS, HTTP, robots, metadata, technology and historical signals without overclaiming.
A Responsible Method for Reconnaissance on Public Web Surfaces
A practical framework for deciding what to collect, when public research becomes active testing, how to define authorization and scope, and when to stop.
How to Turn Weak Signals into Better Questions
A practical OSINT reasoning framework for turning ambiguous observations into testable hypotheses, seeking disconfirming evidence and expressing conclusions with calibrated confidence.