Robots.txt + Sitemap + Meta Analyzer
Inspect robots.txt, sitemap discovery and head meta of a page.
- visibility
- Public
- risk
- Low risk
What this tool does.
Fetches the root robots.txt of the target site, discovers declared sitemaps, and inspects the canonical, title and robots meta of the page itself.
Built for fast SEO triage of a single URL.
Run this tool.
Enter a target to view results from this tool. Review its scope and access requirements before running a search.
Your input is processed by OSINT.dev and, where needed, sent to the provider used by this tool.
Documentation · 2 documents.
Related articles.
Why Robots.txt, Sitemaps and Metadata Still Matter
A practical guide to reading robots.txt, XML sitemaps, canonical links and indexing metadata as separate evidence layers — and using contradictions between them to generate better OSINT questions.
Getting Started with Public Surface Analysis
A practical, passive-first workflow for reading DNS, HTTP, robots, metadata, technology and historical signals without overclaiming.
A Responsible Method for Reconnaissance on Public Web Surfaces
A practical framework for deciding what to collect, when public research becomes active testing, how to define authorization and scope, and when to stop.
How to Turn Weak Signals into Better Questions
A practical OSINT reasoning framework for turning ambiguous observations into testable hypotheses, seeking disconfirming evidence and expressing conclusions with calibrated confidence.