A Responsible Method for Reconnaissance on Public Web Surfaces

Reconnaissance is not a single technique.

It is a spectrum of activities ranging from reading public records and archived pages to sending requests directly to infrastructure and, at the far end, deliberately testing systems for weaknesses.

Those activities are not equivalent.

A responsible analyst should therefore ask a question before choosing a tool:

What level of interaction is actually necessary to answer the research question?

Good reconnaissance begins with restraint.

The objective is not to collect everything that can technically be discovered. It is to gather enough evidence to answer a legitimate question while minimizing impact, avoiding unnecessary collection, and staying within the applicable authorization boundary.

This guide describes a practical framework for doing that.

It is a methodology guide, not legal advice. Laws, contracts, platform policies, and authorization requirements vary by jurisdiction and context.

Reconnaissance is a continuum

The word reconnaissance is used loosely.

In OSINT, it may describe something as passive as reading DNS records.

In security testing, it may describe deliberate enumeration, scanning, or probing before a penetration test.

Those activities should not be treated as interchangeable.

A useful model is to divide reconnaissance into three interaction levels.

Level 1 — Passive or third-party observation

At the lowest-interaction level, the analyst does not deliberately probe the target infrastructure.

Examples include:

• querying public DNS;
• reviewing certificate-transparency records;
• searching public search engines;
• consulting public corporate records;
• reviewing archived web pages;
• examining historical screenshots;
• using public datasets created by third parties;
• reading publicly indexed documentation.

This is often sufficient for OSINT questions involving:

• infrastructure history;
• domain relationships;
• technology changes;
• public branding;
• email configuration;
• certificate issuance;
• historical web presence.

Passive does not mean risk-free.

Public datasets may contain stale information.

Third-party services may have collected data using their own methods.

A historical record may describe an old state rather than the current one.

The analyst still needs to evaluate provenance, freshness, and context.

But when passive evidence can answer the question, there is usually little analytical reason to escalate to more intrusive collection.

Level 2 — Ordinary interaction with a public service

The next level involves contacting infrastructure in the way an ordinary client would.

Examples may include:

• requesting the public homepage;
• following redirects;
• retrieving public response headers;
• requesting robots.txt;
• requesting a declared sitemap;
• reading HTML metadata;
• retrieving a published security.txt;
• making a small number of requests to public resources needed for the research question.

These activities interact directly with the target.

That distinction matters.

Even when a resource is publicly reachable, analysts should consider:

• request volume;
• request frequency;
• server load;
• terms or policies;
• authentication boundaries;
• personal data;
• whether the request is actually necessary.

The responsible default is:

Use the smallest number of ordinary requests necessary to answer the question.

A workflow that needs three requests should not automatically become a crawler that makes three thousand.

Level 3 — Deliberate active security testing

A different boundary is crossed when the objective changes from observing a public surface to deliberately testing a system.

Examples include:

• vulnerability scanning;
• broad port scanning;
• systematic endpoint enumeration;
• credential testing;
• brute-force activity;
• exploit validation;
• authenticated testing;
• payload delivery;
• techniques intended to bypass access controls.

These activities belong to security testing rather than ordinary public-surface OSINT.

They require an appropriate authorization basis and clearly defined scope.

NIST describes Rules of Engagement as the detailed constraints established before security testing begins, giving a test team authority to perform specifically defined activities.

That is a useful principle even outside formal penetration testing:

The more intrusive the technique, the more explicit the authorization and scope must become.

Start with the research question

Before collecting anything, write down the question.

Examples:

Has this organization changed its public web infrastructure?

Which domains appear to support this public service?

Is a historical domain still redirecting to the current site?

What public technologies are visible on the website?

Does the organization publish a vulnerability-disclosure channel?

The question establishes relevance.

Without it, reconnaissance tends to expand indefinitely.

Once the question is defined, write down what evidence would actually answer it.

For example:

Question

Has the website recently migrated?

Potential evidence

• historical archive;
• DNS changes;
• redirect behavior;
• canonical URL;
• public technology indicators;
• certificate history.

That list already tells you something important:

You probably do not need a vulnerability scanner.

Define the authorization boundary

Before moving beyond passive or ordinary public interaction, determine what authority exists for the activity.

In professional security testing, scope is normally explicit.

Useful questions include:

• Which domains are in scope?
• Which IP ranges are in scope?
• Are subdomains included?
• Are third-party services excluded?
• Which testing techniques are allowed?
• Are authenticated systems included?
• Are production systems included?
• Are there request-rate limits?
• Is social engineering allowed?
• Is denial-of-service testing prohibited?
• What data may be accessed or retained?
• Who should be contacted if something unexpected happens?
• When must testing stop?

If those answers are unclear, escalation should stop until they are clarified.

A practical passive-first workflow

For ordinary public-web reconnaissance, use the following sequence.

Step 1 — Define the target precisely

Record:

• hostname;
• URL;
• organization or project being researched;
• date and time;
• research purpose.

Do not silently expand:

example.com

into:

every domain, IP address, employee, cloud account, supplier, and subsidiary that appears related.

Relationship discovery and authorized scope are different concepts.

Step 2 — Collect passive evidence first

Start with sources that do not require deliberate interaction with the target.

Depending on the question:

• DNS records;
• certificate-transparency records;
• search-engine results;
• public archives;
• published corporate information;
• public internet measurement datasets.

Certificate-transparency services such as crt.sh can reveal certificates that have been publicly logged.

Historical archives can reveal earlier versions of websites.

Services such as urlscan.io may expose previously collected public web observations.

These datasets are extremely useful, but they represent observations made at particular times and through particular collection methods.

Treat them as evidence that requires context.

Step 3 — Identify gaps

After passive collection, ask:

What do I still not know?

Perhaps the archive suggests a migration but does not show the current redirect.

Perhaps DNS identifies a hostname, but you do not know whether it serves a public website.

Perhaps public data suggests a technology change, but the current page needs verification.

Only those gaps should drive direct requests.

Step 4 — Make minimal public requests

When direct interaction is justified, begin with normal public resources.

For a website that might mean:

1. request the intended public URL;
2. record the HTTP status;
3. follow the redirect chain;
4. record the final hostname;
5. inspect relevant response headers;
6. retrieve public metadata if needed.

Do not automatically enumerate adjacent endpoints.

Do not treat every discovered hostname as permission to probe it.

The goal is to validate a hypothesis, not expand the attack surface map.

Step 5 — Record observations before interpretation

For each important observation, capture:

• timestamp;
• exact target;
• method;
• result;
• source;
• analyst note.

Separate:

old.example.com returned HTTP 301 to www.example.com

from:

the company migrated its entire infrastructure.

The first is an observation.

The second is an interpretation requiring additional evidence.

robots.txt is not an authorization system

robots.txt is frequently misunderstood in both directions.

The Robots Exclusion Protocol provides instructions to automated crawlers about URI access.

It is not an access-control mechanism.

A Disallow rule does not make a public path secret.

Equally, the absence of a Disallow rule does not grant permission to perform intrusive testing.

Do not reason:

robots.txt allows this path, therefore every form of interaction with it is authorized.

And do not reason:

robots.txt lists this path, therefore I should enumerate everything behind it.

Treat crawler directives as operational signals.

Authorization comes from somewhere else.

security.txt is a reporting mechanism, not automatic permission

Organizations may publish:

/.well-known/security.txt

to provide vulnerability-disclosure information.

RFC 9116 defines this format so researchers can more easily find appropriate contact and policy information.

A security.txt file may contain:

• contact methods;
• policy links;
• encryption information;
• acknowledgements;
• preferred languages;
• disclosure information.

But an important distinction remains:

The existence of security.txt does not automatically authorize security testing.

Likewise, its absence does not automatically prohibit all research.

Permission for testing may instead be defined in the organization's vulnerability disclosure policy or another explicit authorization mechanism.

Read the policy.

Do not infer safe harbor from the presence of a contact address alone.

Vulnerability Disclosure Policies matter

A mature Vulnerability Disclosure Policy can define:

• systems covered;
• systems excluded;
• acceptable research techniques;
• prohibited techniques;
• reporting channels;
• handling expectations;
• disclosure process;
• safe-harbor conditions where provided.

CISA's Vulnerability Disclosure Policy guidance uses this model: scope and acceptable research should be communicated before researchers conduct testing.

For an analyst, the practical rule is simple:

Read the applicable policy before escalating from observation to security testing.

If there is no policy and no explicit authorization, remain on the observational side of the boundary.

Minimize impact

Responsible reconnaissance should be proportionate.

That applies even to apparently harmless automation.

Consider:

Request rate

A human-paced investigation rarely needs high request volume.

Automation should use conservative rate limits unless a higher rate is explicitly permitted.

Concurrency

Hundreds of parallel requests can impose load even if each request is individually ordinary.

Repetition

Repeatedly querying the same endpoint may add no analytical value.

Payload size

Do not deliberately request large resources unless necessary.

Third-party infrastructure

A domain may rely on:

• CDNs;
• SaaS platforms;
• analytics providers;
• cloud services;
• authentication providers.

Authorization concerning one organization does not automatically extend to every third party supporting its website.

Minimize personal-data collection

Technical reconnaissance can unexpectedly expose personal information.

Examples include:

• names;
• email addresses;
• usernames;
• document metadata;
• historical contact details;
• public directory data.

Ask:

Do I actually need this information to answer the research question?

If not, do not retain it merely because it was available.

Good OSINT is not maximal collection.

It is relevant collection.

What to do when you encounter something unexpected

A public-surface investigation may reveal a possible vulnerability or sensitive information even when security testing was not the original objective.

The responsible response is not automatically:

Investigate deeper.

Instead:

1. stop and assess what was observed;
2. avoid expanding access;
3. do not attempt to prove impact by retrieving unnecessary sensitive data;
4. preserve the minimum evidence necessary to explain the observation;
5. look for an applicable Vulnerability Disclosure Policy;
6. check /.well-known/security.txt;
7. use the organization's designated reporting channel;
8. document what you did and did not do.

The key distinction is between:

demonstrating enough to report a potential issue

and:

escalating the issue simply to satisfy curiosity.

Preserve enough evidence — not everything

Good reconnaissance should be reproducible.

For a meaningful finding, record enough context that another analyst can understand what happened.

Useful fields include:

• exact URL or hostname;
• UTC timestamp;
• data source;
• request method where relevant;
• status code;
• redirect destination;
• relevant headers;
• archived capture URL;
• tool name;
• analyst interpretation;
• confidence;
• limitations.

Avoid storing unnecessary raw responses if a smaller summary is sufficient.

Evidence collection should follow the same proportionality principle as reconnaissance itself.

A worked example

Suppose you are researching whether an organization's public website recently moved to a new platform.

Research question

Is the current public website the result of a recent platform migration?

Phase 1 — Passive evidence

You review:

• historical Wayback Machine captures;
• certificate-transparency records;
• public DNS history where available;
• previously collected urlscan.io observations.

You observe that older pages used one hostname and technology pattern while recent observations show another.

That creates a hypothesis.

Phase 2 — Minimal direct verification

You request the old public URL.

It returns a permanent redirect to the current domain.

You inspect the final public page and observe:

• a new canonical URL;
• different technology signals;
• different response headers.

At this point, the evidence may already be sufficient.

Phase 3 — Stop

You do not automatically:

• enumerate every subdomain;
• scan all related IP addresses;
• test administrative paths;
• search for exposed credentials;
• attempt vulnerability exploitation.

Those actions are unnecessary for the research question.

A responsible conclusion might be:

Historical and current public signals are consistent with a migration of the organization's public web platform.

The investigation stops because the question has been answered.

Common mistakes

Mistake 1 — Treating public reachability as unlimited permission

A resource being reachable over the public internet does not mean every possible interaction with it is equivalent.

Observation and deliberate testing are different activities.

Mistake 2 — Expanding scope automatically

Finding:

api.example.com

does not automatically turn it into a target.

Record the relationship first.

Determine whether investigating it is relevant and authorized.

Mistake 3 — Starting with the most powerful tool

Do not begin with scanners simply because they return more data.

Start with the least intrusive technique capable of answering the question.

Mistake 4 — Confusing robots.txt with permission

Crawler instructions are not authorization.

Mistake 5 — Confusing security.txt with permission

A disclosure contact is not necessarily a testing authorization.

Read the associated policy.

Mistake 6 — Overcollecting sensitive information

Finding personal or confidential-looking data does not create a requirement to copy all of it.

Collect the minimum evidence needed.

Mistake 7 — Continuing after the question is answered

Reconnaissance without a stop condition becomes collection for its own sake.

That increases noise, risk, and analytical error.

A simple escalation test

Before moving to a more interactive technique, ask four questions.

1. Is it necessary?

Will this action materially help answer the research question?

If not, stop.

2. Is it within scope?

Is the target actually part of the defined investigation?

If not, stop.

3. Is it authorized?

If the technique moves into active security testing, is there explicit authority for it?

If not, stop.

4. Is it proportionate?

Can the same information be obtained through a less intrusive method?

If yes, use the less intrusive method.

This produces a useful default:

Necessary → in scope → authorized → proportionate.

Fail any one of those gates and reconsider the action.

Related OSINT.dev tools

Several OSINT.dev tools support the lower-interaction parts of this workflow:

• DNS / MX / SPF / DMARC Inspector — public DNS and mail configuration;
• Redirect Chain & Response Inspector — minimal HTTP redirect analysis;
• Security Headers Checker — public response-header inspection;
• Robots.txt + Sitemap + Meta Analyzer — crawler and metadata signals;
• Tech Stack Snapshot — public technology indicators;
• Wayback Machine — historical web observations;
• crt.sh — public certificate-transparency records;
• urlscan.io — previously collected public web scan observations.

The tools do not determine whether an investigation is appropriate.

The analyst does.

The core principle

Responsible reconnaissance is not defined by how little technology you use.

It is defined by how deliberately you choose each action.

A strong workflow moves in one direction:

Question → passive evidence → identify gaps → minimal interaction → corroborate → stop.

Escalation beyond that sequence should happen only when the research objective genuinely requires it and the authorization boundary supports it.

The best reconnaissance is not the investigation that collects the most.

It is the investigation that produces the strongest defensible conclusion with the least unnecessary interaction.

References

Primary and authoritative material supporting the methodology in this guide:

• NIST SP 800-115 — Technical Guide to Information Security Testing and Assessment
  https://csrc.nist.gov/pubs/sp/800/115/final

• NIST CSRC — Rules of Engagement
  https://csrc.nist.gov/glossary/term/rules_of_engagement

• RFC 9309 — Robots Exclusion Protocol
  https://www.rfc-editor.org/rfc/rfc9309.html

• RFC 9116 — A File Format to Aid in Security Vulnerability Disclosure
  https://www.rfc-editor.org/rfc/rfc9116.html

• CISA — Vulnerability Disclosure Policy Template
  https://www.cisa.gov/vulnerability-disclosure-policy-template

• CISA — Coordinated Vulnerability Disclosure Program
  https://www.cisa.gov/resources-tools/programs/coordinated-vulnerability-disclosure-program