2026-09-17, Google · Brand Monitoring · Engineering · Fundamentals

OSINT tools for defenders: mapping your own attack surface and brand exposure

The OSINT tools worth a defender’s time answer one question: what can an outsider learn about our organisation from public sources, and what should we fix? For domains and hosts, start with certificate transparency (crt.sh), theHarvester and SpiderFoot. For internet-exposed services, Shodan. For suspicious pages, urlscan.io. For credential exposure on your email domains, Have I Been Pwned domain search. For lookalike domains, dnstwist. For analysis across all of it, Maltego. Search engines are a source too: operator queries run on a schedule through an API become a feed. Tool facts come from each tool’s own pages, checked 2026-09-17.

OSINT (open-source intelligence) is information collected from publicly available sources. Here it always means collection about assets your organisation owns or is authorised to assess. Nothing here is about investigating individuals.

The tools at a glance

All checked 2026-09-17.

Tool by tool, for defensive use

Certificate transparency and crt.sh

Certificate transparency (CT) logs are, in the project’s words, “append-only” and “publicly auditable”: certificates can be added but not deleted or modified, and anyone can query a log (how CT works, checked 2026-09-17). The same page notes that Chrome and Safari help enforce CT, and that subscribing to a CT monitor for your domain gets you updates when certificates for it are logged.

For defenders that means two things. First, most publicly trusted certificates issued for your domains are visible, including for hostnames you forgot. Second, certificates for lookalike domains are visible too. crt.sh, operated by Sectigo, lets you search by “an Identity (Domain Name, Organization Name, etc), a Certificate Fingerprint (SHA-1 or SHA-256), or a crt.sh ID” (crt.sh).

Use it for subdomain inventory, spotting certificates you did not request, and early warning of lookalike domains. See typosquatting detection.

theHarvester

theHarvester collects “hostnames, email addresses, IP addresses, URLs, ASNs, people, breach names, and structured evidence” from about 60 sources, including certificate transparency, search engines, DNS datasets and code repositories (README, checked 2026-09-17). Its README classifies operations as P0 passive (“queries an existing provider or dataset without directing traffic toward the target”), P1 active DNS, and P2 direct contact, and says P1 and P2 run only when explicitly selected.

That classification is a useful policy tool in itself: an organisation can allow P0 collection broadly for its own domains and require separate approval for anything that touches infrastructure.

Use it for a first pass at hostnames and email addresses associated with your domains.

SpiderFoot

SpiderFoot is an “open source intelligence (OSINT) automation tool” with over 200 modules, a web UI or CLI, and an MIT license (README, checked 2026-09-17). Its README lists target types including IP addresses, domains, hostnames, subnets, ASNs, email addresses, phone numbers, usernames, person names and Bitcoin addresses.

For defensive work, restrict targets to organisational assets (domains, netblocks, ASNs) and review which modules are enabled. Many modules send your target to third-party services, which matters when the target itself is sensitive.

Use it for a broad automated sweep of your domains and netblocks, with results you then triage.

Shodan

Shodan indexes internet-connected devices and services. Its billing page lists a $49 membership as a one-time payment with 100 query credits and 100 scan credits a month and 16 monitored IPs, and API plans from Freelancer at $69 a month (“up to 1 million results per month”, monitoring for 5,120 IPs) to Corporate at $1,099 a month (billing, checked 2026-09-17).

Use it for checking which services on your own IP ranges are visible from the internet, and monitoring those ranges for new exposure.

urlscan.io

urlscan.io scans a URL by having “an automated process … browse to the URL like a regular user and record the activity”, capturing contacted domains and IPs, resources, a screenshot, the DOM and cookies (about, checked 2026-09-17). Its free plan allows 50 private, 1,000 unlisted and 5,000 public scans a day (pricing).

Visibility matters. Its documentation defines public scans as “visible on the frontpage and in the public search results”, unlisted scans as visible to “vetted security researchers and security companies in our urlscan Pro platform”, and private scans as “only visible to you” or to whoever you share the scan ID with (visibility).

Use it for safely inspecting a suspected phishing page that uses your brand, with private visibility when the URL could contain customer data.

Have I Been Pwned

Have I Been Pwned’s domain search returns breached addresses on a domain “only after successfully verifying that the person performing the search is authorised to access assets on the domain”, and “larger domains normally require a subscription” (FAQ, checked 2026-09-17). Its API documentation says an API key is required, and that domain verification can be done in the dashboard or via API by adding a DNS record or sending an email to one of several pre-defined addresses (API v3).

Use it for finding which of your organisation’s email addresses appear in known breaches, to prioritise password resets and phishing awareness.

dnstwist

dnstwist is a “domain name permutation engine for detecting homograph phishing attacks, typo squatting, and brand impersonation” (README, checked 2026-09-17). It generates variants of a domain and checks which are registered, with options for rogue MX detection, HTML similarity via fuzzy hashing, and screenshot similarity via perceptual hashes. It is Apache-2.0 licensed and has a browser version at dnstwist.it.

Use it for a recurring list of registered lookalikes of your domains.

Maltego

Maltego’s pricing page lists a free Basic tier with 200 credits a month and Maltego Graph Community Edition, Entry Standard at €3,000 a year with 10,000 credits a month, Professional Standard at €7,500 a year with 20,000 credits a month, and Enterprise by quote for teams of 5+ (pricing, checked 2026-09-17).

Use it for relating findings from the other tools (domains, certificates, IPs, organisations) in one graph when an investigation into an impersonation campaign gets complicated.

Search results as an OSINT source

Search engines index things your other tools miss: documents on your own domains that should not be public, error pages and directory listings, pages outside your domains that use your brand on login-like pages, and news reports of phishing campaigns against your customers. Analysts find these with operator queries, which are hard to repeat consistently by hand.

This API runs the query on Google and returns organicResults[] with position, title, link, displayedLink, snippet and date. Google interprets operators in the query, such as site:, quoted phrases, -term, inurl: and filetype:; behaviour is Google’s and can change, so test each template by hand. The API does not visit, crawl or scan the linked pages.

Defensive query templates for your own assets

Build an exposure inventory

The script below runs templates for your domains as one async batch, then merges hosts found in results with hosts you exported from crt.sh or theHarvester, and flags hosts that are not in your asset register.

import csv, datetime as dt, os, requests
from urllib.parse import urlsplit

API = "https://api.answerline.dev"
HEADERS = {"Authorization": f"Bearer {os.environ['API_KEY']}"}
HOOK = {"url": "https://hooks.example.com/exposure"}

DOMAINS = ["example.com", "example.io"]
TEMPLATES = {
    "docs":      "site:{d} (filetype:pdf OR filetype:xlsx OR filetype:docx) (internal OR confidential)",
    "subdomains": "site:{d} -www",
    "listings":  "site:{d} (intitle:\"index of\" OR \"stack trace\")",
}

def submit():
    day = dt.date.today().isoformat()
    tasks = [{"taskType": "GOOGLE",
              "payload": {"query": t.format(d=d), "country": "US", "pages": 3 if name == "subdomains" else 1},
              "idempotencyKey": f"exposure|{name}|{d}|{day}", "webhook": HOOK}
             for d in DOMAINS for name, t in TEMPLATES.items()]
    res = requests.post(f"{API}/v1/async/task/batch", json=tasks, headers=HEADERS, timeout=60)
    res.raise_for_status()
    return res.json()["summary"]

def hosts_from_results(result):
    return {(urlsplit(o["link"]).hostname or "").lower() for o in result.get("organicResults", [])}

def unknown_hosts(found, register_csv="asset_register.csv", ct_export="ct_hosts.txt"):
    with open(register_csv) as f:
        known = {row["host"].lower() for row in csv.DictReader(f)}
    with open(ct_export) as f:
        ct = {line.strip().lower().lstrip("*.") for line in f if line.strip()}
    candidates = (found | ct) - known
    return sorted(h for h in candidates if any(h == d or h.endswith("." + d) for d in DOMAINS))

In the webhook handler, verify Webhook-Signature, skip "test": true, deduplicate by task.id, and pass response.result to hosts_from_results (webhooks). A host that appears in search results or certificates but not in your register is either shadow IT, a forgotten environment, or something to investigate. Documents and listings found by the other templates go to the owning team.

The threat intelligence use case extends this to a full template library, first-seen diffing, alerting and precision metrics.

Cost

A Google Search async task is 3 credits for one page and 2 per extra page. Per domain, the inventory above costs 3 credits for docs, 3 for listings and 7 for subdomains at 3 pages: 13 credits. For 2 domains that is 26 credits per daily run, about 780 a month. See pricing.

Ethical and legal boundaries

This is general guidance, not legal advice. Laws on computer misuse, data protection and privacy differ between jurisdictions, and contracts and platform terms add their own limits. Talk to counsel before starting a program.

1. Own assets or written authorisation. Scope collection to domains, IP ranges and brands your organisation owns, or that a client has authorised in writing with an explicit scope. Keep the authorisation with the findings.
2. Prefer passive collection. Querying existing datasets (CT logs, search results, breach indexes) does not touch the target. Active DNS queries, port scans and page scans do. theHarvester’s P0/P1/P2 split is a good model: default to passive, approve active work per engagement.
3. Do not investigate individuals. Employee names and email addresses appear as side effects of domain OSINT. Use them only for the defensive purpose (for example, prioritising resets after a breach), minimise what you keep, and do not build profiles of people. Many tools accept a person’s name or phone number as a target; a defensive program should not use those target types.
4. Treat personal data as personal data even when it is public. Data protection obligations can apply to information that is publicly available. Define retention periods and access controls for findings.
5. Respect services’ terms and verification. Have I Been Pwned requires domain verification before a domain search; follow such processes rather than working around them. Do not use tools to bypass authentication, CAPTCHAs or bot protection.
6. Mind what you disclose. Submitting a URL publicly to a scanning service can reveal it to others. Choose private visibility for anything that might contain customer data or internal paths.
7. Handle found links as hostile. Open suspected phishing or malware pages only in isolated environments.
8. Report, don’t retaliate. Findings about third-party infrastructure go to the host, registrar, platform or law enforcement, not to “hack back”.

A practical weekly routine

1. Export new certificates for your domains and lookalikes from a CT monitor or crt.sh.
2. Run passive theHarvester or SpiderFoot modules against your domains; diff against last week.
3. Run scheduled search templates daily; review new results.
4. Check your IP ranges in Shodan for newly exposed services.
5. Run dnstwist for registered lookalikes; send new ones to brand protection triage.
6. Review Have I Been Pwned domain notifications and act on new breaches.
7. Update the asset register with anything legitimate you discovered.

For request fields and operators, see the Google Search engine page; to send a first query, the quickstart.

Keep reading