Passive Subdomain Discovery & Asset Intelligence Aggregator
╔══════════════════════════════════════════════════════════╗
║ SUBDOMAINX v1.0 ║
║ Passive Subdomain Discovery & Intelligence ║
╚══════════════════════════════════════════════════════════╝
SUBDOMAINX discovers, normalizes, validates, correlates, enriches, scores and exports subdomains for a target domain using only publicly accessible, unauthenticated passive intelligence sources.
API KEYS REQUIRED: NONE
AUTHENTICATION REQUIRED: NONE
- Quick start
- What it does
- Command reference
- Passive sources
- Confidence scoring
- Output formats
- Local database & history
- Diff, baseline and watch mode
- Interactive mode
- Configuration
- Performance profiles
- Project layout
- Running the tests
- Responsible use & security boundaries
- Troubleshooting
git clone https://github.com/mrdineshpathro-dot/SUBDOMAINX.git
cd SUBDOMAINX
pip install -r requirements.txt
python subdomainx.py example.comThere is nothing to configure: no API keys, no accounts, no .env file.
# Deep passive run with DNS + HTTP + technology detection
python subdomainx.py example.com --sources all --resolve --probe --tech
# Authorized-research view + standalone HTML report
python subdomainx.py example.com --bugbounty --resolve --probe \
--output report.html --format html
# Historical / database queries
python subdomainx.py history example.com
python subdomainx.py new example.com
python subdomainx.py removed example.com
python subdomainx.py sources example.com
python subdomainx.py stats example.com
# Compare against a previous export
python subdomainx.py example.com --diff previous.json
# Several targets
python subdomainx.py --input domains.txt --resolve --probeCollection → Normalization → Validation → Deduplication → Source correlation
→ DNS enrichment → HTTP enrichment → Technology detection → Cloud/CDN
classification → Risk & confidence scoring → Interesting host identification
→ Historical comparison → Export / reporting
- Collection — 14 unauthenticated passive source adapters, executed concurrently with per-source rate limiting.
- Normalization — collapses URLs, ports, schemes, trailing dots, wildcard
notation, uppercase, percent-encoding, duplicate separators and IDN into one
canonical hostname. Rejects
javascript:,data:,file:,localhost, IP literals and metadata addresses. - Validation — strict registrable-domain scope enforcement with correct
handling of
co.uk,com.au,co.in,org.uk, … (example.com.evil.comandfooexample.comare rejected). - Deduplication — across case, trailing dot, scheme, port, wildcard notation and source duplication, while preserving provenance.
- Correlation — every host knows which sources saw it, how often, and when it was first and last observed.
- DNS enrichment — A, AAAA, CNAME, MX, NS, TXT, CAA, DNSSEC (optional), internal-looking detection and reliable wildcard detection.
- HTTP enrichment — status, redirect chain, title, server, content type, size, timing and TLS certificate metadata.
- Fingerprinting — technology, CDN and cloud provider classification with explicit evidence and honest confidence levels.
- Scoring — an explainable 0–100 confidence score plus a reconnaissance priority ordering.
- History — first seen / last seen, current vs historical, and change detection between runs.
- Reporting — terminal dashboard, JSON, CSV, TXT and a standalone offline HTML report.
subdomainx <domain> [options]
subdomainx <history|new|removed|sources|stats|targets|show> <domain> [--db FILE]
| Flag | Description |
|---|---|
domain |
root domain (positional) |
--input FILE |
file with one domain per line (# comments allowed) |
--scope FILE |
restrict results to the listed patterns (*.example.com) |
--exclude FILE |
drop hosts matching the listed patterns |
| Flag | Description |
|---|---|
--sources all / --sources crtsh,urlscan |
select sources |
--list-sources |
show the source catalogue and exit |
--filter REGEX |
keep only hosts matching the regex |
--exclude-regex REGEX |
drop hosts matching the regex |
--history |
include archive sources and mark historical hosts |
| Flag | Description |
|---|---|
--resolve / --no-resolve |
DNS enrichment on/off |
--probe / --no-probe |
HTTP probing on/off |
--tech |
technology detection (implies HTTP probing) |
--tls |
capture TLS certificate metadata |
--asn |
enrich resolved IPs with public ASN metadata |
--profile fast|balanced|deep |
performance profile |
| Flag | Description |
|---|---|
--threads N |
global concurrency (default 10) |
--timeout N |
per-request timeout in seconds (default 10) |
--retries N |
retries per request (default 2) |
--rate-limit SECONDS |
minimum interval between requests to one source |
| Flag | Description |
|---|---|
--cache / --no-cache |
enable/disable the local response cache |
--cache-ttl SECONDS |
cache TTL (default 86400) |
--clear-cache |
delete cached responses and exit |
--db FILE |
SQLite path (default subdomainx.db) |
--no-db |
do not write to the database |
| Flag | Description |
|---|---|
--output FILE, -o |
write the report to a file |
--format txt|json|csv|html |
report format |
--output-profile minimal|standard|detailed |
terminal layout |
--bugbounty |
authorized-research view |
--quiet, --verbose, --debug |
verbosity |
--no-color |
disable colour |
--log-format json |
structured JSON logging |
--log-file FILE |
also write logs to a file |
| Flag | Description |
|---|---|
--diff FILE / --baseline FILE |
compare against a previous export |
--watch |
rerun discovery periodically |
--interval SECONDS |
watch interval (default 21600) |
| Flag | Description |
|---|---|
--interactive |
menu driven mode |
--import FILE |
load a previous JSON export |
--config FILE |
load config.toml |
--write-config FILE |
write an example configuration and exit |
--version, --help |
version / help |
| Name | Type | Name | Type |
|---|---|---|---|
crtsh |
Certificate Transparency | otx |
AlienVault OTX passive DNS |
certspotter |
CT issuance API | threatminer |
OSINT passive DNS |
hackertarget |
passive DNS | threatcrowd |
OSINT (often offline) |
rapiddns |
passive DNS | anubis |
community dataset |
dnsdumpster |
public DNS workflow | subdomaincenter |
public API |
bufferover |
passive DNS | urlscan |
URL observations |
wayback |
Internet Archive CDX | commoncrawl |
Common Crawl index |
securitytrails and virustotal are registered but never executed because
they require an API key — the health dashboard shows them as UNSUPPORTED.
See docs/sources.md for endpoints and honest caveats.
SOURCE HEALTH
────────────────────────────────────────────────────────
SOURCE STATUS RESULTS LATENCY ERRORS
crtsh ONLINE 142 820ms 0
hackertarget ONLINE 37 310ms 0
rapiddns ONLINE 68 1240ms 0
urlscan RATE LIMITED 42 980ms 1
commoncrawl ONLINE 19 4100ms 0
virustotal UNSUPPORTED 0 0ms 0
────────────────────────────────────────────────────────
Tracked per source: results, successes, errors, rate-limit events, parser failures, last HTTP status, average response time and last successful query.
| Band | Score |
|---|---|
| VERY HIGH | 95–100 |
| HIGH | 80–94 |
| MEDIUM | 60–79 |
| LOW | 30–59 |
| VERY LOW | 0–29 |
Positive evidence: source diversity, certificate-transparency presence, high trust sources, resolving DNS, live HTTP, repeated historical sightings, DNSSEC.
Negative evidence: wildcard DNS matches, an uncorroborated single source, random-looking labels, historical-only evidence.
Every score comes with the list of factors that produced it, e.g.
+32 4 independent sources reported this host
+12 present in certificate transparency logs
+20 resolves to A/AAAA records
+12 HTTP 200
python subdomainx.py example.com --output hosts.txt --format txt
python subdomainx.py example.com --output hosts.csv --format csv
python subdomainx.py example.com --output report.json --format json
python subdomainx.py example.com --output report.html --format html- TXT — one host per line (
--output-profile detailedadds score/label/category). - CSV — 24 columns including IP, CNAME, status, title, technology, CDN, cloud, category, first/last seen.
- JSON — the documented schema (
tool,version,target,scan,statistics,source_health,changes,hosts), importable again with--import. - HTML — a single self-contained file (embedded CSS/JS, no external resources) with overview cards, a sortable/filterable discovery table, source performance, interesting hosts, changes, asset intelligence and a source correlation graph.
Everything is stored locally in subdomainx.db (SQLite, no server):
python subdomainx.py history example.com # first/last seen per host
python subdomainx.py new example.com # hosts that appeared in the last run
python subdomainx.py removed example.com # hosts missing from the last run
python subdomainx.py sources example.com # per-source performance
python subdomainx.py stats example.com # aggregate statistics
python subdomainx.py targets # every known target
python subdomainx.py show example.com api.example.comAdd --format json to any of them for machine-readable output.
Schema: targets, hosts, sources, observations, dns_records,
http_observations, tls_observations, technology,
historical_observations, scan_runs, run_hosts, source_runs, errors.
python subdomainx.py example.com --diff previous.jsonNEW SUBDOMAINS
────────────────────────────────────────────────────────
admin-new.example.com
api-v3.example.com
REMOVED SUBDOMAINS
────────────────────────────────────────────────────────
old-api.example.com
UNCHANGED
────────────────────────────────────────────────────────
127 hosts
Changes are also detected without a baseline file: the previous run is read
from subdomainx.db and compared host by host (DNS answers, CNAME, HTTP
status, title, technology, sources, confidence).
python subdomainx.py example.com --watch --interval 21600==================================================
SUBDOMAINX WATCH MODE
==================================================
Target: example.com
Interval: 21600 seconds
Previous: 184
Current: 187
NEW:
new-api.example.com
preview.example.com
python subdomainx.py --interactiveSUBDOMAINX Interactive Mode
Target domain: example.com
[1] Passive discovery
[2] DNS enrichment
[3] HTTP probing
[4] Technology detection
[5] Historical analysis
[6] Export results
[7] Exit
config.toml in the working directory is picked up automatically
(--config selects another file, --write-config writes an example).
Every value is optional.
[general]
threads = 10
timeout = 10
retries = 2
profile = "balanced"
[dns]
enabled = true
[http]
enabled = true
max_body_size = 1000000
[cache]
enabled = true
ttl = 86400
[sources]
crtsh = true
hackertarget = true
rapiddns = true
urlscan = true
commoncrawl = truePriority: defaults < config.toml < command line flags. No secrets are ever read or stored.
| Profile | Behaviour |
|---|---|
fast |
passive discovery only (no DNS, no HTTP, no wildcard detection) |
balanced |
passive + DNS + lightweight HTTP (default) |
deep |
passive + DNS + HTTP + TLS + technology + DNSSEC + historical analysis |
subdomainx.py launcher (run from the project root)
pyproject.toml
requirements.txt
subdomainx/
├── cli.py CLI, database commands, interactive & watch mode
├── config.py dataclass config + TOML loading + profiles
├── engine.py pipeline orchestration
├── models.py dataclasses (Host, ScanResult, SourceHealth, ...)
├── sources/ one module per passive source + registry/base (SDK)
├── discovery/ normalize, validate, dedupe, correlate, classify,
│ score, diff
├── enrichment/ dns, http, tls, technology, cdn, cloud, asn
├── storage/ database, cache, migrations
├── reporting/ terminal, json, csv, html
└── utils/ http, logging, helpers
tests/ 340+ unit tests, all network mocked
docs/ architecture, sources, adding-source
examples/ domains.txt, config.example.toml
pip install -r requirements.txt
python -m pytestAll tests are offline: HTTP is served by FakeHTTPClient, DNS by an in-memory
stub resolver and SQLite by temporary files. No test requires internet
access.
SUBDOMAINX is intended for authorized bug bounty reconnaissance, defensive asset inventory, security assessments and OSINT research.
The project explicitly does not implement:
subdomain brute forcing credential attacks password spraying
exploitation RCE SQL injection
XSS SSRF attacks authentication bypass
CAPTCHA solving/bypass WAF bypass stealth scanning
evasion malware persistence
destructive actions
Additional guarantees:
- Only read-only, publicly accessible endpoints are queried.
- A fixed, honest
User-Agentidentifies the tool. - Per-source rate limiting is applied as a courtesy to upstream providers.
- Only lightweight HTTP metadata is collected; response bodies are size-limited and kept in memory only.
- Cookie names may be recorded, never cookie values.
- "Interesting" hosts describe reconnaissance priority, not vulnerabilities.
Only run SUBDOMAINX against domains you own or are explicitly authorized to assess.
| Symptom | Cause / fix |
|---|---|
All sources report FAILED |
no outbound internet access, or a corporate proxy; check with curl https://crt.sh |
A source reports RATE LIMITED |
the provider throttled us; wait, or raise --rate-limit |
UNSUPPORTED for a source |
the source requires an API key, so SUBDOMAINX refuses to run it |
| Results look out of scope | verify the target's registrable domain; use --scope to narrow further |
Everything is HISTORICAL |
the host was only seen in archives; DNS/HTTP enrichment found nothing |
| Database locked | another scan is running; use --db to point at a different file |
Enable --debug (or --log-format json --log-file scan.log) for detailed
diagnostics.
- Python 3.11+
requests,dnspython,tldextract,beautifulsoup4,pydantic,rich
MIT — see LICENSE.