Bash › Recon Automation

Building a recon pipeline

5 min read Intermediate 5 sections

This is where everything comes together. A recon pipeline chains specialised tools with pipes, so a single command turns a domain into a map of its attack surface. The tools do the heavy lifting; Bash is the glue that connects them. Mastering this pattern is the single most valuable Bash skill for security work.

You'll learn to

Chain recon tools with pipes
Deduplicate and save results at each stage
Build a pipeline you'd actually run on a target

The pipeline mindset

Each recon tool does one job and reads or writes plain text — which means you can connect them with pipes. The classic chain:

# Find subdomains, then probe which are alive — one command:
subfinder -d example.com -silent | httpx -silent

Read it: subfinder lists subdomains of the target, -silent keeps the output clean (just the domains), the pipe feeds them to httpx, which checks which ones respond and prints the live ones. Two tools, one flow, and you’ve gone from a domain to a list of live hosts.

Adding stages

# Subdomains -> live hosts -> with status codes and titles:
subfinder -d example.com -silent \
  | httpx -silent -status-code -title -tech-detect

# Collect historical URLs and filter to interesting ones:
gau example.com | grep -E "\.(js|json|env|config)$" | sort -u

# Find subdomains, probe them, and screenshot the live ones:
subfinder -d example.com -silent | httpx -silent | while read -r url; do
  echo "[live] $url"
done

The backslash at the end of a line continues a command onto the next, keeping long pipelines readable. Each stage narrows or enriches the data: domains become live hosts become hosts-with-details.

Saving results so you never lose work

#!/usr/bin/env bash
set -euo pipefail

TARGET="$1"
mkdir -p "recon/$TARGET"

# Stage 1: subdomains (save raw, then dedupe)
subfinder -d "$TARGET" -silent | sort -u > "recon/$TARGET/subs.txt"
echo "[+] $(wc -l < recon/$TARGET/subs.txt) subdomains"

# Stage 2: live hosts (read the saved file, don't re-run stage 1)
httpx -l "recon/$TARGET/subs.txt" -silent > "recon/$TARGET/live.txt"
echo "[+] $(wc -l < recon/$TARGET/live.txt) live hosts"

# Stage 3: historical URLs
gau "$TARGET" | sort -u > "recon/$TARGET/urls.txt"
echo "[+] $(wc -l < recon/$TARGET/urls.txt) URLs"

The crucial habit: write each stage’s output to a file before moving on. A long pipeline that pipes everything together loses all progress if one stage crashes. Saving to files means you can restart at any stage, inspect intermediate results, and never re-run a slow step you’ve already completed.

Checkpoint

Why save each pipeline stage to a file instead of piping everything together in one long chain?

Try it yourself

On a domain you’re authorised to test, build a two-stage pipeline: subdomain enumeration piped into a liveness check, saving each stage to its own file with sort -u. Print a count after each stage with wc -l. Then extend it with a third stage that collects URLs. Notice how each saved file lets you re-run later stages independently.

Summary

A recon pipeline chains single-purpose tools with pipes, turning a domain into a map of live hosts and endpoints. The core chain is subfinder | httpx; real pipelines add URL collection, crawling, and scanning. Save each stage’s output to a file (and sort -u it) so the work is resumable and inspectable rather than lost on a crash. Keep scope tight and authorised — recon sends real traffic. The pipeline, not any single tool, is the product.

Key takeaways

Recon tools read/write plain text, so pipes chain them into one flow.
subfinder -d target -silent | httpx -silent is the core liveness chain.
Save each stage to a file and sort -u it — resumable and deduplicated.
Stay in authorised scope; pipelines send real traffic at scale.

Quick quiz

Next, processing the JavaScript files this recon finds — extracting the endpoints and secrets hidden in an app’s frontend code.

Was this lesson helpful?