Crawley — Reconnaissance | Tools

Description

Crawley is a fast, unix-friendly web crawler that extracts URLs, forms, comments, and other interesting data from websites. It follows the unix philosophy — one tool that does one thing well, and pipes cleanly into other tools.

Installation

BASH

                            # Using Go
go install github.com/s0rg/crawley@latest

# Homebrew
brew install s0rg/tap/crawley
                        

Basic Usage

                        BASH
                        
                            # Crawl a website
crawley https://target.com

# Set depth
crawley -depth 3 https://target.com

# Unique URLs only
crawley https://target.com | sort -u

Advanced Usage

                        BASH
                        
                    

                            # Custom User-Agent
crawley -user-agent "Mozilla/5.0" https://target.com

# Delay between requests
crawley -delay 500ms https://target.com

# Timeout per request
crawley -timeout 10s https://target.com

# Include subdomains
crawley -subdomains https://target.com

# Show all tags (not just links)
crawley -tag script,img,form https://target.com

# Robots.txt respect
crawley -robots-txt https://target.com
                        

Common Workflows

                        BASH
                        
                            # Crawl and extract JS files
crawley https://target.com | grep "\.js$" | sort -u > js_files.txt

# Crawl then scan for vulnerabilities
crawley -depth 3 https://target.com | sort -u | nuclei -t cves/

# Extract all URLs for further testing
crawley -depth 5 -subdomains https://target.com | sort -u > all_urls.txt