Content Discovery

Gobuster:

# Discover Pages
gobuster dir -u http://<ip>/ -w /usr/share/wordlists/dirb/big.txt -x php,html,xml,txt

# Follow paths recursivly
gobuster dir -u http://<ip>/ -w /usr/share/wordlists/dirb/big.txt -e -r -f

# Fast discovery
gobuster dir -u http://<ip>/ -w /usr/share/wordlists/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt -x php,html,xml,txt

GoSpider:

gospider -S domain.txt -o output -c 10 -d 1

Recursive download found sites:

wget -r -i sitelist.txt

## Create a list with all files 
find . -type f -exec echo "{}" \; | sort -u > allfiles.txt