A battle-tested bash script to find XML sitemaps on a website.

You know the drill. You're doing an audit, and the first thing you check is robots.txt for the sitemap location. Most of the time, it's right there. But sometimes... it's not.

So you start guessing. sitemap.xml, sitemap_index.xml, sitemap-1.xml... you poke around for a few minutes and maybe you find it. This script does that guesswork for you. If it comes up empty, you can be confident that further black-box testing is pointless. It's time to pick up the phone and call the client.

It's a simple, two-stage process:

1. The Civilized Method: It first checks /robots.txt for a Sitemap: entry. If it finds one, it prints it, and we're all happy.
2. The Brute-Force Method: If robots.txt comes up empty (or QUIT_ON_FIRST_RESULT is 0), the script methodically tests a list of ~1,700 potential sitemap URLs based on paths i've seen in the wild over the years.

The script checks each candidate URL via HEAD requests, until it receives a 2xx status code and a content type that looks like XML, GZIP, or plain text. (Since Google allows sitemaps in .txt format, we check for that too.)

1. Make it executable:

   chmod +x sitemap-finder.sh

2. Run it against a website:

   ./sitemap-finder.sh 'https://www.example.com/'

===============================================
= sitemap finder started for:                 =
=                                             =
= https://www.example.com                     =
=                                             =
= 2025-07-28 20:11:26h ========================


- checking robots.txt...
- no hint in robots.txt

- starting try & error run...
- testing *.xml...
- Found URL with code 200 and type application/xml: https://www.example.com/en/googlesitemap.xml
- testing *.xml.gz...
- testing *.txt...


= 2025-07-28 20:14:14h ========================
=                                             =
= done.                                       =
=                                             =
= script runtime:       217sec. ~= 3.61min.   =
= sum of http-requests: 1684                  =
= avg. requests / sec:  7.76                  =
===============================================

Real-life example:

You can tweak the script's behavior by editing the file directly.

• QUIT_ON_FIRST_RESULT: By default, this is "0", so the script will keep searching even after it finds a valid sitemap. Set it to "1" if you want it to exit immediately after the first hit.

You probably already have these, but the Sitemap Finder needs:

• bash
• curl
• tput (for the fancy colors)
• bc (for the math at the end)
• perl (for that one little regex)

Q: Why is the script not faster?

A: The Sitemap Finder is already running at a speed that's better than "nice citizen mode" – without turning the scan into a real stress test for the target server. That's because every request goes to the same host, and hammering it with more parallel requests would be, well, not cool (some requests might not get answered correctly by the remote server any more or worse).

So, this is the sweet spot: We're as fast as possible without causing too much trouble on well configured servers. After all its just a couple of sequential requests. Not 100s of thousands as in a DOS attack. If the remote machine cannot deal with that, it cannot deal with internet traffic.

And you – just check your mails or go for a coffee, the script will finish in a few minutes.

Copyright (c) 2025 Daniel Abromeit (https://daniel-abromeit.de/)

Released under the MIT License. See LICENSE for details.