IndieHackedCheck
IndieHackedCheck reads a website only when someone runs one of IndieHacked's site checks on that address. It fetches the files that check needs and nothing else.
Every request is signed, so a host that verifies bots can tell it from a copy using its name. To keep it off your site, name it in robots.txt.
What it reads
For each check it fetches the address entered, the site's robots.txt, its sitemap and its llms.txt. The copy check also reads up to 3 more pages on the same site, linked from the first.
- It runs no JavaScript, fills in no forms and signs in to nothing.
- It follows up to 5 redirects, reads at most 3 MB of a page and gives up after 12 seconds.
- It never connects to an address inside a private network.
The copy check also asks Firecrawl to read its pages after their JavaScript has run. Those requests come from Firecrawl's servers, not from IndieHackedCheck, and Firecrawl's own rules apply to them.
How often
Only when a person asks. One account runs at most 10 checks a day and the same check on the same site once every 5 minutes. All of IndieHacked runs at most 400 a day.
Keeping it off your site
It follows your robots.txt the way search engines do: a group that names it if you have one, otherwise the * group. A check of a page your rules disallow stops before the page is fetched, and the reader is told why. To keep it off the whole site and leave other crawlers as they are, add this group:
User-agent: IndieHackedCheck
Disallow: /While your robots.txt answers with a server error, it reads that as a no, as search engines do.
Checking a request is ours
Every request carries this user agent:
Mozilla/5.0 (compatible; IndieHackedCheck/1.0; +https://indiehacked.com/bot)Every request is also signed with Web Bot Auth, the IETF draft for bots signing their requests with HTTP Message Signatures. The Signature-Agent header reads "https://indiehacked.com", and the public key that checks the signature is here:
https://indiehacked.com/.well-known/http-message-signatures-directoryA request that uses the name without a valid signature didn't come from us.
Something it shouldn't do
Write to us if IndieHackedCheck does anything on your site this page doesn't describe.