What our scanner reads, and how to stop it

What it fetches

• The home page, and the privacy policy it links to, or the usual addresses such as /privacy. Only GET requests, one level deep, no forms, no logins.
• Your robots.txt first. If it disallows our agent, or all agents, from an address, that address is not fetched — and if it closes the site, nothing is fetched and the report says the site was not checked rather than that it found nothing.
• At most one request a second, and never a form, a login, a search or a checkout.
• Public DNS records of the domain (MX, TXT) to see the mail services it uses.
• Each page waits at most 8 seconds. A website is fetched at most once every 24 hours, whoever asks.

How to recognise us

Everything we fetch leaves one address, 206.189.60.74, which resolves to scanner.clausebench.com and back again. Three agents go out from it, and each one links here:

• clausebench-scanner/1.0 (+https://clausebench.com/scanner) — the website check on this site, described above.
• clausebench-monitor/1.0 (+https://clausebench.com/scanner) — the nightly re-read of a vendor’s own legal pages, to notice when one of them changes.
• clausebench-registry/1.0 (+https://clausebench.com/scanner) — filling in a registry record that is missing a link.
• clausebench-relay/1.0 (+https://clausebench.com/scanner) — the same reading, sent through a third-party network, and only after one of the three above was refused by your site. It does not come from the address above, which is why it does not use their name.

One exception, said plainly. When your site refuses the registry reader under its own name, it asks once more under a browser’s User-Agent — from the same address, or through the third-party network above — and only for the pages a vendor publishes for its customers to read: a data processing agreement, a list of sub-processors, a trust or security page. The website check and the nightly monitor never do this.

All of them read your robots.txt first, and a path it closes stays closed whatever name we ask under. The form below stops all of them at once.

If your website turns us away

Some sites sit behind bot protection — DataDome, Cloudflare Bot Management, Akamai, Imperva, HUMAN and the rest — which answers us with a check page instead of the site. The report names the product when it can see which one it is, and says nothing about the website itself: we were not shown it.

The website check does not try to get past one. It sends the User-Agent below, and the browser it renders pages with sends the same — it reads a page the way a visitor does, and it never presents itself as somebody else to get in. A site that wants to keep the check out keeps it out.

So if it is your own website and you want the check to run, the way in is yours to open: allow this User-Agent in whichever product is doing it. It is a string and anyone can send it, so pair it with a rate limit rather than trusting it on its own — we fetch two or three pages, once every 24 hours.

clausebench-scanner/1.0 (+https://clausebench.com/scanner)

If you would rather not open anything, the free check is not the only way in: the product reads a privacy policy you upload as a file, and records it as a document you supplied rather than a page we read.

What we keep

The list of websites entered, the services found and the differences shown in the report. Results are deleted after 90 days, or immediately when the person who ran the scan presses delete. No screenshots and no copies of your pages are stored.

Stop scanning your website

Use an email address at the website's domain. After you confirm from that inbox, the website is excluded from every future scan and stored results for it are removed.