Should we block AI crawlers in robots.txt?

The block nobody chose

Most of the site-wide blocks we find on public sites were never debated. A line went into robots.txt in 2023, when the crawlers were new and the answer engines were not yet how residents looked things up, and nobody has been back to it since. The audit reads that rule per template and tells you which of the 20 AI crawlers we track are allowed in on the service page, the fee schedule, the portal and the news release, and which are not. That is a narrower question than whether the rule was right, and it is the one your teams can act on this quarter.

What the answer looks like with the block in place

Residents ask the engines either way. With your domain closed, the answer about a permit or a benefit is assembled from a commercial guide, a local news story and a neighbourhood thread. The tracking shows exactly that, as cited pages and cited domains split into yours and everyone else's. Your side is missing from an answer people act on the same day they read it.

Splitting the decision by template

• Let the crawlers into the service pages residents ask about
• Keep them out of the areas you have a reason to protect
• Treat archives and record systems as a decision of their own

We do not give legal advice, and if your body has a policy or records position on this, that position wins. What we bring is the fact base: who is allowed in today, template by template, and what each of the five engines is citing about your services now. Whatever you decide goes onto the roadmap with an owner, so the file matches the decision instead of an old default.