Making Government a Trusted Source in AI Systems

Executive Summary

Many people are now finding government information through artificial intelligence (AI) assistants rather than searching a website directly. Someone asking ChatGPT, Claude, or Gemini a question—where to report a pothole, how much they should expect in food assistance, or who to call during a flood—is asking about information from their government. That means those tools must be able to retrieve and summarize information from the source of truth: a government’s website.

As more residents turn to AI assistants instead of visiting a government website directly, their ability to find the right answer increasingly depends on if that assistant can actually reach—and correctly interpret—government content. This starts with understanding how these systems access a website in the first place.

AI answers depend on many factors, including the assistant’s own retrieval, ranking, and model behavior. This guide focuses on three things governments can directly control on their websites: 

1. Access: Can AI crawlers reach a government’s public web content in the first place?
2. Accuracy: Is the information on that page correct and current? 
3. Structure: Is the content provided in a way that AI systems can parse and relay back to the user? 

Weakness in any of these areas can make it less likely that an assistant retrieves, interprets, or cites the government’s information correctly. Governments should address all three topics to make official information more transparent.  

This guide is written primarily for IT offices, chief information officers, chief security officers, communications officers, and web design teams—the people most likely to own or influence the technical and content decisions on government websites.

Recommended Actions

Access, accuracy, and structure are interdependent, and all are necessary. For each pillar, governments should assign clear ownership, take a concrete action, and verify that the change is working.

 

Access: Can AI crawlers reach a government’s public web content in the first place?

• Assign: Name an IT or website security contact. This person should be responsible for maintaining and periodically reviewing crawler-access settings.
• Action: Document and update the AI crawler configuration. Record existing verified-bot detection, response thresholds, geographic rules, and robots.txt settings, then apply the tiered access framework and robots.txt template in this report.
• Verify: Confirm that trusted crawlers can actually access the site. Use a security or platform dashboard to see which crawlers are passing through or being blocked.

 

Accuracy: Is the information you steward correct and current?

• Assign: Name a communications or content owner. This person should be responsible for the accuracy and currency of high-stakes websites.
• Action: Review priority pages for accuracy and currentness. Identify outdated, conflicting, incomplete, or ambiguous information and update it at the source.
• Verify: Test AI assistant answers against authoritative information on those pages. Spot-check common questions and correct significant gaps or inconsistencies in the content.

 

Structure: Is your content provided in a way that AI systems can parse and relay back to users?

• Assign: Name a web or content owner responsible for content structure. This may be the same person responsible for accuracy, depending on how your organization manages its website.
• Action: Review priority pages for structural barriers. Identify important information that is difficult to extract because it is buried in images, complex tables, or other formats that are hard for AI systems to parse.
• Verify: Test whether AI systems can correctly extract and relay key information. Ask representative questions and compare the responses with the page’s actual content.

Fig 1: A single government website can be reached by three different kinds of AI crawlers, each running on its own schedule and shaping a different part of what users eventually see.

A person’s question might get answered from any of the three mechanisms. This is why access, accuracy, and content structure all matter for governments. A well-structured, current page improves the odds that a person receives an up-to-date, accurate answer from a government website across any of these paths.

 

Free Versus Paid Tiers

Free and paid tiers of the same AI assistant don’t always have identical access to state-of-the-art models, usage limits, or retrieval and research features. For example, paid tiers might include live web search by default and more capable or recently-updated models, while a free tier may lean more on training data that can be months old. This isn’t a fixed rule across every provider, but it’s a reasonable explanation why the same question about a government service might receive different answers on paid versus free tiers. Since most users of AI assistants use the free tier, governments should make every effort to ensure that accurate, up-to-date information is accessible for all users. information is accessible for all users.

________ 1Google-Extended is technically a permission setting rather than a separate crawler. It governs whether content Google’s other crawlers already gathered can be used for this training. It is grouped here because the action for a government is the same: allow or restrict access via robots.txt, as described later in this report.

 

 

---

AI Access: Can AI crawlers reach a government’s public web content in the first place?

 

Why Access Matters

Security and public access to information can move in different directions². For example, a setting built to block malicious scraping may also inadvertently block a legitimate AI assistant trying to answer a person’s question. However, one concern with opening a website to more AI crawlers is that the added traffic could strain a government’s hosting capacity or budget. This is a legitimate concern, but not a reason to block access outright. The graduated, tiered approach below lets a government extend access to the crawlers that matter most to residents while still controlling for cost and load from crawlers that do not add value.

 

Assessing Your Current Configuration

This section focuses on Cloudflare, since it’s a common option for government websites of any size. ProudCity, a platform widely used by small local governments, recommends Cloudflare to its government customers and notes its free tier is sufficient for most. Cloudflare’s AI Crawl Control, available on all plans, provides a dedicated view of AI crawler traffic along with allow/block controls.

Many modern Content Delivery Network (CDN) and firewall platforms, including Cloudflare, offer three related mechanisms to check. Understanding which of these your current configuration uses is the first step toward documenting it.

1. Verified-bot detection: A mechanism that distinguishes traffic that has been cryptographically or otherwise verified as coming from a legitimate, named crawler. Verification can rely on published IP ranges, reverse Domain Name System validation, or newer signature-based methods. This is in contrast to traffic that merely claims to be that crawler via its User-Agent string. This distinction matters because a User-Agent string can be spoofed, while verified-bot mechanisms provide stronger identity assurance. On Cloudflare, this shows up as a “Verified Bot” field available to Web Application Firewall (WAF) rules.
2. A graduated response system: Rather than a single allow/block decision, many platforms score incoming traffic on how “bot-like” it looks and respond differently at different thresholds. For example, it can do this by logging low-confidence bot traffic, challenging medium-confidence traffic, and blocking high-confidence bot traffic. On Cloudflare, this is called the “bot score,” which runs from 1 to 99. Granular bot scores may only be available on Enterprise plans with Bot Management. On free and pro plans, users will instead see broad categories (e.g., “automated,” “likely human,” or “verified bot”) in Bot Analytics.
3. Geographic restrictions: Some configurations block traffic from outside an approved list of countries, often for reasons unrelated to AI crawlers (e.g., for reducing spam or fraud traffic). This matters here because if such a rule runs before bot verification in the rule order, it can block a legitimate AI crawler that happens to route its traffic through a country outside the approved list, regardless of whether that crawler is otherwise verified. Check whether a geographic rule exists, and if so, where it sits relative to any bot-verification rule.  

 

What to Document: For each of the three mechanisms above, note whether it exists in your configuration, what it currently does, and who set up it (if known). This becomes the “Current State” record referenced throughout the rest of this guide. The goal is a written answer to “What does our configuration actually do today?“

 

If You’re Not on Cloudflare 

The concepts above aren’t Cloudflare-specific, but implementations differ by provider. Other CDN and firewall platforms may use labels, categories, managed rule groups, or other signals rather than scoring. Look for the equivalent capabilities (e.g., by identifying verified bots, applying differentiated rules, and checking rule order).

 

The Crawler Tiers Framework

A three-tier structure balances resident access against server load and security:

Tier	Description	Recommendation
Tier 1	Established AI providers with broad resident-facing use	Allow when identity is verified
Tier 2	Established, but narrower relevance, or with a documented compliance concern	Allow, monitor, and evaluate case by case (see note below)
Tier 3	Any bot not matching a verified-bot signature, and any crawler not otherwise listed	Subject to a graduated response system (log / challenge / block)

On Tier 2: This tier deserves particular scrutiny rather than a default “allow.” A crawler can be well-established and still be a poor fit, either because its relevance to residents is indirect, or because it has a documented history of not honoring robots.txt directives. Evaluate each Tier 2 candidate on both resident relevance and compliance behavior before allowing it, and recheck periodically.

 

Tier 1 Crawler Reference Table

The table below lists the specific crawlers that make up Tier 1 in the framework above—the crawlers this guide recommends allowing when their identities are verified. It reflects the major AI crawlers active as of mid-2026. This list changes as providers launch new products or adjust crawling strategies. Governments should treat this as a snapshot, not a permanent reference.

Crawler or Crawl Setting Control	Operator	Category	What it does
GPTBot	OpenAI	Training	Trains future ChatGPT models
OAI-SearchBot	OpenAI	Search	Powers ChatGPT’s live search
ChatGPT-User	OpenAI	On-demand	Fetches a page when a user asks ChatGPT to look up something
ClaudeBot	Anthropic	Training	Trains future Claude models
Claude-SearchBot	Anthropic	Search	Powers Claude’s live search
Claude-User	Anthropic	On-demand	Fetches a page when a user asks Claude to look up something
Google-Extended	Google	Training	Controls whether Gemini can train on the site’s content
Googlebot	Google	Search	Standard search indexing
Google-Agent	Google	On-demand	Navigates the web and performs actions at a user’s request

 

Access Recommendations:

1. Assign a named IT team and contact. Identify who owns the configuration and technical implementation going forward.

2. Document what the current AI crawler configuration actually does. Include verified-bot detection, graduated response thresholds, and geographic rules. This becomes the baseline record for future decisions, so the configuration is not dependent on one person’s memory of how it was set up.

3. Update robots.txt. Most websites publish a file called robots.txt at their domain root. This is a plain-text file listing which parts of a site crawlers may access and which they should avoid. It’s a voluntary standard: A crawler operator can choose to honor it or ignore it, and there’s no technical mechanism that forces compliance. Major providers document robots.txt controls for their automated training and search crawlers. This makes robots.txt a useful, low-effort signal even though it isn’t an enforcement mechanism on its own.

The code block below is ready to adapt: Replace the placeholder domain references with your own, confirm it does not conflict with any of the website’s existing robots.txt content, and add in any additional “Disallow” domains for each crawler.

User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /user/login Crawl-delay: 1 User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /user/login User-agent: ChatGPT-User Allow: / Disallow: /admin/ Disallow: /user/login User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /user/login Crawl-delay: 1 User-agent: Claude-SearchBot Allow: / Disallow: /admin/ Disallow: /user/login User-agent: Claude-User Allow: / Disallow: /admin/ Disallow: /user/login User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /user/login Crawl-delay: 1 User-agent: Googlebot Allow: / Disallow: /admin/ Disallow: /user/login User-agent: Google-agent Allow: / Disallow: /admin/ Disallow: /user/login3

 

4. Apply the tier framework deliberately. Default to verified-bot detection for Tier 1 crawlers, and require an explicit check of resident relevance and robots.txt compliance before allowing any Tier 2 crawler.

5. Configure the platform. On Cloudflare: Start with AI Crawl Control, which is available on all plans, to review traffic and set allow/block policies for individual AI crawlers. Then review WAF/custom rules for broader rules (including geographic restrictions) that could still affect verified traffic. Where a verified-bot exception is needed, use Cloudflare’s bot-verification signal. On other platforms, apply these equivalent capabilities or create similar custom rules

6. Confirm the configuration works via the platform’s dashboard. The security dashboard will show what traffic is actually passing through or being blocked. On Cloudflare, use AI Crawl Control and Security Events to verify that the Tier 1 traffic you intended to allow is reaching the site.

________ 2This tension has become more visible as AI capabilities improve. For example, in September 2026, reports described autonomous AI agents—more action-taking systems than the crawlers discussed in this guide—interacting with U.S. government systems in unexpected ways, including using developer keys found in public code repositories. Separately, an OpenAI agent gained non-public access to an Australian government Medicare statistics system, an incident described as the first known breach of a government agency by an AI agent. These incidents are related to this guide only in the broad sense that they involve AI systems interacting with government websites. The recommendations here concern controlled access to information governments already intend to make public; they are not designed to prevent an autonomous agent from bypassing access controls or using credentials found on the open web. Those risks require separate cybersecurity protections for non-public systems.   3The crawl-delay above applies only to the three training crawlers, which sweep a site on a schedule. Search and on-demand crawlers fire when a resident is actively asking a question, a crawl-delay on those just slows down the answer a resident is waiting for, so they’re left without one above.

 

 

---

Accuracy: Is the information you steward correct and current?

 

Why Accuracy Matters

Access is only part of what makes an AI assistant’s answer useful. Another part is whether the content a crawler retrieves is truly correct and current. A page can be fully reachable and still produce a wrong answer if the page contains outdated, inaccurate, or misleading information.

For example, if a person asks an AI assistant for a department’s phone number, but the page listing that phone number also contains other phone numbers the AI assistant may return a number that’s outdated or incorrect. Depending on the context—an emergency line, a benefits hotline—it could have real, tangible consequences.

Recommendations:

1. Assign a named communications or content-owning contact. Content accuracy and structure are a separate problem from access, and typically sit with different teams than the ones managing crawler configuration. Often, this is a communications or web content team, or a specific program office, depending on how site ownership is structured locally. If it is not  immediately clear who owns a page, treat identifying the owner as the first step.
2. Inventory high-stakes pages. Start with pages where errors hold real consequences: emergency contacts, benefit amounts, eligibility criteria, and deadlines. Not every page needs the same scrutiny, but high-priority pages do.  
3. Spot-check current AI-assistant answers against source-of-truth data. For each high-stakes page, ask major AI assistants like ChatGPT, Claude, and Gemini a typical question a resident might ask, and then compare their answers with ones from authoritative sources.  
4. Trace discrepancies back to a cause. For example, a wrong answer can come from an inaccurate or outdated page.  
5. Prioritize by consequence, not by volume. A rarely-viewed page with a wrong emergency number matters more than a frequently-viewed page with a minor outdated statistic.  
6. Recommend fixes and retest. Once corrected, re-run the same AI-assistant queries periodically to see whether the change appears. Timing varies by provider and retrieval path; a corrected page does not guarantee that a particular answer or citation will update immediately.

 

---

Structure: Is your content provided in a way that AI systems can parse and relay back to the user?

 

Why Structure Matters

Content accuracy asks whether the information on a page is up to date and correct. But the structure of the page also matters: Once an AI model reaches a page, can it actually parse and interpret what’s there?

In practice, important information should have a stable, direct URL and appear as ordinary page text that can be reached through crawlable links, rather than requiring form submissions, clicks, or other interactive features to reveal it.

The examples below are illustrative cases, not controlled tests; they show how content structure can plausibly affect what an AI assistant can retrieve and answer.

 

 

Likely Explanation: Mendocino’s budget tool is built as a JavaScript-powered interactive dashboard. This works well for human visitors, who can click through the tool to find the information they need. But the underlying numbers may not be exposed as ordinary page text at a stable URL. AI crawlers and retrieval systems may not reliably render or interact with this kind of dynamic content, making the county’s actual figure harder for an AI assistant to retrieve.

 

 

 

Likely Explanation: Larimer County’s permit guidance is published as ordinary text on a stable webpage. The relevant information is directly available on the page rather than requiring a user to interact with a dashboard or form. That gives search and AI systems a much simpler path to find and retrieve the correct figure.

 

Recommendations:

1. Test whether key information is directly accessible on priority pages. Check whether critical information—like deadlines, eligibility rules, and contact details—appears directly on the page, or only after a user clicks, filters, searches, or otherwise interacts with it.
2. Prefer extractable text and data over images or scans. Publish information as webpage text, HTML tables, or accessible PDFs with real text (not screenshots or scanned documents), so the underlying information can be read and indexed.
3. Avoid making interactive tools the only source for important information. If a dashboard or widget is the only place a key figure or rule appears, publish the same information as ordinary webpage text or in another stable, crawlable format.
4. Retest after major website redesigns or platform migrations. Changes to a website’s framework or content management system can affect whether AI crawlers and retrieval systems can access important information, even when the visible content appears unchanged.

 

 

 

---

Keeping Access, Accuracy, and Structure Up to Date

Revisit the decisions in this guide at least once a year, or sooner if any of the following occur:

• A major new AI system reaches significant adoption and isn’t represented in your crawler tiers. Review whether the crawlers should be allowed, limited, or blocked based on how the system uses public web content.
• A Tier 1 or Tier 2 crawler operator changes its documented behavior or compliance practices. Reassess whether the crawler still belongs in the same tier and whether your current access rules are appropriate.
• Your CDN, firewall, or bot-management provider materially changes its defaults. Vendor updates can alter which bots are allowed or challenged, even if you have not intentionally changed your own settings.
• Federal or state guidance on government AI crawler access is issued. Update your approach if new requirements, standards, or recommended practices affect how public information should be made available.
• A staffing change moves ownership of this configuration to someone new. Make sure the new owner understands the existing decisions, where they are documented, and how to review or change them.
• Major new pages, tools, or content types are added to or removed from your website. Retest whether important information is still accessible, accurate, and structured in ways AI systems can retrieve.
• A significant AI agent incident affecting government systems is reported. Even when unrelated to crawler access specifically, these incidents shape how much scrutiny and urgency agencies apply to their AI-facing configuration.