How to Set Up Your Website for Google and AI Search with an AI Agent: 7 Steps and the Prompts
You can't hand an AI agent your website and say "do SEO." It will produce a long list of plausible fixes, some of them wrong, and it will happily write fifty blog posts nobody needed. If the foundations are broken, none of that work gets found.
What an agent is very good at is the tedious part: reading every template, fetching raw HTML, checking every link, comparing your sitemap to your live pages, and drafting the fix. This guide gives you the order to do it in, a copy-paste prompt for each step, and the check that tells you the step is actually done.
It is written for B2B marketers working with Claude Code, Codex or a similar coding agent, alongside whoever owns the website. The foundations are the same for Google and for AI answers. Google says it plainly: there are "no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary" (Google Search Central). If you want the longer argument for why, read why traditional SEO is the foundation for AI visibility and our traditional SEO foundations guide.
Before you start: set up the agent
The agent needs four things to do real work:
- Your site. The code repository, or an export from your CMS, plus a staging URL if you have one.
- Search data. Read access to Google Search Console and Bing Webmaster Tools, or CSV exports from both.
- Analytics. Read access, or an export of landing pages, referrals and conversions for the last 90 days.
- Logs, if you can get them. Thirty days of server or CDN access logs. They are the only proof of which crawlers actually visit.
And one rule, which every prompt below repeats: report first, change later. The agent inspects and proposes. A person approves. Then the agent makes the change on a branch, and someone reviews the diff before it ships.
Start each session with this prompt so the agent knows the ground rules:
You are helping me fix the search foundations of our website for Google and AI answer engines. Work in this order: inspect, report, propose, wait for my approval, then change. Never publish, deploy, or edit robots.txt, firewall or DNS settings without my explicit approval. Never invent facts about our product, pricing, customers or results; if a page needs a fact you can't find in our own materials, list it as a question for me. Mark anything you could not check as "not checked" with the reason. Give file paths and URLs for every finding.
Step 1: Make sure crawlers can reach and read your pages
Every later step depends on this one. Three things commonly break it.
robots.txt rules aimed at the wrong bot. AI vendors run separate crawlers for separate jobs. OpenAI documents GPTBot for training, OAI-SearchBot for ChatGPT search results and ChatGPT-User for fetches a person triggers. Sites that block OAI-SearchBot "will not be shown in ChatGPT search answers," and OpenAI says each setting is independent, so you can block training and still allow search (OpenAI crawler documentation). Anthropic and Perplexity document the same split; see our guides to Claude and ChatGPT for the user agents, and the GPTBot glossary entry for the OpenAI details.
A firewall or CDN that challenges bots. A bot-protection rule can return a challenge page to an AI crawler while your site looks fine in a browser. Search Console won't show it, because Googlebot is often treated differently.
Content that only exists after JavaScript runs. Vercel's December 2024 analysis of its network found that the major AI crawlers it measured, including GPTBot, ClaudeBot and PerplexityBot, did not execute JavaScript (Vercel). If your product details load client-side, many AI crawlers see an empty page.
Prompt:
Audit crawler access for our site. (1) Read robots.txt and list every rule that affects Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Flag any rule that blocks a search or user-triggered crawler. (2) For our 20 most important URLs (homepage, product, pricing, integrations, top blog posts), fetch the raw HTML without running JavaScript and report whether the main content, headings, prices and internal links are present in it. (3) If we have server or CDN logs, count requests and status codes by those user agents for the last 30 days and flag any 403, 429 or 503 responses. Report only; propose fixes; change nothing.
Done when: every page you care about returns its main content in raw HTML, robots.txt blocks only what you decided to block, and AI crawlers get 200 responses in the logs.
Step 2: Fix internal links and site structure
Crawlers discover pages by following links. Google is specific about what counts: it "can only crawl your link if it's an <a> HTML element ... with an href attribute" (Google Search Central). Buttons and JavaScript click handlers that act like links don't count.
AI crawlers are stricter still. In a 41-day server-log test we covered, six AI crawlers found none of 293 pages that were linked only through JavaScript, while Google found 48%. After the links were converted to plain HTML, GPTBot crawled 250 of those pages in under three hours (our write-up of the experiment).
Structure matters too. Every important page should be reachable from navigation or a hub page within a few clicks, and every blog post should link to the product or guide page it supports. Pages with no internal links pointing to them ("orphan pages") are easy to miss for people and crawlers alike.
Prompt:
Map our internal links. (1) List every navigation, footer and in-content link that is not a real element in the server-rendered HTML. (2) Find orphan pages: URLs in our sitemap or CMS that no other page links to. (3) Find broken internal links and redirect chains. (4) Propose a simple structure: which hub pages should link to which product, guide and blog pages, with the exact source page, destination URL and link text for each new link. Don't propose new pages or empty categories. Report and propose only.
Done when: navigation and key in-content links are plain HTML links, no important page is an orphan, and there are no broken internal links.
Step 3: Clean up your sitemap and structured data
These are different jobs, and both are worth doing properly.
Your sitemap is the list of URLs you want found. Reference it in robots.txt with a Sitemap: line, and keep it limited to canonical, indexable pages that return 200. Google's sitemap documentation covers the formats and the protocol (Google Search Central). Keep <lastmod> honest: it should change when the main content changes, not on every deploy.
Structured data describes what a page is. Organization, Product or SoftwareApplication, Article and BreadcrumbList cover most B2B sites. Treat it as accuracy, not a ranking trick: Google says no special schema is needed for its AI features, and FAQ rich results no longer appear in Search. The markup must match the visible text. Our schema markup guide has the types and examples.
Prompt:
Audit our sitemap and structured data. (1) Compare the sitemap with our live pages: list URLs in the sitemap that redirect, return errors, are noindexed or are not canonical, and important live pages missing from it. Check that robots.txt references the sitemap. (2) Check whether lastmod values reflect real content changes or are regenerated on every build. (3) For each page template, list the JSON-LD types present, and flag missing Organization, Product or SoftwareApplication, Article and BreadcrumbList markup where it fits, plus any markup that contradicts the visible page text. Propose the corrected markup for one page of each template. Change nothing yet.
Done when: the sitemap lists only canonical 200 URLs, robots.txt points to it, and each template's markup is valid and matches what the page says.
Step 4: Put the facts buyers ask about on crawlable pages
AI answers are built from what engines can find and trust. When a buyer asks "which tools integrate with Salesforce and cost under $X," the answer can only include you if those facts are on a page an engine can read.
The usual gaps on B2B sites: pricing hidden behind "contact us," integrations shown only as logos in an image, the ideal customer described only in a PDF, and no page that compares you honestly with the alternatives. Our guides on where LLMs get their information and how to get cited by ChatGPT explain why these facts matter.
This is the step where the agent most needs supervision. It can find the gaps. Only you can supply the facts.
Prompt:
Here are 15 questions our buyers ask before they book a demo: [paste them]. For each question, find the page on our site that answers it, quote the sentence that answers it, and say whether that sentence is in the raw HTML. If no page answers it, or the answer is only in an image, PDF or script, list it as a gap. For each gap, propose where the answer should live (an existing page first, a new page only if nothing fits) and write the question I need to answer with the real fact. Don't write any claims about our product yourself.
Done when: each core buyer question has a clear, crawlable answer on a page you would want a buyer to land on.
Step 5: Register and verify with Google and Bing
Verify the site in Google Search Console and Bing Webmaster Tools, and submit your sitemap to both. Both are free, and both now report on AI features.
- Google: Search Console counts traffic from AI Overviews and AI Mode in the Performance report. To be shown as a supporting link in those features, "a page must be indexed and eligible to be shown in Google Search with a snippet" (Google Search Central). See our AI Mode glossary entry for how it is counted.
- Bing: the AI Performance report in Bing Webmaster Tools, in public preview since February 2026, shows how often your pages are cited in "Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations" (Bing Webmaster Blog). It's one of the few free first-party views of AI citations.
Prompt:
Using the Search Console and Bing Webmaster Tools exports I've given you: (1) confirm the sitemap is submitted and processed in both, and list any errors; (2) list important pages that are not indexed, with the reason each tool gives; (3) list our top 20 pages by impressions in Search Console, and from Bing's AI Performance report, which of our pages are cited and for which grounding queries. Flag important pages that appear in neither. Report only.
Done when: the site is verified in both tools, sitemaps are processed without errors, and every important page is indexed or has a known, accepted reason not to be.
Step 6: Measure what happens after the click
Rankings and citations are not the goal. Pipeline is. Set up measurement before you start changing content, so you can tell what worked.
- A real conversion event. A completed demo request, signup or qualified inquiry, tested end to end. A click on "Book a demo" is not a booked demo.
- AI assistant referrals. Visits from chatgpt.com, perplexity.ai, copilot.microsoft.com, gemini.google.com and similar, grouped as one channel in your analytics.
- Crawler activity. AI crawler requests and status codes from your logs, by month.
- A fixed set of buyer questions. The same questions, asked in the AI engines your buyers use, on a regular schedule. Answers vary from run to run, so one screenshot proves little; a consistent set over time does.
Our guide on how to measure AI search performance defines each metric and how to report it.
Prompt:
Check our measurement setup. (1) List the conversion events in our analytics and tell me which ones represent a completed action versus a click; flag any that can't be verified. (2) Propose a channel grouping that separates AI assistant referrals from other referrals, listing the referrer domains it should include. (3) Build a monthly report template with Search Console clicks and impressions for our top pages, Bing AI Performance citations, AI referral sessions and conversions, and AI crawler requests from logs. Don't change any tracking; propose it.
Done when: you can say, for last month, how many conversions came from search and from AI assistants, and which pages they landed on.
Step 7: Run it as a weekly loop, with a person approving
Once the foundations are in place, the agent becomes a weekly maintenance crew:
- Weekly: check new and changed pages for raw-HTML content, links, markup and sitemap entries; check crawler errors in the logs; review Search Console and Bing for newly unindexed pages.
- Monthly: re-run the buyer questions, update the report, and pick the three pages most worth improving based on impressions, conversions and answer gaps.
Prompt for the weekly run:
Weekly site check. Compare this week's pages with last week's. For every new or changed URL: confirm it returns 200, has its main content and links in raw HTML, is in the sitemap with an accurate lastmod, and has valid structured data that matches the page. Check logs for new 4xx or 5xx responses to search or AI crawlers. Check Search Console and Bing for newly unindexed pages. Give me a short list of problems, each with the URL, the evidence and the proposed fix. After I approve a fix and it ships, fetch the live page and confirm the fix is there before you mark it done.
What not to let the agent do, ever:
- Publish at volume. Fifty thin pages a week doesn't fix a weak site; it adds pages that crawlers and buyers ignore.
- Chase shortcuts. An llms.txt file is harmless but unproven. A Common Crawl analysis we covered found 68% of 584,107 llms.txt files were template output (our write-up), and Google says you don't need AI text files to appear in its AI features.
- Invent facts. Every claim about your product, pricing or customers comes from you.
- Mark work done without checking the live page.
The whole checklist on one page
| Step | What the agent checks | Done when |
|---|---|---|
| 1. Crawler access | robots.txt rules by user agent, raw HTML of key pages, firewall responses in logs | Key content is in raw HTML; AI and search crawlers get 200s |
| 2. Internal links | Non-<a href> links, orphan pages, broken links |
Key pages linked with plain HTML links from hubs |
| 3. Sitemap and schema | Sitemap vs live pages, lastmod, JSON-LD by template | Only canonical 200 URLs; markup matches the page |
| 4. Buyer facts | Where each buyer question is answered, in crawlable text | Every core question has a crawlable answer |
| 5. Google and Bing | Verification, sitemap processing, indexing, Bing AI citations | Verified; important pages indexed |
| 6. Measurement | Conversion events, AI referral channel, crawler counts, question set | Conversions by channel and landing page are visible |
| 7. Weekly loop | New and changed pages, errors, unindexed pages | A short weekly list, each fix verified live |
What the agent can't tell you
An agent can make your site easy to crawl, index and cite. It can't tell you what ChatGPT, Gemini, Perplexity and Google's AI features actually say when your buyers ask about your category, which competitors they name instead of you, or which third-party sources they rely on. That takes a repeatable measurement across engines and runs.
If you want a quick look at where you stand, the free AI Visibility Snapshot shows what the major AI engines say about your company and which competitors they name, with no account needed.
Frequently asked questions
Can an AI agent do SEO for my website on its own?
Not well, and not safely. An agent can read your code or CMS, check pages, find broken links and draft fixes much faster than a person. It cannot decide your positioning, confirm facts about your product, or judge which pages matter to buyers. Use it to inspect, report and draft, and keep a person approving every change that goes live.
Which AI agent should I use?
Any coding agent that can read your site's code or export, run commands and fetch pages will work for this guide. Claude Code and Codex are common choices. What matters more than the tool is access: give it the repository or a CMS export, read access to Search Console and Bing Webmaster Tools data, and your analytics.
Do I need special optimization to appear in Google's AI Overviews or AI Mode?
Google says no. Its documentation states there are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary. To be shown as a supporting link, a page must be indexed and eligible to appear in Google Search with a snippet. That is why the SEO foundations in this guide come first.
If I block GPTBot, will my site disappear from ChatGPT search?
Not according to OpenAI. GPTBot governs whether crawled content may be used for training. OAI-SearchBot is the crawler used to show sites in ChatGPT's search results. OpenAI says each setting is independent, so you can allow OAI-SearchBot and disallow GPTBot.
Do AI crawlers run JavaScript?
Do not count on it. Vercel's December 2024 analysis of its network found that the major AI crawlers it measured, including GPTBot, ClaudeBot and PerplexityBot, did not execute JavaScript. Content and links that only appear after client-side rendering are at risk. Check the raw HTML your server returns.
Why do internal links have to be real HTML links?
Google says it can generally only crawl a link if it is an <a> element with an href attribute. In a 41-day server-log test we covered, six AI crawlers discovered none of 293 pages that were linked only through JavaScript, while Google found 48%. After the links were converted to plain HTML, GPTBot crawled 250 of them in under three hours.
Does schema markup make AI engines cite my pages?
There is no published evidence that it does on its own. Google says structured data helps it understand a page, and also says no special schema is needed for its AI features. Add accurate Organization, Product or SoftwareApplication, Article and BreadcrumbList markup because it describes your business clearly, and keep it consistent with the visible text.
Should I publish an llms.txt file?
It is cheap and harmless, but do not expect measurable impact. Google says you do not need AI text files to appear in its AI features, and a Common Crawl analysis we covered found that 68% of 584,107 llms.txt files were template output. Fix crawler access, links and page content first.
Why register with Bing Webmaster Tools if most of my traffic is from Google?
Because Bing now reports AI citations. Its AI Performance report, launched in public preview in February 2026, shows how often your pages are cited in Microsoft Copilot, AI-generated summaries in Bing and select partner integrations. It is free first-party data on AI answers that you cannot get elsewhere.
What should I measure after the fixes ship?
Four things: Search Console performance (Google counts AI Overviews and AI Mode traffic there), Bing Webmaster Tools AI Performance citations, AI assistant referrals and real conversions in your analytics, and a fixed set of buyer questions you re-check in the AI engines your buyers use. Add AI crawler hits from server or CDN logs.
How long does this take?
The audit steps take an agent minutes to an hour on a typical B2B site. The fixes take as long as your team's review and release process. Plan one step per working session, review the agent's report, approve the changes you agree with, and then move to the next step.
Is it safe to give an AI agent access to my production site?
Give it the least access the step needs. For audits, read access to the repository, a staging copy and data exports is enough. Ask it to propose changes as a branch or a pull request, never to publish directly, and review every diff before it ships.
What should the agent never do?
Publish pages without review, generate large numbers of thin pages, invent facts about your product or customers, change robots.txt or firewall rules without approval, or claim a fix worked without checking the live page. Each prompt in this guide tells the agent to stop for approval and to mark anything it could not check.
Does this replace an SEO audit tool or an AI visibility platform?
No. The agent fixes and checks your own site. It cannot see how AI engines answer your buyers' questions across many runs, which competitors they name, or which sources they cite. Pair the fixes with a repeatable way to measure AI answers over time.
Where should I start if I only have one hour?
Run step 1. A robots.txt line, a firewall rule or client-side rendering that hides your pages from crawlers makes every later step pointless. Then fix the internal links in step 2, because pages crawlers cannot discover cannot be indexed or cited.
See where AI leaves your brand out
Put this into practice. Genmark AI shows you exactly how ChatGPT, Gemini, Perplexity and the other major engines answer about your brand — then helps you create the content that earns the citation.
Free · No account needed · Results by email
Need a team to execute? Explore Expert-Led Growth