Technical Requirements

These are the minimum technical requirements a site has to meet before any AEO content or authority work pays off. Treat this as a deployment checklist, not a wish list.

Crawler access

• Robots.txt does not block major AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Bingbot).
• WAF and CDN rules allow these user agents at adequate rates.
• IP-based allowlisting (where used) covers each engine’s published IP ranges and is updated automatically.
• Reverse DNS verification works for crawlers that support it.
• HTTP responses to crawler requests do not differ from responses to browser requests (no cloaking).

Verify with: curl -A "GPTBot" https://example.com/path for each engine, on a representative sample of URLs.

Rendering

• Server-rendered HTML or pre-rendered HTML contains the primary content of every cite-worthy page.
• Pages do not require client-side JavaScript execution to expose their main content.
• For SPAs: pre-rendering or server-side rendering is implemented and verified for crawler user agents.
• Lazy-loaded content critical to the page is exposed without user interaction (no infinite scroll required to read).
• Hydration mismatches between server and client are minimal and do not affect cite-worthy content.

Structured data

• Every page that should be cited has appropriate schema markup.
• Schema is JSON-LD (preferred) or Microdata. Avoid RDFa.
• Schema validates against Schema.org and against Google’s Rich Results Test where applicable.
• Identifier properties (@id, url, sameAs) are consistent across the site.
• dateModified reflects actual modification, not page generation time.
• Organization schema is present on the homepage with name, url, logo, and sameAs linking out to verified social and reference profiles.

Discovery files

• robots.txt exists at the root and is reachable.
• sitemap.xml is current, points only to canonical URLs, and is submitted to Google Search Console and Bing Webmaster Tools.
• llms.txt exists at the root and lists canonical pages.
• humans.txt is optional but harmless.

URL and canonical handling

• One canonical URL per piece of content. No duplicate content across URLs; align with URL structure for AEO.
• is set correctly on every page, pointing to itself or to a clearly preferred version.
• Redirects are 301 (permanent) for permanent moves, 302 only for temporary ones.
• No redirect chains longer than two hops.
• Trailing slashes, www vs non-www, and HTTP vs HTTPS are normalized to a single canonical form.

Performance

• Time-to-first-byte under 800ms at the median.
• Largest Contentful Paint under 2.5 seconds.
• 5xx error rate under 0.1% over rolling 7-day windows.
• 4xx error rate (excluding intentional 404s) under 0.5%.
• No sustained crawler rate-limiting that drops requests.

Logging and monitoring

• AI crawler traffic is logged separately or filterable in standard log analysis tools.
• Alerts exist for: sustained drop in crawler request volume, sustained rise in 5xx for crawler traffic, schema validation failures in production.
• Search Console (Google) and Bing Webmaster Tools are connected and reviewed at least weekly.

Documentation

• Internal documentation exists for which crawlers are allowed, why, and where the rules live.
• Schema implementation patterns are documented for content authors.
• The llms.txt update process is documented.

Audit cadence

• Full technical audit: quarterly.
• Smoke test (key pages, core schema, crawler reachability): monthly.
• Post-deploy check (any deploy touching infrastructure, CDN, or routing): every deploy.

Implementation example

AwesomeShoes Co. adds this checklist to its release process after a CDN change previously caused a citation drop. The technical program manager needs each deployment to pass objective AEO readiness gates, not subjective review.

Implementation discussion: release tooling runs crawler-header smoke tests, schema validation, canonical checks, and performance thresholds before production rollout. The platform owner blocks deployment when critical checks fail, while SEO signs off only after live verification on priority URLs. This keeps technical quality measurable and operationally reliable.