Technical SEO audit: what to check and what to fix first

A technical SEO audit answers one question: can search engines and AI assistants find, fetch, understand and index the pages you care about? You don’t need a dedicated SEO engineer for it. You need Search Console, Bing Webmaster Tools, a site crawler, and the discipline to fix causes rather than symptoms. This guide covers what to check, roughly in order of impact, and ends with a severity rubric and a checklist.

When to audit and how to scope it

Run a full audit:

• Before and after a site migration, redesign, CMS change or domain move.
• When organic traffic drops and content or demand changes don’t explain it.
• When Search Console shows a sudden jump in non-indexed pages.
• Once or twice a year as routine maintenance, with a lighter monthly check on the Page indexing report.

Scope it before you crawl. List the templates that drive revenue (product, pricing, solution, comparison, blog, docs) and the hostnames in play. A B2B site audit is mostly an audit of a handful of templates: one broken template breaks every page built on it. Record a baseline of indexed pages, clicks and impressions so you can show what the fixes changed.

If clicks are the problem but rankings are fine, this is the wrong audit; see Search Console CTR optimization.

Crawlability: robots.txt and robots directives

What robots.txt does

robots.txt tells crawlers which paths they may fetch. A few rules that trip teams up:

• It must sit at the root of each host, and it only applies to that host, protocol and port. blog.example.com needs its own file.
• A crawler obeys only one group: the one whose User-agent most specifically matches it. If you add a User-agent: Googlebot group, Googlebot ignores everything under User-agent: *, so repeat any shared rules. Bing documents the same behaviour for Bingbot.
• Within a group, the most specific (longest) matching path wins; on a tie, the least restrictive rule applies.
• Google reads up to 500 KiB of the file and generally caches it for up to 24 hours, so changes are not instant.
• If robots.txt returns a 5xx error, Google stops crawling the site for the first 12 hours, then uses the last cached copy for up to 30 days. A broken robots.txt endpoint is a site-wide outage for crawling.

What robots.txt does not do

robots.txt does not keep a page out of search results. Google says plainly that it is not a mechanism for keeping a page out of Google: a blocked URL can still be indexed from links elsewhere and shown without a description. Search Console even has a status for this, “Indexed, though blocked by robots.txt”. Bing’s guidelines say the same: robots.txt controls crawl access, not indexing.

To keep a page out of the index, use noindex, either as a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP header (the header is the way to do it for PDFs and other non-HTML files). The page must stay crawlable: if robots.txt blocks it, the crawler never sees the noindex. So never block and noindex the same URL.

Search crawlers vs AI crawlers

Each AI vendor runs several crawlers with different jobs. Blocking a training crawler is a policy choice; blocking a search crawler removes you from that product’s answers.

Anthropic says its bots honour standard robots.txt directives and support the non-standard Crawl-delay extension. Vendors change these crawlers, so re-check the official pages linked in Sources before you edit production rules. For how this fits into AI search visibility more broadly, see GEO technical implementation.

A common finding here isn’t in robots.txt at all: a CDN “block AI bots” toggle or WAF rule returns 403 to search bots that robots.txt allows. Check your logs for each user agent and the status codes it got.

Indexing: read the Page indexing report first

The Search Console Page indexing report is the fastest way to see what Google actually did with your URLs. Google itself says not to expect every URL on your site to be indexed. Duplicates and thin utility pages are fine outside the index. The question is whether your important pages are in it.

The statuses that matter most in an audit:

• Crawled – currently not indexed: Google fetched the page and chose not to index it (it may later, without a resubmit). On important pages, suspect thin or near-duplicate content. Improve or consolidate; repeated indexing requests won’t fix it.
• Discovered – currently not indexed: Google knows the URL but hasn’t crawled it, sometimes delaying to avoid overloading your site. Large counts suggest slow servers, masses of low-value parameter URLs competing for crawl, or weak internal links to those pages.
• Duplicate without user-selected canonical: Google treats the page as a duplicate and you haven’t declared a canonical. Declare one.
• Duplicate, Google chose different canonical than user: Google disagrees with your canonical. Look for conflicting signals (redirects, internal links and sitemaps pointing at a different URL than your canonical tag does).
• Alternate page with proper canonical tag: working as intended. No action needed.
• Soft 404: the page returns 200 but looks like an error or empty page. Return a real 404 or 410, or fix the content.
• URL marked ‘noindex’ and Blocked by robots.txt: confirm each one is intentional. If a revenue page shows up here, it’s a critical finding.

Use URL Inspection to confirm individual cases. The indexed result is the version Google last indexed; the live test fetches the page now. A passing live test only means the page can be crawled and parsed, not that it will be indexed. Inspection also shows the user-declared canonical next to the Google-selected one, the quickest way to diagnose canonical conflicts. Request indexing has a daily limit per property, so for many changed URLs submit a sitemap with accurate lastmod instead.

Repeat these checks in Bing Webmaster Tools: Bing’s index also feeds Copilot.

Canonicals and duplicate URLs

Most B2B sites have more duplicate URLs than their owners think. Typical sources:

• Protocol and host variants: http:// vs https://, www vs bare domain. Google generally prefers HTTPS, but you should make the choice explicit with redirects.
• Parameters: tracking parameters like ?gclid= or ?utm_source=, sort and filter parameters, session IDs.
• Trailing slashes: Google treats /pricing and /pricing/ as separate URLs (the bare root is the exception: example.com and example.com/ are the same). Pick one form and redirect the other.
• Staging hosts: a staging subdomain left crawlable.

Canonical signals, from strongest to weakest:

1. Permanent redirects: a strong signal that the target is canonical. Use them when you retire a duplicate.
2. rel="canonical": also a strong signal. Put it in the <head> or send it as an HTTP Link header (useful for PDFs). Use absolute URLs, and give each canonical page a self-referencing canonical.
3. Sitemap inclusion: a weak signal.

Keep the signals consistent: the canonical tag, internal links, sitemap entries and redirects should all point at the same URL. Don’t use noindex to choose between duplicates on your own site, because noindex removes the page from search entirely. Use a canonical instead.

hreflang basics

If you publish localized versions, hreflang tells Google which version to show to which audience. The rules that audits catch most often:

• Each version must list itself and every alternate. If two pages don’t reference each other, Google ignores the annotations.
• Use full URLs, including the protocol.
• Codes are language first (ISO 639-1, e.g. de), optionally followed by a region (ISO 3166-1 Alpha 2, e.g. de-AT). A region alone is invalid.
• Add x-default for a fallback such as a language selector.
• Pick one method (HTML tags, HTTP headers or sitemap). Using all three adds maintenance work and no benefit.

Google doesn’t use hreflang or the HTML lang attribute to detect a page’s language, so a page in English tagged as German is still English to Google.

Redirects

Google treats redirects in two groups:

• Permanent: 301 and 308 (and an instant meta refresh). Google treats the target as canonical. Use these when the move won’t be reverted.
• Temporary: 302, 303 and 307 (and a delayed meta refresh). The source URL can stay in results; the target is not treated as canonical.

The difference between 301 and 308 (and between 302 and 307) is HTTP semantics, not SEO: 308 and 307 require the client to keep the request method, so a POST stays a POST. For search purposes, 301 and 308 behave the same.

JavaScript redirects only work if Google renders the page, and rendering can fail. Use them only when server-side or meta refresh redirects aren’t possible. AI crawlers that don’t run JavaScript won’t follow them at all.

What to look for:

• Temporary redirects used for permanent moves: a 302 left in place for years after a rebrand.
• Chains: Googlebot generally follows up to 10 hops, but Google recommends redirecting straight to the final URL, ideally no more than 3 hops and fewer than 5. Chains often build up when one migration’s redirects point at the next migration’s old URLs.
• Loops: A → B → A. The page can’t be reached at all.
• Internal links to redirected URLs: point them at the final URL.

Status codes and soft 404s

How Google handles response codes affects how you should serve errors:

• 4xx (except 429): Google doesn’t use the content, and previously indexed URLs that now return 4xx get dropped. 404 and 410 have the same effect. Return them for pages that are truly gone.
• 5xx and 429: Google slows down crawling. URLs that are already indexed are kept at first but dropped if the errors persist. Recurring 5xx errors on important templates are a high-severity finding even if they’re intermittent.
• Soft 404s: a 200 response for an empty or error-like page, such as an empty search result, a “product not found” template or a thin tag archive. Return a real 404 or 410, or give the page real content.

Crawl the site and filter for non-200 responses on URLs that are linked internally or listed in sitemaps.

XML sitemaps

A sitemap is a hint, not a command, but a clean one helps discovery.

• Limits: 50,000 URLs or 50 MB uncompressed per file. Split larger sites into several sitemaps and list them in a sitemap index.
• Contents: only canonical, indexable URLs that return 200. No redirects, no noindexed pages, no parameter variants. Use absolute URLs.
• lastmod: set it to the date of the last significant change to the page. Google uses it only when it is consistently and verifiably accurate, so a CMS that stamps every URL with today’s date throws the signal away. Cosmetic changes like a copyright year don’t count.
• priority and changefreq: Google ignores both. Don’t spend time tuning them.
• Submission: submit in Search Console (and Bing Webmaster Tools) and add a Sitemap: line to robots.txt.

Tip: split sitemaps by template (products, blog, docs) so you can filter the Page indexing report by sitemap and see which template has the problem.

JavaScript rendering

Google processes JavaScript pages in three phases: crawl, render, index. A few details matter for audits:

• Pages that return a non-200 status, or that robots.txt blocks, may not be rendered at all.
• If the original HTML contains noindex, Google may skip rendering, so JavaScript that removes a noindex tag may not work. Don’t ship noindex in the initial HTML of a page you want indexed.
• Links must be real <a href> elements. Google can’t reliably extract URLs from <a> elements without an href, and doesn’t recommend javascript: links.

Google still recommends server-side rendering or pre-rendering, partly because not all bots can run JavaScript. That covers most AI crawlers. Vercel’s analysis of crawler traffic (December 2024) found that none of the major AI crawlers it studied rendered JavaScript, including GPTBot, ClaudeBot and PerplexityBot. They fetched script files but didn’t execute them. Google’s own AI features use Googlebot’s rendering. Bing’s guidelines also warn against hiding critical content behind client-side rendering.

The test: curl the page (or load it with JavaScript disabled) and check that the main copy, headings, internal links, canonical and structured data are in the raw HTML. If your pricing table only appears after hydration, most AI assistants never see it.

Internal linking and orphan pages

Google’s guidance is direct: every page you care about should have a link from at least one other page on your site. An orphan page, one that’s in your sitemap or analytics but not linked from anywhere, is hard to discover and gets little weight.

To find orphans, compare three lists: URLs found by crawling from the home page, URLs in your sitemaps, and URLs that get organic traffic in Search Console. Anything in the second or third list but not the first is orphaned. Then check:

• Important pages are a few clicks from the home page or a hub page, not buried in pagination.
• Navigation and related-content links are crawlable <a href> links, not click handlers.
• Anchor text is descriptive (“SOC 2 compliance checklist”), not “click here” or “learn more”.
• Internal links point at canonical, final URLs, not redirects or parameter variants.

Structured data

Structured data helps search engines understand a page. It doesn’t guarantee a rich result. Audit for correctness rather than coverage:

• Only mark up content that’s visible on the page. Hidden markup breaks Google’s structured data guidelines.
• Use JSON-LD, Google’s recommended format.
• Validate with the Rich Results Test and watch the structured data reports in Search Console for errors after template changes.
• Don’t build plans around retired features. Google stopped showing HowTo rich results in 2023 and FAQ rich results in May 2026, and removed the documentation for both. Existing FAQ or HowTo markup does no harm, but it no longer earns a special result in Google Search.

Page experience

Speed and stability problems belong in the audit, but they have their own diagnostics and fixes. Note any templates with poor field data in the Core Web Vitals report and work through them with Core Web Vitals. Also confirm every page is served over HTTPS with no mixed-content warnings.

Site migrations checklist

Before launch:

1. Export every old URL with traffic, backlinks or a sitemap entry.
2. Map each to its closest new equivalent, not to the home page.
3. Implement permanent (301 or 308) redirects straight to the final URL, without chains through intermediate URLs.
4. Give every new page a self-referencing canonical, and check that staging noindex tags and robots.txt blocks aren’t shipping to production.
5. Prepare the new robots.txt and XML sitemaps.

At launch and after:

1. Update internal links to the new URLs right away.
2. Submit the new sitemaps; remove the old ones once Google is using the new ones.
3. For a domain or subdomain change, use Search Console’s Change of Address tool, for every verified variant of the old domain. Don’t use it for HTTP to HTTPS, www changes or path-only changes.
4. Keep the redirects for as long as possible, and generally for at least a year.
5. Watch the Page indexing report, server logs and 404s daily for the first weeks. Expect rankings to fluctuate for a while before they settle.

Severity rubric: what to fix first

Rate each finding on how badly it blocks crawling or indexing and how many important pages it affects:

Turning findings into tickets

• One ticket per root cause, not per URL. “Blog template omits self-canonical (1,240 URLs)” is fixable. A spreadsheet of 1,240 rows isn’t.
• Write it for the engineer: the affected template or rule, a few example URLs, current vs expected behaviour, and how to verify (a curl command, URL Inspection, a crawl filter).
• Name the owner. CDN rules, CMS templates and the redirect map often belong to different people.
• Define done in data: a Page indexing status, a clean recrawl, response codes in logs. Recheck a few weeks after release, since Google reprocesses pages over time.

Audit checklist

Sources

• Google Search Central, Introduction to robots.txt
• Google Search Central, How Google interprets the robots.txt specification
• Google Search Central, Robots meta tag, data-nosnippet and X-Robots-Tag specifications
• Google, Google’s common crawlers (Googlebot, Google-Extended)
• OpenAI, Overview of OpenAI crawlers
• Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity, Perplexity crawlers
• Bing, Bing Webmaster Guidelines
• Bing, How to create a robots.txt file
• Search Console Help, Page indexing report
• Search Console Help, URL Inspection tool
• Google Search Central, How to specify a canonical URL
• Google Search Central Blog, To slash or not to slash
• Google Search Central, Localized versions of your pages (hreflang)
• Google Search Central, Redirects and Google Search
• IETF, RFC 9110: HTTP Semantics, redirection 3xx
• Google Search Central, How HTTP status codes and network errors affect Google Search
• Google Search Central, Build and submit a sitemap
• Google Search Central, Understand JavaScript SEO basics
• Vercel, The rise of the AI crawler (December 2024)
• Google Search Central, Link best practices
• Google Search Central, General structured data guidelines
• Google Search Central, Latest documentation updates (HowTo and FAQ rich result removals)
• Google Search Central, Site moves with URL changes