There’s a single line of HTML that can erase your website from Google. It’s ten characters of configuration, it’s invisible to everyone who visits the site, and it ships on more relaunches than anyone in this industry wants to admit.
We’ll come back to that line. First, the situation it creates — because if you’ve ever typed your own service into Google, scrolled two pages, and not found yourself, you know the feeling. The site looks great. It loads. Customers who have the link say nice things. And to Google, it may as well not exist.
Here’s the reframe that makes the whole problem tractable: being invisible on Google is almost never a punishment, and almost never a mystery. It’s a plumbing failure — and plumbing can be inspected, joint by joint.
Google doesn’t search the web. It searches its copy of the web
When someone types a query, Google doesn’t go out and scan the internet. That would take years. It searches its index — a colossal, pre-built filing system containing a processed copy of every page it has decided to keep.
Think of it as an enormous library. Your page can only be recommended to a reader if a librarian first visited it (crawling), then actually read it (rendering), then decided it was worth filing on the shelf (indexing). Only then does it compete for the top spots (ranking).
Nearly everyone who says “my SEO isn’t working” is worrying about step four — ranking — while their actual failure sits at step one, two, or three. And you can’t rank a page that never made it onto the shelf. No amount of “content strategy” fixes a filing problem.
One very 2026 footnote: this pipeline now matters twice. AI search tools — Google’s AI Overviews, and assistants that lean on search indexes for their answers — draw from the same crawled-and-indexed plumbing. A page Google can’t see is a page AI answers can’t cite either. The stakes of the boring technical layer went up, not down.
Step one — crawling: can Googlebot even reach you?
Googlebot discovers pages by following links and reading sitemaps, then requests them like any visitor would. Mundane things break this constantly:
robots.txt is a small text file at your domain root that tells crawlers where they’re not welcome. It’s also home to a genuinely counterintuitive trap: robots.txt blocks crawling, not indexing. Block a page there and Google can still index its bare URL from links elsewhere — it just can’t read the page. The result is that ghostly search listing with no description, which helps nobody. (And a stray Disallow: / — one character of overreach — quietly walls off the entire site.)
Status codes are the three-digit answers your server gives to every request: 200 means “here you go”, 301 means “moved permanently, follow me”, 404 means “nothing here”, 500 means “I broke”. Googlebot takes these literally. A page that intermittently returns 500s gets crawled less. A “page not found” screen that returns 200 instead of 404 — a soft 404 — teaches Google your site says “success” when it means “failure”, which erodes trust in everything else it says.
Redirect chains — where A points to B points to C points to D — waste the crawler’s patience and dilute the signal each hop was supposed to pass along. They accumulate silently over years of restructures and https migrations.
Orphan pages are pages no other page links to. Sitemap or not, a page with zero internal links is a room with no corridor leading to it. Crawlers, like guests, mostly find rooms via corridors.
And a structural note: Google uses mobile-first indexing — it’s predominantly the mobile rendering of your site that gets read and filed. If your mobile view hides half the content that desktop shows, the hidden half is what Google may never weigh.
Step two — rendering: what Google actually sees
Fetching HTML isn’t the same as seeing the page. Modern sites often ship a nearly empty HTML shell plus a bundle of JavaScript that builds the visible page in the browser. Google can handle this — it renders pages with an up-to-date Chromium, same engine as Chrome — but with two caveats that bite real businesses.
First, rendering is expensive, so it can be deferred. Your raw HTML gets read immediately; the JavaScript-built version may be processed later, in a queue. Content that exists only after scripts run is on the slow path.
Second — and this is the one that actually hurts — Googlebot doesn’t interact. It doesn’t click your “Load more” button, open your accordions’ lazy-loaded panels, or fill in your search box. Content that requires a human gesture to fetch simply never enters the record.
An illustrative composite (not a named client, but assembled from cases every technical SEO has met): a company’s beautiful single-page app renders its entire service catalogue client-side, fed by an API call. In a browser: gorgeous. In the raw HTML Google fetches first: one <div id="app"></div> and a spinner. The fix wasn’t more blog posts. It was server-side rendering the content so the librarian could read the book without operating machinery.
The two-minute diagnostic here is timeless: right-click, View Page Source — the raw HTML, not the DevTools-rendered view — and search for a sentence you consider important. If it isn’t there as text, you’re betting your visibility on the rendering queue.
Step three — indexing: the quiet filter
This is where that ten-character line lives: noindex. Placed in a page’s meta robots tag (or sent as an X-Robots-Tag header), it tells Google: crawl me if you like, but do not file me. It exists for good reasons — you genuinely don’t want your staging environment or thank-you pages indexed.
The classic disaster writes itself: an agency builds the new site on a staging domain, correctly sets noindex sitewide, launches to the live domain… and the tag comes along for the ride. Nothing looks wrong. The site works perfectly. It simply evaporates from Google over the following weeks, one recrawled page at a time — a slow-motion vanishing that gets misdiagnosed as “an algorithm update” more often than you’d think.
Even without noindex, indexing is a decision, not a right. Google Search Console has two statuses that tell you Google is declining, politely:
- “Discovered – currently not indexed” — we know this URL exists; we haven’t bothered to crawl it yet.
- “Crawled – currently not indexed” — we read it, and chose not to shelve it.
That second one is Google’s quality verdict in a trench coat. Near-duplicate pages, thin boilerplate, ten town-name landing pages that are 95% identical with the place name swapped — the doorway page pattern — are exactly what it declines. The shelf space is infinite; the librarian’s respect is not.
A related lever: the canonical tag, which tells Google “of these similar pages, this one is the original”. It’s a strong hint, not a command — and a mis-set canonical pointing every page at your homepage is a self-inflicted noindex in all but name. Structured data (the JSON-LD kind Google recommends) won’t rescue a page from any of this, but it helps a filed page get understood — and become eligible for richer listings.
There’s also a duplication problem most owners don’t know they have: your site probably answers on several addresses at once. http:// and https://, www. and bare domain, with and without a trailing slash — that’s potentially eight spellings of every page. If your server doesn’t redirect them all to one canonical form, Google meets a crowd of near-identical twins and has to guess which one is real, splitting whatever authority your links have earned between them. The fix is one round of permanent 301 redirects plus consistent canonical tags — an afternoon of configuration that some sites go years without.
How professionals actually diagnose it
None of the above requires guesswork, because Google will simply tell you — in Google Search Console, the free control panel that too many businesses have never opened. (Its data also feeds most serious SEO tooling.) The professional workflow is unglamorous:
- URL Inspection on the page that matters most. Paste in your money page. GSC reports whether it’s indexed, when it was last crawled, what the canonical resolved to, and — under “View crawled page” — the actual HTML Google captured. This one screen settles most arguments about what Google “sees”.
- The Page indexing report. Every excluded URL, grouped by reason: blocked by robots.txt,
noindex, redirect, duplicate without canonical, crawled-not-indexed. Triage is usually obvious — one reason dominates. - A full crawl of the site with a spider like Screaming Frog, which walks your site the way Googlebot would and surfaces the redirect chains, orphan pages, soft 404s, and stray meta tags no human clicks their way into finding.
- A rendering check comparing raw HTML against the rendered result, for the JavaScript problems from step two.
One caution while you investigate: the site:yourdomain.com search operator is a rough sketch, not an audit — Google itself says its counts aren’t reliable. Use Search Console’s numbers, not the operator’s.
“Could Google have penalised us?”
It’s the question owners actually lie awake on, so let’s handle it directly: true penalties — Google calls them manual actions — are real but rare, reserved for things like bought links and deceptive content, and they are not silent. Search Console has a dedicated Manual actions report, and Google notifies you there when one is applied. If that report says “No issues detected” — and for the overwhelming majority of small-business sites it does — you haven’t been penalised. You’ve been filtered or missed, which is mechanical, diagnosable, and fixable with everything above. The distinction matters because the remedies are opposite: penalties need reconsideration requests; plumbing needs a wrench.
Notice what’s absent from that workflow: keyword magic, content hacks, anything secret. Technical SEO at this layer is closer to plumbing inspection than to marketing — which is why it’s checkable, and why “we don’t know why it isn’t ranking” from a provider should raise an eyebrow. The pipeline leaves receipts.
The 20-minute self-audit
You can run the first pass yourself, today, with no tools installed:
- Search
site:yourdomain.comin Google. Rough sketch only — but zero results is a fire alarm. - Open (or set up) Google Search Console — free, and verification takes minutes. Check the Page indexing report: how many pages indexed vs excluded, and for which reasons?
- Run URL Inspection on your homepage and your most valuable page. Indexed? Crawl date? Canonical as expected?
- View source on your money page and text-search for your most important sentence and the word
noindex. The first should be present; the second absent. - Visit
yourdomain.com/robots.txt. Read it as a bouncer’s guest list. Anything disallowed that shouldn’t be? A sitemap line present? - Check your sitemap (commonly
/sitemap.xml): does it exist, is it current, is it submitted in Search Console? - Load the site on your phone and ask what’s missing versus desktop — remembering that the mobile version is the one being filed.
- Type a page-not-found URL on purpose (
yourdomain.com/xyz123) and check it actually returns a 404, not a cheerful 200.
Twenty minutes, and you’ll either have peace of mind or a short, specific list of blockages — either of which beats another month of wondering.
If that list surfaced something you don’t want to untangle alone, Zayn runs free technical visibility audits — the full crawl-render-index inspection, reported in plain English with the actual blockers ranked by impact. No retainer required to hear the diagnosis. Book a call and we’ll show you what Google sees when it looks at your site.
The algorithm isn’t ignoring you. It just hasn’t been allowed to meet you.