Let's say you've built a website. You've invested in the design, bought a domain, and so on. But a website isn't only what you see on the screen. A lot happens behind the scenes. You have to understand how the internet works, as well as how Search Engine Optimization (SEO) works.

Speed is a key part of it, too. A slow site can have very weak SEO, because most people won't spend much time waiting for a page to load, especially these days.

Site speed is a well-known topic with many problems of its own to consider, like Largest Contentful Paint (LCP), Cumulative Layout Shift (CLS), and server speed. But I'll leave those for another article.

This one is about the problems that stop search engines from reaching and reading your pages.

In my opinion, technical SEO is much harder to apply than regular SEO, which focuses on content and keywords. The technical side usually requires a developer's knowledge, and might need some changes that can feel a bit uncomfortable if you don't have a technical background.

A technical problem can stay unnoticed for months, because you usually don't see it. It's not a design problem. You only find it when you run specific checks, for example with Google PageSpeed Insights.

In this tutorial, you'll find and fix the most common ones on a small practice site. I didn't want to take a real website and shame its owners, so I built a practice site with the problems already in it.

Table of Contents

• Prerequisites
• What Is Technical SEO?
• How Common Are These Problems on Real Websites?
• How to Set Up the Practice Site
• How to Check a Page's HTTP Status Code
  • What Is a Soft 404?
• How to Find and Fix Redirect Chains
• How to Make Sure robots.txt Isn't Blocking Important Pages
• How to Find an Accidental noindex
  • How to Check the Robots Meta Tag
  • How to Check the X-Robots-Tag Header
• How to Use Canonical Tags Correctly
• How to Fix Missing and Duplicate Titles and Meta Descriptions
• How to Make Your Links Crawlable
• How to Check Whether Your Content Needs JavaScript to Appear
• Can You Check a Page in Five Minutes?
• Conclusion

Prerequisites

You don't need to be an experienced programmer to follow along. You'll need:

• A web browser, such as Chrome, Firefox, Edge or Safari. In Safari, first turn on Show features for web developers under Settings > Advanced.
• A terminal: Terminal on macOS, PowerShell on Windows, or any terminal on Linux.
• curl, a small program that downloads web pages. It comes with macOS, most Linux distributions, and Windows 10 (version 1803 and later) and Windows 11.
• Node.js, to run the practice site. The code was tested with Node.js 26.8 and curl 8.7.

On Windows PowerShell, type curl.exe instead of curl, because there curl is a shortcut to a different command. Replace | grep -i "text" with | Select-String "text", and /dev/null with NUL.

What Is Technical SEO?

Google has programs called crawlers. They search the internet all the time for new websites and new pages. When Google discovers a new address, it may crawl the page and process what's on it (the text, images, and videos).

If Google decides the page is worth indexing, it may add it to a very big database of web pages, called the index. Google can then consider the pages in its index for its search results, when they're relevant to what someone searches for. Google's guide to how Search works calls these three steps crawling, indexing, and serving.

So crawling doesn't guarantee indexing, and indexing doesn't guarantee a place in the results. The same guide says Google doesn't guarantee any of the three stages, even for a page that follows all of its rules.

Fixing technical problems won't necessarily make your site rank first on Google. But it does strengthen your chances of ranking higher, because every problem you fix makes your site better.

Technical SEO has to come together with regular SEO, like keywords, backlinks, engagement on social media, and so on.

How Common Are These Problems on Real Websites?

The numbers below come from audits run with the site-auditing tool I build. Each website counts once, using its first audit, and this data comes from October 25th.

Crawls check many pages on one site (a median of 10 pages here). Single-page audits check one page, usually the home page.

Problem	How often	Sample
At least one page with no canonical tag	64.2% of sites	53 crawled sites, Aug 2025 to Oct 2026
Two or more pages with exactly the same title	61.8% of sites	34 crawled sites with two or more pages that loaded
At least one page with no meta description	49.1% of sites	53 crawled sites
At least one link leading to a page that returns 404	18.9% of sites	53 crawled sites
Page built mostly with JavaScript	17.7% of pages	277 single-page audits, May to Oct 2026
At least one page with no title	13.2% of sites	53 crawled sites
A link that search engines may not be able to follow	8.3% of pages	278 single-page audits
A robots.txt file with errors, or one that failed to load	7.9% of sites	252 single-page audits, Jul to Oct 2026
Page blocked by noindex or by robots.txt	4.8% of pages	251 single-page audits, Jul to Oct 2026

These are all technical problems, but they don't all weigh the same. Missing canonical tags are at the top, but Google doesn't require a canonical tag, so a page without one isn't broken by itself. It matters when the same or very similar content is reachable at more than one URL, as you'll see in the canonical section.

Duplicate titles are something I had myself: when you write code, it's easy to miss things like that. That's why there are tools that check for these problems.

Let's go over each issue in the table to make sure they're all clear:

• Missing canonical tag: