🤖 AI 资讯

Let's say you've built a website. You've invested in the design, bought a domain, and so on. But a website isn't only what you see on the screen. A lot happens behind the scenes. You have to understand how the internet works, as well as how Search Engine Optimization (SEO) works.

Speed is a key part of it, too. A slow site can have very weak SEO, because most people won't spend much time waiting for a page to load, especially these days.

Site speed is a well-known topic with many problems of its own to consider, like Largest Contentful Paint (LCP), Cumulative Layout Shift (CLS), and server speed. But I'll leave those for another article.

This one is about the problems that stop search engines from reaching and reading your pages.

In my opinion, technical SEO is much harder to apply than regular SEO, which focuses on content and keywords. The technical side usually requires a developer's knowledge, and might need some changes that can feel a bit uncomfortable if you don't have a technical background.

A technical problem can stay unnoticed for months, because you usually don't see it. It's not a design problem. You only find it when you run specific checks, for example with Google PageSpeed Insights.

In this tutorial, you'll find and fix the most common ones on a small practice site. I didn't want to take a real website and shame its owners, so I built a practice site with the problems already in it.

Table of Contents

• Prerequisites
• What Is Technical SEO?
• How Common Are These Problems on Real Websites?
• How to Set Up the Practice Site
• How to Check a Page's HTTP Status Code
  • What Is a Soft 404?
• How to Find and Fix Redirect Chains
• How to Make Sure robots.txt Isn't Blocking Important Pages
• How to Find an Accidental noindex
  • How to Check the Robots Meta Tag
  • How to Check the X-Robots-Tag Header
• How to Use Canonical Tags Correctly
• How to Fix Missing and Duplicate Titles and Meta Descriptions
• How to Make Your Links Crawlable
• How to Check Whether Your Content Needs JavaScript to Appear
• Can You Check a Page in Five Minutes?
• Conclusion

Prerequisites

You don't need to be an experienced programmer to follow along. You'll need:

• A web browser, such as Chrome, Firefox, Edge or Safari. In Safari, first turn on Show features for web developers under Settings > Advanced.
• A terminal: Terminal on macOS, PowerShell on Windows, or any terminal on Linux.
• curl, a small program that downloads web pages. It comes with macOS, most Linux distributions, and Windows 10 (version 1803 and later) and Windows 11.
• Node.js, to run the practice site. The code was tested with Node.js 26.8 and curl 8.7.

On Windows PowerShell, type curl.exe instead of curl, because there curl is a shortcut to a different command. Replace | grep -i "text" with | Select-String "text", and /dev/null with NUL.

What Is Technical SEO?

Google has programs called crawlers. They search the internet all the time for new websites and new pages. When Google discovers a new address, it may crawl the page and process what's on it (the text, images, and videos).

If Google decides the page is worth indexing, it may add it to a very big database of web pages, called the index. Google can then consider the pages in its index for its search results, when they're relevant to what someone searches for. Google's guide to how Search works calls these three steps crawling, indexing, and serving.

So crawling doesn't guarantee indexing, and indexing doesn't guarantee a place in the results. The same guide says Google doesn't guarantee any of the three stages, even for a page that follows all of its rules.

Fixing technical problems won't necessarily make your site rank first on Google. But it does strengthen your chances of ranking higher, because every problem you fix makes your site better.

Technical SEO has to come together with regular SEO, like keywords, backlinks, engagement on social media, and so on.

How Common Are These Problems on Real Websites?

The numbers below come from audits run with the site-auditing tool I build. Each website counts once, using its first audit, and this data comes from October 25th.

Crawls check many pages on one site (a median of 10 pages here). Single-page audits check one page, usually the home page.

These are all technical problems, but they don't all weigh the same. Missing canonical tags are at the top, but Google doesn't require a canonical tag, so a page without one isn't broken by itself. It matters when the same or very similar content is reachable at more than one URL, as you'll see in the canonical section.

Duplicate titles are something I had myself: when you write code, it's easy to miss things like that. That's why there are tools that check for these problems.

Let's go over each issue in the table to make sure they're all clear:

• Missing canonical tag: