The Problem
Duplicate URLs aren’t a “SEO issue”. They’re a system design issue.
When I’m building a content-heavy app solo, URLs multiply fast. Trailing slash vs no slash. ?utm_source= junk. Filters like ?state=ca&role=icu. Sort options. Even framework-level behavior (redirects, notFound(), middleware) can create multiple reachable URLs for the same document.
Google doesn’t ask permission. It picks a canonical on its own if you don’t.
My failure mode was predictable: I’d ship a feature, traffic would go up, then Search Console would show duplicates, “Discovered — currently not indexed”, and soft 404s. Worse, the wrong URLs would rank (parameterized junk), and the ones I cared about wouldn’t.
So I treated it like any other production bug: define an invariant. One document → one indexable URL.
Options I Considered
There are a few common approaches. None is perfect.
| Approach | Pros | Cons | Best For |
Canonical only (<link rel="canonical">) | Keeps alternates crawlable; consolidates signals | Google may ignore it; duplicates still consume crawl budget | Mild duplication where alternates are still useful |
noindex only (<meta name="robots" content="noindex">) | Hard stop for indexing; fast cleanup | Doesn’t consolidate signals well; still crawled unless blocked | Thin pages or internal utility pages |
| Redirect alternates to one URL (301/308) | Strongest consolidation; simplest mental model | Breaks some UX (filters/back button); can cause redirect chains | When alternates are truly equivalent |
Canonical + noindex + robots rules (hybrid) | Defensive; handles messy real-world URLs | Easy to over-block; requires discipline in routing | Apps with filters, facets, and lots of generated URLs |
I started with canonical-only. It worked until it didn’t.
Here’s why canonical-only failed for me:
- Parameterized URLs often got indexed anyway. Google treated them as distinct enough.
- Canonical mistakes are easy. One bug in a shared layout and you emit the wrong canonical for thousands of pages.
- Crawl budget isn’t theoretical when you have lots of pages. Duplicates dilute attention.
Redirect-only was tempting. But I didn’t want to redirect every filter combination.
Faceted URLs are tricky:
- Some facets are trash (sort order, tracking params).
- Some facets are legitimate landing pages (state, role, category).
If you redirect everything, you kill valid long-tail entry points. If you redirect nothing, you get duplication.
So I went hybrid.
What I Chose (and Why)
I chose canonical + selective noindex + robots.txt rules for known junk params, plus a hard rule: every indexable page must emit an explicit canonical.
Ranked reasons:
- Fail-safe behavior. If a duplicate URL slips through, it still won’t get indexed.
- Control. I decide which facets deserve indexing. Not Google.
- Incremental rollout. I can add rules per route type without rewriting routing.
What I gave up:
- I gave up the simplicity of “canonical everywhere and pray”. Now I maintain explicit allow/deny logic.
- I gave up indexing for some URLs that might’ve been valuable. That’s on me to evaluate.
Step 1: Normalize the canonical URL
In Next.js, the easiest trap is building canonicals from req.url or searchParams. Don’t.
I treat canonical as a pure function of the route params that define the document.
export function canonicalUrl(baseUrl: string, pathname: string) {
const cleanPath = pathname === "/" ? "/" : pathname.replace(/\/+$/, "");
return new URL(cleanPath, baseUrl).toString();
}
export function isIndexablePath(pathname: string) {
if (pathname === "/") return true;
if (/^\/states\/[a-z]{2}$/.test(pathname)) return true;
if (/^\/cities\/[a-z-]+$/.test(pathname)) return true;
if (/^\/categories\/[a-z-]+$/.test(pathname)) return true;
if (/^\/jobs\/[0-9]+$/.test(pathname)) return true;
return false;
}
Step 2: Emit canonical + robots per page (or layout)
If you’re on the App Router, generateMetadata() is the cleanest place to do this.
import type { Metadata } from "next";
import { canonicalUrl, isIndexablePath } from "@/app/lib/seo";
const BASE_URL = process.env.NEXT_PUBLIC_BASE_URL!;
export async function generateMetadata(
{ params }: { params: Promise<{ slug?: string[] }> }
): Promise<Metadata> {
const { slug } = await params;
const pathname = "/" + (slug?.join("/") ?? "");
const canonical = canonicalUrl(BASE_URL, pathname);
const indexable = isIndexablePath(pathname);
return {
alternates: { canonical },
robots: indexable
? { index: true, follow: true }
: { index: false, follow: true },
};
}
export default function Page() {
return null;
}
That follow: true is intentional. I still want discovery through internal links even if the page itself isn’t indexable.
Step 3: Kill obvious junk at the robots layer
Robots.txt isn’t a noindex mechanism anymore (Google stopped respecting noindex in robots years ago). But it’s still useful for crawl control.
I block parameters that should never be crawled.
# public/robots.txt
User-agent: *
Disallow: /*?utm_
Disallow: /*&utm_
Disallow: /*?ref=
Disallow: /*&ref=
Disallow: /*?sort=
Disallow: /*&sort=
# Let everything else be crawlable
Allow: /
This doesn’t prevent indexing if there are external links pointing at a URL (Google can index a URL it can’t crawl). That’s why I still rely on noindex for anything that’s reachable and shouldn’t be indexed.
Step 4: Make 404s real 404s
Soft 404s were another source of garbage URLs showing up. If a page doesn’t exist, return a real 404.
In App Router, notFound() does the right thing if you don’t swallow it and render a 200.
import { notFound } from "next/navigation";
async function getJob(id: number) {
const res = await fetch(`${process.env.API_BASE_URL}/jobs/${id}`, {
cache: "no-store",
});
if (res.status === 404) return null;
if (!res.ok) throw new Error(`Failed to fetch job ${id}: ${res.status}`);
return res.json() as Promise<{ id: number; title: string }>;
}
export default async function JobPage(
{ params }: { params: Promise<{ id: string }> }
) {
const { id } = await params;
const jobId = Number(id);
if (!Number.isInteger(jobId) || jobId <= 0) notFound();
const job = await getJob(jobId);
if (!job) notFound();
return (
<main>
<h1>{job.title}</h1>
<p>Job #{job.id}</p>
</main>
);
}
That tiny Number.isInteger(jobId) check prevented a whole class of /jobs/abc garbage from turning into “valid” pages.
How It Worked in Production
This was one of those fixes where you don’t get to celebrate immediately. Google takes its time. Also, Search Console reporting lags.
But the signals were clear.
- Duplicate canonical issues dropped from 46 to 0 in 9 days.
- Soft 404s dropped from 12 to 0 after I fixed
notFound() usage and stopped returning 200s for missing entities.
- “Discovered — currently not indexed” URLs went down by 50+ after I blocked crawl traps (
utm_, sort, ref) and noindexed non-landing facet pages.
The surprise: canonical correctness mattered more than I expected.
I had one bug where I accidentally emitted the same canonical for every city page because I computed it in a layout using the parent route path. Google didn’t just ignore the canonical. It started clustering pages together. Rankings got weird. Pages dropped.
After I moved canonical generation to the leaf route and made it purely derived from route params, the clustering stopped.
This wasn’t “SEO”. It was a distributed system resolving conflicting identifiers.
When This Doesn't Work
This setup breaks when you actually want faceted navigation to be indexable at scale.
If your business depends on long-tail combinations (think /laptops?brand=lenovo&ram=32gb&cpu=amd), blanket noindex on parameterized URLs will kneecap you. In that world, you need a real facet strategy: allow-list specific combinations, generate clean path-based landing pages, and control internal linking so you don’t create infinite crawl graphs.
Also: if your canonical logic depends on runtime headers (host, protocol) behind proxies/CDNs, you’ll generate mismatched canonicals (http vs https). That’s a mess. Use an explicit BASE_URL and stick to it.
Key Takeaways
- Treat URLs like primary keys. One document should have one indexable identifier.
- Canonical-only is optimistic. Canonical + selective
noindex is defensive.
- Robots.txt controls crawl. It doesn’t guarantee deindexing.
- Make 404s real 404s. Soft 404s are just duplicate-content bugs wearing a different hat.
- Keep an allow-list for indexable routes. If you can’t explain why a URL should rank, it shouldn’t be indexable.
Closing
I’ve settled on a rule: if a URL can be generated by a user clicking around (filters, sorting, tracking params), it’s guilty until proven innocent.
What’s your rule for deciding which faceted URLs become first-class landing pages, and which ones get canonical + noindex?