Robots.txt and Sitemap Examples for Content Sites

An earlier version of this guide relied on generic example.com examples. This time, I wanted to check what SeoGeo Tech actually serves. On October 8, 2026, I checked the public robots.txt, followed its sitemap pointer, and looked for this article in the resulting page sitemap. This is a record of those responses, not a claim about what Google has indexed.

First, the live robots.txt file

These are selected lines from the file served at the root of the WordPress site:

User-Agent: *
Allow: /
Allow: /wp-content/uploads/
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /*?s=

Sitemap: https://seogeo.tech/sitemap.xml

The file also disallows comment feeds and feed paths; the excerpt above is not a replacement robots.txt file. It permits crawling of public article paths and names the sitemap URL. It does not say whether a page should be indexed. The /wp-admin/ rule is a crawler instruction, not a login or security control. I would not copy these paths to another site without checking its own URL structure.

Then, follow the sitemap rather than guessing its shape

At /sitemap.xml I found a sitemapindex, not a list of article URLs. Its child is /page-sitemap.xml; that child includes https://seogeo.tech/articles/robots-txt-sitemap-examples/. The article URL itself returned HTTP 200 and its canonical element pointed back to that same URL when I checked it. That is the route I can verify: robots pointer → sitemap index → child sitemap → live page and canonical.

Here is the relevant child node from the live sitemap index. I have only normalized the indentation:

<sitemap>
  <loc>https://seogeo.tech/page-sitemap.xml</loc>
  <lastmod>2026-09-30T15:52:37+00:00</lastmod>
</sitemap>

And this is the entry for this article in the live page sitemap:

<url>
  <loc>https://seogeo.tech/articles/robots-txt-sitemap-examples/</loc>
  <lastmod>2026-07-09T14:35:15+00:00</lastmod>
</url>

These timestamps reflect the sitemap responses I inspected before publishing this revision. After the update goes live, I will check whether the article’s sitemap entry reflects the substantive content change.

The robots.txt comments include a “Yoast block” label, while the sitemap response identifies Rank Math as its generator. Those labels describe the current file output; they do not, on their own, prove that two SEO plugins are active or that the sitemap is broken. If I need to change either file, I must first confirm which WordPress setting or server rule actually owns it. The local scripts/build.mjs also produces a static dist/sitemap.xml, but that build artifact is not the live WordPress sitemap index I just inspected.

What I would check after the next release

curl -fsSL https://seogeo.tech/robots.txt
curl -fsSL https://seogeo.tech/sitemap.xml
curl -fsSL https://seogeo.tech/page-sitemap.xml

I would confirm that the sitemap pointer still resolves, find the intended canonical URL in the child sitemap, then fetch that page and compare its canonical element. If the sitemap lists a redirect or an obsolete URL, the next step is to correct the sitemap source, not to add another robots rule. If a public article is missing, I would check its WordPress visibility and sitemap inclusion settings before assuming a crawl problem. I did not change the live sitemap during this article review.

What these files cannot answer

A robots.txt rule manages crawler access; it is not a reliable way to keep a URL out of Google Search. For a page that should not be indexed, Google’s noindex guidance requires the page to remain accessible so the crawler can see the rule. A sitemap entry suggests a preferred URL, but it does not force Google to crawl, index, or select it as canonical. Google’s sitemap documentation and canonical documentation make those boundaries explicit.

I also do not copy a sitemap <lastmod> timestamp into an article’s visible “Updated” label. WordPress and the sitemap plugin may record publication-system changes at a different time from an editorial revision. The right check is whether the timestamp accurately reflects a significant change to that page, not whether it makes the page look fresher.