If the public robots.txt response still shows old rules after you save an edit, identify which system serves that response before changing more settings.
Before investigating why a file fails to refresh, it is worth clarifying the scope of these directives. According to Google’s robots.txt introduction, the primary purpose of a robots exclusion file is to manage crawler traffic and prevent server overload rather than serve as a mechanism to secure private data or guarantee index removal. Directives manage crawling, not indexation. When a public response does not reflect your intended crawl permissions, diagnosing the technical architecture serving the request allows you to identify where the stale output originates.
Understand How WordPress Serves Virtual Robots Files
WordPress can generate a robots.txt response dynamically. Seeing text at the public URL therefore does not prove that a matching file exists on disk. Start by identifying which system handles that request on your hosting setup.
Under the hood, the internal function WordPress do_robots() builds this dynamic output and applies the robots_txt filter. Plugins, custom code snippets, and active themes use this filter to append, strip, or rewrite rules on the fly. If you adjust settings in an SEO plugin that relies on this internal filter, the change affects generated output; check that plugin’s documentation to understand what it actually saves.
However, dynamic handling works only if the web server passes the request to the application. If something sits upstream or intercepts the path before it touches dynamic code, updates within the administrative dashboard produce no visible changes on the public URL.
Potential Layers Serving Your Robots Directives
For the separate question of what the rules should accomplish, see our robots.txt review guide.
Layer 1: A Physical File on the Server Root
One possibility is a physical file in the public web root. Depending on server configuration, that file may be served before WordPress handles the request. Ask your host which directory and routing rule apply to this domain.
If a physical file exists on the hosting storage, the server engine may send that file straight to the requester. In such a setup, any filter hooked into dynamic core functions never runs because the request never reaches the application layer. An administrative user editing directives via a plugin dashboard may believe their settings are broken, when in reality the server is faithfully delivering an abandoned file left behind by a migration or an earlier manual upload.
Layer 2: Application-Level Plugins and Object Caches
If the request reaches WordPress, inspect the components responsible for its generated output. At this stage, multiple plugins might compete for the robots_txt hook. If you have two different tools configured—such as an SEO utility and a legacy security or caching plugin—one might overwrite the directives added by the other.
An application or hosting cache is another possibility. Check whether the specific robots.txt response is cached, where that copy is stored and how it is refreshed. Do not assume that every cache on the site stores the same material.
Layer 3: Web Application Firewalls and Edge CDN Caches
Web traffic frequently passes through intermediary proxies, managed reverse proxies at your host, or edge content delivery networks (CDNs). Edge systems cache static assets across distributed locations to reduce server load.
If an edge service caches this path, it may return an older response after the origin changes. That is a hypothesis to test using the provider’s documented cache behaviour and the response you capture. The filename alone does not establish whether caching is active.
Step-by-Step Diagnostic Method
Step 1: Check for a Static File in Hosting Storage
Connect to your hosting server using Secure File Transfer Protocol (SFTP), SSH, or your host’s control panel file manager. Identify the domain’s public web root with your host; it may differ from the directory containing the WordPress application.
- If a physical file named
robots.txtis present, download a local backup copy to your computer. Once the backup is secured, examine its contents. If the static text matches the stale public response, that is a useful clue, but matching text alone does not prove the route; confirm the serving configuration. - If no file appears in that directory, ask whether another document root, server rule or proxy serves the path.
Step 2: Inspect HTTP Response Headers
A direct inspection of the HTTP headers sent alongside the response provides concrete diagnostic clues. You can inspect headers using command-line utilities like curl or through your web browser developer tools.
To retrieve the body and response headers, substitute your own public URL in this read-only command:
curl -sS -D - https://example.com/robots.txtReview the output for the following indicators:
| Header Name | Observed Value Indicator | Diagnostic Interpretation |
|---|---|---|
Server |
Cloudflare, Fastly, or Custom Edge | A clue about the responding system; confirm it with hosting records. |
CF-Cache-Status or X-Cache |
HIT |
Consult the provider’s definition of HIT and compare the returned body. |
X-Cache |
MISS or BYPASS |
Does not rule out another cache upstream; check the provider’s meaning. |
Last-Modified |
A static date and time | A response timestamp, not proof that a physical file exists. |
Content-Type |
text/plain; charset=utf-8 |
Plain-text content; both physical and generated responses can use it. |
Canada Create™ builds and optimizes WordPress sites for Toronto businesses. Tell us your goals and we will recommend the right setup.
If cache headers indicate a cache hit, purge the cache specifically for that individual path within your CDN or host control panel, then re-check the URL.
Applying Safe Corrections
Once you identify the layer delivering the stale text, implement the appropriate correction methodically:
- If a physical file was overriding dynamic rules: If you intend to use a plugin to manage your rules, plan the handover with your host, preserve the existing rules and verify your backup before renaming or removing the file. If you prefer using a static file for predictability, edit that physical file directly rather than adjusting plugin interfaces.
- If dynamic hooks are conflicting: Temporarily deactivate non-essential SEO or rule-rewriting plugins in a staging environment to observe which tool retains control over the dynamic output.
- If edge caching retained stale headers: Use a documented refresh or path-specific purge for robots.txt, then verify the result. Avoid changing caching for unrelated paths.
After completing an update, request the live URL again and record the response headers, body and timestamp again to confirm the public response matches your expected syntax. If the output remains stubborn, contact your hosting provider’s technical support with your captured header traces, asking specifically whether their server configuration serves a fallback physical file or enforces custom rewrite rules for the path.
Frequently Asked Questions
Why is my WordPress robots.txt not updating?
A physical robots.txt file overrides the virtual one, or a caching layer or CDN serves an old copy.
How do I edit robots.txt in WordPress?
Through your SEO plugin, or by editing the physical file in the site root.
What should robots.txt contain?
Usually just a sitemap reference and a few disallow rules. Do not block CSS, JavaScript or important pages.
Who can audit my robots.txt?
Our SEO services include robots.txt and indexing checks.
