When creating a new WordPress site, it’s important to keep it hidden from search engines until it’s fully prepared for visitors. We follow this practice for all our new projects, as it is crucial for a seamless and professional launch.
Managing what search engines can access is vital for safeguarding private content and ensuring your site’s future SEO success. This gives you full control over when your site becomes publicly visible.
With years of experience, we have identified the most effective methods for controlling site visibility. These straightforward techniques are the same ones we apply to our own websites and recommend to our audience.
Let’s go through the precise steps to prevent search engines from crawling your WordPress site.
Reasons to Prevent Search Engines from Crawling Your WordPress Site
For the majority of WordPress websites, search engines serve as the primary source of traffic. You might wonder, why would anyone want to limit search engine access?
Here are some scenarios where you may prefer not to have search engines index your website:
- When starting out, you might not know how to set up a local development environment or a staging site, and instead, you may develop your website while it is live. In such cases, you wouldn’t want Google to index your site while it is still under construction or in maintenance mode.
- Many individuals use WordPress to create private blogs that they prefer not to have indexed by search engines due to their personal nature.
- WordPress can also be utilized for project management or as an intranet, where you would want to keep internal documents hidden from public access.
A common myth is that if no links direct to your domain, search engines will never discover your website. This isn’t entirely accurate.
For instance, there could be links to your site from a previous owner of the domain. Additionally, numerous websites exist that simply compile lists of domain names, which may include yours.
Before we explore the methods, let’s clarify two key terms:
- Crawlingrefers to the process by which search engine bots, such as Googlebot, visit your website to gather its content.
- Indexingis the process of adding that content to a vast search database, allowing it to appear in search results.
You can think of it like a librarian who reads a new book (crawling) and then includes it in the library’s catalog (indexing). Some of the methods outlined below will completely stop crawling, while others will only prevent indexing. We’ll clearly indicate which method does what!
Now, let’s explore how to prevent search engines from crawling your website. We will discuss four effective methods:
- Requesting Search Engines to Avoid Crawling Your WordPress Site
- Requesting Search Engines to Avoid Crawling Specific Pages
- Securing Your Entire Site with Password Protection via cPanel
- Using a Plugin to Password Protect Your WordPress Site
- Common Questions (FAQs)
- Expert Resources on Search Engine Crawling and Indexing
Method 1: Requesting Search Engines to Avoid Crawling Your WordPress Site
This is the easiest method, but it does not completely prevent your website from being crawled.
WordPress includes a built-in option that allows you to tell search engines not to index your site. Simply navigate to Settings » Reading and check the box labeled ‘Search Engine Visibility’.
When this box is checked, WordPress adds the following line to your website’s header:
<meta name='robots' content='noindex,nofollow' />
This tag instructs search engines not to display this page in search results (noindex), while allowing them to follow the links on this page to discover additional content (noindex.
WordPress automatically updates your site’s robots.txt file to include a disallow rule:
User-agent: *
Disallow: /
These lines instruct web crawlers not to index your web pages. However, it’s ultimately up to search engines to comply with this request or disregard it. While most search engines honor this request, there remains a possibility that some pages or images from your site could still be indexed.
To completely prevent search engines from indexing or crawling your website, you will need to password-protect your WordPress site using Methods 3 or 4.
Method 2: Requesting Search Engines Not to Index Specific Pages
You may want search engines to crawl and index your website, but exclude a particular post or specific page from appearing in search results.
The simplest way to achieve this is by using the All in One SEO (AIOSEO) plugin. It is the top SEO tool for WordPress and is trusted by over 3 million businesses.
In this tutorial, we will utilize the free version of AIOSEO, which includes the SEO Analysis tool. There is also a premium version available that provides additional features such as sitemap tools, a redirection manager, schema markup, a robots.txt editor, and more.
The first step is to install and activate the AIOSEO plugin on your website. You can find detailed instructions on how to install and configure the plugin by following our comprehensive guide on setting up All in One SEO for WordPress.
After setting up the plugin, you can instruct search engines not to index specific posts and pages. However, it’s important to note that it is ultimately up to the search engines to honor this request or disregard it.
To prevent a post or page from being indexed, simply edit the desired content. Scroll down to the AIOSEO Settings section at the bottom of the WordPress editor and click on the ‘Advanced’ tab.
You will see that the article is currently using the default robots settings.
To modify this, toggle the ‘Use Default Settings’ option to the off position.
Now, you can check the ‘No Index’ box. Once the post is published or updated, search engines will be requested not to index it.
For more information, check out our in-depth AIOSEO review.
Method 3: Protecting Your Entire Site with a Password Using cPanel
If your WordPress hosting provider gives you cPanel access to manage your hosting account, you can secure your entire site using cPanel.
Log in to your cPanel dashboard and click on the ‘Directory Privacy’ icon located in the ‘Files’ section.
Locate the folder where you installed WordPress, which is typically the public_html folder.
Next, click the ‘Edit’ button next to that folder.
Important Note:If you have multiple WordPress sites installed within thepublic_htmldirectory, click on thepublic_htmllink to navigate through those sites, and then edit the folder for the specific website you wish to password-protect.
This will take you to a screen where you can enable password protection.
Check the box that says ‘Password protect this directory’ and click the ‘Save’ button. You can also customize the name for the protected directory if desired.
A confirmation message will appear, indicating that the access permissions for the directory have been updated.
Next, click the ‘Go Back’ button.
You will be directed to a screen where you can create a username and password required to access this directory.
To set up password protection, enter a username and password, then confirm your password. It’s important to store your username and password securely, such as in a password manager app.
After clicking the ‘Save’ button, you will have successfully enabled password protection for your WordPress site.
Now, whenever a visitor or search engine accesses your website, they will be required to enter the username and password you created earlier to gain access.
Method 4: Using a Plugin to Password Protect WordPress
If you’re using a managed WordPress hosting service, you might not have access to cPanel. In this case, you can utilize a WordPress plugin to password-protect your site.
Here are two of the most popular options:
- SeedProd is the leading coming soon and maintenance mode plugin for WordPress, utilized by over 2 million websites. It offers comprehensive access control and permissions features that enable you to hide your website from everyone, including search engines. We provide a detailed step-by-step guide for using SeedProd.
- Password Protected is a straightforward solution for password-protecting your WordPress site with a single password (no user accounts required). Check out our step-by-step guide on how to implement password protection on a WordPress site.
Frequently Asked Questions (FAQs)
Here are the answers to some frequently asked questions about preventing search engines from crawling WordPress websites.
1. What is the difference between blocking crawling and blocking indexing?
Blocking crawling (using robots.txt) is similar to placing a ‘do not enter’ sign, instructing search engine bots not to access your site’s content. Blocking indexing (using a noindex tag) informs search engines not to include your site in their search results, even if they have crawled it.
2. Is the built-in WordPress ‘Discourage search engines’ option sufficient to hide my site?
For temporary development purposes, this option is often adequate. However, it is merely a request, and some search engines may not comply. For complete privacy or to safeguard sensitive information, using password protection is the only reliable method.
3. Will password-protecting my site negatively impact my SEO when I am ready to launch?
No, it will not negatively affect your SEO. When your site is password-protected, search engines cannot access it, meaning it has no SEO value (positive or negative). Once you remove the password protection and allow indexing, search engines will start to crawl and rank it as usual.
4. How can I verify if Google has already indexed my site?
The simplest method is to perform a site search on Google. Visit google.com and enter site:yourdomain.com in the search field, replacing ‘yourdomain.com’ with your actual website URL. If you see any results, it means your site has been indexed.
Comprehensive Guides on Search Engine Crawling and Indexing
We hope this article has assisted you in preventing search engines from crawling or indexing your WordPress site. You may also find these related guides on search engine crawling and indexing helpful:
- How to Conceal a WordPress Page from Google
- How Can I Get My WordPress Site Listed on Google? (Beginner’s Guide)
- What Caused My WordPress Site to Be De-Indexed from Google?
- How to Request Google to Recrawl URLs of Your WordPress Site
- How to Optimize Your WordPress Robots.txt File for SEO
- Understanding the WordPress SEO Crawl Budget Issue and How to Resolve It
If you enjoyed this article, please subscribe to our YouTube Channel for WordPress video tutorials. You can also connect with us on Twitter and Facebook.
