Looking for a robots.txt file with no index no follow code? Read on.
As I was working on my web hosting company for The Great Build, I realized that my site had more pages indexed than I had created.
On checking the pages, I realized they were automatically generated by WHMCS when I installed it. The pages included Knowledge Base and Affiliate pages of different languages.
Since my target customers are just Kenyans, I did not see why the pages should be there. First off, none of my target customers will be accessing them.
Secondly, the pages do not have a lot of content. This can be problematic in the grand scheme of things when it comes to search engine optimization.
Therefore, the best option would be to “no index” the content.
I have installed WHMCS in a folder named “clients”. Therefore, I wanted to no-index this entire folder so that none of WHMCS pages (billing, checkout, cart, etc.) are indexed on Google.
This is what I did.
Step 1: Create a robots.txt file. If you already have an existing file in the root folder of your domain name, great. You can use the existing file.
Step 2: Enter this information in the file:
User-agent: *
Disallow: /clients/*
NB: replace “clients” with the name of your folder.
Step 3: Create a .htaccess file where you have installed WHMCS (in my case, it is in the “clients” folder) and enter the following:
Header set X-Robots-Tag “noindex, nofollow”
This code will prevent Google search bots from indexing content in that folder.
Step 4: This is an optional step. If there are any WHMCS pages that had already been indexed, you can either remove them manually or let them be dropped off the Google index.
