SiteCozy

  • My account

Should you block admin-ajax.php in robots.txt?

2018-11-07

Without hesitating I answer “YES!”. I don’t think any wordpress theme owners use admin-ajax.php to hide URL links on web pages. You can check it quite easily by visiting your wordpress website, doing a right click, click on Inspect, network and reload the page. If you see no admin-ajax.php it means that the theme and the plugin do not use admin-ajax.php for loading this page. Then, if you see admin-ajax.php in the list, this may mean that a plugin uses admin-ajax.php. Anyways if a plugin prompts admin-ajax.php to get a link. I think it is not a good thing because it looks like a hidden link to me.  

I am not aware of any situations where a theme owner would prompt admin-ajax.php to redirect visitors to a different webpage. I do understand that we could use ajax to get the schedule of a flight in real time. Ajax is used for data that updated in real time. This could be the retrieval of an API to update the weather forecast or flight schedule. This type of information should not be crawled by Googlebot because it is changing all the time. 

For your information, on one of our website, we saw that googleBot didn’t stop crawling admin-ajax.php. It returned a HTTP 200 code but Googlebot stopped visiting other webpages of the website. We decided to block admin-ajax.php in robots.txt. We did the same thing here:

https://sitecozy.com/robots.txt

We created a robots.txt file manually and we added the following directives:

User-agent: *
Disallow: /wp-admin/

Related Posts:

  • Why we uninstalled Ajax search forms
  • Htaccess code for a WordPress subdomain multisite
  • Htaccess code for a wordpress subfolder multisite
Download our Broken link checker freeware here Buy a license key for the Sitecozy broken link checker

Customer Login

Lost password?

Categories

  • Webmaster advice
  • SEO advice
  • Web hosting
  • SiteCozy link checker KB
  • WordPress theme & plugin reviews
  • All articles
Disclosure: We are a professional review site that receives compensation from the companies whose products we review. We test each product thoroughly and give high marks to only the very best. We are independently owned and the opinions expressed here are our own.

Copyright Sitecozy 2018