Default Robots.txt in WordPress - Should i change it??
-
I have a WordPress site as using theme Genesis i am using default robots.txt. that has a line Allow: /wp-admin/admin-ajax.php, is it okay or any problem. Should i change it?
-
Yes, we're a news site as well and in our case we want to make sure the low quality pages on TNW aren't indexed.
-
Thank you both for your response.
@Martijn your robots.txt is really a nice example but for my new site is it good practice to block this areas??
@Peter To be a safe side I was using the same robots.txt...
-
In addition of Martijn here is mine robots.txt:
User-agent: *
Disallow:Sitemap: http://peter.nikolow.me/sitemap_index.xml
But using Yoast - categories, tags, most of archives and other generated pages are disabled for indexing.
-
Hi Peter,
Usually I would say it's not enough as the robots.txt is forgeting about excluding the search pages and in most cases you want to make sure the WP core files are not included + tag pages. Take a look at our robots.txt to see what we've included there: http://thenextweb.com/robots.txt then you'll notice we include for example these:
User-agent: *
Disallow: ?p=
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /wp-admin/*
Disallow: /wp-register.php
Disallow: /wp-content/themes/icetea/includes/*
Disallow: /tag/
Disallow: ?s=
Disallow: /search/*Other cases in our robots.txt are very specifically in there because of our site and may not apply to others.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Changing website with country-specific TLD to unlisted. How?
Help: I have a website with a country-specific TLD. Google Console sees the target country as the same for the TLD, but tells me that I can 'select Unlisted' if I "want to ensure that your site is not associated with any country or region." The question is, how? I cannot see how to edit the country in the new console? Any ideas/help?
Intermediate & Advanced SEO | | twofourseven0 -
Robots.txt, Disallow & Indexed-Pages..
Hi guys, hope you're well. I have a problem with my new website. I have 3 pages with the same content: http://example.examples.com/brand/brand1 (good page) http://example.examples.com/brand/brand1?show=false http://example.examples.com/brand/brand1?show=true The good page has rel=canonical & it is the only page should be appear in Search results but Google has indexed 3 pages... I don't know how should do now, but, i am thinking 2 posibilites: Remove filters (true, false) and leave only the good page and show 404 page for others pages. Update robots.txt with disallow for these parameters & remove those URL's manually Thank you so much!
Intermediate & Advanced SEO | | thekiller990 -
Can't find X-Robots tag!
Hi all. I've been checking out http://www.unthankbooks.com/ as it seems to have some indexing problems. I ran a server header check, and got a 200 response. However, it also shows the following: X-Robots-Tag:
Intermediate & Advanced SEO | | Blink-SEO
noindex, nofollow It's not in the page HTML though. Could it be being picked up from somewhere else?0 -
Google: How to See URLs Blocked by Robots?
Google Webmaster Tools says we have 17K out of 34K URLs that are blocked by our Robots.txt file. How can I see the URLs that are being blocked? Here's our Robots.txt file. User-agent: * Disallow: /swish.cgi Disallow: /demo Disallow: /reviews/review.php/new/ Disallow: /cgi-audiobooksonline/sb/order.cgi Disallow: /cgi-audiobooksonline/sb/productsearch.cgi Disallow: /cgi-audiobooksonline/sb/billing.cgi Disallow: /cgi-audiobooksonline/sb/inv.cgi Disallow: /cgi-audiobooksonline/sb/new_options.cgi Disallow: /cgi-audiobooksonline/sb/registration.cgi Disallow: /cgi-audiobooksonline/sb/tellfriend.cgi Disallow: /*?gdftrk Sitemap: http://www.audiobooksonline.com/google-sitemap.xml
Intermediate & Advanced SEO | | lbohen0 -
Wordpress Duplicate Content
We have recently moved our company's blog to Wordpress on a subdomain (we utilize the Yoast SEO plugin). We are now experiencing an ever-growing volume of crawl errors (nearly 300 4xx now) for pages that do not exist to begin with. I believe it may have something to do with having the blog on a subdomain and/or our yoast seo plugin's indexation archives (author, category, etc) --- we currently have Subpages of archives and taxonomies, and category archives in use. I'm not as familiar with Wordpress and the Yoast SEO plugin as I am with other CMS' so any help in this matter would be greatly appreciated. I can PM further info if necessary. Thank you for the help in advance.
Intermediate & Advanced SEO | | BethA0 -
Are free WordPress templates bad for SEO?
Hello, I often build sites using WordPress. The other day I read an article in which the author stated that the sites built using the free WordPress template are not SEO-friendly. Could someone please confirm this statement? Does this mean that I need to develop themes myself or buy WordPress templates? Thank you for your help.
Intermediate & Advanced SEO | | salvyy0 -
Robots.txt 404 problem
I've just set up a wordpress site with a hosting company who only allow you to install your wordpress site in http://www.myurl.com/folder as opposed to the root folder. I now have the problem that the robots.txt file only works in http://www.myurl./com/folder/robots.txt Of course google is looking for it at http://www.myurl.com/robots.txt and returning a 404 error. How can I get around this? Is there a way to tell google in webmaster tools to use a different path to locate it? I'm stumped?
Intermediate & Advanced SEO | | SamCUK0 -
Google WMT Change of address.
Hey all. I've got two domain names. -rt112media.com -route112media.com http://rt112media.com is my main address. The preferred domain. My Google WMT account has 4 sites listed that I am a verified owner on. 1.)rt112media.com 2.)www.rt112media.com 3.)route112media.com 4.)www.route112media.com I recent changed the address in WMT for route112media.com to have the preferred domain rt112media.com. I did this on Nov 23. It is currently saying their is a request and I can withdrawl. Now I'm needing to set the preferred for www.rt112media.com to be rt112media.com but it will not let me because of the pending route112media.com. The problem is I'm getting a ton of errors under my www.rt112media.com diagnostics. I'm not getting any under rt112media.com but I am thinking it is still effecting my page rank. I have the 301's for all set up as a 301 wildcard to rt112media.com. They all redirect to the preferred domain I wan (rt112media.com.) Suggestions?
Intermediate & Advanced SEO | | Route112Media0