Default Robots.txt in WordPress - Should i change it??
-
I have a WordPress site as using theme Genesis i am using default robots.txt. that has a line Allow: /wp-admin/admin-ajax.php, is it okay or any problem. Should i change it?
-
Yes, we're a news site as well and in our case we want to make sure the low quality pages on TNW aren't indexed.
-
Thank you both for your response.
@Martijn your robots.txt is really a nice example but for my new site is it good practice to block this areas??
@Peter To be a safe side I was using the same robots.txt...
-
In addition of Martijn here is mine robots.txt:
User-agent: *
Disallow:Sitemap: http://peter.nikolow.me/sitemap_index.xml
But using Yoast - categories, tags, most of archives and other generated pages are disabled for indexing.
-
Hi Peter,
Usually I would say it's not enough as the robots.txt is forgeting about excluding the search pages and in most cases you want to make sure the WP core files are not included + tag pages. Take a look at our robots.txt to see what we've included there: http://thenextweb.com/robots.txt then you'll notice we include for example these:
User-agent: *
Disallow: ?p=
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /wp-admin/*
Disallow: /wp-register.php
Disallow: /wp-content/themes/icetea/includes/*
Disallow: /tag/
Disallow: ?s=
Disallow: /search/*Other cases in our robots.txt are very specifically in there because of our site and may not apply to others.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there any benefit to changing 303 redirects to 301?
A year ago I moved my marketplace website from http to https. I implemented some design changes at the same time, and saw a huge drop in traffic that we have not recovered from. I've been searching for reasons for the organic traffic decline and have noticed that the redirects from http to https URLs are 303 redirects. There's little information available about 303 redirects but most articles say they don't pass link juice. Is it worth changing them to 301 redirects now? Are there risks in making such a change a year later, and is it likely to have any benefits for rankings?
Intermediate & Advanced SEO | | MAdeit0 -
Will changing category URLs on site hurt SEO?
Hi Moz Community, We're looking to replace some URLs on our Wordpress site and I want to make sure we won't hurt our SEO with the changes. The site is lushpalm.com When we originally launched our site we created pages (which are linked to in our main menu) to essentially display our categories. We did this as a workaround because we didn’t like the URL to have the word “category” in it. Now we would like to make some changes and we want to make sure we’re not going to hurt our SEO in any way by accidentally duplicating content or otherwise. We want to fix our structure and now link to our category pages from our main menu, BUT we want to change the URL of the category page so that it doesn’t have “category” in it, essentially renaming it the name of the page currently linked to in our main menu. So basically, the category lushpalm.com/category/surf-trips, would be renamed with the URL lushpalm.com/surf-trips and the current page that is at lushpalm.com/surf-trips would be therefore replaced. My questions are: If we did this, would that mean that the previous “lushpalm.com/category/surf-trips” would cease to exist? Or is there some imprint of that out on the web? And if it is then would it re-direct to the new page? Would replacing the current page URL with a category hurt our current SEO in any way? Would this change cause any duplicate pages somehow? Thanks so much for your help!
Intermediate & Advanced SEO | | TaraLP1 -
Best practice for disallowing URLS with Robots.txt
Hi Everybody, We are currently trying to tidy up the crawling errors which are appearing when we crawl the site. On first viewing, we were very worried to say the least:17000+. But after looking closer at the report, we found the majority of these errors were being caused by bad URLs featuring: Currency - For example: "directory/currency/switch/currency/GBP/uenc/aHR0cDovL2NlbnR1cnlzYWZldHkuY29tL3dvcmt3ZWFyP3ByaWNlPTUwLSZzdGFuZGFyZHM9NzEx/" Color - For example: ?color=91 Price - For example: "?price=650-700" Order - For example: ?dir=desc&order=most_popular Page - For example: "?p=1&standards=704" Login - For example: "customer/account/login/referer/aHR0cDovL2NlbnR1cnlzYWZldHkuY29tL2NhdGFsb2cvcHJvZHVjdC92aWV3L2lkLzQ1ODczLyNyZXZpZXctZm9ybQ,,/" My question now is as a novice of working with Robots.txt, what would be the best practice for disallowing URLs featuring these from being crawled? Any advice would be appreciated!
Intermediate & Advanced SEO | | centurysafety0 -
Please suggest! Overstuffed URLs - to change, or not to change?
Hello guys, We are in doubts about changing URLs on our http://www.fiberscope.net/ store The site was hit by Google in January and has dropped from top 3 to top 20 for most of its keywords. Now we're working on some redesign, including removing of over optimized content from it. It looks like our categories and content pages URLs are overstuffed with keywords. Here is a dilemma. Should we rewrite them and set up 301 redirect or just leave as is? I look trough the seomoz Q&A and find out that a lot of guys got in troubles after redirecting PLEASE, Suggest!
Intermediate & Advanced SEO | | Meditinc.com0 -
Wordpress Photography Site + eCommerce Plugin
Anyone know of good photography sites set up on Wordpress with an eCommerce plugin used for selling photos or services? Just looking for ideas. I've found good referrals in the Moz archives for Wordpress eCommerce plugins. Now I'm looking for WP photography sites employing eCommerce. Thanks for sharing.
Intermediate & Advanced SEO | | AWCthreads0 -
Avoiding keyword cannibalisation in Wordpress Structure
Hi all, I've been planning the WordPress structure for a client for a long time, and soon need to get started. I keep coming back to this problem / question, whereby there will be a static Page with a name like "Christmas Collection", but there also has to be a Category called "Christmas" to accomodate new content. Is this likely to cause serious keyword cannabalisation? It's a non-ecommerce site To elaborate we need to have the following static Pages: christmas collection easter collection halloween Collection and each of these Page's will be populated with related Posts (based on category ID) after the Page's content. And these corresponding Categories need to be similarly named. As there's no apparent relation between Pages and Categories, I keep coming back to the concern that having a Page named "Christmas Collection" and a Category called "Christmas" is a big SEO NO NO. Any thoughts, opinions, workarounds much appreciated! Regards, c
Intermediate & Advanced SEO | | councilflat0 -
Wordpress site architecture conflicts and how to go about fixing them
I am attempting to figure something out with a site I'm trying to fix. So the problem is that I've got two categories that are basically related keywords. I set this up when I first started doing this work and didn't know what I was doing. So that site at one time was ranking on the first page for a specific term (example: 'project manager salary' and posted in the category 'project manager salary'. But they we added 'project manager salary in Vermont' and all other 50 posts for all states in a different category called, 'project manager salaries and benefits'. So my question is this: Would this cause some kind of keyword rank cannibalization? How do I fix this properly? Thanks! Michael
Intermediate & Advanced SEO | | mtking.us_gmail.com0 -
Robots.txt is blocking Wordpress Pages from Googlebot?
I have a robots.txt file on my server, which I did not develop, it was done by the web designer at the company before me. Then there is a word press plugin that generates a robots.txt file. How Do I unblock all the wordpress pages from googlebot?
Intermediate & Advanced SEO | | ENSO0