Default Robots.txt in WordPress - Should i change it??
-
I have a WordPress site as using theme Genesis i am using default robots.txt. that has a line Allow: /wp-admin/admin-ajax.php, is it okay or any problem. Should i change it?
-
Yes, we're a news site as well and in our case we want to make sure the low quality pages on TNW aren't indexed.
-
Thank you both for your response.
@Martijn your robots.txt is really a nice example but for my new site is it good practice to block this areas??
@Peter To be a safe side I was using the same robots.txt...
-
In addition of Martijn here is mine robots.txt:
User-agent: *
Disallow:Sitemap: http://peter.nikolow.me/sitemap_index.xml
But using Yoast - categories, tags, most of archives and other generated pages are disabled for indexing.
-
Hi Peter,
Usually I would say it's not enough as the robots.txt is forgeting about excluding the search pages and in most cases you want to make sure the WP core files are not included + tag pages. Take a look at our robots.txt to see what we've included there: http://thenextweb.com/robots.txt then you'll notice we include for example these:
User-agent: *
Disallow: ?p=
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /wp-admin/*
Disallow: /wp-register.php
Disallow: /wp-content/themes/icetea/includes/*
Disallow: /tag/
Disallow: ?s=
Disallow: /search/*Other cases in our robots.txt are very specifically in there because of our site and may not apply to others.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt advice
Hey Guys, Have you ever seen coding like this in a robots.txt, I have never seen a noindex rule in a robots.txt file before - have you? user-agent: AhrefsBot User-agent: trovitBot
Intermediate & Advanced SEO | | eLab_London
User-agent: Nutch
User-agent: Baiduspider
Disallow: / User-agent: *
Disallow: /WebServices/
Disallow: /*?notfound=
Disallow: /?list=
Noindex: /?*list=
Noindex: /local/
Disallow: /local/
Noindex: /handle/
Disallow: /handle/
Noindex: /Handle/
Disallow: /Handle/
Noindex: /localsites/
Disallow: /localsites/
Noindex: /search/
Disallow: /search/
Noindex: /Search/
Disallow: /Search/
Disallow: ? I have never seen a noindex rule in a robots.txt file before - have you?
Any pointers?0 -
Robots.txt Disallowed Pages and Still Indexed
Alright, I am pretty sure I know the answer is "Nothing more I can do here." but I just wanted to double check. It relates to the robots.txt file and that pesky "A description for this result is not available because of this site's robots.txt". Typically people want the URL indexed and the normal Meta Description to be displayed but I don't want the link there at all. I purposefully am trying to robots that stuff outta there.
Intermediate & Advanced SEO | | DRSearchEngOpt
My question is, has anybody tried to get a page taken out of the Index and had this happen; URL still there but pesky robots.txt message for meta description? Were you able to get the URL to no longer show up or did you just live with this? Thanks folks, you are always great!0 -
URLs with parameters + canonicals + meta robots
Hi Moz community! I'm posting a new question here as I couldn't find specific answer to the case I'm facing. Along with canonical tags, we are implementing meta robots on our pages (e-commerce website with thousands of pages). Most of the cases have been covered but I still have one unanswered case: our products are linked from list pages (mostly categories) but they almost always include a tracking parameter (ie /my-product.html?ref=xxx) products urls are secured with a canonical tag (referring only to the clean url /my-product.html) but what would be the best solution regarding the meta robots? For now we opted for a meta robot 'noindex, follow' for non canonical urls (so the ones unfortunately linked from our category/list pages), but I'm afraid that it could hurt our SEO (apparently no juice is given from URLs with a noindex robots), and even maybe prevent bots from crawling our website properly ... Would it be best to have no meta robots at all on these product urls with parameters? (we obviously can't have 'index, follow' when the canonical ref points to another url!). Thanks for your help!
Intermediate & Advanced SEO | | JessicaZylberberg0 -
There is no change of address pending for your site
Hi everyone. Google has recently started using our old .net instead of .com in the SERPS. I went to do a change of address for the old http://www.sqlsentry.net and it gives me the "There is no change of address pending for your site." I've tried for both www and non-www and still get the same result. All pages seem to be redirecting to the new site with no issues. Is there anything else I can do to change this? Or is there a step I'm missing? Thanks!
Intermediate & Advanced SEO | | Sika220 -
SEO is changing - how has your day to day changed?
I'm sure we all read on our alternatives to Google Reader that SEO is changing - "here's what we must do to be relevant in 2014". I find these articles boring and uninformative. I suspect I'm not alone. The reason I'm not their biggest fan is because I feel like I've invested 10 minutes into an article that I have no actual guidance from. Therefore, I thought I'd ask the real SEO's, you guys, what has actually changed for you? Are you now not creating content with the aim of getting links? If you run a commercial website, what are you doing different to rank your product pages - directly or indirectly? Please share with the group. I'm sure many like me are still brainstorming and creating content they think will grab people's attention and gain them links, whilst also pushing their Facebook, Twitter, Youtube profiles, etc etc. What has changed about this?
Intermediate & Advanced SEO | | purpleindigo0 -
Hit By Penguin...Wait for recovery or do i change domains?
Hey Guys Would very much appreciate all opinions on our following situation, we have an .uk based ecommerce sports nutrition site www.cardiffsportsnutrition.co.uk Previously we worked with an SEO, that to put it simply did not follow webmaster guidelines(money anchor heavy, bad links etc), we reached some very good ranks too quickly and subsequently after the first penguin we where hit. We didn't receive any link warning or manual penalties just what i am assuming algorithmic...Rankings and traffic drop significantly, but not business ending. Since the first penguin we have done very little to no SEO, some unique content, re-writing of product pages, lots of social activity and didnt really lose much traffic after that, some small ups and down after refresh s and a slight slow decline on some keywords. Come Penguin 2.0...things that where still ranking for have now dropped even further, impressions in webmasters is now down over 50% and we have a had a wkly but not drastic drop in traffic since then. Over the last couple of months we have obtained some good quality links, have added lots of great unique content that has been shared significantly and generated some great traffic to our blog, added more unique product pages and category pages. But organically things are starting to look pretty grim apart from our brand keywords and everything is still in a slow decline and no increase in impressions in webmasters either jsut small drops We have been working to remove the poor quality and toxic links that the previous SEO built,getting anchor text corrected and collating information on the whole process ready to submit a file of links to disavow tool. which we are planning to do within the next couple of wks. Now i have read some successful stories and some not so successful one, so im starting to think of how to deal with this worse case scenario, If our domain is too damaged by the previous SEO guys We have the same domain name but on the .com that will help us carry over our brand name directly, but my concern is even though we have not had any manual penalty and not 301'd the .com back to the .co.uk or any other form of link will the penalties be carried over to the new domain just on the basis of brand association. We wouldn't plan to redirect any of the .co.uk traffic back to the .com but rather focus on our already strong ablate less converting traffic from the likes of twitter and facebook and run a small PPC campaign for some brand keyword to help buffer the traffic loss. While we focus on building good quality links and putting up plenty of new quality content on the site on the new domain that does not have any poor quality links back to it. What im trying to avoid is carry on spending time money and effort on the .co.uk domain for the next 3/4 months and continue to lose traffic slowly and then have to switch the domain anyway. I plan to wait and see for the next 4-6wks after we run the dissavow to but October time would be time i would have to make a decision and go for it. Any advice or opinions would be appreciated marc
Intermediate & Advanced SEO | | CSN0 -
Should I block wordpress archive and tag?
I use Wodpress and Wordpress SEO by Yoast. I've set ip up to add noindex meta tag on all archive and tag pages. I don't think its useful to include thoses pages in search results because there's quite a few. Especialy the tag archive. Should I consider anything else or change my mind? What do you think? Thanks
Intermediate & Advanced SEO | | Akeif0 -
Changing Servers + Effect on SEO
Hi, I am currently with a very slow server. Our website takes quite a while to load, FTP is very slow and content changes with Wordpress are slow because even the database connection takes a lot of time. However, my website ranks very well. Traffic has doubled in the last year. Our domain has been registered with this company for over 10 years. I am wondering if changing to a different hosting provider would have an effect on my rankings due to the change in IP.
Intermediate & Advanced SEO | | MangoMan160