Default Robots.txt in WordPress - Should i change it??
-
I have a WordPress site as using theme Genesis i am using default robots.txt. that has a line Allow: /wp-admin/admin-ajax.php, is it okay or any problem. Should i change it?
-
Yes, we're a news site as well and in our case we want to make sure the low quality pages on TNW aren't indexed.
-
Thank you both for your response.
@Martijn your robots.txt is really a nice example but for my new site is it good practice to block this areas??
@Peter To be a safe side I was using the same robots.txt...
-
In addition of Martijn here is mine robots.txt:
User-agent: *
Disallow:Sitemap: http://peter.nikolow.me/sitemap_index.xml
But using Yoast - categories, tags, most of archives and other generated pages are disabled for indexing.
-
Hi Peter,
Usually I would say it's not enough as the robots.txt is forgeting about excluding the search pages and in most cases you want to make sure the WP core files are not included + tag pages. Take a look at our robots.txt to see what we've included there: http://thenextweb.com/robots.txt then you'll notice we include for example these:
User-agent: *
Disallow: ?p=
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /wp-admin/*
Disallow: /wp-register.php
Disallow: /wp-content/themes/icetea/includes/*
Disallow: /tag/
Disallow: ?s=
Disallow: /search/*Other cases in our robots.txt are very specifically in there because of our site and may not apply to others.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
WordPress posts Title field inserts title into blog posts like a headline but doesn't ad H1 tag how to change?
I have a Wordpress website which is just using the Default theme, when I post in the blog, whatever I put in the "Title" field at the top of the editor is automatically is placed within the body of the blog post, like a headline, but it doesn't include any H1 tags that I can see. If I add my own headline within in the blog editor, it still inserts the Title like a headline. I am using the Yoast SEO Plugin and also write the meta title there, should I just leave the Wordpress title field blank so it doesn't insert into the blog post? Or is that inserted Title being recognized as an H1 even though I don't see h1 tags anywhere? Hope this isn't too confusing.
Intermediate & Advanced SEO | | SEO4leagalPA1 -
Pages getting into Google Index, blocked by Robots.txt??
Hi all, So yesterday we set up to Remove URL's that got into the Google index that were not supposed to be there, due to faceted navigation... We searched for the URL's by using this in Google Search.
Intermediate & Advanced SEO | | bjs2010
site:www.sekretza.com inurl:price=
site:www.sekretza.com inurl:artists= So it brings up a list of "duplicate" pages, and they have the usual: "A description for this result is not available because of this site's robots.txt – learn more." So we removed them all, and google removed them all, every single one. This morning I do a check, and I find that more are creeping in - If i take one of the suspecting dupes to the Robots.txt tester, Google tells me it's Blocked. - and yet it's appearing in their index?? I'm confused as to why a path that is blocked is able to get into the index?? I'm thinking of lifting the Robots block so that Google can see that these pages also have a Meta NOINDEX,FOLLOW tag on - but surely that will waste my crawl budget on unnecessary pages? Any ideas? thanks.0 -
Robots.txt vs noindex
I recently started working on a site that has thousands of member pages that are currently robots.txt'd out. Most pages of the site have 1 to 6 links to these member pages, accumulating into what I regard as something of link juice cul-d-sac. The pages themselves have little to no unique content or other relevant search play and for other reasons still want them kept out of search. Wouldn't it be better to "noindex, follow" these pages and remove the robots.txt block from this url type? At least that way Google could crawl these pages and pass the link juice on to still other pages vs flushing it into a black hole. BTW, the site is currently dealing with a hit from Panda 4.0 last month. Thanks! Best... Darcy
Intermediate & Advanced SEO | | 945010 -
Changing a subdomain to a full domain to rank for a keyword
We have been attempting to get our blogsite to rank for our business name(Instabill). We are now considering changing the url from blog.instabill.com to something like instabillblog.com. I have following concerns about the change; Will changing the domain really be that helpful (i.e. will the change get our blog on page one for the term instabill) We have over 350 pages of content on our blog. Will changing the domain have possible negative effects ( I was thinking of using url updater in webmaster tools and creating a permanent 301 redirect from the older url to the new) Having never changed a url for a site with this much content and seo value for my company I would like to know the following from someone who has made mistakes here before; what not to do what steps you would take to make the transition easier Any help here will be greatly appreciated. cheers, Instabill
Intermediate & Advanced SEO | | Instabill0 -
Should I change Permalinks?
I have a site that is doing well in rankings, and its also been accepted in to Google news. I have around 800 article live on the site. My current permalink structure is - /%year%/%monthnum%/%postname%/ What I am wondering, would it be better to change it to - /%postname%/ The reason I ask is that most of the well ranking sites and sites built by SEO companies all seem to have the /%postname%/ Will I get a benefit from changing this? I know that I could republish content have the /%postname%/ permalinks. Are there any other benefits? If the is a benefit, I'd perefr to change it as soon as possible before the site grows even more Thanks in advance for anyone who can help
Intermediate & Advanced SEO | | JohnPeters0 -
Should I remove the ?replytocom variables in wordpress?
I'm using Yoast's wordpress plugin and there is an option to remove the replytocom variables. I'm curious what everyone's thoughts were on that, and if I should do it. Here's the site if you need to see it. Thanks!
Intermediate & Advanced SEO | | NoahsDad0 -
How To Create Dynamic WordPress Tags
Does anyone know how to make WordPress "tag" pages automatically generate a description based on the posts included in the tag? I have a lot of tags, and most of them rank well for long tail keywords. However I have noticed that although they have a dynamically generated "title meta tag" they do not generate a "description meta tag". I know WordPress lets you customize the description for each tag, but I have way to many for that. I need the description meta to be auto generated from the posts that are being tagged, rather than not including one at all. Does anyone know how to do this?
Intermediate & Advanced SEO | | MyNet0 -
Robots.txt & url removal vs. noindex, follow?
When de-indexing pages from google, what are the pros & cons of each of the below two options: robots.txt & requesting url removal from google webmasters Use the noindex, follow meta tag on all doctor profile pages Keep the URLs in the Sitemap file so that Google will recrawl them and find the noindex meta tag make sure that they're not disallowed by the robots.txt file
Intermediate & Advanced SEO | | nicole.healthline0