Correct robots.txt for WordPress
-
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong?
I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right.
Thank you so much.
-
Me too, can you upload or screenshot the actual file that you are using
-
I have edited it down to
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /contact/ Disallow: /refer/ It didn’t help. I get a “Blocked by robots.txt” message after submitting the URL for indexing in google webmaster tools. I’m really puzzled.
-
Hi, in addition to the answer that effectdigital gave; another option,optimised for WordPress:
User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml -
Just seems overly complex and like there's way more in there than there needs to be
I'd go with something that 'just' does what you have stated that you want to achieve, and nothing else
User-Agent: *
Disallow: /wp-content/plugins/
Disallow: /comments
Disallow: /*?s=
Disallow: /*&s=
Disallow: /search
See if that helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicated Content on Wordpress Mobile&Desktop versions- Is it bad for SEO?
Hello, I use a Wordpress theme for displaying mobile and desktop versions separately. The problem is that if you use tools like screaming frog or if you look at the code (view source), you can detect duplicated content. But if you're browsing on your mobile you will see only the content I have created for the mobile version and the same if you're looking at the desktop version, you will only see the desktop content.
On-Page Optimization | | AlphaRoadside
Is this creating an SEO problem? If it is, please let me know why and if it has a solution. Thanks in advance.0 -
Wordpress: Why do the URLs of my posts keep changing to match the posts titles?
I'll try to keep this brief. The URLs of my posts keep snapping back to exactly match their post titles, no matter how often I try to change them. e.g. title: How to Tie Your Shoelaces desired URL: tie-shoelaces BUT actual URL: how-to-tie-your-shoelaces Anyone come across/ resolved this issue before?
On-Page Optimization | | GerardAdlum0 -
No-index or 301 - custom wordpress archives
Hi, We are using custom wordpresss posts to showcase different services we provide, these are then grouped within custom archives. These custom archive pages are our main keyword landing pages and traffic generators (although ranking is fairly average). It's come to my attention that these custom posts have also been listed in the wordpress standard categories, with an identical title, excerpts and near identical urls as the custom archives. This appears to have been the case for quite a while. We are concerned that this is may causing duplicate content issues and unsure how to proceed. We have been advised to simply no-index the redundant 'standard' categories but as they have been indexed for some time we are cautious of causing any upset with search engines (although the categories are indexed they are not ranking for any major keywords) Are we best 301ing the redundant category to the custom archives or using a canonical tags or simply no-indexing the categories like other archive pages? Any advice is aprreciated Many thanks BC
On-Page Optimization | | benct0 -
Wordpress / Full URL In Menu Box
I came across an article online (not Moz) that says adding the full url in a menu is a Google standard. So when you make a menu link you put "www.example.com/page" instead of "/page". What are your thoughts on this? Any real reason to? Y840pbN lrwZPDj
On-Page Optimization | | InfinityTechnologySolutions0 -
Telling Google SERP's my correct currency.
Hi, I'm having a problem with Google SERP results showing my currency as USD, when it should be CAD. An example of a page with this problem is - http://www.absoluteautomation.ca/fgd400-sensaphone400-p/fgd400.htm - can anyone see where Google is getting USD from on there? I don't see it anywhere in the coding. Thanks in advance!
On-Page Optimization | | absoauto0 -
Help: my WordPress Blog generates too many onpage links and duplicate content
I have a WordPress Blog since November last year (so I'm pretty new to WordPress) and the effects on ranking for some keywords are really good. So I thought tag clouds are good. Crawl Diagnostics tell me now that I have too many onpage links for example my author page breaks the record: 256
On-Page Optimization | | inlinear
http://inlinear.com/blog/author/inlinear/ I think thats because there are links for each word in the tag cloud generated ... On this page (and many other pages) WordPress displays (teasers) the beginning of each post (read more ...) producing duplicate content and even new canonical tags.... The page titles are also too long because I installed "All in One SEO Pack" and now this plugin and wordpress itself mixes titles together ... But what can I do to avoid all this. Is there a PlugIn that can help... I think millions of blogs will have the same problems... I my blog yet has very few content. Thanks for your answers :))0 -
When it comes to duplicate page content how do I deal with correcting it. Its a dynamic e commerce site.
I am under the impression that with ecommerce sites this happens often and that there's a plug in or just simply not worry about it since queries will often find similar conent.
On-Page Optimization | | Wayne_c0 -
Best practice for Meta-Robots tag in categories and author pages?
For some of our site we use Wordpress, which we really like working with. The question I have is for the categories and authors pages (and similiar pages), i.e. the one looking: http://www.domain.com/authors/. Should you or should you not use follow, noindex for meta-robots? We have a lot of categories/tags/authors which generates a lot of pages. I'm a bit worried that google won't like this and leaning towards adding the follow, noindex. But the more I read about it, the more I see people disagree. What does the community of Seomoz think?
On-Page Optimization | | Lobtec0