Correct robots.txt for WordPress
-
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong?
I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right.
Thank you so much.
-
Me too, can you upload or screenshot the actual file that you are using
-
I have edited it down to
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /contact/ Disallow: /refer/ It didn’t help. I get a “Blocked by robots.txt” message after submitting the URL for indexing in google webmaster tools. I’m really puzzled.
-
Hi, in addition to the answer that effectdigital gave; another option,optimised for WordPress:
User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml -
Just seems overly complex and like there's way more in there than there needs to be
I'd go with something that 'just' does what you have stated that you want to achieve, and nothing else
User-Agent: *
Disallow: /wp-content/plugins/
Disallow: /comments
Disallow: /*?s=
Disallow: /*&s=
Disallow: /search
See if that helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Need Wordpress Front-end Plugin With Moz API
Hi Guys,
On-Page Optimization | | mrezair
I'm looking for Moz SEO Front-end Wordpress Plugin To Audit My Visitors Website And Show Results in my site. Like This plugin for Moz DA Checker: https://www.724ws.net/domain-authority-checker/ It's not important to be a free plugin or Premium one. I need to increase leads and traffic by it. Any suggestion will be appreciated.0 -
Google Webmaster Tools Not Showing Correct Data?
Hi, I am experience a weird issue. Google webmaster tools suggested me some HTML improvements a few weeks ago. The suggestions were about duplicate Title Tags and Short Meta Descriptions. I changed the Title Tags and Meta Descriptions. But after 3 Google Updates, webmaster still shows the same suggestion. Please advise Thanks
On-Page Optimization | | Kashif-Amin0 -
Wordpress 503 errors
We've launched a new website and i'm see a lot of 503 errors when i run Screaming Frog Seo Spider. Does anyone know how to solve this problem? wwww.besteluieraanbiedingen.nl Thanks!
On-Page Optimization | | Happy-SEO1 -
Title Tags not Displaying in SERPs Correctly
We have two websites that are very similar. One is .co.nz and the other is a .com.au. I've noticed some of our page titles are displaying differently in SERPs even though the title tags are formatted the same:e.g. The title: "PC Games | at Mighty Ape Australia" shows as: "PC Games | at Mighty Ape Australia - MightyApe.com.au"It's showing with extra text "- MightyApe.com.au". Whereas the NZ result for the title "PC Games | at Mighty Ape NZ" is fine i.e. displays as: "PC Games | at Mighty Ape NZ"Does anyone who why this might be happening?P.S. It's odd that in the Australian results the site links don't show up even though the sites are structured the same.
On-Page Optimization | | kevinliao0 -
Correct .htaccess settings for canonical url?
I want to forward all urls to http:www.mysite.com but am a little confuse because I am getting duplicate content error: Pages with Duplicate Page Content as of Jan 15http://titanappliancerepair.com/ 1 duplicatehttp://titanappliancerepair.com 1 duplicatehttp://titanappliancerepair.com/index.html 1 duplicate*****************************************************************What should I put ion htaccess file so I can forwardhttp://titanappliancerepair.com/index.htmlhttp://titanappliancerepair.comhttp://titanappliancerepair.com/to http://www.titanappliancerepair.comor what is the correct way to do it?I'm confused because when I enter http://titanappliancerepair.com/ in browser it showshttp://titanappliancerepair.com so how can it be considered duplicate content?.Can someone help?I have godaddy and they have gave me this code to put RewriteEngine on
On-Page Optimization | | webbutler13
rewritecond %{http_host} ^coolexample.com [nc]
rewriterule ^(.)$ http://www.coolexample.com/$1 [r=301,nc]What is correct?0 -
WordPress - duplicate content
I'm using WordPress for my website. However, whenever I use the post section for news, I get a report back from SEOmoz saying that there's duplicate content. What it does is it posts them in the Category and Archive section. Does anyone know if Google sees this as duplicate content and if so how to stop it? Thanks
On-Page Optimization | | AAttias0 -
Advantages of wordpress over dreamweaver
I have a site that I have built with dreamweaver (local service business not a lot of new content needing to be added) and I am curious how much of an advantage for me converting it to wordpresss would be. I see that cs5 works with wordpress and that lynda.com has a tutorial on integrating the two. I would like to integrate a blog into my site, but it is not totally critical that I do so. I would just like to hear what some of you experienced users have to say about the advantages of wordpress vs dreamweaver from an SEO prospective.
On-Page Optimization | | ayetti0 -
Optimizing Wordpress Categories
I’m trying to optimize the category links in the wordpress blog http://blog.reputationforward.com. I’m in the process of installing the Yeost SEO plugin and am going to revise the Category Title Tags, Meta Descriptions and slugs. I’m also in the process of using Tags so that I can get the posts that appear in a category to be specific to that category (currently the publisher is listing posts in multiple categories: See “Business Management Category”). But other than that, is there anything else I can do to optimize each category?
On-Page Optimization | | EricVallee341