Correct robots.txt for WordPress
-
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong?
I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right.
Thank you so much.
-
Me too, can you upload or screenshot the actual file that you are using
-
I have edited it down to
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /contact/ Disallow: /refer/ It didn’t help. I get a “Blocked by robots.txt” message after submitting the URL for indexing in google webmaster tools. I’m really puzzled.
-
Hi, in addition to the answer that effectdigital gave; another option,optimised for WordPress:
User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml -
Just seems overly complex and like there's way more in there than there needs to be
I'd go with something that 'just' does what you have stated that you want to achieve, and nothing else
User-Agent: *
Disallow: /wp-content/plugins/
Disallow: /comments
Disallow: /*?s=
Disallow: /*&s=
Disallow: /search
See if that helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Meta title not showing up correctly on SERP
We have an issue with some clients on a Wordpress CMS where title tags implemented on a Yoast SEO plugin for the Homepage are not coming up as we'd implemented them. However, the source code shows we'd implemented them correctly according to what we'd wanted.**For example, this is the title tag we implemented in the CMS:Towing Services Alberta | Jack's TowingSource code shows:Towing Services Alberta | Jack's TowingHowever, SERP results shows:**Jack's Towing | Towing Services Alberta This is not an issue with the rest of the other pages and there isn't a global template for our client's sites. It's perplexing that it's only happening on the Homepage and this is across for 3 of our clients' sites.Even more perplexing, recently we've noticed that SERP is only showing the client's business name as the title tag and this is across for 2 out of the 3 clients we'd mentioned above. Nothing has changed in the back-end.Would appreciate some insight on this issue!
On-Page Optimization | | Gavo4 -
Two Robots.txt files
Hi there Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one. Thanks in advance! Cheers,
On-Page Optimization | | TrulyTravel
Satla0 -
Google Webmaster Tools Not Showing Correct Data?
Hi, I am experience a weird issue. Google webmaster tools suggested me some HTML improvements a few weeks ago. The suggestions were about duplicate Title Tags and Short Meta Descriptions. I changed the Title Tags and Meta Descriptions. But after 3 Google Updates, webmaster still shows the same suggestion. Please advise Thanks
On-Page Optimization | | Kashif-Amin0 -
Best way to separate blogs, media coverage, and press releases on WordPress?
I'm curious what some of your thoughts are on the best way to handle the separation of blog posts, from press releases stories, from media coverage. With 1 WordPress installation, we're obviously utilizing the Posts for these types of content. It seems obvious to put press releases into a "press release" category and media coverage into a "media coverage" category.... but then what about blog posts? We could put blog posts into a "blog" category, but I hate that. And what about actual blog categories? I tried making sub-categories for the blog category which seemed like it was going to work, until the breadcrumbs looked all crazy. Example: Homepage > Blog > Blog > Sub-Category Homepage = http://www.example.com First 'Blog' = http://www.example.com/blog Second 'Blog' = http://www.example.com/category/blog Sub-Category = http://www.example.com/category/blog/sub-category This just doesn't seem very clean and I feel like there has to be a better solution to this. What about post types? I've never really worked with them. Is that the solution to my woes? All suggestions are welcome! EDIT: I should add that we would like the URL to contain /blog/ for blog posts /media-coverage/ for media coverage, and /press-releases/ for press releases. For blog posts, we don't want the sub-category to be in the URL.
On-Page Optimization | | Philip-DiPatrizio0 -
How Do I Get Rel=Author to Work Via Wordpress SEO by Yoast?
I have read Yoast's tutorial and others on how to get rel=author to work properly on my site (http://injuryattorneyshouston.com). Here are the results I get from the Google Structured Data Testing Tool: "Authorship is working for this webpage." "rel=author markup has successfully established authorship for this webpage." "Publisher markup is verified for this page." All seems well except for this: hatom-feed | hatom-entry: | |
On-Page Optimization | | cbizzle
| Error: At least one field must be set for HatomEntry.Error: Missing required field "entry-title".Error: Missing required field "updated".Error: Missing required hCard "author". |
| Error: At least one field must be set for HatomEntry.Error: Missing required field "entry-title".Error: Missing required field "updated".Error: Missing required hCard "author". | Does anyone know how to fix this?0 -
Keywords vs Tags In Wordpress
Whats the difference between Keywords vs Tags In Wordpress? Also sine meta keywords don't matter in a website do they matter on a blog?
On-Page Optimization | | splashmedia0 -
Problem with Plugin: Wordpress SEO by Yoast
Hi Hi, I'm using the Plugin "Wordpress SEO by Yoast". Unfortunatelly, this plugin generates a sourcecode looking like this: name='description' content='XXXXXX.'/> Seomoz can't read this and tells me, that my pages have no description. Google doesn't matter and reads it correct. Is there anything I can do about this? Many greetings André
On-Page Optimization | | waynestock0 -
Wordpress when to use posts or pages
Hi Guys, I have a network of EMD sites that currently use a homepage and then we have a blog page which has 5-6 posts on. Is this the best way to do it with sites under 10-20 pages? Or should we create say 3-4 new pages/categories and drop the posts relevant to each page/category in there? Thank you Jon
On-Page Optimization | | imrubbish0