Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pros and Cons of using rel=next on blog posts
Hi there, at the bottom of my website's blog posts the rel=prev and rel=next tags are used on links that point to the previous article that was posted and the next article that was posted. Often these articles are not 'linked' in terms of their content or message. Is this the correct use of rel=next/prev and if not what are the possible negative effects. Many thanks.
Technical SEO | | Bee1590 -
Wordpress BackupBuddy adding ?doing_wp_cron= in URLS
Hi Has anyone found WordPress Backup Buddy causing a problem with SEO. I understand why it does it, but wondered if anyone experienced issues with this? Only sometimes it adds /?doing_wp_cron=****** on to the end of a URL Thanks Tom
Technical SEO | | TomPryor831 -
Wordpress: Should your blog posts be noindex?
Wordpress defaults all blog posts to no index/nofollow Is this how it should be handled? I understand the nofollow from the page.com/blog to the page.com/blog/blogtitle But why noindex? We have Yoast installed and this is the default.
Technical SEO | | cschwartzel0 -
Duplicate Title Tags On Blogs
Hello Mozzers, I have a client who has a blog and articles section on their website, due to the amount of content they have in these section they have multiples pages so it would be like.. www.bluewidgets.com/blog/p1 www.bluewidgets.com/blog/p2 www.bluewidgets.com/blog/p3 www.bluewidgets.com/blog/p4 All obvioulsy with seperate content in each blog/article posts but the meta descriptions on title tags for this p1,p2,p3,p4 and so on are all the same, is this a major issue? and if so how do I fix it?
Technical SEO | | Prestige-SEO0 -
Redirect blogspot blog ?
Hi We have an ecom site selling childrens clothing. We startet up with a blogspot blog due to the lack of that option from our provider. I have a few links in to the blogspot blog and realized today that the PA is 39 - the PA of homepage of ecom site is 40. Ive lately been using the ..../blog on my ecom site. The functionality so far of it is so bad that i am considering using the blogspot blog again. How can i best utilize the strength of our blog at blogspot. Just keep posting w anchor text to relevant pages our main site? Is redirecting a possibility at all ? Dan Lærum
Technical SEO | | danlae0 -
Related post script for non wordpress sites
Guys am looking for a related posts script or tool that can read my site map or and post related articles under each of my articles. There are plugins like yarrp, linkwithin but they are for wordpress. I need something i can you use with a normal html website.
Technical SEO | | Emeka0 -
Which do you believe & why? Wordpress posts or pages?
I hear a lot of conflicting opinions regarding when to use pages over posts. If you were using wordpress mainly as a static site and less of a blog would you use pages over posts? Let's say it is a yacht review site. Would you set up most of the content (boat review pages) as pages or posts? I hear some say in that case you would want to use mostly pages and use posts for news related items. Then there are others that suggest only making the few (contact, about, privacy, terms) as pages and everything else created as posts organized into categories. Any thoughts? Also, is there a tool to find out if a website that is not yours is using a page or a post? Thanks
Technical SEO | | PEnterprises0 -
Which is best blogging platform from SEO POV?
I am curious to find out what is the blogging platform of choice for enterprize level companies (employees more than 500, revenue more tan 150M). What would be the best solution from SEO point of view? I have used Wordpress in the past for small companies and feel that is the best. We are currently using Telligent. Is anybody using it?
Technical SEO | | Amjath0