Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My wordpress website is facing indexing problems
Hello, I am facing indexing issues on one of my which is about budget bushcraft knife, Four months have been passed I built my site and published almost 8 articles so far. But i am worried no single keyword ranked on google till yet as I checked through MOZ site explorer. Can anyone guide me on what should I need to do now? Thanks
Technical SEO | | Ewerurt0 -
No H1 tag in wordpress, what are the implications?
When checking my blog posts I noticed that they don't contain any H1 tags. This confused me as I'd heard that wordpress automatically puts H1 tags on your blog post titles. Is this something that might negatively affect my rankings? If an H1 tag is absent, does the H2 tag take center stage so to speak? Is this something my wordpress theme does and how can I fix it? Cheers! Anton
Technical SEO | | antonshakr0 -
Who uses WordPress Tags anymore?
Just curious if people are still using WordPress Tags. I wonder if with the direction Google is going the last couple years, having sites that get bloated with extraneous Tag archives just decreases the quality of the site.
Technical SEO | | WilliamBay2 -
Using category pages in Wordpress
In our niche we have one main keyword, which represents the entire category. We are using wordpress. I am trying to understand the best URL structure and wonder if the below is a good approach: http://domain.com/keyword This category page will be written to contain an article on the subject. The posts that are put into that category will subsequently appear on this page, below that article. Each of those posts would be targeting a related keyword. e.g. I would write a post which has, as the main target keyword: "MainKeyword training" and another post, which would be targeting "MainKeyword techniques" ... (and so on). Thanks for your advice. Andrew
Technical SEO | | seowhiskey0 -
Which domain should i set up a blog on?
I have a client who uses a .com for there website in Australia. Were now building an external blog which will be on a subdomain. We recently discovered they also own the Australian version of there domain name. Should we build there blog on: blog.currentdomain.com 2) blog.newdomain.com.au Thanks
Technical SEO | | acs1110 -
Wordpress multilanguage sitemaps
Hi, I have a multilingual wordpress site. which is in Bulgarian and English - translated using qtranslate. The xml sitemap of the 2 languages is in one sitemap file- all the links for the Bulgarian and English version are in one file. (Our web is using this plugin - http://wordpress.org/extend/plugins/google-xml-sitemaps-v3-for-qtranslate Do you have any idea how can I make separate xml sitemap for every language? I ask you here because may be you have identical problems with your multilanguage wordpress website. You can see the sitemap with 2 languages links in one sitemap here: http://cholakovit.com/ sitemap.xml Cholakov IT I have read from this article that it is better practise and also it will help with geo-targetazing your web site: http://www.seomoz.org/blog/multiple-xml-sitemaps-increased-indexation-and-traffic
Technical SEO | | vladokan0 -
Moving a blog from unique domain to root /blog/ but on 2 different servers? HELP!
I have a main site hosted on one server, I have the blog hosted on another server - BOTH of which my team has FULL control over. I ultimately want the blog to reside on the root domain: www.mysite.com/blog/ My network team is saying "DNS will not allow this to happen, the resolution will ultimately have to be on blog.website.com" Has anyone out there done this? Is it even possible? HELP!
Technical SEO | | BCA0 -
Ping Services for Wordpress
I have been trying to research adding ping services to the default list in wordpress, which only includes http://rpc.pingomatic.com/ A lot of the articles that I have come across seem to be quite old, can anyone tell me if this is still relevant and if so, which services do you find to be worthwhile using? Thanks
Technical SEO | | pugh0