Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blog not ranking for my name
Hi everyone. I'm new here so apologies if I'm not asking an appropriate question - just let me know! Can anyone help me figure out why my blog (https://www.jamescrowley.net/) isn't ranking at all for my name? I've run it through the standard Moz audit tools and it hasn't picked up any major issues. It ranks fine for my name plus " CTO", but doesn't appear anywhere in the top 50 without that qualifier. I realise there are many other 'James Crowley's to compete with but weirdly even my GitHub profile page appears to rank higher (https://github.com/jamescrowley) I moved the domain a while back (18 months+) and I used to rank highly, but it never seems to have recovered (all the standard redirects are in place, and told Google at the time about the move). Any suggestions would be very much appreciated!
Technical SEO | | james.crowley0 -
Referencing links in Articles and Blogs
Hi I am wondering if the <sup>tag in html is picked up by google as a reference point?</sup> I.e when you put a superscript in word it puts a small number next to your sentence. Then you have a list of reference at the end of the blog/article does google recognise this?
Technical SEO | | Cocoonfxmedia0 -
Home Page Blog Snippets - Duplicate Content Help?
Afternoon Folks- I have been asked to contribute to a new site that has a blogfeed prominently displayed on the home page. It's laid out like this: Logo | Menu HOME PAGE SLIDER Blog 1 Title about 100 words of blog 1 Text Blog 2 Title about 100 words of blog 2 Text Blog 3 Title about 100 words of blog 3 Text Footer: -- This seems like an obvious duplicate content situation but also a way I have seen a lot of blogs laid out. (I.E. With blog content snippets being a significant portion of the home page content) I want the blogs to rank and I want the home page to rank, so I don't feel like a rel canonical on the blog post's is the correct option unless I have misunderstood their purpose. Anyone have any ideas or know how this is usually handled?
Technical SEO | | CRO_first0 -
Ajax Crawling | Blocked URLs Spike
http://www.zando.co.za/women/shoes/ (for example) Hello, I'm concerned that WMT is reporting a large spike in blocked URLs - now reporting more blocked URLs than good URLs. Our product recommendations get generated via an Ajax call and these autogenerated, unique, URLs are rendered in the /recommendations/ folder which sits in the root of our site: http://www.zando.co.za/recommendations/ I can't see how I can prevent Google from calling the Ajax - I can only assume that's what's happening.This is what the code typically looks like:
Technical SEO | | RocketZando0 -
Wordpress site, combine Blog without hurting SEO - Need Expert Advice
Hi, I come from the old html days of Frontpage and then moved to Dreamweaver. I first worked with Wordpress at version 2.7 and was not all that impressed, but then recently I worked in the new version and was extremely impressed. So my knowledge of Wordpress is VERY limited and plan to build future sites with it. I need to know the best way to solve an issue for a customer. The client is http://www.nextgenrestoration.com/ Site was built years ago with Frontpage. The popularity of Blogs was hot so someone told them that if they add new content it would be better to use a blog, so they added a blog. So you have the following: www.nextgenrestoration.com (main site) then they installed wordpress in a folder (blog) www.nextgenrestoration.com/blog Original person that built the site quit. New person took over and said the main site needed to changed to Wordpress because they did not have Frontpage and all they knew was Wordpress. Main site was converted to Wordpress. They wanted to keep the original design so they did not use a stock template, they just built it with their design. I guess from looking at the Editor, they manually went in and put the design in to match. Now.. this last month, the person that had changed
Technical SEO | | Force7
the site to Wordpress quit. So I got involved because the new person they hired could not add content to the main website. If you add a page, it does not show up, you have to manually go in the php and add the link to the category. The new person knows how to use Wordpress but she knows nothing about PHP so is lost when it comes to manually adding content to the site. Here was my Thoughts. The main site needs to be rebuilt in a stock template so it automatically creates new pages, blog posts. I have to make sure that if we change the
main website that we could keep all the same links and page names. The girl
that built the site, if you hover over the links that she put it under ‘florida’,
that must be a category. But we would need to keep the same page names. I know
we could do a 301 redirect but this guy cannot lose traffic. He is already down
in hits after the last Panda update. My thought was, rebuild the main site in a stock template so
someone can actually add content easily to the site. Also build a new blog
section so it all matches. (personally the existing design looks old and dated and needs updating) If you look at the site now. The blog looks totally
different and it is not helping if a customer comes to the blog but cannot see
the navigation for the whole site. My thought was to just leave the old blog, it has a LOT of backlinks. But just add a new blog to the main site and all new content goes there. The old blog would stay just make sure we did build in some call to action so it sends them to the main site. Also, we found we cannot create a Blog on the
wordpress we have installed in the main directory. I am guessing because it
wants to name it /blog? I want to be sure we give this client the best advice on what to do without
hurting his existing seo and traffic. As you can tell, I am not qualified to really give the best advice since I am so new to Wordpress. This is a small company that really needs some help. Thanks in advance for your time! Force70 -
Wordpress Category Archives
Wordpress question here. Can anyone tell me if there is an SEO advantage to creating a page filtered to show results from an individual category as opposed to simply linking to the category archive? The content is identical in both cases.
Technical SEO | | waynekolenchuk0 -
How to block google robots from a subdomain
I have a subdomain that lets me preview the changes I put on my site. The live site URL is www.site.com, working preview version is www.site.edit.com The contents on both are almost identical I want to block the preview version (www.site.edit.com) from Google Robots, so that they don't penalize me for duplicated content. Is it the right way to do it: User-Agent: * Disallow: .edit.com/*
Technical SEO | | Alexey_mindvalley0 -
How to Redirect all inactive Feed to a specific Wordpress page
Hi Guys, I've been doing much cleaning on my blog lately and deleted numerous categories including their posts with low quality content. After deleting the categories, Google Webmaster Tools is reporting some 404 errors about the RSS Feeds for the deleted categories. I've created a 404.php file inside my theme and placed the following code header("HTTP/1.1 301 Moved Permanently");
Technical SEO | | Trigun
header("Location: http://www.mysite.com/My404Page/", true, 301);
exit();
?> this have catched all 404 errors and redirected them to the specific page. Unfortunately, it could not catch the inactive feed urls. Is there a way to do this so that all inactive feeds will be redirected to my 404 page? Thanks in advance....0