Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Canonicalize or Block?
Hi Mozers, We have staff profile pages w/ one main URL and then URLs with query parameters and jump links to take you to different parts of the page. The longer URLs with parameters canonicalize to the main pages but should they also be nonidexed? Thanks, Yael
Technical SEO | | yaelslater0 -
Non Published Wordpress Pages
Hi, Is there any negative SEO consequences from having too many pages private or not published. Can it like slow the site down or does it not matter? Someone in my dept. has so many pages started/not complete and besides being messy, I wonder if it has any negative impact on the site. Thanks
Technical SEO | | aua1 -
Blogger to Wordpress 301 and Meta Refresher Redirect
Hi Everyone! So my client has a blogger that she has developed a good amount of link equity for. It is a hersite.blogspot.com (she doesn't own her own domain yet). She is moving to the Wordpress platform though and the only way we can do a redirect is through a meta refresh redirect (since she doesn't have access to the servers on blogger). I went to Google Webmasters to do a change of address and found that the 301 checker said it couldn't find any 301 redirect, which is disappointing. What we're planning is telling all the places that link to the blog to change their links to the new blog but other than that what does anyone recommend to keep this link strength? Thanks!
Technical SEO | | mattdinbrooklyn0 -
Updating old blog posts in Wordpress to appear more recent?
I'm doing work for a law firm that has a lot of blog post content from 2010-2011 ranking for long-term keywords. These pages are displaying date snippets in SERPs, but because legal information can change year to year, I don't want the content to appear as though it's 2-3 years old. The date of the post is in the URL structure, so I can't change the publication date w/o changing the URL. So my question is twofold: is there a way to show an updated date snippet in search results, or block the date snippet from showing, even if the date is in the URL? Or are there other options - creating pages for each of these posts and 301ing them to the page that has a cleaner URL, etc.? Thanks in advance for your help!
Technical SEO | | dchristensen30 -
Wordpress Pods and Wordpress SEO by Yoast
Hi I am optimising a new site that has been built in Wordpress using Pods. The Yoast Wordpress SEO plugin is not recognising any content on the site - has anyone any ideas on how to get around this - does it matter - is it the plugin that is at fault rahter than the set up of the site?
Technical SEO | | Highlandgael1 -
Wordpress and Redirects?
I want to update my permalinks - actually I want to change the URL's to fit the content and keywords better. I can choose "edit" the URL, but don't I need a redirect? I don't see any htaccess Plugin installed.......is that what I need to be able to change my URL's in Wordpress?
Technical SEO | | cschwartzel0 -
302 Redirects & WordPress
Hi Mozzers- We started a WordPress blog for one of my clients to help improve the domain authority, increase visits, etc. a few months ago. I ran a crawl report a few days ago and noticed that the /trackback/ pages are getting redirected to the appropriate page using a 302 redirect not a 301 redirect. I've search through the forum looking for other similar questions and found one. The answer suggested that I use the TAC plugin and check for any malicious code. I did this and found nothing. Another response suggested using the Redirection plugin. I haven't tried this yet. Has anyone encountered this and if so, how did you deal with it. Thank you! LHC
Technical SEO | | lhc670 -
Ping Services for Wordpress
I have been trying to research adding ping services to the default list in wordpress, which only includes http://rpc.pingomatic.com/ A lot of the articles that I have come across seem to be quite old, can anyone tell me if this is still relevant and if so, which services do you find to be worthwhile using? Thanks
Technical SEO | | pugh0