Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We switched the domain from www.blog.domain.com to domain.com/blog.
We switched the domain from www.blog.domain.com to domain.com/blog. This was done with the purpose of gaining backlinks to our main website as well along with to our blog. This set us very low in organic traffic and not to mention, lost the backlinks. For anything, they are being redirected to 301 code. Kindly suggest changes to bring back all the traffic.
Technical SEO | | arun.negi0 -
Unexplainable 404 in my Wordpress Monitor
Hi Guys, We started a new website (far from complete) and i'm fixing some SEO problems. I have a SEO 404 monitor plugin and he gives me strange 404's that i can't find: https://compleetverkleed.nl/feestkleding/www.compleetverkleed.nl/feestkleding
Technical SEO | | Happy-SEO
https://compleetverkleed.nl/rode-peper-pak/www.compleetverkleed.nl/feestkleding
https://compleetverkleed.nl/new-kids-kleding/www.compleetverkleed.nl/feestkleding
https://compleetverkleed.nl/fee-kostuum/www.compleetverkleed.nl/feestkleding The 404 error code comes due to a double domain URL, and it happens only on the /feestkleding subdomain. When i check my original page, for example https://compleetverkleed.nl/rode-peper-pak/ there is only 1 internal link to /feestkleding but when i click on it, it works fine. Does anyone know how to fix this problem or where to look to see why this is happening? (it happens on every page, not the 4 above > starting to come through in webmastertools also) 2.
https://compleetverkleed.nl/pa_categorie-sitemap.xml
https://compleetverkleed.nl/product_shipping_class-sitemap.xml
https://compleetverkleed.nl/featured_item_category-sitemap.xml
https://compleetverkleed.nl/post_tag-sitemap.xml
https://compleetverkleed.nl/featured_item_tag-sitemap.xml This 404's are completely new for me and i don't know where to find this 😞 Thanks!2 -
To integrate a blog tool onto site - or build a blog solution - what's better for SEO?
Currently looking at adding a blog to our company site subdirectory and wanted to know if there was a SEO distinction between the following methods: Integrating a bolt-on blog tool with the site to create the blog VS. just using the current site infrastructure to build blog functionality. What's better for SEO? (and if tool integration is the overwhelming response - which tool?). Cheers.
Technical SEO | | Oxfordcomma0 -
Tag archives in wordpress
I have duplicate content issue on my site, because i allow to index tags in my wordpress. And the content overlaps on them. What could be a solution to this? How do i fight it, if still want my tag pages to be indexed in Google, but i don't want to to influence my traffic negatively? Currently i have 596 tags! 🙂 Site:
Technical SEO | | pycckuu
richclubgirl.com My idea was to put canonical tag for the post i want to rank from the most popular tag pages (with biggest page authority). Would love to hear from You!1 -
Where to host my blog
If I cannot have my blog on my main site, what is the second best option? It is currently on Blogger, but I am wondering if it would be worth it to move it to blog.mainsite.com. Thanks!
Technical SEO | | AlightAnalytics0 -
What to do about "blocked by meta-robots"?
The crawl report tells me "Notices are interesting facts about your pages we found while crawling". One of these interesting facts is that my blog archives are "blocked by meta robots". Articles are not blocked, just the archives. What is a "meta" robot? I think its just normal (since the article need only be crawled once) but want a second opinion. Should I care about this?
Technical SEO | | GPN0 -
WordPress Pretty Permalinks vs Site Speed
A couple of issues at play here as I wrestle with the best permalink structure for a site I'm toying with now. 1. I know that WordPress wants a unique number in the post to improve performance and db calls. 2. I know that for basic on-page SEO, most of us would opt for CATEGORY/POST or maybe even just post. I constantly change those. It's a bad habit, but sometimes you want the killer headline and a decent title in the post. So here is the issue: I can rewrite or use a plugin (anyone have a favorite) the permalinks to speed up site performance. We all know Google wants that. Maybe the permalink becomes /1234-foo But you know, a number in front of the URL just isn't awfully user friendly. If someone wants to read the foo post, it's nice to send them directly there. So would you trade off a slowdown in site speed for the prettiest permalinks for usability and SEO? And since you're asking a WP question, has anyone heard of a hard cap on static pages where the database starts dragging? The site I have in mind has 400 each posts and pages. Would moving platforms to Drupal or Joomla allow handling that many pages more effectively? Thanks for contributing and any help you can give. George
Technical SEO | | georgebounacos0 -
Mobile SEO or Block Crawlers?
We're in the process of launching mobile versions of many of our brand sites and our ecommerce site and one of our partners suggested that we should block crawlers on the mobile view so it doesn't compete for the same keywords as the standard site (We will be automatically redirecting mobile handsets to the mobile site). Does this advice make sense? It seems counterintuitive to me.
Technical SEO | | BruceMillard0