Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Question on Pagination - /blog/ vs /blog/?page=1
Question on Pagination Because we could have /blog/ or /blog/?page=1 as page one would this be the correct way to markup the difference between these two URL? The first page of a sequence could start with either one of these URLs. Clarity around what to do on this first page would be helpful. Example… Would this be the correct way to do this as these two URLs would have the exact content? Internal links would likely link to /blog/ so signal could be muddy. URL: https://www.somedomain.com/blog/
Technical SEO | | jorgensoncompanies
<link rel="canonical" href="https://www.somedomain.com/blog/?page=1"> URL: https://www.somedomain.com/blog/?page=1
<link rel="canonical" href="https://www.somedomain.com/blog/?page=1"> Google is now saying to just use the canonical to the correct paginated URL with page number. You can read that here:
https://developers.google.com/search/docs/advanced/ecommerce/pagination-and-incremental-page-loading But they do not clarify what to do on /blog/?page=1 vs /blog/ as they are the exact same thing. Thanks for your help.0 -
Multilingual Blog Structure
Hi I have a domain in 20 languages. I want to integrate a wordpress blog (in subfolders) in the 3 most important languages like EN-ES-FR (actually they will be 3 independent blogs) and I want to know which structure is the best one. OPTION 1 domain/en/blog/post1 domain/es/blog/post1 domain/fr/blog/post1 OPTION 2 domain/blog_en/post1 domain/blog_es/post1 domain/blog_fr/post1 Last question. For the rest of the 17 languages of my domain, can I put a link the english blog or is not recommended because maybe too many pages will be linking to the blog? Thank you
Technical SEO | | andromedical0 -
Proper way of handling wordpress urls and redirects?
I have recently changed some of my urls in wordpress + adding a sub category in the url structure. from
Technical SEO | | Livet
www.mydomain.se/category/subcategory to www.mydomain.se/category/subcategory/subcategory2/ My products are stil under www.mydomain.se/products/cool-product and are not affected. Should I 301 the old url (www.mydomain.se/category/subcategory) to the new (www.mydomain.se/category/subcategory/subcategory2/)? If that is so, can someone recommend a good 301 redirection plugin? Thanks!0 -
Post vs page in Wordpress?
Hello there, I have a Wordpress site and would like to know if it is better to have 600 posts or 600 pages in terms of efficiency in the site. I would like to publish the content as pages, as I can have subapges,etc... and keep the path: www.website.com/page/subpage1... in terms of good SEO. This structure of using pages rahter than posts allow me to keep the path as stated above (with a category/post path I could not manage in this sense as a pile of articles is displayed although the path category/post in terms of SEO I understand would be good too). Thank you very much for your thoughts here as I would go for a page structure. Antonio
Technical SEO | | aalcocer20030 -
Yoast WordPress SEO settings please help
Hello 🙂 Can you please look at these screenshots of my Yoast WordPress SEO settings http://www.zaslike.com/files/h5149mi5435dspiswfm.jpg http://www.zaslike.com/files/5dlhmjxfh2j0hqswesha.jpg http://www.zaslike.com/files/fmx1pwih240gwiofh86s.jpg http://www.zaslike.com/files/w7tyvlhgr5vhv149b9a.png http://www.zaslike.com/files/l9lo37jfpeqmrpufke8.png are they good ? Do i need to change something or correct ? please help Thank you !!!! :))))
Technical SEO | | wolfinjo0 -
Best Strategy to Rank Wordpress Category
Does anyone some suggestions for best practices when trying to rank categories using Wordpress? Unless you filter it, a category provides a complete list of all posts in that category. I can add a good title and description but am not sure beyond that. Has anyone tried using a sticky post in the category and putting keyword body text in that? Lookin' for ideas.
Technical SEO | | waynekolenchuk1 -
Canonical tags/wordpress permalink question
Need help: Do canonical tags do the exact same thing that wordpress already does with it’s permalink function? Or are these 2 separate things? thank you.
Technical SEO | | bonnierSEO1 -
Thesis Vs Yoast Wordpress SEO Plugin
Hi All. I've noticed some issues between the thesis themes and Yoast's plugin, namely, thesis doesn't recognise Yoast's plugin in for title tag and meta description. Is there a way to overide this?
Technical SEO | | PerchDigital0