Blocked By Meta Robots
-
Hi
I logged in the other day to find that over night I received 8347 notices saying certain pages are being kept out of the search engine indexes by meta-robots.
I have not changed my robots.txt in years and I certainly didn't block Google from visiting those pages.
Is this a fault on Roger Mozbot behalf? Or is there a bot preventing 8000+ pages being indexed?
Is there a way to find out what meta-robot is doing this and where? And how I can get rid of it?
I usually rank between #3 and #5 for the term 'sex toys' on google.com.au, but I now rank #7 to #9 so it would seem some of my pages/content is being blocked.
My website is www.theloveshop.com THIS IS AN ADULT TOYS SITE.
There is no porn videos or anything like that on it, but just in case you don't wish to look at sex toys or are around kids I thought I would mention it.
Blake
-
Hi Nakul
Thank you very much for the suggestions and I will start checking the code on those pages today. Thanks for heads up on the .com.au site as well I will be changing the hosting today to Aus, and thanks to you that site may get pushed up the rankings for a highly competitive term that I have been fighting to get for years
As for following my competitors links, I have been doing that for awhile and it would seem most of them either buy their links or have them on pages that are irrelevant to our niche, one of my competitors has even acquired more links than SEOmoz.com but has no were near the content. In saying that I am by no means a pure white hat SEO, more of a grey hat if such a thing exists, simply because I find it hard to get links to an adult site without it being spammy of some sorts. So I have started writing articles and linking to products mentioned, but trying to make the articles interesting/funny at the same time. Hopefully soon I will start to see some improvement.
Again thank you for your help and I will let you know how I go with those pages blocked by bots.
Blake
-
The meta robots block sounds like it's a block from your CMS or something in your header include file or something. I would check on the pages that show as blocked to see if there's any noindex sort of code in those pages. Other then that, shift in rankings could be based on your competition optimizing for the same set of keywords. I would recommend to spend some time analyzing the backlinks of the competition and see if you can gain some of those kinds of links.
On a side note: It also looks like you own both the .com and the .com.au site. The .com is hosted in AU and the .com.au is hosted in USA. Did you intend to do it the other way around ? I would imagine your rankings for the .com.au get better if it was hosted in Australia instead of USA.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Eliminate render-blocking CSS in above the fold content with shopify
Hello, It seems that my shopify website needs to eliminate render-blocking CSS in above the fold content to get better mobile speed and I don't see or find anything about how this could be done. I have succeeded with a wordpress site but the way things are restricted with shopify, I have not been able to do the same. The other thing would be to leverage browser caching which I have no clue how it can be done with shopify either. Thanks!
On-Page Optimization | | Mabo320 -
Google is NOT showing up the right META DESCRIPTION
Hi, Recently I changed titles and meta descriptions of some pages. The problem is that google have updated the titles but not the meta descriptions.. Ive also checekd the source code of the google cached version of the pages crawled and the meta description reflect the changes i did...but the changes don't appear in google. Do you have some solution/advice for the issue? Tx so Much
On-Page Optimization | | tourtravel0 -
How do i block an entire category/directory with robots.txt?
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now. The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc. I'm not really sure how i'd type it into the robots.txt file, or where to place the file. any help would be appreciated thanks
On-Page Optimization | | bricerhodes0 -
Same Meta Description on all pages harmful?
As the question title suggests, is it harmful (on Google) for a site to use the same meta description sitewide? On every single page. If not, is there any particular benefit to crafting a unique meta description for every page besides CTR on google searches. What do you think? Thanks in advance!
On-Page Optimization | | danielpett0 -
Meta Description Question
Is it ok to put your domain name in the meta description if that is the name of your company? so if my company name is BlaBla.com and I want to have a meta description like Shop for your next Widget at BlaBla.com. We have Widget in 30 colors. Free shipping for orders over $50. and I am obviosuly promoting the widget page is this ok. I am concerned that Google will not like the domain name in the meta description. After Peguin I am paranoid about my shadow.....LOL
On-Page Optimization | | freestone0 -
Right way to block google robots from ppc landing pages
What is the right way to completely block seo robots from my adword landing pages? Robots.txt does not work really good for that, as far I know. Adding metatags noindex nofollow on the other side will block adwords robot as well. right? Thank you very much, Serge
On-Page Optimization | | Kotkov0 -
Curious meta-tag
Has anyone met this meta tag? <meta name="ranking" content="Top10, Page=1" /> I imagine it has no effect but I'm curios where did it came from, i saw it on many sites
On-Page Optimization | | seo.academy0 -
How many meta tags are appropriate?
How many meta tags are appropriate? There is a page that ranks above me that has many more than I do. It has (I have removed the content of each to give a description of the type of content): I only use meta tags for keywords and description, robots. Should I have more? Thanks.
On-Page Optimization | | scanlin0