Blocked By Meta Robots
-
Hi
I logged in the other day to find that over night I received 8347 notices saying certain pages are being kept out of the search engine indexes by meta-robots.
I have not changed my robots.txt in years and I certainly didn't block Google from visiting those pages.
Is this a fault on Roger Mozbot behalf? Or is there a bot preventing 8000+ pages being indexed?
Is there a way to find out what meta-robot is doing this and where? And how I can get rid of it?
I usually rank between #3 and #5 for the term 'sex toys' on google.com.au, but I now rank #7 to #9 so it would seem some of my pages/content is being blocked.
My website is www.theloveshop.com THIS IS AN ADULT TOYS SITE.
There is no porn videos or anything like that on it, but just in case you don't wish to look at sex toys or are around kids I thought I would mention it.
Blake
-
Hi Nakul
Thank you very much for the suggestions and I will start checking the code on those pages today. Thanks for heads up on the .com.au site as well I will be changing the hosting today to Aus, and thanks to you that site may get pushed up the rankings for a highly competitive term that I have been fighting to get for years
As for following my competitors links, I have been doing that for awhile and it would seem most of them either buy their links or have them on pages that are irrelevant to our niche, one of my competitors has even acquired more links than SEOmoz.com but has no were near the content. In saying that I am by no means a pure white hat SEO, more of a grey hat if such a thing exists, simply because I find it hard to get links to an adult site without it being spammy of some sorts. So I have started writing articles and linking to products mentioned, but trying to make the articles interesting/funny at the same time. Hopefully soon I will start to see some improvement.
Again thank you for your help and I will let you know how I go with those pages blocked by bots.
Blake
-
The meta robots block sounds like it's a block from your CMS or something in your header include file or something. I would check on the pages that show as blocked to see if there's any noindex sort of code in those pages. Other then that, shift in rankings could be based on your competition optimizing for the same set of keywords. I would recommend to spend some time analyzing the backlinks of the competition and see if you can gain some of those kinds of links.
On a side note: It also looks like you own both the .com and the .com.au site. The .com is hosted in AU and the .com.au is hosted in USA. Did you intend to do it the other way around ? I would imagine your rankings for the .com.au get better if it was hosted in Australia instead of USA.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Meta description for Privacy Policy?
Hello guys, Quick question about optimizing other pages on my woocommerce e-commerce store. Do I need to optimize pages like the cart page, checkout page, privacy policy, return policy, shipping policy, etc? Strictly talking about on page SEO for these pages, like meta titles, description. Thanks!
On-Page Optimization | | The_Kiwi_Man0 -
Duplicate meta and title in Google Webmaster Tools not updated?
Our canonical tags were removed by mistake for some time and our duplicate meta and title warnings in Google webmaster tools html improvements went up. We fixed the issue a week ago. I made sure the sitemap is picking up the canonical URL. Still the number went up after we fixed it (from around 5000 to around 7000 now) but when I click on details, it shows around 2000 in detail page. How long does it take for Google to update this? RB
On-Page Optimization | | rbai0 -
Regarding Google Title 'Width' and changing Meta Titles w/o Penalty?
A vast majority of pages on my site are now too wide (the character count was fine prior to the March update). I want to go through and update them so they display properly and are not too wide.However, I am concerned, as my understanding was that changing Meta Titles is dangerous and can have a negative effect on your rankings and can cause real issues. Is this an opportunity to change my Titles all-together without any kind of penalty? Or can I only trim the end part? In summary: 1. Can I edit all of my Meta Titles without affecting my rankings? 2. If no, how do I edit them properly to fit within the proper width and not cause any issues? 3. If yes, I can go through and change all my Meta Titles to whatever extent and optimize them to reflect latest best practices? There are changes I wanted to make to all my meta titles but I've been afraid to... due to fear of rankings drops etc Any help with this would be greatly appreciated
On-Page Optimization | | lawfirm0 -
Page Title & Meta Description Getting Cut Off In The SERPs
Hi Guys, I am trying to figure out why my page titles and meta d tags are getting cut off in Goofle SERPS. My page titles are 70 characters or under (including spaces) and my meta Dd's are 155 characters or under (including spaces) so I cannot work out why They are getting cut off. Is there something I have missed?! Thanks, Meaghan
On-Page Optimization | | StoryScout0 -
I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
On-Page Optimization | | absoauto0 -
Global Meta Data and Titles
I am working on an e-commerce website with Volusion. I have 1 main keyword that I am really trying to focus on for my home page. Last week I was in position #7 for this keyword, and I have never been off the first page of Google for it. Using Volusion you can Globally set a Title and meta description for any pages that do not already have a title or description. I noticed my Global title had the same keyword as I was using for my home page and thought that, that might be taking away some importance from my keyword, so I changed the Global Title to a different keyword. When I looked at my SEO moz rankings today, which crawled last night it said My keyword was #24 on Google. I checked and as of now I am in position #11. Like I said before, I have never not been on the first page for this keyword. Does anyone think that changing the Global title may have been the cause for this??
On-Page Optimization | | DTOSI0 -
Are missing Meta Tags a big deal on a blog?
I show 272 missing Meta Description Tags after the crawl completed on my website. Pretty much all of those are from my website hosted wordpress blog. Should I be concerned about this?
On-Page Optimization | | webestate0 -
Confused about meta tags.
(Ok, this might be a stupid question.) I am a little confused about the best practice for the Meta Tag. I know they don't have a real effect on ranking. But I am getting "mixed signals" as to their general use. I've pasted the 2 examples of below that are confusing me. From my Campaign's Crawl Diagnostics: (I have missing Meta Tags) "Meta description tags, while not important to search engine rankings, are extremely important in gaining user click-through from search engine result pages (SERPs). These short paragraphs are the webmaster's opportunity to advertise content to searchers and let them know exactly what the given page has with regard to what they're looking for" From The Keyword Optimization Tool: (Avoid Using Meta Keyword Tool) "Search engines have, for years, ignored the meta keywords tag as a ranking signal. Although it technically does not harm rankings, it can be used by competitors as a method to extract your targeted terms and thus, we recommend against its use. " I would love to hear opinions on this topic. I am trying to decide wether or not to use unique meta tags on every blog post.
On-Page Optimization | | NerdsOnCall0