Whole site blocked by robots in webmaster tools
-
My URL is: www.wheretobuybeauty.com.auThis new site has been re-crawled over last 2 weeks, and in webmaster tools index status the following is displayed:Indexed 50,000 pagesblocked by robots 69,000Search query 'site:wheretobuybeauty.com.au' returns 55,000 pagesHowever, all pages in the site do appear to be blocked and over the 2 weeks, the google search query site traffic declined from significant to zero (proving this is in fact the case ).This is a Linux php site and has the following: 55,000 URLs in sitemap.xml submitted successfully to webmaster toolsrobots.txt file existed but did not have any entries to allow or disallow URLs - today I have removed robots.txt file completely URL re-direction within Linux .htaccess file - there are many rows within this complex set of re-directions. Developer has double checked this file and found that it is valid.I have read everything that google and other sources have on this topic and this does not help. Also checked webmaster crawl errors, crawl stats, malware and there is no problem there related to this issue.Is this a duplicate content issue - this is a price comparison site where approx half the products have duplicate product descriptions - duplicated because they are obtained from the suppliers through an XML data file. The suppliers have the descriptions from the files in their own sites.Help!!
-
After reviewing your site, I can see that:
- Your site is well indexed by Google.
- Your site has very thin content.
- Your site is mostly duplicate content.
I think you'll need a content strategy to achieve and maintain significant rankings. You'll probably need to noindex your entire site except for a few pages that you add significant and valuable content to. (Unless you have the resources to write good content for your entire site.)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Web Site Ranking
Hi Folks, I made some changes on my website www.gemslearninginstitute.com and published it two days ago. It was ranking on Google first page for a few keywords. I did not touch the pages which were ranking on first page. Since then I am not seeing the website ranking on the Google. Does it take a few days to rank again? How can I ensure that next time if I update the website or publish some blog on my website then it should not effect the ranking. Secondly, if I would like to rank in three different cities then do I need to create separate pages for each city or how should I proceed with this. Thanks
Intermediate & Advanced SEO | | fslpso0 -
Google Indexing our site
We have 700 city pages on our site. We submitted to google via a https://www.samhillbands.com/sitemaps/locations.xml but they only indexed 15 so far. Yes the content is similar on all of the pages...thought on getting them to index the remaining pages?
Intermediate & Advanced SEO | | brianvest0 -
How do I add https version of site to Bing webmaster tools?
I could add my site to Google Webmaster tools with no problems, but when I try to add it in Bing webmaster tools it just redirects me to what I already have. Everything is staying the same but the switch from http to https. Anyone else experienced this? This is what I just received back from Bing and it doesn't seem right- I understand that you switched to the https version of your site and you're now trying to use the Site Move tool. However, in order to do this, you must verify the https version of your site first. You cannot do this because it just redirects you to the dashboard. We thank you for reporting this to us. We've investigated on this matter and can see that you're already put a redirect from the http to the https version of your site. We also checked the /BingSiteAuth.xml file and this also redirects to the https version. At this point, we suggest that you remove the current website (http version) that is verified through Bing Webmaster Tool and add your https domain. When done, use the Site Move tool. Thoughts?
Intermediate & Advanced SEO | | EcommerceSite1 -
Our quilting site was hit by Panda/Penguin...should we start a second "traffic" site?
I built a website for my wife who is a quilter called LearnHowToMakeQuilts.com. However, it has been hit by Panda or Penguin (I’m not quite sure) and am scared to tell her to go ahead and keep building the site up. She really wants to post on her blog on Learnhowtomakequilts.com, but I’m afraid it will be in vain for Google’s search engine. Yahoo and Bing still rank well. I don’t want her to produce good content that will never rank well if the whole site is penalized in some way. I’ve overly optimized in linking strongly to the keywords “how to make a quilt” for our main keyword, mainly to the home page and I think that is one of the main reasons we are incurring some kind of penalty. First main question: From looking at the attached Google Analytics image, does anyone know if it was Panda or Penguin that we were “hit” by? And, what can be done about it? (We originally wanted to build a nice content website, but were lured in by a get rich quick personality to rather make a “squeeze page” for the Home page and force all your people through that page to get to the really good content. Thus, our avenge time on site per person is terrible and Pages per Visit is low at: 1.2. We really want to try to improve it some day. She has a local business website, Customcarequilts.com that did not get hit. Second question: Should we start a second site rather than invest the time in trying to repair the damage from my bad link building and article marketing? We do need to keep the site up and running because it has her online quilting course for beginner quilters to learn how to quilt their first quilt. We host the videos through Amazon S3 and were selling at least one course every other day. But now that the Google drop has hit, we are lucky to sell one quilting course per month. So, if we start a second site we can use that to build as a big content site that we can use to introduce people to learnhowtomakequilts.com that has Martha’s quilting course. So, should we go ahead and start a new fresh site rather than to repair the damage done by my bad over optimizing? (We’ve already picked out a great website name that would work really well with her personal facebook page.) Or, here’s a second option, which is to use her local business website: customcarequilts.com. She created it in 2003 and has had it ever since. It is only PR 1. Would this be an option? Anyway I’m looking for guidance on whether we should pursue repairing the damage and whether we should start a second fresh site or use an existing site to create new content (for getting new quilters to eventually purchase her course). Brad & Martha Novacek rnUXcWd
Intermediate & Advanced SEO | | BradNovi0 -
SEO Tools
Anyone have any experience and thoughts about the woo rank website and seo tool?
Intermediate & Advanced SEO | | casper4341 -
10yr old Domain, Conflicting Message from Webmaster tools/Google search
This is the first time I have encountered this and am quite frankly a little baffled on how to proceed. We have some domains that are 10 years old, and do get some hits / impressions and they have a lot of content. So I redid the site in wordpress etc... Anyway, on Google the sites show up as www. , and on Webmaster tools,- the www. shows no impressions or anything, while the non-www domain shows up in google webmaster tools with data. The question is, if google displays the site as www. and webmaster tools shows data for non www. Which one do I proceed with, finding info on this has been pretty hard to do. Any input is appreciated, Thanks in advance:)
Intermediate & Advanced SEO | | choiceenergy0 -
Is my site being penalized?
I launched http://rumma.ge in February of this year. Because I'm using a domain hack (the Georgian domain), I'd really like to rank for just the word "rummage". After launching, I was steady at around page 4/5 on searches for "rummage". However since then I've tumbled out of the first 100. In fact I can't even find the site in the first 20 pages on Google for that search. Even a search for my exact homepage title text doesn't bring up the site, despite the fact that the site is still in the index. I'm wondering if one of the following could be the root cause: We have a ccTLD (.ge)--not sure about the impacts of this, but seems like it might not be the root cause because we were ranking for "rummage" when we first launched. Tried running an Adwords campaign but the site was flagged as a "bridge page" (working on getting this addressed). I'm wondering if this could have carryover impacts into natural search rankings? We've tried doing some press and built up a decent number of backlinks over the past couple of months, many of which had "rummage" in the anchor text. This was all organic, but happened over the span of a month which may be too fast? Am I being penalized? Beyond checking indexing of the site, is there a way to tell if I've been flagged for some bad behavior? Any help or thoughts would be greatly appreciated. I'm really confused by this since I feel like I've been doing things right and my rankings have been travelling downward. Thanks!! Matt
Intermediate & Advanced SEO | | minouye0 -
Making a whole site website SSL (https)
Our IT department wants to make a change to our website and serve all the pages under SSL (https). This will be happening at the same time as a move from classic ASP to ASP.Net so the file extensions for non re-written urls will change (this doesn't equate to many). They will be ensuring everything is 301 redirected correctly. Even with this I can't help being very nervous about the change. We have tens of thousands of links to the website gained over many years, and I understand even with 301'ing them they will lose some of their value. We receive tens of thousands of natural visitors per day. Has anyone done anything like this before, or have any advice on whether it is the right thing to do?
Intermediate & Advanced SEO | | BigMiniMan0