Development site crawled
-
We just found out our password protected development site has been crawled. We are worried about duplicate content - what are the best steps to take to correct this beyond adding to robots.txt?
-
Unfortunately, robots.txt won't prevent your site from being crawled and indexed if there is a link from an external site pointing to yours. What you need to do is use
on all your development pages. I don't know how big your site is, so this may or may not be a lot of work. Do this, then after the next Google crawl, your pages will be dropped from the SERPs.
-
Thanks Stephen & Kyle! We had the site behind a login, so we're not sure how this happened. Any idea?
-
Put the site behind a login
-
Oops! That sounds unfortunate, Marcy. How did that happen?
Once you have added the correct rules to the robots.txt - I'm guessing you're using "Disallow: /" - you can request, if your development site is registered in Google Webmaster Tools, that Google remove the site from its index.
www.google.com/webmasters/tools/url-removal
Hope that helps,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Improving Crawl Efficieny
Hi I'm reading about crawl efficiency & have looked in WMT at the current crawl rate - letting Google optimise this as recommended. What it's set to is 0.5 requests every 2 seconds, which is 15 URLs every minute. To me this doesn't sound very good, especially for a site with over 20,000 pages at least? I'm reading about improving this but if anyone has advice that would be great
Intermediate & Advanced SEO | | BeckyKey1 -
New site failing to rank - could this be why?
I have a new client site that is not appearing anywhere in the top 100 for its main keywords. ASSUMING that this is not an issue with optimization or link quality, I am wondering whether it might be the following... The client's company has a parent company whose website has decent authority. This website links to the new (client) website. In addition, the 2 press releases we have done include links to both companies, since one was an outgrowth of the other. This is all 100% natural, so my inclination is that this is not causing the issue. But does anyone have any experience to suggest otherwise? That having website A linking to website B, and 50+ press release websites linking to both, could be causing the algorithm to throttle website Bs ability to rank? Thanks in advance!
Intermediate & Advanced SEO | | zakkyg0 -
Site Redesign - Inbound Links
Hello all. What would be some of the best practices or good resources on site redesign while maintaining inbound links? We would hate to have the natural, organic links to the site we have generated over the past 3 years to all of a sudden become broken. The domain is not changing but the URL structure very well may. For example, www.domain dot com/blog/postabouttopic which has many inbound links may move to www.domain dot com/news/blog/postabouttopic Is it a matter of simply using 301 redirects from the old pages to the new pages? Is there any issues to be aware of when having hundreds of 301 redirects? Is there a best practice? A good site that explains this in detail? Thank you for your time! Have a great day!
Intermediate & Advanced SEO | | S2RSolutions0 -
SEO Priorities for Ecommerce Sites
Hello All! What is the best way to rank SEO tasks by PRIORITY for Ecommerce sites to improve?? It can be quite overwhelming with all the types of projects/tasks needed to improve organic rankings... How would you rank the most CRITICAL tasks to spend the MOST TIME on to the tasks you spend less on. Appreciate your input in advance 🙂 Thank you! Mark
Intermediate & Advanced SEO | | wickerparadise0 -
How to remove bad link to your site?
Hello, Our website www.footballshirtblog.co.uk recently suffered a major Google penalty, wiping out 6 months of hard work. We went from getting 6000-10000 hits a day to absolutely nothing from Google. We have been baffled by the penalty as we couldn't think of anything we've done wrong. After some analysis of Open Site Explorer, it seems I may have found the answer. There is a ton of bad links pointing to us. A few example domains are: ru.gg/ gogopzh.com/ 0575bbs.com/ This is nothing to do with us and so I can only assume some competitor has done this. As we were only about 4-5 months old, I guess Google has punished us. What do we do now? This is not a situation I have experienced before and would really appreciate your expert advice.
Intermediate & Advanced SEO | | ukss19840 -
Site: on Google
Hello, people. I have a quick question regarding search in Google. I use search operator [site:url] to see indexing stauts of my site. Today, I was checking indexing status and I found that Google shows different numbers of indexed pages depends on search setting. 1. At default setting (set as 10 search result shows) > I get about 150 pages indexed by Google. 2. I set 100 results shows per page and tried again. > I get about 52 pages indexed by Google. Of course I used same page URL. I really want to know which data is accurate. Please help people!!
Intermediate & Advanced SEO | | Artience0 -
Press Release Sites
Ok, I am getting a lot of conflicting information about press release sites. i have been doing press release's for a while (mostly manually), I have also tried a few companies that claim to do it well (never do). After the Panda update the PR sites I have been using are just not as effective. Does anyone else have this problem or are there better PR sites that can be recommended.
Intermediate & Advanced SEO | | TomBarker820 -
Key page of site not ranking at all
Our site has the largest selection of dog clothes on the Internet. We're been (every so slowly) creeping up in the rankings for the "dog clothes" term, but for some reason only rank for our home page. Even though the home page (and every page on the domain) has links pointing to our specific Dog Clothes page, that page doesn't even rank anywhere when searching Google with "dog clothes site:baxterboo.com". http://www.google.com/webhp?source=hp&q=dog+clothes+site:baxterboo.com&#sclient=psy&hl=en&site=webhp&source=hp&q=dog+clothes+site:baxterboo.com&btnG=Google+Search&aq=f&aqi=&aql=&oq=dog+clothes+site:baxterboo.com&pbx=1&bav=on.2,or.r_gc.r_pw.&fp=f4efcaa1b8c328f Pages 2+ of product results from that page rank, but not the base page. It's not excluded in robots.txt, All on site links to that page use the same URL. That page is loaded with more text that includes the keywords. I don't believe there's duplicated content. What am I missing? Has the page somehow been penalized?
Intermediate & Advanced SEO | | BBPets0