Development site crawled
-
We just found out our password protected development site has been crawled. We are worried about duplicate content - what are the best steps to take to correct this beyond adding to robots.txt?
-
Unfortunately, robots.txt won't prevent your site from being crawled and indexed if there is a link from an external site pointing to yours. What you need to do is use
on all your development pages. I don't know how big your site is, so this may or may not be a lot of work. Do this, then after the next Google crawl, your pages will be dropped from the SERPs.
-
Thanks Stephen & Kyle! We had the site behind a login, so we're not sure how this happened. Any idea?
-
Put the site behind a login
-
Oops! That sounds unfortunate, Marcy. How did that happen?
Once you have added the correct rules to the robots.txt - I'm guessing you're using "Disallow: /" - you can request, if your development site is registered in Google Webmaster Tools, that Google remove the site from its index.
www.google.com/webmasters/tools/url-removal
Hope that helps,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Enquiries stopped after site move
Hello, I really hope someone can help. We recently moved our website from a shared server with one host to a VPS with another. At the same time we decided it would be right to switch from the .co.uk to the .com and also purchase an SSL. Since the switch we have had zero enquiries (3 weeks ago) when we would normally average one a day. According to Google Analytics, I cannot see that traffic has been to adversely effected and rankings, though dropping very slightly have not dramatically fallen. We have tested the site rigirously and there are no issues with it we can see. I ensured at domain level that there was a 301 redirect on the .co.uk site as well. Does anyone have any suggestions as to why this would be the case? And/or whether switching it all back to where it was with the .co.uk would be a foolish idea? Many thanks!
Intermediate & Advanced SEO | | Opus4Marketing0 -
Domain Change Before or After Site Revamp?
In the last year traffic to our site has dropped in half and ranking has dropped significantly. Very little no content has been added in that time. We would now like to improve ranking by adding new content. 2 domains effectively exist for the site. The existing domain is www.nyc-officespace-leader.com. But www.metro-manhattan.com redirects to www.nyc-officespace-leader.com. Our company is Metro Manhattan Office Space, Inc.. We registered www.metro-manhattan.com and created the redirect to www.nyc-officespace-leader.com in 2012. www.nyc-officespace-leader.com was registered in 2006. Many links to the site show www.metro-manhattan.com and I believe this may be a source of confusion for Google. Would it be best to make the domain consistent at this time by redirecting it once and for all and to do so before adding new content? If this is done correctly can we avoid taking a hit on ranking? Note: -www.nyc-officespace-leader.com is the old domain.
Intermediate & Advanced SEO | | Kingalan1
-www.metro-manhattan is the new domain but has existed since 2012 and has been redirecting to the old domain since then
-The company name is Metro Manhattan Office Space (similar in branding to the new domain) Am I correct in assuming that having the 2 domains may be causing issues with Google involving domain authority? Change the domain before adding content or add content before?0 -
Site: inurl: Search
I have a site that allows for multiple filter options and some of these URL's have these have been indexed. I am in the process of adding the noindex, nofollow meta tag to these pages but I want to have an idea of how many of these URL's have been indexed so I can monitor when these have been re crawled and dropped. The structure for these URL's is: http://www.example.co.uk/category/women/shopby/brand1--brand2.html The unique identifier for the multiple filtered URL's is --, however I've tried using site:example.co.uk inurl:-- but this doesn't seem to work. I have also tried using regex but still no success. I was wondering if there is a way around this so I can get a rough idea of how many of these URL's have been indexed? Thanks
Intermediate & Advanced SEO | | GrappleAgency0 -
SEO site Review
Does anyone have suggestions on places that provide in depth site / analytics reviews for SEO?
Intermediate & Advanced SEO | | Gordian0 -
Depth of Links on Ecommerce Site
Hi, In my sitemap, I have the preferred entrance pages and URL's of categories and subcategories. But I would like to know more about how Googlebot and other spiders see a site - e.g. - what is classed as a deep link? I am using Screaming Frog SEO spider, and it has a metric called level on it - and this represents how deep or how many clicks away this content is.. but I don't know if that is how Googlebot would see it - From what Screaming Frog SEO spider software says, each move horizontally across from Navigation is another level which visually doesnt make sense to me? Also, in my sitemap, I list the URL's of all the products, there are no levels within the sitemap. Should I be concerned about this? Thanks, B
Intermediate & Advanced SEO | | bjs20100 -
Crawl errors in GWT!
I have been seeing a large number of access denied and not found crawl errors. I have since fixed the issued causing these errors; however, I am still seeing the in webmaster tools. At first I thought the data was outdated, but the data is tracked on a daily basis! Does anyone have experience with this? Does GWT really re-crawl all those pages/links everyday to see if the errors still exist? Thanks in advance for any help/advice.
Intermediate & Advanced SEO | | inhouseseo0 -
Critique My Site For SEO
Hi Everyone, I was wondering if someone might critique my site and let me know what you think. I've done pretty much everything I know to do proper seo for my site. I'd love to hear some critiques about what I am doing wrong. I'm not sure if my titles are okay, being that they are similar amongst pages. The other thing is that for all the javascript buttons on the top I have no followed them since they don't have any anchor text. The way google will crawl my page is through the links in the footer. I was thinking of moving them throughout the body of the page since I hear google isn't giving as much weight to footer links. I also wanted to hear what you think about putting a blog on my site and updating with fresh content as opposed to creating a separate blog and then linking back to my website with anchor text. Thanks for all the help. And glad to be a member Bill
Intermediate & Advanced SEO | | wsh150 -
Getting rid of a site in Google
Hi, I have two sites, lets call them site A and site B, both are sub domains of the same root domain. Because of a server config error, both got indexed by Google. Google reports millions of inbound links from Site B to Site A I want to get rid of Site B, because its duplicate content. First I tried to remove the site from webmaster tools, and blocking all content in the robots.txt for site B, this removed all content from the search results, but the links from site B to site A still stayed in place, and increased (even after 2 months) I also tried to change all the pages on Site B to 404 pages, but this did not work either I then removed the blocks, cleaned up the robots.txt and changed the server config on Site B so that everything redirects (301) to a landing page for Site B. But still the links in Webmaster Tools to site A from Site B is on the increase. What do you think is the best way to delete a site from google and to delete all the links it had to other sites so that there is NO history of this site? It seems that when you block it with robots.txt, the links and juice does not disappear, but only the blocked by robots.txt report on WMT increases Any suggestions?
Intermediate & Advanced SEO | | JacoRoux0