Indexation question
-
Hi Guys, i have a small problem with our development website. Our development website is website.dev.website.nl This page shouldn't be indexed bij Google but unfortunately it is.
What can i do to deindex it and ask google not to index this website. In the robots.txt or are there better ways to do this?
Kind regards Ruud
-
In addition to Tom's response, you can also use Google Webmaster to remove the site out of the index a little faster:
Follow these steps to remove a site or directory from Google's search results:
- On the Webmaster Tools home page, click the site you want.
- On the left-hand menu, click Optimization and then click Remove URLs.
- Click New removal request.
- Type the URL of the directory you want removed from search results and then click Continue. (How to find the right URL.)
- The URL is case-sensitive—use exactly the same characters and capitalization that the site uses.
- If you want to remove the whole site, you can leave this blank.
- Click Remove directory.
- Click Submit Request.
Make sure you aren't accidentally linking to the dev site from your live site. I have seen this happen too many times.
-
Hi Ruud
As you've said, I'd go ahead and add a robots.txt file. One thing to remember is that each subdomain requires its own robots.txt file.
In addition, I would add "noindex, follow" to the header of the pages you don't want crawl.
You can find out the best way to implement these changes with the Moz Robotstxt guide.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site not getting indexed by googlebot.
The following question is in regards to http://footeschool.org/. This site is not getting indexed with google(googlebot) This only happens when the user agent is set googlebot. This is a recent issue. We are using DNN as CMS. Are there any suggestion to help resolve this issue?
Technical SEO | | bcmull0 -
Anything new if determining how many of a sites pages are in Google's supplemental index vs the main index?
Since site:mysite.com *** -sljktf stopped working to find pages in the supplemental index several years ago has anyone found another way to identify content that has been regulated to the supplemental index?
Technical SEO | | SEMPassion0 -
Carwling and indexing problems
hi, i have noticed since my site was upgraded that google is taking a long time to publish my articles. before the upgrade google would publish the article straight away, but now it takes an average of around 4 days. the article i am talking about at the moment is here http://www.in2town.co.uk/celebrities-in-the-news/stuart-hall-has-his-prison-sentence-for-sex-crimes-doubled-to-30-months now i have a blog here on blogger and the article was picked up within six mins http://showbizgossipandnews.blogspot.co.uk/2013/07/stuart-hall-has-his-prison-sentence-for.html so i am just wondering what the problem is and what i need to solve this my problem is, my site is mostly a news site so it is no good to me if google is publishing new stories every four days, any help would be great.
Technical SEO | | ClaireH-1848860 -
Questions about Redirects
Hi, I am trying to make sure that I can determine if a site has a 301 redirect set up to redirect the site from domain.com to www.domain.com and am hoping that you can confirm the following for me, or let me know if I am off track: is http://www.internetofficer.com/seo-tool/redirect-check/ a reliable way to check if a 301 redirect is set up? is Screaming Frog SEO Spider a good tool to use to see if a redirect is in place? if I search for site:www.domain.com and site:domain.com, I should only get results for the site being indexed, not for the site that has the 301 redirect set up, right? For example, if www.domain.com is set up to redirect to domain.com, then I should get no search results for site:www.domain.com and only show indexed pages for domain.com. If I search for site:www.domain.com and site:domain.com and get results for both, then does this mean that the redirect is not set up? if a redirect is set up from www.domain.com to domain.com, should the crawl report should only show one page crawled on www.domain.com? if a crawl report shows same number of pages for www.domain.com as for domain.com, does that mean that redirect is not set up properly? Thanks in advance for your help! Carolina
Technical SEO | | csmm0 -
Question about content on ecommerce pages.
Long time ago we hired a seo company to do seo in our website and one of the things they did is that they wrote long text on the category pages of our products. Example here: http://www.theprinterdepo.com/refurbished-printers/wide-format-laser-refurbished-printers Now my marketing person is saying that if its possible to put the text below the items, technically I will find out how to do it, but from your seo experience, is it good or bad? What about if we short those texts to one paragraph only? Thanks
Technical SEO | | levalencia10 -
Redirect question
I would like to redirect http://example.com/index.html to http://www.example.com/ Is the code below correct ? RewriteEngine on RewriteCond %{HTTP_HOST}^example.comRewriteRule (.*) http://www.example.com/$1 [R=301,L] RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index.html\ HTTP/ RewriteRule ^index.html$ http://www.example.com/ [R=301,L]
Technical SEO | | seoug_20050 -
Why googlebot indexing one page, not the other?
Why googlebot indexing one page, not the other in the same conditions? In html sitemap, for example. We have 6 new pages with unique content. Googlebot immediately indexes only 2 pages, and then after sometime the remaining 4 pages. On what parameters the crawler decides to scan or not scan this page?
Technical SEO | | ATCnik0 -
Over 1000 pages de-indexed over night
Hello, On my site (www.bridgman.co.uk) we had a lot of duplicate page issues as reported by the Seomoz site report tool - this was due to database driven URL strings. As a result, I sent an excel file with all the duplicate pages to my web developer who put rel canonical tags on what I assumed would be all the correct pages. I am not sure if this is a coincidence, or a direct result of the canonical tags, but a few days after (yesterday) the amount of pages indexed by google dropped from 1,200 to under 200. The number is still declining, and other than the canonical tags I can't work out why Google would just start de-indexing most of our pages. If you could offer any solutions that would be greatly appreciated. Thanks, Robert.
Technical SEO | | 87ROB0