Indexation question
-
Hi Guys, i have a small problem with our development website. Our development website is website.dev.website.nl This page shouldn't be indexed bij Google but unfortunately it is.
What can i do to deindex it and ask google not to index this website. In the robots.txt or are there better ways to do this?
Kind regards Ruud
-
In addition to Tom's response, you can also use Google Webmaster to remove the site out of the index a little faster:
Follow these steps to remove a site or directory from Google's search results:
- On the Webmaster Tools home page, click the site you want.
- On the left-hand menu, click Optimization and then click Remove URLs.
- Click New removal request.
- Type the URL of the directory you want removed from search results and then click Continue. (How to find the right URL.)
- The URL is case-sensitive—use exactly the same characters and capitalization that the site uses.
- If you want to remove the whole site, you can leave this blank.
- Click Remove directory.
- Click Submit Request.
Make sure you aren't accidentally linking to the dev site from your live site. I have seen this happen too many times.
-
Hi Ruud
As you've said, I'd go ahead and add a robots.txt file. One thing to remember is that each subdomain requires its own robots.txt file.
In addition, I would add "noindex, follow" to the header of the pages you don't want crawl.
You can find out the best way to implement these changes with the Moz Robotstxt guide.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Indexing without content
Hello. I have a problem of page indexing without content. I have website in 3 different languages and 2 of the pages are indexing just fine, but one language page (the most important one) is indexing without content. When searching using site: page comes up, but when searching unique keywords for which I should rank 100% nothing comes up. This page was indexing just fine and the problem arose couple of days ago after google update finished. Looking further, the problem is language related and every page in the given language that is newly indexed has this problem, while pages that were last crawled around one week ago are just fine. Has anyone ran into this type of problem?
Technical SEO | | AtuliSulava1 -
Google Not Indexing Pages (Wordpress)
Hello, recently I started noticing that google is not indexing our new pages or our new blog posts. We are simply getting a "Discovered - Currently Not Indexed" message on all new pages. When I click "Request Indexing" is takes a few days, but eventually it does get indexed and is on Google. This is very strange, as our website has been around since the late 90's and the quality of the new content is neither duplicate nor "low quality". We started noticing this happening around February. We also do not have many pages - maybe 500 maximum? I have looked at all the obvious answers (allowing for indexing, etc.), but just can't seem to pinpoint a reason why. Has anyone had this happen recently? It is getting very annoying having to manually go in and request indexing for every page and makes me think there may be some underlying issues with the website that should be fixed.
Technical SEO | | Hasanovic1 -
App Indexing
Can anyone please check if our app is indexed or not? Also check if deep linking done is correct or not rel="alternate" href="android-app://in.instafresh.app/http/www.instafrsh.com/" /> Website - http://instafrsh.com/ App - https://play.google.com/store/apps/details?id=in.instafresh.app
Technical SEO | | Obbserv0 -
Do Sitespect links get indexed?
I put a link on one of my websites using sitespect because the next release is not for a few weeks. The reason for the link is to pass domain authority (SEO Juice) to the linked site. In my next release I will add the link in the actual code, but am hoping that from now till then google will crawl and index this link. So the question is, will google crawl and index links adding to webpages via sitespect? Here is the code: | * [http://www.](<a class=)yourdomain.com" class="" >YourDomain |
Technical SEO | | AlyssaN
| | | Link to Sitespect: http://www.sitespect.com/0 -
No index on subdomains
Hi, We have a subdomain that is appearing in the search results - I want to hide this as it looks really bad. If I were to add the no index tag to the sub domain would URL would this affect the whole domain or just that sub domain? The main domain is vitally important - it is just that sub domain I need to hide. Many thanks
Technical SEO | | Creditsafe0 -
Update index date
If I update the content of a page without changing the initial url and google crawls my new page, will the index date (that appears in the SERP) change to the latest update? In positive case how many change should I do to consider an update? tks
Technical SEO | | fabrico230 -
Summarize your question.Sitemap blocking or not blocking that is the question?
Hi from wet & overcast wetherby UK 😞 Ones question is this... " Is the sitemap plus boxes blocking bots ie they cant pass on this page http://www.langleys.com/Site-Map.aspx " Its just the + boxes that concern me, i remeber reading somewherte javascript nav can be toxic. Is there a way to test javascript nav set ups and see if they block bots or not? Thanks in advance 🙂
Technical SEO | | Nightwing0 -
301 Redirect Questions
I have a site I built on a wisiwig editing platform that will not allow a 301 redirect. The site has already been remade and I need to point it to another domain. To do the redirect, can I change it to another domain host that will allow a 301 or will that make me loose the authority of the site? I may not be able to move the content of the site. Please help.
Technical SEO | | photoseo10