Indexation question
-
Hi Guys, i have a small problem with our development website. Our development website is website.dev.website.nl This page shouldn't be indexed bij Google but unfortunately it is.
What can i do to deindex it and ask google not to index this website. In the robots.txt or are there better ways to do this?
Kind regards Ruud
-
In addition to Tom's response, you can also use Google Webmaster to remove the site out of the index a little faster:
Follow these steps to remove a site or directory from Google's search results:
- On the Webmaster Tools home page, click the site you want.
- On the left-hand menu, click Optimization and then click Remove URLs.
- Click New removal request.
- Type the URL of the directory you want removed from search results and then click Continue. (How to find the right URL.)
- The URL is case-sensitive—use exactly the same characters and capitalization that the site uses.
- If you want to remove the whole site, you can leave this blank.
- Click Remove directory.
- Click Submit Request.
Make sure you aren't accidentally linking to the dev site from your live site. I have seen this happen too many times.
-
Hi Ruud
As you've said, I'd go ahead and add a robots.txt file. One thing to remember is that each subdomain requires its own robots.txt file.
In addition, I would add "noindex, follow" to the header of the pages you don't want crawl.
You can find out the best way to implement these changes with the Moz Robotstxt guide.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hi - I have a question about IP addresses
- would it hurt link juice to host a blog on a different server to the rest of your website? I have a web host saying they can't run Wordpress as they won't support PHP for "security reasons" - one solution would be to set up Wordpress on a different server and redirect domain.com/blog there (I presume this is do-able?). But I don't know if that affects the SEO adversely?
Technical SEO | | abisti21 -
Some SEO 2016 questions
Hello MOZ Community, I have some questions where the following is still working for seo in 2016: Is an exact keyword in the domain still a good start? If a domain contains the most important keyword does one still need subfolders with that keyword in the url? Do you need multiple subpages so the main url becomes stronger? Is linkbuilding still the number one factor? Thank you for your thoughts!
Technical SEO | | mhenze0 -
Question about creating friendly URLs
I am working on creating new SEO friendly URLs for my company website. The products are the items with the highest search volume and each is very geo-specific
Technical SEO | | theLotter
There is not a high search volume for the geo-location associated with the product, but the searches we do get convert well. Do you think it is preferable to leave the location out of the URL or include it?0 -
How should i knows google to indexed my new pages ?
I have added many products in my ecommerce site but most of the google still not indexed yet. I already submitted sitemap a month ago but indexed process was very slow. Is there anyway to know the google to indexed my products or pages immediately. I can do ping but always doing ping is not the good idea. Any more suggestions ?
Technical SEO | | chandubaba1 -
Header Tag Question
While reviewing code on a site, I found the following: <h1 class="<a class="attribute-value">logo</a>"> <a id="<a class="attribute-value">logo</a>" href="[http://siteexampleh1.com](view-source:http://dmbinc.com/)"><span>Example of most important content on this page- Companyspan>a> h1> Is this the correct way to place code for an h1 tag? The content is cached within the page and is hidden to the viewer. The content that is assigned as the h1, is a logo. Majority of code I have been reviewing does not use this setup. The code would instead read as ( This is heading 1 ). Can anyone provide insights on this? Thanks!
Technical SEO | | jfeitlinger0 -
Spider Indexed Disallowed URLs
Hi there, In order to reduce the huge amount of duplicate content and titles for a cliënt, we have disallowed all spiders for some areas of the site in August via the robots.txt-file. This was followed by a huge decrease in errors in our SEOmoz crawl report, which, of course, made us satisfied. In the meanwhile, we haven't changed anything in the back-end, robots.txt-file, FTP, website or anything. But our crawl report came in this November and all of a sudden all the errors where back. We've checked the errors and noticed URLs that are definitly disallowed. The disallowment of these URLs is also verified by our Google Webmaster Tools, other robots.txt-checkers and when we search for a disallowed URL in Google, it says that it's blocked for spiders. Where did these errors came from? Was it the SEOmoz spider that broke our disallowment or something? You can see the drop and the increase in errors in the attached image. Thanks in advance. [](<a href=)" target="_blank">a> [](<a href=)" target="_blank">a> LAAFj.jpg
Technical SEO | | ooseoo0 -
Wordpress question
I was curious when i run an OSE report on certain websites and their name.wordpress.com shows up with a PA of whatever and a DA of 100. But when I created my wordpress site and post on it, it only has a PA and DA of 1. is this because SEOmoz has not indexed it yet? It is a month old. http://shiftinsurance.wordpress.com/ Can anyone help pls?
Technical SEO | | greasy0 -
SEO MOZ technical questions
Hi there, I would be very grateful if you can provide me with an explanation to the following so I understand it better - what do these heading mean? Domain Authority: (out of 100) Domain MozRank: Domain MozTrust: Total Links: Ext. Followed Links: Linking Root Domains: Followed Linking Root Domains: Linking C-Blocks: Thanks very much guys, much apprciated. Thanks Gareth
Technical SEO | | GAZ090