Google Indexing Development Site Despite Robots.txt Block
-
Hi,
A development site that has been set-up has the following Robots.txt file:
User-agent: *
Disallow: /
In an attempt to block Google indexing the site, however this isn't the case and the development site has since been indexed.
Any clues why this is or what I could do to resolve it?
Thanks!
-
Hi so I'm assuming your on IIS (I'm no expert on ISS I think you will need to configure the web.config) and I'm just going to step back now and get my coat as I only have experience with Apache
-
Thanks for your help! Much appreciated
-
It's generally best to noindex/nofollow using the meta robots tag in the header. If it's not too much of a stretch for you, you can also password protect the test site. The over-so-lovely and charming Googles will still display results blocked by robots.txt - though it won't generally cache the content. If you would like, you can hookup the test site with Webmaster Tools and remove the URL(s) from the index.
-
Its my understanding that htaccess is PHP based and as we code in .net we don't have a htaccess file.
Do you know of this this happening before because its not something that I've heard of.
-
You would need to block access via htaccess rather than robots file as the robots.txt is only advisory
If you are using wordpress I use this simple plugin JF3 Maintenance Redirect
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sudden site drop google, not banned or penalised?
Hi all, We've been working on our site weldingmart.com for a while. 4 weeks ago we got a sudden drop from google rankings even with our own brand name. No clear cause found, and decided to walk through all technicalities of the SEO fundament. thus we did the following
Technical SEO | | jkossel
-> Setup google webmasters tools
no issues found, a few 404's, few 410's sure thats all ok,
-> Setup robots.txt to only index homepages, lister pages, content and product detail pages (disabled all filters and search queries) Also we banned russian and spammy bots for performance-sake.
-> Added sitemaps, and around 14k pages seemed to be already indexed.
-> When searching for "site:weldingmart.com" i can find 14k pages.
-> we had a low 33/100 page speed score and improved this to 76/100 So we did a lot of clean up and improved a lot of items. but still 2-weeks in. we still have no ranking improvements. Before we went down we had around 100 clicks a day from google. now 5 avg. by the way i think a main issue is the low link count of course but still googling your own name should return us in top3 right. Is there something we are missing, do we need more time. I just want to verify that we do not mis anything!1 -
New SEO manager needs help! Currently only about 15% of our live sitemap (~4 million url e-commerce site) is actually indexed in Google. What are best practices sitemaps for big sites with a lot of changing content?
In Google Search console 4,218,017 URLs submitted 402,035 URLs indexed what is the best way to troubleshoot? What is best guidance for sitemap indexation of large sites with a lot of changing content? view?usp=sharing
Technical SEO | | Hamish_TM1 -
2 sitemaps on my robots.txt?
Hi, I thought that I just could link one sitemap from my site's robots.txt but... I may be wrong. So, I need to confirm if this kind of implementation is right or wrong: robots.txt for Magento Community and Enterprise ...
Technical SEO | | Webicultors
Sitemap: http://www.mysite.es/media/sitemap/es.xml
Sitemap: http://www.mysite.pt/media/sitemap/pt.xml Thanks in advance,0 -
Google Seeing Way More Pages Than My Site Actually Has
For one of my sites, A-1 Scuba Diving And Snorkeling Adventures, Google is seeing way more pages than I actually have. It sees almost 550 pages but I only have about 50 pages in my XML. I am sure this is an error on my part. Here is the search results that show all my pages. Can anyone give me some guidance on what I did wrong. Is it a canonical url problem, a redirect problem or something else. Built on Wordpress. Thanks in advance for any help you can give. I just want to make sure I am delivering everything I can for the client.
Technical SEO | | InfinityTechnologySolutions0 -
Why seomoz.org still in Google index?
I searched in Google, the number of URLs indexed left in the seomoz.org domain since it changed to moz.comI am surprised that after all this time more than 15,000 URLs indexed:https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=site%3Aseomoz.org%20inurl%3Aseomoz.org If I clicked on any of the results it will be redirect (301) to the new domain, so it is working, but Google still keep these URLs in the index.
Technical SEO | | Yosef
What could be the reason?Will not cause duplicated content issue on moz.com?0 -
Domain recommendations for non indexed sites
For complicated internal reasons (which I swear have validity) we are starting websites that will then disallow from Google for ~1 year. Is there any value in purchasing "premium" domains with SEO juice or will we lose the value in the time we are hiding the website from Google?
Technical SEO | | theLotter0 -
Google sitemap just for a part of site?
Hi, I am about reorganize (content and seo-wise) a part of a larger site and I wondered if it is possible to use a Google sitemap just for some but not all pages of a site? Does anyone know if this has any impact on pages that are not included in the sitemap? Thanks
Technical SEO | | haest0 -
Is it a good idea to make 301 from a site which you know google has banned certain keywords for to a new site with similar content
Here is a short question re. 301. I read Dovers article on how to move an old domain to a new one. Say you have been a little inexperienced regarding linkbuilding and used some cheap service in the past and you have steadily seen that certain keywords have been depreciating in the SERP - however the PR is still 3 for the domain - now the qustion is should you rediect with a 301 in .htaccess to a new domain when you know that google does not like certain keywords with respect to the old site. Will the doom and gloom carry over to the new site?
Technical SEO | | Kofoed0