Correct use for Robots.txt
-
I'm in the process of building a website and am experimenting with some new pages. I don't want search engines to begin crawling the site yet. I would like to add the Robot.txt on my pages that I don't want them to crawl. If I do this, can I remove it later and get them to crawl those pages?
-
Lewis,
Thank you for the clarification!
-
Hi Eric
The guidance above means that Google when it looks to crawl your site won't its not a message to Google telling it never to come back.
Once everything is sorted, remove whichever approach you took to block the search engines and supply a sitemap to Google via the Webmaster tools. Your site should be crawled in no time after that.
Hope this helps.
-
Damian,
Thanks for your answer, that helps. If I add either one of the above items to my web page, and then remove it at a later date, will the search engines crawl and rank my site (at sometime after they are removed)? In other words, and I know this sounds stupid, but does a search engine see a Robots.txt file and never visit it again?
-
Hey Eric,
If you want to create and work on pages but you don't want them indexed you can add the following to the page in the section (the pages will still be crawled):
If you want NONE of your pages to be crawled (I.E the whole website) you can add the following to your robots.txt file:
User-agent: * Disallow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does a similar CMS fabric/theme used by 2 colleagues, cause SEO issues?
Does using the same CMS fabric/theme with substantially different content for each website but the same business address (on google places and the websites) cause SEO issues? For example: 2 colleagues with somewhat similar services located at the same business addresses desire to use the same CMS fabric /theme but will have different content on the sites. Will this hurt their SEO / should they use a different website skin/theme?
Web Design | | toti5880 -
When rel canonical tag used, which page does Google considers for ranking and indexing? A/B test scenario!
Hi Moz community, We have redesigned our website and launched for A/B testing using canonical tags from old website to new website pages, so there will be no duplicate content issues and new website will be shown to the half of the website visitors successfully to calculate the metrics. However I wonder how actually Google considers it? Which pages Google will crawl and index to consider for ranking? Please share your views on this for better optimisation. Thanks
Web Design | | vtmoz0 -
Why use a wwwP subdomain naming convention
While working through a series of crawl reports and competitive insights for a site, I noticed one of the competitors had switched from a WWW-version to a wwwP-version. Looking back at the snapshot I took of this during the same time period in 2014, I noticed a significant drop in PA/DA by 20+/-. I'm curious to know if anybody else has experienced something similar, and if anybody can provide insights on why a change like this would even be made? I'll preface it with, everything we could see that this competitor was doing from the outside, was legitimate and propelling them in a positive direction.
Web Design | | dodgejd0 -
Duplicate Title Issues using # anchor tags
Our homepage navigation uses anchor tags (?TabNumb=1#, ?TabNumb=1# etc) rather than directly linking to different pages to decrease load time (and simplify the build process I owuld imagine). These anchor links are showing up as duplicate titles in Moz. I am pretty sure if I were to use noindex or rel tags, that could have a negative affect on my search results. Any way to tackle this outside of a complete redesign of the structure? http://www.dedoose.com/about-us/?TabNum=2# as an example
Web Design | | sbnjl0 -
How to know if a wordpress theme is coded correctly for Seo
Hi, So I am curious if there is a tool to see if a site is coed properly for Google? I am running Avada, a standalone theme, yet I am also using a cache plugin. But when I search my code, its all like on one huge line. So I am curious if there is a way to verify or check if a theme is coded correctly? Thank you
Web Design | | Berner1 -
Does IE9 use doucument mode IE9 Standards or Quirks???
Trying to set up the styling on a new Shopping Cart, and having trouble with Internet Explorer (IE) When I view it in IE, I click on F12 to identify the code viewer (like FireBug) and it lets me choose to view it using the browser IE9 or earlier. So far, so good... But right next to the browser selector is the "Document Mode" selector and sometimes it says "Quirks Mode" and sometimes it says "IE9 Standards" (when using IE9 browser) - and parts of the page styling goes crazy when I switch back and forth from Quirks to Standards. Help! Which does IE9 and IE8 actually use? Does it have to work for both? Is there a way to force it to one or the other? Any guidance or tips is appreciated. Thanks, Greg
Web Design | | GregB1230 -
Using (duplicate) content in different contexts
I have three distinct hosting products, each solving three different problems. While these three products have different features in terms of functionality, they are all built on the same platform. Now, in terms of marketing some features of the platform, f.ex. High Availability, is significant to all of the products. How do I go about to include information about this feature on all product pages without getting penalized for duplicate content? Is there a way to tell Google that parts of the content on the pages for product 1-3 is duplicated with intent, or duplicated from f.ex. a page that explains the technical aspects of the platform?
Web Design | | SYSE0 -
Is anyone here managing or doing SEO for a site using GoECart?
We are preparing to update/migrate to a new ecommerce platform. We are in the process of choosing right now. One of the things we know we want is faceted navigation, but I am well aware of the problems this presents for SEO. Are any of you amazing people here using, managing or have experience with GoECart? I am interested to know your feedback, particularly from an SEO viewpoint. Thanks in advance! Dana
Web Design | | danatanseo0