Correct use for Robots.txt
-
I'm in the process of building a website and am experimenting with some new pages. I don't want search engines to begin crawling the site yet. I would like to add the Robot.txt on my pages that I don't want them to crawl. If I do this, can I remove it later and get them to crawl those pages?
-
Lewis,
Thank you for the clarification!
-
Hi Eric
The guidance above means that Google when it looks to crawl your site won't its not a message to Google telling it never to come back.
Once everything is sorted, remove whichever approach you took to block the search engines and supply a sitemap to Google via the Webmaster tools. Your site should be crawled in no time after that.
Hope this helps.
-
Damian,
Thanks for your answer, that helps. If I add either one of the above items to my web page, and then remove it at a later date, will the search engines crawl and rank my site (at sometime after they are removed)? In other words, and I know this sounds stupid, but does a search engine see a Robots.txt file and never visit it again?
-
Hey Eric,
If you want to create and work on pages but you don't want them indexed you can add the following to the page in the section (the pages will still be crawled):
If you want NONE of your pages to be crawled (I.E the whole website) you can add the following to your robots.txt file:
User-agent: * Disallow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Anybody with a business site using Webflow CMS or another non-WordPress CMS/ sitebuilder?
If you have built your business site using Webflow CMS, kindly share your insights on how good their CMS is for SEO and how easy it is to rank a site. If you are not familiar with Webflow but have had a positive experience with another non-WordPress CMS/ sitebuilder, kindly share your opinions.
Web Design | | Blu_Ocean99.0 -
Block parent folder in robot.txt, but not children
Example: I want to block this URL (which shows up in Webmaster Tools as an error): http://www.siteurl.com/news/events-calendar/usa But not this: http://www.siteurl.com/news/events-calendar/usa/event-name
Web Design | | Zuken0 -
How to correct error in customized posttype WP site
Hi folks Can anybody help me. I foolishly, dogedly followed a Lynda.com tutorial for developing an 'online portfolio in WP'. Little did I know that my initial assumption - to use the 'twenty twelve' rather than the 'twenty eleven' theme would land me in such deep water. I was attempting to learn php on my own. All went well, until, --- the index page for the customized post type. Now I have two beautiful customized posttypes, 'companies' 'coverage' and no idea how to create an index page for either. I can't do the next step! I have tried every permutation - changing the permalink settings, changing them back, desperately searching for any handle to the nebulous links within the menu section. The only thing I can do (and have done for now) is to link the menu item 'company' and the menu item 'coverage' to a single post. Then the poor visitor has to scroll through the posts individually. I tried contacting the tutor and Lynda.com, to no avail! I have searched forums and found this is a common problem, but because I am so confused and novice to php they might as well be speaking Chinese. To compound my problems, looking through 'Wordpress SEO' for Yoast, I am painfully aware I can't go to the first basic step and fix the peramilinks to 'Postname' as that just makes my flakey menu collapse like a pack of cards. Help!
Web Design | | catherine-2793880 -
Image Replacement Using Cufon (Javascript), SEO effects
Hello Friends, I am using Cufon for image replacement in my website. May I know, does Cufon have any negative seo effects? Will this affect the search engine ranking or loading time of my website? What are the advantages and disadvantages of using cufon in a website in seo perspective?
Web Design | | zco_seo0 -
Nofollow links to resources used to save bandwidth?
I have a site on volusion, www.ecowindchimes.com. Until recently I was doing fairly well (top 3 for keyword(s) for 6 years) in serps. I was hit by the an update around july of last year, and did a full page redesign in november. My site has been losing ranking for its main keyword "wind chimes".
Web Design | | sbetzen
One change I noticed is that no-follow was removed (when the designers added a lightbox popup for the sound) from the many many links I have for my sound files of the wind chimes (I house them on a separate server to save bandwidth, which is expensive at volusion). The webaddress the sound-files are on doesn't even have a page... it is just there for the files. (there are ~100 files linked to on almost every page of the site where a product listing shows). Should I go through and no-follow all of these links again? Is that hurting me?
I suspect it is, but it is a lot of work for nothing if that is not the problem.0 -
What backup application to use?
I'm just looking for a backup application in Wordpress. I've found BackupWordpress y BackWPup. What of these ap. is the best in terms of performance and functionality? Is there any plugin better?Thanks in advanced.
Web Design | | aalcocer20030 -
Hey on some of my report cards its saying im not using rel canonical correctly how do i change this on my site?
on some of my report cards its saying certain things featured on my services page are actually linking to my blog or something. and its saying im not using rel canonical correctly. can you help me out?
Web Design | | ClearVisionDesign0 -
How to best correct cannibalization?
I apologize if this has already been answered, but after reading several posts on cannibalization, I can't seem to find what I am looking for. The site in question is www.urbanitystudios.com and in particular the term "western wedding invitation". We rank in the top 30 for this term in Google, but Google has indexed a particular product, versus our western wedding invitation collection page. The product that is indexed for this term: http://www.urbanitystudios.com/Designs/western-wedding-invitations-p-1527.html The page that we would rather be indexed: http://www.urbanitystudios.com/Designs/western-wedding-invitations-c-95_179_181.html After running an onpage report in SEOmoz tools for the collection page, we recieve an A grade, but get a warning on the cannibalization line item. As you can see, we name each product within that collection as "Western Wedding Invitation-x" (and have done this for other product categories...not good). After a good head slap, we realized that we are confusing Google as to what should be the main page. If we rename our products, the product's URL will change-Do we do a 301 for those products? If we rename our products, do we take out the words "Western Wedding Invitation" entirely or can we say "x-Western Wedding Invitation"? Or. because cannibalization is deemed a "low priority" in the reports, do we let things be and work on getting links to the collections page vs the individual product? Any insight would be most appreciated.
Web Design | | UrbanityStudios0