Correct use for Robots.txt
-
I'm in the process of building a website and am experimenting with some new pages. I don't want search engines to begin crawling the site yet. I would like to add the Robot.txt on my pages that I don't want them to crawl. If I do this, can I remove it later and get them to crawl those pages?
-
Lewis,
Thank you for the clarification!
-
Hi Eric
The guidance above means that Google when it looks to crawl your site won't its not a message to Google telling it never to come back.
Once everything is sorted, remove whichever approach you took to block the search engines and supply a sitemap to Google via the Webmaster tools. Your site should be crawled in no time after that.
Hope this helps.
-
Damian,
Thanks for your answer, that helps. If I add either one of the above items to my web page, and then remove it at a later date, will the search engines crawl and rank my site (at sometime after they are removed)? In other words, and I know this sounds stupid, but does a search engine see a Robots.txt file and never visit it again?
-
Hey Eric,
If you want to create and work on pages but you don't want them indexed you can add the following to the page in the section (the pages will still be crawled):
If you want NONE of your pages to be crawled (I.E the whole website) you can add the following to your robots.txt file:
User-agent: * Disallow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Multiple sites using same text - how to avoid Google duplicate content penalty?
Hi Mozers, my client located in Colorado is opening a similar (but not identical) clinic in California. Will Google penalize the new California site if we use text from our website that features his Colorado office? He runs the clinic in CO and will be a partner of the clinic in CA, so the CA clinic has his "permission" to use his original text. Eventually he hopes to go national, with multiple sites utilizing essentially the same text. Will Google penalize the new CA site for plagiarism and/or duplicate content? Or is there a way to tell Google, "hey Google, this new clinic is not ripping off my text"?
Web Design | | CalamityJane770 -
Moving servers which means moving ip address but using the same URL. Would it harm the website's SEO?
Hello everyone, The server (in-house) which we use to host our website is a bit old. We are using CDN77 for our static content. What if I move all our website to the CDN service? meaning I use their storage capability and just have our url point to the IP address they provide. Would that hurt our rankings?
Web Design | | Edgar-Cerecerez0 -
Using More Info javascript:toggleDisplay tag for More info text
Is there any harm in using javascript so a user can "toggle" open or closed additional text on a website? For example, if a user wants to read more about something, they can click on "More Info" and the text would then appear. Google is able to read the text, because I chose a random 8 word section of the text within the More Info and pasted it into a Google Search and the website showed up in search results. Just wondering if using this technique would have any negative impact. Here's what the code would look like:
Web Design | | EEE3
<a <span="">title</a><a <span="">="Show Tables" href="</a><a class=" " target="_blank">javascript:toggleDisplay('table1')</a>">More Info style="display: none;" id="table1"> this is where the text would be, and from this section was where I grabbed text to search with in google. Then in the footer, here is the script needed so the more info will work: I am by no means an expert in coding/html/javascript. Thanks!0 -
How does using a CMS (i.e. Wordpress/Drupal) affect backlinks and SEO?
So I need to build a website with over 100 pages in it. Elements of the design will probably be moved around and or tested so I need to use a CMS. It's pretty much a review site so while the content will remain static I'd like to employ A/B testing to mess with conversion rates. Wordpress has a plugin for that even. So I'm just wondering, since CMS pages are pretty much created on spot and not retrieved from a library, how this affects backlinks and anchor text? How exactly does the external website point to yours if the URL is dynamically generated? Or am I misunderstanding something? Please recommend any extra resources as well if you can.
Web Design | | seochump0 -
Anyone used bugherd.com for onsite seo purposes?
Just as the title says, has anyone used bugherd.com for SEO purposes? I was thinking it could be used to show client changes that need to be made regarding the website. Example could be if you are looking at a CRO prospective, you may want to change/add some graphics or text to improve conversions. It seems like a nifty tool to show the changes you want made and to keep track of them. It integrates with basecamp also 🙂
Web Design | | KyleChamp0 -
How to best correct cannibalization?
I apologize if this has already been answered, but after reading several posts on cannibalization, I can't seem to find what I am looking for. The site in question is www.urbanitystudios.com and in particular the term "western wedding invitation". We rank in the top 30 for this term in Google, but Google has indexed a particular product, versus our western wedding invitation collection page. The product that is indexed for this term: http://www.urbanitystudios.com/Designs/western-wedding-invitations-p-1527.html The page that we would rather be indexed: http://www.urbanitystudios.com/Designs/western-wedding-invitations-c-95_179_181.html After running an onpage report in SEOmoz tools for the collection page, we recieve an A grade, but get a warning on the cannibalization line item. As you can see, we name each product within that collection as "Western Wedding Invitation-x" (and have done this for other product categories...not good). After a good head slap, we realized that we are confusing Google as to what should be the main page. If we rename our products, the product's URL will change-Do we do a 301 for those products? If we rename our products, do we take out the words "Western Wedding Invitation" entirely or can we say "x-Western Wedding Invitation"? Or. because cannibalization is deemed a "low priority" in the reports, do we let things be and work on getting links to the collections page vs the individual product? Any insight would be most appreciated.
Web Design | | UrbanityStudios0 -
Using "#" anchors to display different content
If I have a page that has an area on the page that acts like a widget and has three different tabs. These tabs provide 3 different types of information relevant to the page subject matter. By default when someone goes to the page one of the tabs is showing but you have to click on the others to see the info on them. Is it OK to use domain.com/topic#TAB1, domain.com/topic#TAB2, domain.com/topic#TAB3 to create shortcut links so that people can land on the page and have that predetermined tab showing. I'm wondering what search engines might think. Essentially all the content of all three tabs is there for people to see but they'd have to click to see the other tabs. I don't consider the content to be hidden. But I'd like to hear people's thoughts.
Web Design | | Business.com0
