How to block search bots in crawling my site except for homepage?
-
Thanks in advance!
-
Robots.txt exclusion is definitely the easiest way to go. The URLs within the site might still be "indexed", but they will not be crawled and if they ever showed up in a search, they would look like something like this: http://i.imgur.com/xU6mDYA.png
-
You can use robots meta great guide here - http://moz.com/learn/seo/robotstxt
or if you wanted to be a bit different you can hide content behind a login screen or flash etc. things robots cant get through, I'd recommend the robots though
Hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Traffic drop on this site
I am SEO'ing this site but need some assistance in the analysis. it was doing not too bad but in the last 4 months the google traffic has really fallen off, i suspect the keywords may need improving but any tips or observations would be great.
Intermediate & Advanced SEO | | crowng0 -
Is there any problem if we migrate the entire site to HTTPS except for the blog ?
Hello guys,
Intermediate & Advanced SEO | | newrankbg
I have a question to those of you, who have migrated from HTTP to HTTPS. We are planning to migrate the site of our customer to Always SSL. In other words, we want to redirect all site pages to HTTPS, except for the blog. Currently, the whole site is using the HTTP protocol (except the checkout page).
After the change, our customer's site should look like this: https://www.domain.com
http://www.domain.com/blog/ The reasons we do not want to migrate the blog to HTTPS are as follows: The blog does not collect any sensitive user information, as opposed to the site. We all know that on-site algorithms like Panda are having sitewide effect. If the Panda doesn’t like part of the blog (if any thin or low quality content), we do not want this to reflect on the rankings of the entire website. Having in mind that for Google, HTTP and HTTPS are two different protocols, a possible blog penalty should not reflect the web site, which will use HTTPS. Point 2 is the reason I am writing here, as this is just a theory. I would like to hear more thoughts from the experts here. Also, I would like to know your opinion, regarding this mixed use of protocols – could this change lead to a negative effect for any of the properties and why? For me, there should be no negative effect at all. The only disadvantage is that we will have to monitor both metrics – the blog and the site separately in webmaster tools. Thank you all and looking forward for your comments.0 -
Regional and Global Site
We have numerous versions of what is basically the same site, that targets different countries, such as United States, United Kingdom, South Africa. These websites use Tlds to designate the region, for example, co.uk, co.za I believe this is sufficient (with a little help from Google Webmastertools) to convince the search engines what site is for what region. My question is how do we tell the search engines to send traffic from other regions besides the above to our global site, which would have a .com TLD. For example, we don't have a Brazilian site, how do we drive traffic from Brazil to our global .com site? Many thanks, Jason
Intermediate & Advanced SEO | | Clickmetrics0 -
Is this ok for content on our site?
We run a printing company and as an example the grey box (at the bottom of the page) is what we have on each page http://www.discountbannerprinting.co.uk/banners/vinyl-pvc-banners.html We used to use this but tried to get most of the content on the page, but we now want to add a bit more in-depth information to each page. The question i have is - would a 1200 word document be ok in there and not look bad to Google.
Intermediate & Advanced SEO | | BobAnderson0 -
Noindex Mobile Site?
So I wanted to get everyone's opinion. Have a client in online retail on ASP and their developers built a mobile site a while back before we took the client on. For the sake of this post, just assume, resources are limited anddevelopers are not good (constantly break things we request to get fixed). They never installed analytics on the mobile site, so all I have to go off of is referral data on the main stores GA account for m.example.com However if I look to see what is indexed by doing site:m.example.com am not seeing many pages. The mobile site has a ton of internal links in GWT and am questioning its negative impact as there are no canonicals, no mobile sitemap present. In the ideal world, I would implement proper Mobile SEO practices but given the resources of no dev budget and devs not being good, I was thinking about noindexing the mobile site since I can RDP into the site and access robots. Thoughts?
Intermediate & Advanced SEO | | Sean_Dawes0 -
Search Spike over the weekend
I own a website, and recently had a HUGE spike in organic search traffic from Google over the weekend. It seems to be going back down. Anyone have any idea why? Anyone have any similar experiences? odd-traffic.png
Intermediate & Advanced SEO | | alhallinan1 -
Is it good to change an ecommerce site homepage everyday?
my website is www.theprinterdepo.com but now my boss says that it should change everyday, its en ecommerce site, that the products should change everyday, even some times more than day. Is it good for seo? taking into account that google does not crawl all sites every day I suppose.
Intermediate & Advanced SEO | | levalencia10 -
ReLaunching a very old site
Hi, I am in the process of re-vamping a website that hasn't been touched for years and whose rankings slowly dropped. Any best practice in how to do it making sure that there's not any more loss and - hopefully - it could go back to the old glory? The website is http://www.nlp-world.com Thanks
Intermediate & Advanced SEO | | pdmonline0