How to block search bots in crawling my site except for homepage?
-
Thanks in advance!
-
Robots.txt exclusion is definitely the easiest way to go. The URLs within the site might still be "indexed", but they will not be crawled and if they ever showed up in a search, they would look like something like this: http://i.imgur.com/xU6mDYA.png
-
You can use robots meta great guide here - http://moz.com/learn/seo/robotstxt
or if you wanted to be a bit different you can hide content behind a login screen or flash etc. things robots cant get through, I'd recommend the robots though
Hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Image Search - Is there a way to influence the related icons at the top of the image search results?
Google recently added related icons at the top of the image search results page. Some of the icons may be unrelated to the search. Are there any best practices to influence what is positioned in the related image icons section? Thank you.
Intermediate & Advanced SEO | | JaredBroussard1 -
Redirecting an Entire Site to a Page on Another Site?
So I have a site that I want to shut down http://vowrenewalsmaui.com and redirect to a dedicated Vow Renewals page I am making on this site here: https://simplemauiwedding.net. My main question is: I don't want to lose all the authority of the pages and if I just redirect the site using my domain registrar's 301 redirect it will only redirect the main URL not all of the supporting pages, to my knowledge. How do I not lose all the authority of the supporting pages and still shut down the site and close down my site builder? I know if I leave the site up I can redirect all of the individual pages to corresponding pages on the other site, but I want to be done with it. Just trying to figure out if there is a better way than I know of. The domain is hosted through GoDaddy.
Intermediate & Advanced SEO | | photoseo10 -
Search console site verification
I've been going on the assumption that when verifying a website in search console, it's always good to register and verify all variants of the site URL: http https www non-www However, if you create redirects to the preferred URL, is it really necessary to register/virfy of the other three? If so, why?
Intermediate & Advanced SEO | | muzzmoz0 -
Some site's links look different on google search. For example Games.com › Flash games › Decoration games How can we do our url's like this?
For example Games.com › Flash games › Decoration games How can we do our url's like this?
Intermediate & Advanced SEO | | lutfigunduz0 -
New site causes massive drop off in ranking, old site restored how long to recover?
Hello, We launched and updated version of our site, mainly design changes and some functionality. 3 days after the launch we vanished from the rankings, previous page one results were now out of the top 100. We have identified some of the issues with the new site and chose to restore the old well ranking site. My question is how long might it take for the ranking to come back, if at all? The drop happened on the third day and the site was restored on the third day. We are now on day 6. Using GWT with have used fetch as Google and resubmitted the site map. Any help would be gladly received. Thanks James
Intermediate & Advanced SEO | | JamesBryant0 -
Crawling issue
Hello, I am working on 3 weeks old new Magento website. On GWT, under index status >advanced, I can only see 1 crawl on the 4th day of launching and I don't see any numbers for indexed or blocked status. | Total indexed | Ever crawled | Blocked by robots | Removed |
Intermediate & Advanced SEO | | sedamiran
| 0 | 1 | 0 | 0 | I can see the traffic on Google Analytic and i can see the website on SERPS when i search for some of the keywords, i can see the links appear on Google but i don't see any numbers on GWT.. As far as I check there is no 'no index' or robot block issue but Google doesn't crawl the website for some reason. Any ideas why i cannot see any numbers for indexed or crawled status on GWT? Thanks Seda | | | | |
| | | | |0 -
Homepage bombed from rankings
I'm working on a site that has historically had issues ranking the homepage. We cleaned up some on page issues and then it went into a high and low pattern - page 4 then page 12 etc (was static around page 9 before), settling at page 6. The link profile was not good and there were a high level of links that should have been no-follow as they were clearly looking paid for - we addressed this along with some other poor links. This effectively dropped ranking down to page 23, but not unexpected considering the very big drop in followed links. Meanwhile we have embarked on a fresh steady link building strategy with nice clean links, varied anchor text coming from varying DA domains, smattered with a few no-follow links - strongly focussing on being as natural as possible. At the Penguin update the homepage has totally disappeared. Frustratingly just after the update (same day) we removed a 301ed old domain from the profile. This was the old company URL which we discovered had a lot of spam linking associated with it. An oversight - there were other 301 domains that were removed some time ago which were totally unrelated to the main site and we were told all other domains were simply bought and redirected to stop hijacking - all but this were. Considering the work we have done would it be good assumption this domain 301 could be the underlying factor? So far organic traffic is steady, in fact a tad up. What would you guys do?
Intermediate & Advanced SEO | | MickEdwards0 -
Site comparison - what is wrong with me?
www.bcspeakers.com/ vs www.psbspeakers.com/ with the search term "speakers" why does BC speakers show up in around #50-60 and PSB is not in the top #1000? From all metrics on seomoz PSB kicks BC in every area by a large margine! can anyone see why BC is listed for that keyword and PSB is not?
Intermediate & Advanced SEO | | kevin48030
