How to block search bots in crawling my site except for homepage?
-
Thanks in advance!
-
Robots.txt exclusion is definitely the easiest way to go. The URLs within the site might still be "indexed", but they will not be crawled and if they ever showed up in a search, they would look like something like this: http://i.imgur.com/xU6mDYA.png
-
You can use robots meta great guide here - http://moz.com/learn/seo/robotstxt
or if you wanted to be a bit different you can hide content behind a login screen or flash etc. things robots cant get through, I'd recommend the robots though
Hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Mobile site crawl returns poorer results on 100% responsive site
Has anyone experienced an issue where Google Mobile site crawl returns poorer results than their Desktop site crawl on a 100% responsive website that passes all Google Mobile tests?
Intermediate & Advanced SEO | | MFCommunications0 -
How to optimise for voice search ?
Hello, Let's imagine I do a search for barcelona in the keyword tool. One the the words that comes back with high relevance is barcelona city. I am writing a my content but as a human it sounds more natural to write the city of barcelona. My problem is the the city of barcelona is not given by the keyword explorer, what should I do stick with barcelona city use the city of barcelona even though it is not in the list of keywords ? Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Old site penalised, we moved: Shall we cut loose from the old site. It's curently 301 to new site.
Hi, We had a site with many bad links pointing to it (.co.uk). It was knocked from the SERPS. We tried to manually ask webmasters to remove links.Then submitted a Disavow and a recon request. We have since moved the site to a new URL (.com) about a year ago. As the company needed it's customer to find them still. We 301 redirected the .co.uk to the .com There are still lots of bad links pointing to the .co.uk. The questions are: #1 Do we stop the 301 redirect from .co.uk to .com now? The .co.uk is not showing in the rankings. We could have a basic holding page on the .co.uk with 'we have moved' (No link). Or just switch it off. #2 If we keep the .co.uk 301 to the .com, shall we upload disavow to .com webmasters tools or .co.uk webmasters tools. I ask this because someone else had uploaded the .co.uk's disavow list of spam links to the .com webmasters tools. Is this bad? Thanks in advance for any advise or insight!
Intermediate & Advanced SEO | | SolveWebMedia0 -
Only homepage is ranking after site re-launch
We've been moving all our sites over to a new platform (Demandware) this year. In the process, they've all gotten updated designs (from the same template), on-page optimizations, etc. Since they're all on the same platform and are essentially copies from one template, any technical issues found have been fixed across all sites. The problem I'm seeing is there are a few sites that haven't really seen much/any recovery from the site launch, and these are sites that were done 4-5 months ago. There's one in particular that's especially concerning, since it's showing issues that none of the other sites seem to have. In my Moz reports, it looks like of all the keywords that are ranking, they're only ranking the https version of the homepage (and from what I'm seeing, the https version wasn't picked up and ranked until the beginning of October, which was also the time that WMT shows a huge drop in clicks and impressions). I've crawled the site (ScreamingFrog), done a site search in Google (all pages look to be indexed), etc. and I haven't come across any specific problems there that would suggest a technical issue. We're wondering if it might be a link authority problem, since this site had the most dramatic change in navigation. The navigation used to be product based (Boots, Shoes, etc.) and is now broken up by gender. I've noticed that a few other pages that are ranking are dual gender pages that also existed on the old site, whereas all of these new categories aren't ranking at all and I'm not seeing this happen with any of our other sites. I've gone down a bunch of different paths trying to figure this out, but I haven't come up with any concrete answers as to why this is happening and how to fix it. Any thoughts as to what else I can look into or try for this?
Intermediate & Advanced SEO | | WWWSEO0 -
Site dropped after recovery
Hi everybody! I've been working for http://www.newyoubootcamp.com for some time now. They came to me as they had dropped heavily for their main term, "boot camp". This turned out to be due to a manual penalty, which was in part due to their forum being hacked, as well as some bad link building. Here's an example of the dodgy forum links - http://about1.typepad.com/blog/2014/04/tweetdeck-to-launch-as-html5-web-app-now-accepting-beta-testers.html. The anchor is "microsoft". They've all been 410'd now. Also, we cleaned up the other bad links as best we could, and got through the manual penalty. The site then returned to #5 for "boot camps", below its pre-crash peak of #2, but OK. Over the past few weeks, it has started to slide though. I'm certain it is not down to a lack of quality links - this site has great PR and links from national newspapers and magazines. There's been a few on-site issues too, but nothing outrageous. I'm getting a bit stumped though, and any fresh eyes would be much appreciated!
Intermediate & Advanced SEO | | Blink-SEO0 -
Why is my site not getting crawled by google?
Hi Moz Community, I have an escort directory website that is built out of ajax. We basically followed all the recommendations like implementing the escaped fragment code so Google would be able to see the content. Problem is whenever I submit my sitemap on Google webmastertool it always 700 had been submitted and only 12 static pages had been indexed. I did the site query and only a number of pages where indexed. Does it have anything to do with my site being on HTTPS and not on HTTP? My site is under HTTPS and all my content is ajax based. Thanks
Intermediate & Advanced SEO | | en-gageinc0 -
Robots.txt Blocked Most Site URLs Because of Canonical
Had a bit of a "Gotcha" in Magento. We had Yoast Canonical Links extension which worked well , but then we installed Mageworx SEO Suite.. which broke Canonical Links. Unfortunately it started putting www.mysite.com/catalog/product/view/id/516/ as the Canonical Link - and all URLs with /catalog/productview/* is blocked in Robots.txt So unfortunately We told Google that the correct page is also a blocked page. they haven't been removed as far as I can see but traffic has certainly dropped. We have also , at the same time had some Site changes grouping some pages & having 301 redirects. Resubmitted site map & did a fetch as google. Any other ideas? And Idea how long it will take to become unblocked?
Intermediate & Advanced SEO | | s_EOgi_Bear0 -
Are Links from Homepage safe?
I have the option of getting a link from 2 websites home pages. My site is a financial service website. I want to know if it is safe to do so. The first site is a hotel site, so their content has no relation to my site And the second is a Google News website about finances They have offered me a section in the sidebar to have a sentence or 2 about my site with a link. Can you tell me would these be safe to get? I don't want to be hit by any penalty
Intermediate & Advanced SEO | | JohnPeters0