Best way to block a search engine from crawling a link?
-
If we have one page on our site that is is only linked to by one other page, what is the best way to block crawler access to that page?
I know we could set the link to "nofollow" and that would prevent the crawler from passing any authority, and we can set the page to "noindex" to prevent it from appearing in search results, but what is the best way to prevent the crawler from accessing that one link?
-
Hi there,
I'm assuming you are trying to do pagerank sculpting (or something related..) - which was made a little more tough in recent years. I'll base my answer around this assumption, so feel free to correct me if this isn't the case.
There are several methods to make a link uncrawlable:
- AJAX - Googlebot will not read any calls through AJAX. If you can load your link through an external call, it would be completely hidden.
- Javascript - Obfuscate links with Javascript that masks the link. You can do any number of solutions here, including using tags with a title of your URL, which upon clicking, goes that that URL. Simple and effective.
- Redirects - I haven't tested this last idea, and it may not work. You might be able to redirect to another page in your website, which is then set to not be indexed. Then redirect to the intended page through a query string. In theory it should work, but obviously not as good as the previous methods I described.
Let me know if you have questions. I'd be glad to help further.
Cheers!
-
Noindex/nofollow should be good enough, but if you want to be sure it doesn't get indexed, you could can also include <meta name="robots" content="NOINDEX, NOFOLLOW"> in the head section of the page to be blocked. You can also exclude the page in your robots.txt file. </meta name="robots">
You can find a simple robots.txt generator in Google Webmaster Tools if you need to block particular pages or directories. The robots.txt file should be in the root directory of your site and look something like this:
User-agent: * Disallow: /file-you-want-to-hide.html
You can also request removal of specific URLs in Webmaster Tools if it has already been indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Internal Linking
Hi I've been looking over my pages and it says for this page for example http://www.key.co.uk/en/key/1-6kw-halogen-heater I have too many links, I think it was about 178. These links are from the menu and bottom of the page - how much of an issue is this for internal linking structure? I wouldn't want to remove the menus or change them too much. Thank you!
Intermediate & Advanced SEO | | BeckyKey0 -
Getting too many links on Google search results, how do I fix?
I'm a total newbie so I apologize for what I am sure is a dumb question — I recently followed Moz suggestions for increasing visibility on my site for a specific keyword by including that keyword in more verbose page descriptions for multiple pages. This worked TOO well as now that keyword is bringing up too many results in Google for these different pages on my site . . . is there a way to compile them into one result with the subpages like for instance, the attached image for a search on Apple? Do I need to change something in my robots.txt file to direct these to my main page? Basically, I am a photographer and a search for my name now brings up each of my different photo gallery pages in multiple results, it's a little over the top. Thanks for any and all help! CNPJZgb
Intermediate & Advanced SEO | | jason54540 -
Block web archieve/way back machine
Hi i want to block web archive/wayback machine from indexing my site and creating a record of it on their database. Any ideas on how to do this? Cheers,
Intermediate & Advanced SEO | | Mikey008
Superpak2 -
We are switching our CMS local pages from a subdomain approach to a subfolder approach. What's the best way to handle this? Should we redirect every local subdomain page to its new subfolder page?
We are looking to create a new subfolder approach within our website versus our current subdomain approach. How should we go about handling this politely as to not lose everything we've worked on up to this point using the subdomain approach? Do we need to redirect every subdomain URL to the new subfolder page? Our current local pages subdomain set up: stores.websitename.com How we plan on adding our new local subfolder set-up: websitename.com/stores/state/city/storelocation Any and all help is appreciated.
Intermediate & Advanced SEO | | SEO.CIC0 -
Best way of connecting with key influencers
Hi, I've identified a list of key influencers within my niche. I now want to connect with them, specifically to ask them to provide regular content for our site (they'll hopefully promote it via their own channels as well as us doing some promotion). The reason I want to do this is because it's easier to move Mohammed than the mountain! - I want to piggy-back on their success! How would you go about doing this? Some of the influencers are follwing us on Twitter and I've had some nice discussions with them over the last few months or so (our Twitter has only been going since about March this year). Would you reach out via Twitter, or call them up? I really don't want to mess this up! Thanks, Amelia
Intermediate & Advanced SEO | | CommT1 -
Best practice to disavow spammy links
Hi Forum, I'm trying to quantify the logic for removing spammy links.
Intermediate & Advanced SEO | | Mark_Ch
I've read the article: http://www.seomoz.org/blog/how-to-check-which-links-can-harm-your-sites-rankings. Based on my pivot chart results, I see around 55% of my backlinks at zero pagerank. Q: Should I simply remove all zero page rank links or carry out an assessment based on the links (zero pagerank) DA / PA. If so what are sensible DA and/or PA metrics? Q: What other factors should be taken into consideration, such as anchor text etc.0 -
Should product searches (on site searches) be noindex?
We have a large new site that is suffering from a sitewide panda like penalty. The site has 200k pages indexed by Google. Lots of category and sub category page content and about 25% of the product pages have unique content hand written (vs the other pages using copied content). So it seems our site is labeled as thin. I'm wondering about using noindex paramaters for the internal site search. We have a canonical tag on search results pointing to domain.com/search/ (client thought that would help) but I'm wondering if we need to just no index all the product search results. Thoughts?
Intermediate & Advanced SEO | | iAnalyst.com0 -
Footer Link
Hello, Some of my hosted clients don't mind if I put a footer link on the bottom of their website. I would like to put a footer link that looks like Seomoz's - http://imgur.com/GrC8y Basically it would look like so: "Powered by "my company name". The world's #1 "keyword" provider (LOGO goes here) Here are my questions: 1. Would this hurt or help my rankings? 2. Should the logo be hosted by my clients so that a different ip is hosting my logo (where my image name will get picked up)? Or is it best to host it myself? 3. If my company name and keyword are getting linked, is that one link too many? 4. Is it a good idea to use a different keyword so that other keywords get picked up by SERPs, or should I set myself up on one keyword ? Thank you so much! Shawn
Intermediate & Advanced SEO | | Shawn1240