Adding the link masking directory to robots.txt?
-
Hey guys,
Just want to know if you have any experience with this.
Is it worthwhile blocking search engines from following the link masking directory.. (what i mean by this is the directory that holds the link redirectors to an affiliate site:
example:
mydomain.com/go/thislinkgoes to
amazon.com/affiliatelinkI want to know if blocking the 'go' directory from getting crawled in robots.txt is a good idea or a bad idea?
I am not using wordpress but rather a custom built php site where i need to manually decide on these things.
i want to specifically know if this in any way violates guidelines for google. it doesn't change the custom experience because they know exactly where they will end up if they click on the link.
any advice would be much appreciated.
-
Iredeto,
I do this on a few of my sites and it works out well. Saves on crawl budget, keeps google from accessing my affiliate links, and keeps any pagerank from passing through the links, which keeps me in-line with Google's webmaster policy on links.
One thing to keep in mind is that Google may show those URLs in the search results with a message that they can't show the content because it has been blocked. If you want to keep this from happening you may have to remove that directory GWT using the URL Removal tool. If they get re-indexed in 90 days (or whatever the reset time frame is) you will have to do it again. Hopefully that won't be an issue once you get them all removed the first time and block the folder.
Using rel = "nofollow" tags on the hyperlinks going to that directory wouldn't hurt either.
-
Hi!
I know cases where you can hide the contents of a folder to Google to not affect the rest of the website.
Note that you are facing a machine that knows how to crawl and index content.
Also be sure to block with noindex nofollow meta tags and removing URLs from Google index if at some point you had not locked.
Best regards!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Domain has been redirected our site; but many incoming links from sub domain. Will this hurts?
Hi all, This is the scenario: Our website is newwebsite.com. Our old website is oldwebsite.com which has been redirected to newwebsite.com (years back). But one of the old website's sub domain has a lot of back links to our current website like: seo.oldwebsite.com to newwebsite.com. Will this scenario hurts with any wrong linking? Thanks
Algorithm Updates | | vtmoz0 -
Linking from high ranking sub domain pages to less ranking main domain pages to benefit latter
Hi all, We have our product guide pages on sub domain which are years old, so have some backlinks and high ranking for the beand related queries. Now we created new guide pages on our main website and we want these new pages to rank top beating the old pages from sub domain. Again we can't deindex or rel canonical to solve the issue as there are some part of users still using the old pages. We are planning to give a link from every old page of sub domain to same new page on main domain. Will this linking increases the authority of new pages technically and helps in ranking better? Like we give a link to "Moz guide 1" page to "Moz guide 2" page to rank latter better. Thanks
Algorithm Updates | | vtmoz0 -
Flat Structure URL vs Structured Sub-directory URL
We are finally taking our classifieds site forward and moving into a much improved URL structure, however, there is some disagreement over whether to go with a Flat URL structure or a structured sub-directory. I've browsed all of the posts and Q&A's for this going back to 2011, and still don't feel like I have a real answer. Has anyone tested this yet, or is there any consensus over ranking? I am in a disagreement with another SEO manager about this for our proposed URL structure redesign who is for it because it is what our competitors are doing. Our classifieds are geographically based, and we group by state, county, and city. Most of our traffic comes from state and county based searches. We also would like to integrate categories into the URL for some of the major search terms we see. The disagreement arises around how to structure the site. I prefer the logical sub-directory style: [sitename]/[category]/[state]/[county]/
Algorithm Updates | | newspore
mysite.com/for-sale/california/kern-county/
or
[sitename]/[category]/[county]-county-[stateabb]/
mysite.com/for-sale/kern-county-ca/ I don't mind the second, except for when you look at it in the context of the whole site: Geo Landing Pages:
mysite.com/california/
mysite.com/los-angeles-ca-90210/ Actual Search Pages:
mysite.com/for-sale/orange-ca/[filters] Detail Pages:
mysite.com/widget-type/cool-product-name/productid I want to make sure this flat structure performs better before sacrificing my analytics sanity (and ordered logic). Any case studies, tests or real data around this would be most helpful, someone at Moz must've tackled this by now!0 -
After penguin 2.0, 20-25% drop sitewide, no google unatural links message, What could be causing it?
Hi,Since Penguin 2.0 we've taken a 20-25% knock but not recieved an unatural link message from Google. After sending a bunch of removal requests, I decided to submit a disavow file anyway two weeks ago and tried to make sure I rooted out some links that were built way back when our site started and link building best practice was a bit shadier. Analysis of our backlink profile points to about 40-50% links coming from general directories, wondering if perhaps their weight has been adjusted and this is why the drop occured? Having said that we have some high quality links from government sources and highly trusted sites so not too spammy. Can anyone shed some light or offer suggestions? Thanx
Algorithm Updates | | Mulith0 -
Do links count in syndicated content?
If I write a press release that goes viral and is syndicated all over do each of those links to my site in the syndications of the press release count and pass page rank with Google? Or does Google only count the link in the original press release? I heard that Google counts all the links for a time then eventually counts only one link from the original content and discounting all the other links as duplicate content. Any truth to this? Thanks mozzers! Ron10
Algorithm Updates | | Ron100 -
Doing Directory Submission are Worth Now a days ?
Hello Guys, **Doing directory submission these days are worth ? ** What are the best factors of link building ? Please suggest me link building strategies to rank my keywords well in search engines.
Algorithm Updates | | sumit600 -
Is building a WordPress plugin a bad link-building strategy now?
One strategy I was considering for a new site was developing a WordPress plugin that would have the side-effect of generating lots of back links. Given Google's recent over optimization update, this sounds like it could be a really bad idea. The nature of the plugin would be such that it would probably be used on very new blogs with low quality.
Algorithm Updates | | JonDiPietro0 -
FLASH vs HTML links in SEO
In terms of a small flash slideshow and having text and links on various slides within, is such text and links as easily index-able (or even at all) compared to static html text on a webpage?
Algorithm Updates | | heritageseo0