Ranking for our member's company names without giving them all away!
-
Hi,
We have a directory of 25,000 odd companies who use our site.
We have a strong PR site and want to rank a page for each company name. Some initial testing on one or two company names brings us to #2 after the company's own web site in the format: "Company Name Reviews and Feedback" - so it works well.
We want to do this for all 25,000 of our members, however we do not wish to make it easy for our competitors to scrape through our member database!!
e.g. using: www.ourdomain.com/randomstring/company-name-(profile).php
unfortunately with the above performing a search on google for site:domain.com/()/()(profile).php
would bring up all records.
Are there any tried and tested ways of achieving what we're after here?
Many Thanks.
-
Bottom line, you cannot make data available online without offering a means for a user to grab that data.
You said you "don't wish to make it easy" so I will share some ideas:
-
EGOL's suggestion is good and not that hard to implement. I am not sure if your site requires registration but you can set it up so guests can view a maximum of ?20 member pages or whatever amount you deem to be a reasonable number.
-
There are more complicated methods by which you can establish a script that will block any IP or user who pulls too many pages too quickly.
-
The real challenge is your sitemap. If all that is required is the company's name, your sitemap is all someone needs. In this case there is simply nothing I can think of you can do.
-
If the sitemap isn't a challenge, another idea is to present the data in a method that is not easy to read. You can leave the description information in HTML but present the company name in Flash, for example.
Bottom line, if you want to rank well, the site has to be easy to crawl. If the crawl data offers enough information for others to steal, there is simply no reasonable method that can be used to prevent automated tools from grabbing it.
-
-
If you have links into all of these 25000 pages then people and robots will be able to find them.
If you want to keep robot scrapers out then you can use a white list of robots that allows the robots of search engines and other allowed automated visitors in but instructs others to keep out. No guarantees that they will not find a way in with this but it might help. Human scraping will still get through.
You could probably also devise a way to throttle the number of pageviews per visitor or per IP but that would take some creative programming.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Re: Inbound Links. Whether it's HTTP or HTTPS, does it still go towards the same inbound link count?
Re: Inbound Links. If another website links to my website, does it make a difference to my inbound link count if they use http or https? Basically, my site http://mysite.com redirects to https://mysite.com, so if another website uses the link http://mysite.com, will https://mysite.com still benefit from the inbound links count? I'm unsure if I should reach out to all my inbound links to tell them to use my https URL instead...which would be rather time consuming so just checking http and https counts all the same. Thanks.
Intermediate & Advanced SEO | | premieresales0 -
Help, no organic traffic recovery after new site launch (it's been 6 months)!
I worked with a team of developers to launch a new site back in March. I was (and still am) in charge of SEO for the site, including combining 4 sites into 1. I made sure 301 redirects were in place to combine the sites and pretty much every SEO tactic I can think of to make sure the site would maintain rankings following launch. However, here we are 6 months later and YoY numbers are down -70% on average for organic traffic. Anyone mind taking a look at http://www.guestguidepublications.com and seeing if there's a glaring mistake I'm missing?!?!?! Thanks ahead of time!
Intermediate & Advanced SEO | | Annapurna-Digital1 -
Something happened within the last 2 weeks on our WordPress-hosted site that created "duplicates" by counting www.company.com/example and company.com/example (without the 'www.') as separate pages. Any idea what could have happened, and how to fix it?
Our website is running through WordPress. We've been running Moz for over a month now. Only recently, within the past 2 weeks, have we been alerted to over 100 duplicate pages. It appears something happened that created a duplicate of every single page on our site; "www.company.com/example" and "company.com/example." Again, according to our MOZ, this is a recent issue. I'm almost certain that prior to a couple of weeks ago, there existed both forms of the URL that directed to the same page without be counting as a duplicate. Thanks for you help!
Intermediate & Advanced SEO | | wzimmer0 -
Silo Architecture - need an expert's advice
I understand the concept of silo architecture. What I don't understand is how to build the site navigation. I see experts talking about silos, but their sites have pervasive top level navigation. In theory, your top level nav breaks your silos. If I have 20 pages of supporting content all linked to my silo page, and the top nav is on the supporting content pages, then those pages all link to the pages in the top nav - silo broken, and link juice diluted. it would seem to me that the only way to build a true silo is to strip out all of the navigation on a supporting page, and only have it link to: 1. The silo landing page 2. Other supporting pages in the silo. is this what Bruce Clay does? I've seen Rand's lectures on silos as well. Is this what he is doing? I recently saw a video by the Network Empire team, and they'd also have a pervasive nav. Can someone please explain this?
Intermediate & Advanced SEO | | CsmBill0 -
Wordpress.com content feeding into site's subdomain, who gets SEO credit?
I have a client who had created a Wordpress.com (not Wordpress.org) blog, and feeds blog posts into a subdomain blog.client-site.com. My understanding was that in terms of SEO, Wordpress.com would still get the credit for these posts, and not the client, but I'm seeing conflicting information. All of the posts are set with permalinks on the client's site, such as blog.client-site.com/name-of-post, and when I run a Google site:search query, all of those individual posts appear in the Google search listings for the client's domain. Also, I've run a marketing.grader.com report, and these same results are seen. Looking at the source code on the page, however, I see this information which leads me to believe the content is being credited to, and fed in from, Wordpress.com ('client name' altered for privacy): href="http://client-name.files.wordpress.com/2012/08/could_you_survive_a_computer_disaster.jpeg">class="alignleft size-thumbnail wp-image-2050" title="Could_you_survive_a_computer_disaster" src="http://client-name.files.wordpress.com/2012/08/could_you_survive_a_computer_disaster.jpeg?w=150&h=143" I'm looking to provide a recommendation to the client on whether they are ok to continue moving forward with this current setup, or whether we should port the blog posts over to a subfolder on their primary domain www.client-site.com/blog and use Wordpress.org functionality, for proper SEO. Any advice?? Thank you!
Intermediate & Advanced SEO | | grapevinemktg0 -
Soft 404's from pages blocked by robots.txt -- cause for concern?
We're seeing soft 404 errors appear in our google webmaster tools section on pages that are blocked by robots.txt (our search result pages). Should we be concerned? Is there anything we can do about this?
Intermediate & Advanced SEO | | nicole.healthline4 -
Pinging SE's - is this spam?
Hi, Just read on ViperChill that Matt Cutts told Glen (owner of ViperChill) that ping services can help your blog posts. Now lets say you have a list of 10 that you ping and you put an article up everyday, thats 300 pings a month, is that not spammy? Here is the link to the post: http://www.viperchill.com/future-of-blogging/ If you scroll down your see a screen print of Google search box, its the para above and below this screen print.
Intermediate & Advanced SEO | | activitysuper1 -
Should I 301 Poorly Worded URL's which are indexed and driving traffic
Hi, I'm working on our sites structure and SEO at present and wondering when the benefit I may get from a well written URL, i.e ourDomain / keyword or keyphrase .html would be preferable to the downturn in traffic i may witness by 301 redirecting an existing, not as well structured, but indexed URL. We have a number of odd looking URL's i.e ourDomain / ourDomain_keyword_92.html alongside some others that will have a keyword followed by 20 underscores in a long line... My concern is although i would like to have a keyword or key phrase sitting on its own in a well targeted URL string I don't want to mess to much with pages that are driving say 2% or 3% of our traffic just because my OCD has kicked in.... Some further advice on strategies i could utilise would be great. My current thinking is that if a page is performing well then i should leave the URL alone. Then if I'm not 100% happy with the keyword or phrase it is targeting I could build another page to handle the new keyword / phrase with the aim of that moving up the rankings and eventually taking over from where the other page left off. Any advice is much appreciated, Guy
Intermediate & Advanced SEO | | guycampbell0