Ranking for our member's company names without giving them all away!
-
Hi,
We have a directory of 25,000 odd companies who use our site.
We have a strong PR site and want to rank a page for each company name. Some initial testing on one or two company names brings us to #2 after the company's own web site in the format: "Company Name Reviews and Feedback" - so it works well.
We want to do this for all 25,000 of our members, however we do not wish to make it easy for our competitors to scrape through our member database!!
e.g. using: www.ourdomain.com/randomstring/company-name-(profile).php
unfortunately with the above performing a search on google for site:domain.com/()/()(profile).php
would bring up all records.
Are there any tried and tested ways of achieving what we're after here?
Many Thanks.
-
Bottom line, you cannot make data available online without offering a means for a user to grab that data.
You said you "don't wish to make it easy" so I will share some ideas:
-
EGOL's suggestion is good and not that hard to implement. I am not sure if your site requires registration but you can set it up so guests can view a maximum of ?20 member pages or whatever amount you deem to be a reasonable number.
-
There are more complicated methods by which you can establish a script that will block any IP or user who pulls too many pages too quickly.
-
The real challenge is your sitemap. If all that is required is the company's name, your sitemap is all someone needs. In this case there is simply nothing I can think of you can do.
-
If the sitemap isn't a challenge, another idea is to present the data in a method that is not easy to read. You can leave the description information in HTML but present the company name in Flash, for example.
Bottom line, if you want to rank well, the site has to be easy to crawl. If the crawl data offers enough information for others to steal, there is simply no reasonable method that can be used to prevent automated tools from grabbing it.
-
-
If you have links into all of these 25000 pages then people and robots will be able to find them.
If you want to keep robot scrapers out then you can use a white list of robots that allows the robots of search engines and other allowed automated visitors in but instructs others to keep out. No guarantees that they will not find a way in with this but it might help. Human scraping will still get through.
You could probably also devise a way to throttle the number of pageviews per visitor or per IP but that would take some creative programming.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's the best redirect to use for a newer version of a blog post?
For example: suppose you have a post "The Best Games to Play for YouTube Gamers in 2016" and you want to make this a yearly series. Should you 301 the 2016 version to the new 2017 one? Should you use the canonical attribute? If 2016 isn't in the URL, should you make the 2017 one the new URL?
Intermediate & Advanced SEO | | Edward_Sturm0 -
We have set up our H1 to contain the product name - it used to be our Company name all the time - would this affect our sales
We noticed recently on our site - that our H1 tag was the Company Name - we changed this to be the product name - our products would be searched for by all or part of the description. Our sales have dropped of the days since we changed it, could it be a result of this change ? Is it best to have the H1 tag as the product name ?
Intermediate & Advanced SEO | | CostumeD0 -
Add noindex,nofollow prior to removing pages resulting in 404's
We're working with another site that unfortunately due to how their website has been programmed creates a bit of a mess. Whenever an employee removes a page from their site through their homegrown 'content management system', rather than 301'ing to another location on their site, the page is deleted and results in a 404. The interim question until they implement a better solution in managing their website is: Should they first add noindex,nofollow to the pages that are scheduled to be removed. Then once they are removed, they become 404's? Of note, it is possible that some of these pages will be used again in the future, and I would imagine they could submit them to Google through Webmaster Tools and adding the pages to their sitemap.
Intermediate & Advanced SEO | | Prospector-Plastics0 -
Are This Site's Backlinks Hurting Us?
Google WMT reports more than 198,000 backlinks to our site (www.audiobooksonline.com) from http://dilandau.eu/? We have never been notified by Google of any penalty, malware notification... but continue to struggle to get our page 1 Google ranking back since Panda. Could these backlinks be hurting our Google ranking? Should I implement a disavow rule for http://dilandau.eu/?
Intermediate & Advanced SEO | | lbohen0 -
How long does it take before URL's are removed from Google?
Hello, I recently changed our websites url structures removing the .html at the end. I had about 55 301's setup from the old url to the new. Within a day all the new URL's were listed in Google, but the old .html ones still have not been removed a week later. Is there something I am missing? Or will it just take time for them to get de-indexed? As well, so far the Page Authority hasn't transfered from the old pages to the new, is this typical? Thanks!
Intermediate & Advanced SEO | | SeanConroy0 -
Is Google's reinclusion request process flawed?
We have been having a bit of a nightmare with a Google penalty (please see http://www.browsermedia.co.uk/2012/04/25/negative-seo-or-google-just-getting-it-painfully-wrong/ or http://econsultancy.com/uk/blog/10093-why-google-needs-to-be-less-kafkaesque for background information - any thoughts on why we have been penalised would be very, very welcome!) which has highlighted a slightly alarming aspect of Google's reinclusion process. As far as I can see (using Google Analytics), supporting material prepared as part of a reinclusion request is basically ignored. I have just written an open letter to the search quality team at http://www.browsermedia.co.uk/2012/06/19/dear-matt-cutts/ which gives more detail but the short story is that the supporting evidence that we prepared as part of a request was NOT viewed by anyone at Google. Has anyone monitored this before and experienced the same thing? Does anyone have any suggestions regarding how to navigate the treacherous waters of resolving a penalty? This no doubt sounds like a sob story for us, but I do think that this is a potentially big issue and one that I would love to explore more. If anyone could contribute from the search quality team, we would love to hear your thoughts! Cheers, Joe
Intermediate & Advanced SEO | | BrowserMediaLtd0 -
How can I change my website's content on specific pages without affecting ranking for specific keywords?
My client's website (www.nursevillage.com) content has not been touched for 4 years and we are currently ranking #1 for "per diem nursing". They do not want to make any changes to the site in fear that it might decrease our rankings. We want to try to use utilize that keyword ranking on specific pages (www.nursevillage.com/nv/content/careeroptions/perdiem.jsp ) ranking for "per diem nursing" and try redirecting traffic or placing some banners and links on that page to specific pages or other sites related to "per diem nursing" jobs so we can get nurses to apply to our new nursing jobs. Any advice on why "per diem nursing" is ranking so high for us and what we can change on the site without messing up our ranking would be greatly appreciated. Thanks
Intermediate & Advanced SEO | | ryanperea1000 -
How do you rank in the "brands for:" section in Google's search results ?
There's a "brands for:" section that appears above the first organic listing for certain search queries. For example, if you search for "dedicated servers" in Google, you will see that a "brands for:" appears. How do you get listed there? Thanks, Brian
Intermediate & Advanced SEO | | InMotionHosting0