Best way to block spambots in htaccess
-
I would like to block Russian Federation, China and Ukraine spam as well as semalt and buttonsforwebsite. I have come up with the following code, what do you think?
For the countries:
BLOCK COUNTRY DOMAINS
RewriteCond %{HTTP_REFERER} .(ru|cn|ua)(/|$) [NC]
RewriteRule .* - [F]And for buttons-for-website.com and semalt-semalt.com:
BLOCK REFERERS
RewriteCond %{HTTP_REFERER} (semalt|buttons) [NC]
RewriteRule .* - [F]or should it be:
BLOCK USER AGENTS
RewriteCond %{HTTP_USER_AGENT} (semalt|buttons) [NC]
RewriteRule .* - [F]Could I add (semalt|buttons|o-o-6-o-o|bestwebsitesawards|humanorightswatch) or is that too many?
-
Hi
I think you're on the right track.
A very good blog post by Jared Gardner has recently addressed this question on the Moz Blog.
Hope it helps,
Don
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
PDF best practices: to get them indexed or not? Do they pass SEO value to the site?
All PDFs have landing pages, and the pages are already indexed. If we allow the PDFs to get indexed, then they'd be downloadable directly from google's results page and we would not get GA events. The PDFs info would somewhat overlap with the landing pages info. Also, if we ever need to move content, we'd now have to redirects the links to the PDFs. What are best practices in this area? To index or not? What do you / your clients do and why? Would a PDF indexed by google and downloaded directly via a link in the SER page pass SEO juice to the domain? What if it's on a subdomain, like when hosted by Pardot? (www1.example.com)
Reporting & Analytics | | hlwebdev1 -
How many ways to use Event in Google Tag Manager for Event Tracking?
Hello Experts, How many ways to use Event in Google Tag Manager for Event Tracking? As per me there are 5 ways given below are they correct? 2nd thing if yes can you please please tell me procedure of using all or which is the best one to use? HTML 5 Data Attributes Classic Google Analytics Example: _gaq.push(['_trackPageview', '/downloads/pdfs/corporateBrief.pdf']);
Reporting & Analytics | | bkmitesh
3) Universal Analytics Example: ga(‘send’, ‘pageview’, ‘page path’); query string? or is it possible without any coding on website we can configure id's in google tag manager? Thanks! BK Mitesh0 -
Best way to handle ignored Rel=Canonical?
My Google Analytics is reporting organic traffic for URL's with a QueryString attached, even though there's a Canonical tag that points to the preferred (non-QueryStringed) version. Would the best way to handle this be the GWT URL Parameters Tool? I'm fairly unfamiliar with the tool, but after some research, it looks like this might be the best way to go. Does anyone have any good/bad advice for using the tool? Thanks!
Reporting & Analytics | | GalcoIndustrial1 -
800,000 pages blocked by robots...
We made some mods to our robots.txt file. Added in many php and html pages that should not have been indexed. Well, not sure what happened or if there was some type of dynamic conflict with our CMS and one of these pages, but in a few weeks we checked webmaster tools and to our great surprise and dismay, the number of blocked pages we had by robots.txt was up to about 800,000 pages out of the 900,000 or so we have indexed. 1. So, first question is, has anyone experienced this before? I removed the files from robots.txt and the number of blocked files has still been climbing. Changed the robots.txt file on the 27th. It is the 29th and the new robots.txt file has been downloaded, but the blocked pages count has been rising in spite of it. 2. I understand that even if a page is blocked by robots.txt, it still shows up in the index, but does anyone know how the blocked page affects the ranking? i.e. while it might still show up even though it has been blocked will google show it at a lower rank because it was blocked by robots.txt? Our current robots.txt just says: User-agent: *
Reporting & Analytics | | TheCraig
Disallow: Sitemap: oursitemap Any thoughts? Thanks! Craig0 -
Ways to analyze a 1M rows dataset of search queries
Hi, I have this large dataset, about 1 million search queries with visits, bounce rate and a few other metrics. I'm trying to explore this data to find keyword "buckets" (such as include product name, location name, transactional objective, informational, etc.), as well as explore the density of certain keywords (keywords as in instances of a single word amongst all queries) My idea was to use Excel and a macro to split all queries in separate words (also clearing punctuation and uppercase/lowercase), then storing this word in a new worksheet, adding to another column the visit counts from the row where the word was extracted (as to give a sense of weight). Before adding the word to the new worksheet, the script will look if the word already existed, if so it would just add the current value of visits to the existing visit counts etc. In the end it will create sort of a "dictionary" of all the keywords in all search queries ranked by weight (= visits from search query including this keyword) This would help me get started I believe, because I can't segment and analyze 1M raw search queries... My issue is: this VBA has been running on my (fast) PC for the last 24hr and it doesn't seem to get to an end. Obviously excel+VBA is not the best way to do text mining and manipulation in such a large dataset (although it's just a 30mb file) What would you do if you had this dataset and would like to mine the text/semantic as I am doing? Any idea of tools? process? I'm considering dumping this data into a MySQL db and doing the processing through PHP (the only backend language I'm versed in), and getting the "summified" data stored into another table, which I'll then be able to export to a Excel for analysis. But I'm afraid that I'll be facing memory limit issues and such... In the meantime, I'm definitely interested into knowing what you guys would do if you had this data and wanted to simply start exploring its constituencies Thanks!
Reporting & Analytics | | briacg0 -
Is there a way to calculate Domain Authority and Domain Trust on a large scale?
Hi there! Since on of our websites is hit by the Penguin update I'm trying to find spammy and low quality external links from our domain. I'm using Xenu to get all the external links. Now I want to know about the quality of the external websites our domain links to. Is there a way to calculate domain authority and domain trust on a large scale in Excel or Google spreadsheet? Kind regards, Mark
Reporting & Analytics | | StephWeigert0 -
What is the best way to track mobile sites in Google Analytics?
Hello! I am wondering what the pros and cons of using the regular Google Analytics tracking code on a mobile site versus the tracking documentation from Google specifically on it found at http://code.google.com/mobile/analytics/docs/web/ which is still in labs mode. Does the mobile specific tracking have the same features as the regular one to be able to track events and report the same statistics? Thanks for the help on this one!
Reporting & Analytics | | CabbageTree0 -
What's the best way to track moble traffic / sales from a subdomain in Google Analytics?
Hello, I've been trying to figure out the best way to track traffic and sales for a subdomain mobile site and would appreciate any feedback or guidance. I've set up an advanced segment but have found that I'm not able to source traffic since the segments get disabled when trying to view top content pages. I've also setup an additional sub-profile under the main domain profile in google analytics and have set up an advanced filter using the following guildelines http://code.google.com/apis/analytics/docs/tracking/gaTrackingSite.html In the Profile Settings page, click the Add Filter link. Choose Add New Filter and provide the filter a name. Choose Custom Filter and select Advanced on the Filter type settings. Under Advanced settings: FieldA should be set to Hostname FieldB should be set to Request URI Set the values for both Field A and Field B to (.*), which is an expression that captures all characters. Set the Output To --> Constructor option to Request URI and provide $A1$B1 as the value for that choice. I'm unsure if I've set up the profile / filter correctly, the traffic stats are different than the segment. The mobile profile is also tracking all ecommerce transactions instead of just mobile. Here's the code I have in place. we're using the Traditional code Thanks, Jamie
Reporting & Analytics | | marketing_zoovy.com0