Baidu Spider Spam
-
Baidu Spider hits my UK site every 5 minutes of every day for the past 2 years.
It has no consideration whether a domain exists or not.
I know this because looking at etc/httpd/logs/error_log, i am getting every 5 minutes hits from Baidu spider trying to access a domain which points to my server which no longer exists.
Given that I have absolutely no trade with China, and given that the only spam comments I get on my wordpress blog originate from China, do you think it's a good idea to either do a China country block in my .HTACCESS or block out Baidu spider?
Baidu is consuming bandwidth and is clogging my error_logs!!!
Why is it that Google, Bing, Yahoo etc... can all crawl my site nicely, but Baidu just abuses?
-
Hi, ive tried cloudflare before.
Problem is that i am using SSL for some of my pages, so Cloudflare doesn't play nice unless I pay them.
Also, I am using amazon cdn - does that work with cloudflare or is it a bit ott?
I will take a look at your links and thanks!
-
I just remembered another tool that you can easily add to your site and simply block the bots by implementing to not trust this hostname or IP
in fact with cloud flare can block anything looking for that old domain
Is a free service and very good DNS I would utilize it if you must.
Sincerely,
Thomas
-
the complete block is here
Required robots.txt code:
Baidu (CN)
Info: http://www.baidu.com/search/spider.htmRequired robots.txt code:
User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: /http://searchenginewatch.com/article/2067357/Bye-bye-Crawler-Blocking-the-Parasites
http://forums.oscommerce.com/topic/382923-baiduspider-using-multiple-user-agents-how-to-stop-them/
-
?It should respect the robots so may be some one pretending to be Baidu I would try HTACCESS if you're not looking to go near China etc.
-
make sure you're not running an odd plug-in that maybe causing a caching issue I know it sounds strange but I've heard of this before and it was because of an all-in-one event calendar plug in.
If it's not something like that I definitely agree with what Chris's said Good call on that Chris.
however if there is no domain you will have to implement the robots.txt on whatever your server is currently running.
If you want a free tool that will allow you to create a solid block here's one below however Chris has done a great job of creating one.
http://www.internetmarketingninjas.com/seo-tools/robots-txt-generator/
sincerely,
Thomas
-
User-agent: Baiduspider
User-agent: baiduspider
User-agent: Baiduspider+
Disallow: /Baidu spider is blocked, but it doesn't seem to care!
-
Have you tried blocking it in robots ?
#Baiduspider
User-agent: Baiduspider
Disallow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Mobile Usability Issues after Mobile Frist
Hi All A couple months ago we got an email from google, telling us - Mobile-first indexing enabled for https://www.impactsigns.com/ Ran the test on MOZ, Mobile usability shows 100% Last week got an email from google - New Mobile usability issues detected for impactsigns.com Top new issues found, ordered by number of affected pages: Content wider than screen Clickable elements too close together I can not seem to figure out what those issues are, as all content is visible. How important are these 2 issues? Since we are now on the mobile first side?
Algorithm Updates | | samoos0 -
On page vs Off page vs Technical SEO: Priority, easy to handle, easy to measure.
Hi community, I am just trying to figure out which can be priority in on page, off page and technical SEO. Which one you prefer to go first? Which one is easy to handle? Which one is easy to measure? Your opinions and suggestions please. Expecting more realistic answers rather than usual check list. Thanks
Algorithm Updates | | vtmoz0 -
What does it mean to build a 'good' website.
Hi guys. I've heard a lot of SEO professionals, Google, (and Rand in a couple of whiteboard Friday's) say it's really important to build a 'good' website if you want to rank well. What does this mean in more practical terms? (Context... I've found some sites rank much better than they 'should' do based on the competition. However, when I built my own site (well-optimised (on-page) based on thorough keyword research) it was nowhere to be found (not even top 50 after I'd 'matched' the backlink profile of others on page 1). I can only put this down to there being 'good quality website' signals lacking in the latter example. I'm not a web developer so the website was the pretty basic WordPress site.)
Algorithm Updates | | isaac6630 -
I'm Pulling Hairs! - Duplicate Content Issue on 3 Sites
Hi, I'm an SEO intern trying to solve a duplicate content issue on three wine retailer sites. I have read up on the Moz Blog Posts and other helpful articles that were flooded with information on how to fix duplicate content. However, I have tried using canonical tags for duplicates and redirects for expiring pages on these sites and it hasn't fixed the duplicate content problem. My Moz report indicated that we have 1000s of duplicates content pages. I understand that it's a common problem among other e-commerce sites and the way we create landing pages and apply dynamic search results pages kind of conflicts with our SEO progress. Sometimes we'll create landing pages with the same URLs as an older landing page that expired. Unfortunately, I can't go around this problem since this is how customer marketing and recruitment manage their offers and landing pages. Would it be best to nofollow these expired pages or redirect them? Also I tried to use self-referencing canonical tags and canonical tags that point to the higher authority on search results pages and even though it worked for some pages on the site, it didn't work for a lot of the other search result pages. Is there something that we can do to these search result pages that will let google understand that these search results pages on our site are original pages? There are a lot of factors that I can't change and I'm kind of concerned that the three sites won't rank as well and also drive traffic that won't convert on the site. I understand that Google won't penalize your sites with duplicate content unless it's spammy. So If I can't fix these errors -- since the company I work conducts business where we won't ever run out of duplicate content -- Is it worth going on to other priorities in SEO like Keyword research, On/Off page optimization? Or should we really concentrate on fixing these technical issues before doing anything else? I'm curious to know what you think. Thanks!
Algorithm Updates | | drewstorys0 -
Remove spam url errors from search console
My site was hacked some time ago. I've since then redesigned it and obviously removed all the injection spam. Now I see in search console that I'm getting hundreds of url errors (from the spam links that no longer work). How do I remove them from the search console. The only option I see is "mark as fixed", but obviously they are not "fixed", rather removed. I've already uploaded a new sitemap and fetched the site, as well as submitted a reconsideration request that has been approved.
Algorithm Updates | | rubennunez0 -
Is this spamming keywords into a url?
My company has previously added on extensions to a url like the example below http://www.test.com/product-name/extra-keywords My question is since there is no difference between the pages http://www.test.com/product-name and http://www.test.com/product-name/extra-keywords and you don't leave the product page to reach the extra-keyword page is this really necessary? I feel like this is probably not a best practice. Thanks for any suggestions.
Algorithm Updates | | Sika220 -
Spam Back Link Removal Problem.
I have just paid a lot of money to have spam back links removed from directories owned by the same person, the links were on pages that were set up for me without me knowing, at the end of each url is my domain name, the links have been removed on the page leaving a directory page with no other links on however the url is still there with my domain name at the end of the url and in each search box is my domain name, I have asked for the pages to be removed altogether as I did say before I paid the money I did not want my domain name on any of his directories, he has come back and said leaving my domain name in the urls is not a problem as far as Google is concerned, can anyone please advise, I can ask for a refund from PayPal, there are over 768 links on different sections of a number of directories. Thank you inadvance.
Algorithm Updates | | Palmbourne0 -
Is This Keyword Stuffing/Spamming?
We are a custom patch company--we make patches for many different types of clients. I have a gallery of patches for almost every kind of client, and they all have their own pages. If I put navigation on the home page such as what I show below, will Google consider that to be too much? Boy Scout Patches | Motorcycle Patches | Fire Patches | Police Patches | Military Patches | Sports Patches | Business and Organization Patches | Paintball Patches | Scooter Patches | In Memory Patches They would all be links to different pages, and there would be literally 50-60 more! Would it be better to remove the word patches from all of the links? And then another question comes up: too many on-page links?
Algorithm Updates | | UnderRugSwept0