Welcome to the Q&A Forum

There's a possibility someone in your company saw suspicious traffic from an actor spoofing the Baidu user agent. It can get so aggressive that it will eventually bog down your response time through sheer number of requests. But the problem is that same actor, or someone else with malicious intent can simply spoof another user agent or IP.

But the main problem is, the site is straight e-commerce. It could get international business, so why take such a ham fist approach? Even if blocking Baidu gave the desired result, the dev/admin would still have to block individual IP blocks as they come in. It would make more sense to invest in server resources so it can handle the load, or look into DDos Mitigation.

So yeah, it's strange. Though it's more likely a lack of understanding than anything malicious.

I checked the link profile in Majestic. It seemed to hug the X axis in a bad way. There were likely precious few links of any value. When the vast majority of links appear to be garbage, you're pretty much back at ground zero. And that's given a clean bill of health.

Best of luck, cleanup can be a thankless job.

Why not just adopt WordPress? ; ) I generally don't work with the .Net platform in the wild, unless I'm prospecting. You may or may not believe some of the crazy things that platform does out of the box. So make sure to get a full crawl of the site and sort out any issues prior to dev work. I recommend Screaming Frog in most instances. But if you have a gigantic site and/or you may not be able to identify problems readily, you may find Deep Crawl worth the price.

At any rate, just make sure you have a pre and post launch crawl for comparison. That alone can save you hours of time, should something go a awry. If nothing else the crawls will help a consultant, should you need one in the future.

Since you're making major site changes, it's also a good idea to get some site speed benchmarks. (Get benchmarks under various traffic loads, if possible.) It's possible that you can end up with a slower site, even though it's responsive. There are a number of ways that can happen, but at least you'll know if you have a speed problem.

I recommend GTmetrix and Pingdom for the above tasks. Here are some really simple fixes from Feed The Bot that should help with speed, once you're on an Apache server.

Just remember, redesigns are a great time to catch any loose ends you 'didn't have time for' or were 'minor problems'. Those minor problems stack up to considerable wins, once they're righted. You're on the right path with a responsive redesign. Remember to preserve the source order of the site between versions (desktop, mobile, tablet) as much as possible. That will help with the speed as well.

Run your site through SEM Rush. It's generally pretty accurate if you need a better understanding of what pages draw traffic and for which queries. After that, it's a good idea to use your analytics tools to identify pages that draw traffic.

One really good link may be driving a ton of traffic to a specific page, but it may not necessarily rank very well. The site might lose a significant amount of referrals due to that link. So make sure to check where your referral sources go as well.

When all of that is said and done, it's time to look at links. Use a few different sources such as Webmaster Tools (Google and Bing), Majestic SEO, Ahrefs and OSE. Majestic and Ahrefs are the favorites, as far as percentage of links and freshness.

When you have the Keep and Cull list, make sure to redirect the keepers to relevant pages. If the page was about donuts, send it to the new donuts page. Don't send it to the bagels page. Google dislikes that. Blanket redirects are generally sub-optimal.

As for the culled pages, you might try to offer a useful 404 page. Something that helps them find something similar to the topic of the page that went away. When all of the dust clears, 410 the culled pages. Alternately, you can skip straight to a 410 if you're only worried about bots wasting crawl budget on a bunch of 404 results.

Rest assured, that I don't scrape/hammer so hard that it would knock your site down for a period. I often throttle it back to 1 thread and two URI per second. If I forget to configure it, the default is 5 threads at two URI per second. So yeah, maybe a bit of the moz effect.

Chrome Incognito Settings:

Just the typical/vanilla/default incognito settings. It should accept cookies, but they generally wouldn't persist after the session ends.

I didn't receive a message regarding cookies prior to the block notification.

On a side note, I don't allow plugins/extensions while using incognito.

Fun w/ Screaming Frog:

It's hard to say if the 8.5 hour later instance was my instance of Screaming Frog. The IP address would probably tell you the traffic came out of San Antonio, if it was mine. I didn't record the IP at the time, but I remember that much about it. Otherwise it's back in the pool.

Normally Screaming Frog would display notifications, but in this instance the connection just timed out for requested URLs. It didn't appear to be a connectivity issue on my end, so... yeah...

Fun w/ Scraping and/or Spoofing:

Screaming Frog will crawl CSS and JS links in source code. I found it a little odd that it didn't.

I also ran the domain through the Google Page Speed tool for giggles, since it would be traffic from Googlebot. It failed to fetch the resources necessary to run the test. Though cached versions of pages seemed to render fine, with the exception of broken images in some cases. Though I think that may have something to do with the lazy load script in indexinit.js, but I didn't do much more than read the code comments there.

In regard to the settings for the crawler, I had it set to allow cookies. The user agent was googlebot, but it wouldn't have came from the typical IPs. Basically just trying to get around the user agent and cookie problem with an IP that hadn't been blocked. You know, quick - dirty - and likely stupid.

Fun w/ Meta Robots Directives:

A few of the pages that had noindex directives appeared to lack genuine content, in line with the purpose of the site. So I left that avenue alone and figured it was intentional. The noarchive directive should prevent a cache link. I was just wondering if one or more somehow made into the mix, for added zest. Apparently not.

While I'm running off in an almost totally unrelated direction, I thought this was interesting. Apparently Bingbot can be cheeky at times.

Fun w/ The OP:

It looks like Ryan had your answer, and now you have an entirely new potential problem which is interesting. I think I'm just going to take up masonry and carpentry. Feel free to come along if you're interested.

Yes, I got it from Majestic SEO. View page source then ctrl+f and type in 'daisy'. A lot of the domains with a lot of exact match anchors - hidden behind giant images.

The registrant of the site probably has no idea this all happened.

I'll PM my public IP through Moz. I don't really have any issue with that. Oddly enough, I'm still blocked though.

I thought an okay, though slightly annoying, middle ground would be to give me a chance to prove that I'm not a bot. It seems cases like mine may be few and far between, but it happened.

It turns out that our lovely friends at The Googles just released a new version of reCAPTCHA. It's a one-click-prove-you're-not-a-bot-buddy-okay-i-will-friend-who-you-calling-friend-buddy bot check. (One click - and a user can prove they aren't a bot - without super annoying squiggle interpretation and entry.)

I don't speak fluent developer, but there are PHP code snippets hosted on this GitHub repo. From the the documentation, it looks like you can fire the widget when you need to. So if it works like I think it could work, you can have a little breathing room to figure out the possible session problem.

I've also rethought the whole carpenter/mason career path. After much searches on the Yahoos, I think they may require me to go outside. That just isn't going to work.

I'm that guy, at least I am today. Never link simply for the 'SEO Juice'. If you follow that strategy, you will get your client/yourself in trouble in short order.

Is this link legit? Will it drive traffic to the target site? Is it even remotely related?

All of the numbers can be easily gamed. If you handle the situation intelligently, you'll have a happy client - without worry of penalties.

Good to hear you may be getting closer to the root of the problem. Apologies that it took so long to get back to you here. I had 'things'.

I followed the steps and you should be able to determine the outcome. Spoiler Alert: No block, this time.

It's a whole other can of worms, but should you need more human testing on the cheap; you may find Mechanical Turk attractive. One could probably get a couple hundred participants for under a couple hundred dollars, with a task comparable to the one above.

Just a thought...

Semantic markup isn't cruise control for rankings. It's just a way to spoon feed the search engines, per se. I wouldn't use addressLocality.

Edit: I wouldn't use addressLocality for a serviceArea. (Jumped the gun on ending that sentence.

Hi Evelien,

Does organic traffic appear to be attributable to any particular country or countries? This may sound strange, but I wonder if a competitor pulled out of the market. It appears just about every competitor you have in google.be got a pretty nice organic increase around that time, which has continued. Kruidvat seems to have the lion's share now.

Last I knew, Luxembourg was something of a tax shelter. With the recent changes in VAT, I wonder if a significant competitor or competitors found it difficult to continue operations. But that's just a 'shot in the dark' on my part.

That's interesting, but how are the buildings sites relevant? None of them seem to get much traffic over the last couple of years either. The free docs sites appear to have an appreciable amount of estimated traffic, but what of the next Panda update?

Now, for the thought. I checked Sony.com for PDF files via operator. The tenth result was this. It had a Moz PA of 35. The rest were 1s. I couldn't get any backlink information from Majestic or Ahrefs. So, it appears all of our tools are dreadfully broken in this scenario. The best thing to do is more testing.

I'm going to guess that you have something that looks like this in your .htaccess file:
RewriteRule ^blog/$ http://blog.website.com [L,NC,R=301]

WARNING

You can knock your site down with the slightest syntax error when you mess with the htaccess file. Proceed with caution.

Read this first.

Let us know what you find.

Don't do this: "At less than 4 hours away from Baltimore, you’re loved one will never be more than a drive away."

That statement generally means a couple of states away in that region. Also, yes, they are mostly duplicate content with some mild variations. That would generally be considered low quality, but things happen in certain niches. Though the local campaign will more than likely fail.

I'm not a big fan of shipping people with neurological disorders all around the country, unless it is to their benefit.

So, best of luck!

Yes.

Try:

https://goo.gl/

EGOL, you da best.

What is?

If the content isn't there, you can setup a 410. That will tell the search engines and users that the pages are gone for good. GWT will also show broken links to your site as well. So you will want to distinguish between the inbound 404s and the pages that are no longer there.

There's a possibility that the pages have been gone for so long, they're no longer indexed. So I'm not really sure how much good a 301 will do from a link perspective. However, if you have access to referral data - you may find some of those inbound 404s are worth redirecting to a relevant page.

TL;DR Follow Google's Guide on Recovering a Hacked Site

Orrrrr.... deze.

First, are you 100% sure it's clean? Some people get lucky, and have a clean backup. Others aren't so lucky.

Some hacks come in 'low and slow'. In other words, the site could have been compromised well before any outward signs became apparent. So definitely consult a security professional, if you haven't already done so. (Sometimes your host will 'clean' it for you, but miss critical points, and you're back in the same spot later.)

I suppose it couldn't hurt to remove URLs created by the hack. But if they're all gone, resolving as 404 ,they will drop out of the index in a couple weeks or sooner. The main thing is that the site is clean.

If you're sure it's clean, submit the site for review. The guides I've linked mention that it will take several weeks. And once Google has determined the site is clean, you should see any of the 'hacked' and/or 'malware' snippets disappear within a few days.

And I seriously recommend having a security professional review the site and hosting. A security professional will not only help you determine the cause of the problem, but also with mitigation of future risks. Best of luck.

It would appear that it's not such a big deal. Generally the ccTLD is a good indicator of intent. But you can actually target your country of choice in Google Webmaster Tools. That would count as yet another signal that your client is targeting New Zealand.

A similar question was recently answered in the Moz forum. I would look there as well.

Some things you can do to target your location:

• Google Webmaster Tools
• Meta Data
• Contact Information
• Schema Markup

The only thing I would be concerned about is if there was a better web host in New Zealand, or if competitors were using a CDN. That would generally make their sites faster, one through geography and one through infrastructure. Either way, that's a strong ranking factor where you may lose out.

Other than that, it looks like you and your client should be alright in regard to your question.

Schema (structured data) is supported by all major search engines in the Western hemisphere, and a two or three search engines elsewhere. The search engines, Google in particular, may use this structured markup to display various things about your business in search engine result pages. While it's not known to me as a ranking factor, it seems it would be advantageous to tell search engines everything about the business in a language they can easily understand.

Without knowing more about the case, I can't say in a 'yay or nay' sense whether it may be worth the effort. If someone in your ranks understands or can learn Schema markup, go for it, you have nothing to lose. Though there's no guarantee applying the markup will result in a unique snippet for any given query.

Ultimately, it seems prudent to do so. Though it may take a little time to figure out what markup you'll need.

Personally, I've never put a blog on a sub. More people are saying there isn't any difference, but more people may be saying that because a few more have said as such. Lovely how that works?

It was about four years ago we were told that EMDs were losing their potency. Upon hearing that, I jumped right on the bandwagon. Bad move, at least for a little bit. It would be years before a non-EMD, all things equal, could easily compete with an EMD.

What is said and what is are often at chronological odds.

Are you talking about this post?

Perhaps a new URL for the same pages could cause the Googles to 'reconsider' the content, for better or worse. Hopefully it would be for the better. I think you're going to do it anyway, so please take notes. It has the makings of a post I would love to read.

Thanks for elaborating. But what we're debating may prove something else entirely. It seems weird, but it's a very smart dumb machine.

I'm guessing you want to escape regex characters with a ? XD I just want to say again, that regex101 is a good resource. So thanks for sharing that.

I looked at the site. It appears to be primarily 'curated content' with citations. But this be physics. You can't hit the wall without the wall hitting back.

There are quite a few ads as well. cough scraper cough

Though I agree he should definitely wait a moment before disavowing. It's just that the site does appear to be pretty low quality. It does seem like Panda bait. But apparently content gets curated/scraped all the time.

So it really depends upon the target site.

I'm not going to lie, the /jobs/surrey/dorking page made me giggle.

You know, it might not be the links you gained, but the links you lost. Somewhere between February and July, it appears the site lost an RSS feed link from The Guardian. Which would probably account for the /blog problem. There was also a pretty good .edu link that disappeared.

With a smaller portfolio of links, if you lose a couple of good links - you'll feel it. It also looks like there may have been a little 'guest blogging' going on. It's not uncommon for the host blog to take the content and pull the link when you're not looking. Who knows, maybe they did the site a favor?

I've been reasonably assured that the UK and US versions are comparable, and Panda updated a couple of times in May. So, the more likely culprit may be the user profiles on the site. It might be a good idea to cull anything that hasn't been used in a while.

I could see where the site might look thin in places. Perhaps you could partner with some kind of video transcription service? That seems like a logical step.

This may not be exactly what you had in mind, but why not handle it like any typical WordPress blog installation?

Just show a preview snippet with a link to the full profile. You would avoid duplicate content. You could have what you want in effect.

If it was my deal, I would go with a static home page that explains what the whole thing is about. There would be navigation to the Difference Makers. That would essentially be your post type where you showed a snippet of the profile that leads to a post that displays the profile.

If there's an actual blog in there for some reason, you could likely handle either or with a custom post type.

But if you don't like the static home page idea, I suppose you may be able to get away with a preview that links to an actual post/bios on the home page.

I would be interested to see what anyone else thinks. (Not sarcasm, really want to see what others think about it.)

All of the numbers are borderline arbitrary. I would simply state; "Don't be stupid." That will get you further than anything.

That's odd that neither method works. There are still a few more methods you can try. You can find them here. If you could update how that works for you, that would be appreciated.

Whenever I see things like this, I always wonder how often people get fired for system problems beyond their control. At any rate, once you re-verify your account, the historic data should still be there. So no worries on that part.

I'm just going to leave this here. ; ) It would seem that all of the typical means of citation can be recognized as such. Perhaps too readily?

I would generally prefer CSS over JS for navigational elements, but that probably isn't the problem here. Google can crawl JavaScript and attribute links fine. And per SEM Rush, it looks like the site is enjoying a pretty sharp uptick in organic traffic recently. That would seem to be at odds with big indexation problems.

I'm not so sure if it's my network, I'm on a sub par connection now, but I noticed that some CSS and JS files were timing out when I crawled the site. That could lead to a big problem. I would advise that someone check the server log files and see if those files are regularly timing out. Ideally one would want their CSS and JS files combined/concatenated where possible, to reduce the possibility of any such rendering issues.

More on that from SE Roundtable

I checked the cache for the EN version of a few of those pages, and they appear to be cached fine.

cache:https://f5.com/products/security/distributed-denial-of-service-ddos-protection yields, which is pretty much what we want.

http://imgur.com/a/zqsDk

But I do see some problems that could lead to problems with indexation/display. The site has a number of different languages/translations. However, I noticed that the hreflang attribute was missing. It's strongly recommended that hreflang is implemented. You're good on the language meta tag Bing recommends, though.

That would cause some problems, especially on a site that large. I've researched Radware, their competitor, years ago. F5 seems like the type of organization that would pay for a decent translation. (my German and Spanish are so limited, I couldn't discern the quality of the translations) But if it is automatically generated, that would more than likely lead to indexation problems as well.

Another thing I see is that each translation is marked as canonical. This could also cause problems with display and link equity.

Here's more on internationalization from Moz and Google.

I would also look for ways to build internal links to the important products (DDoS Mitigation is supposed to be a huge money maker now.) on the home page, in the body. Not just in boilerplate (nav... footer... etc....) areas.

Edit: Forgot to mention that the mobile menu doesn't appear to directly link important products. I would make sure the experience is the same across devices.