Welcome to the Q&A Forum

Hi Martijn,

Thanks for the reply.

Yes, I've enabled bot filtering on all views except my unfiltered view. My site gets so much bot traffic that the data would be unusable without it.

As for filtering the traffic, I'm not aware of any way to retroactively filter out User-Agent unless I use GTM to create a custom dimension as described here: https://www.addsearch.com/docs/analytics/addsearch-bot-google-analytics/

Perhaps this is something I should do in case something like this ever happens again in the future.
I'm still hoping for other suggestions - particularly ways to ask the Google Analytics team to include "whatstuffwherebot" as part of their bot filtering.

Best,
Akira

Starting yesterday, Aug 26, 2020, I noticed a new bot crawling our site with user agent whatstuffwherebot.

Google Analytics is counting these hits as human traffic, completely throwing off my numbers - yesterday, Analytics reported nearly triple my typical number of visitors.

As of now, Search Console only shows data through Aug 25 so I don't know if Search Console is also affected.

Is anybody else seeing something similar?

Does anybody know what the whatstuffwherebot bot is? I don't get any results when I search on Google or Bing.

For what it's worth, the traffic is coming from Columbus, OH, running over Amazon AWS via 278 different IP addresses so far. Also, WordFence (my WordPress security plugin) correctly identifies these hits as bot traffic.

Dr. Egol - thank you for the thoughtful answer. I'll give it a try!

Considering Google's renewed emphasis on E-A-T, and since Google+ authorship is going away, what is the best practice for establishing authorship of content?

Is it still rel="author" but with a link to one's bio on one's website?

Should this be done on every page of a site? If so, can it be in the footer or would it be more effective immediately below the page content?

Question Deleted

Thanks, Peter. That's useful to know.

I ultimately found an easier solution. My situation is that I'm changing some URLs and I want to keep consistent tracking of my funnels/goals.

So if I changed a URL from say page1.htm to page2.htm, I was thinking I needed to define the associated funnel to be page1.htm OR page2.htm. However, I discovered that if I simply changed the funnel from the former to the latter, all of the historical data for page1.htm would be combined with new data collected for page2.htm. Thus, the logical operator wasn't necessary.

Trenton,

Thanks for the suggestion. I'm trying it your way and will test shortly!

Best,

Akira

When setting up a Funnel within Google Analytics, is it possible to use logical operators (e.g. OR, AND) in the first (required) step of the funnel? For example, suppose I want to track users who visit page1.html AND page2.html before proceeding to the destination goal.

I've entered two pages separated by the OR operator, and neither the "Verify this Goal" nor "Save" produces an error message - is it safe to assume that this is working as I intend?

Thanks in advance!

This is happening to me also. Using the latest Outlook in Office 365.

Thanks for all of the responses - this forum is a great resource!

Over 95% of my revenue comes from organic search; less than 5% comes from AdWords PPC (all other sources account for about 1-2%).

My ROI on AdWords is roughly zero. It's negative if you include opportunity costs.

My question is: if I pause all of my AdWords campaigns, is there ANY chance that my organic rankings (and organic click-through rates) will suffer?

This is really two questions. First, could Google retaliate to my reduced ad spending by dropping my rankings? Second, will searchers think differently about my organic link if they don't also see the accompanying paid link on the SERP?

Thanks for the detailed suggestions!

As a follow up: what metric do you use to decide which offenders to go after, and which ones to ignore? I simply don't have time to go after everybody who has copied my content so I need a way to prioritize.

There are two obvious situations where action is warranted: first, when the infringement is committed by a competitor in my industry, and second, when the infringing content outperforms my own site in the SERPs. What else would you suggest?

Thanks again.

Over the years, tons of original content from my website (written by me) has been scraped by 200-300 external sites. I've been using Copyscape to identify the offenders. It is EXTREMELY time consuming to identify the site owners, prepare an email with supporting evidence (screen shots), and following up 2, 3, 15 times until they remove the scraped content. Filing DMCA takedowns are a final option for sites hosted in the US, but quite a few of the offenders are in China, India, Nigeria, and other places not subject to DMCA. Sometimes, when a site owner takes down scraped content, it reappears a few months or years later. It's exasperating.

My site already performs well in the SERPs - I'm not aware of a third party site's scraped content outperforming my site for any search phrase.

Given my circumstances, how much effort do you think I should continue to put into policing scraped content?

I've found a solution and am posting it here in case anybody else is having the same problem:

RewriteRule ^([0-9]{4})/([0-9]{2})/([^/]+)/[0-9]+ /blog/$1/$2/$3/ [L,R=301]

Sorry in advance if this isn't the best place to ask this question.

Google Webmaster Tools has recently identified a ton of "Not Found" pages, which are actual pages with some digits appended at the end.

For example, suppose an actual page on my blog is:

(A) http://www.example.com/blog/2012/09/my-post-title/

This page works just fine.

However, GWT has identified the following page as a "not found" page:

(B) http://www.example.com/blog/2012/09/my-post-title/9157586677/1846732913010

This appears to be happening to hundreds of posts on my site. In each case, the "9157586677" portion of the URL is identical, but the remaining 13 digits change from page to page.

I haven't been able to determine exactly what is causing this to happen - it's probably a social plug-in for Wordpress, or perhaps Disqus, but I'm not sure which one. I'll go through a process of elimination to narrow it down over the coming week.

As a quick fix, I'd like to create a ModRewrite rule so that requests for (B) get 301 redirected to (A). Since there are hundreds of posts, I need to do this in a way that works regardless of what's in the "/2012/09/my-post-title/" part of the URL.

Unfortunately, mod-rewrite is outside of my area of expertise. Can somebody please suggest how I can handle this? Thanks in advance.

PS - As for tracking down the cause, I've looked at the source of the pages in the "Linked From" area of GWT and the Not Found link is nowhere to be found. That is why I assume the bad link is being generated by some javascript that is a part of one of my plug-ins.

Update: It seems like Disqus is the source of these phantom links. There's considerable discussion here. I'll continue searching for a long-term solution. Meanwhile, I'd still appreciate help with the mod-rewrite question above. Thanks again.

There are about a dozen PDFs and they are indexed.

I agree that it's probably better to play it safe and keep these files on www.

Thanks!

I've set up a CDN to speed up my domain. I've set up a CNAME to map the subdomain cdn.example.com to the URL where the CDN hosts my static content (images, CSS and JS files, and PDFs).

www.example.com and cdn.example.com are now two different IP addresses.

Internal links to my PDF files (white papers and articles) used to be www.example.com/downloads but now they are cdn.example.com/downloads

The same PDF files can be accessed at both the www and the cdn. subdomain. Thus, external links to the www version will continue to work.

Question 1: Should I set up 301 redirects in .htaccess such as:

Redirect permanent /downloads/filename.pdf http://cdn.example.com/downloads/filename.pdf

Question 2: Do I need to do anything else in my .htaccess file (or anywhere else) to ensure that any SEO benefit provided by the PDF files remains associated with my domain?

Question 3: Am I better off keeping my PDF files on the www side and off of the CDN?

Thanks,

Akira

I already do. So maybe it's not necessary to worry about the blog's root page?

Thanks.

I like the idea of excerpts, and I do use them on the home page of my website (say, www.example.com).

But what about on the root page of the blog, www.example.com/blog - which is not the home page of my website?

Do you think I should do the same? (Kind of like how www.seomoz.org/blog is set up)?

Suppose the "Home" page of your blog at www.example.com/domain/ displays your 10 most recent posts.

Each post has its own permalink page (where you have comments/discussion, etc.).

This obviously means that the last 10 posts show up as duplicates on your site.

Is it good practice to use NOINDEX, FOLLOW on the blog root page (blog/) so that only one copy gets indexed?

Thanks,

Akira

Some SEO folks (such as http://www.stonetemple.com/articles/css-and-seo.shtml) advocate using CSS so that unique content in the body of your page appears earlier in the file than non-unique content (such as, for example, the content of an elaborate drop-down menu system).

What say you? Do search engines now disregard position on page, and look at your pages holistically?

I think Guy Warner nailed it. I wrote the copy for my home page back in 2004. I've tweaked it over the years, but it remained largely the same. During that time, lots of others have been copying the text for their own websites. Now, if you search for an exact match, a few of the other sites come up even higher in the SERP than my own site!

I've done my best to get folks to remove the copies (and serving DMCA complaints when necessary) but I think my time is better spent simply rewriting the home page.

It's frustrating on one hand, but it's also good to finally have an idea of what the problem might be.

Thanks, Guy!

Hi Takeshi,

My competitors for the specific keywords I'm talking about have stayed fairly static in the rankings. I haven't noticed any on-page changes for any of them during this period (I check frequently) and I haven't noticed any major changes in backlink profiles (I use Raven and Majestic in addition to SEOmoz Pro).

Over the same period, my site's on- and off-page SEO factors did not change either, and yet my site dropped >10 spots relative to competing sites for these specific keywords.

Meanwhile, other pages of my site continued to rank well for other keywords - so if it was an algo change, it's an algo change that only affected my home page for certain keywords, while leaving my other pages and my competitor's pages alone.

Strange, no?

I'm sure it IS algorithm-related. The strange thing is that it doesn't seem to have affected all of my pages/rankings equally. I'll PM you shortly.

There's actually been a steady and gradual decline in organic traffic since around early February.

I'd rather not say which website or what keywords in a public forum, but I'd be happy to discuss it privately.

The Google Webmaster Tools "site performance" chart shows a period of slow performance for the first half of April. There was also a brief website outage about 2 weeks ago - I don't think it lasted more than 10 minutes.

Thanks for your thoughts on this.

I've been banging my head trying to solve a mystery for a few months, with little luck. Maybe one of you has an answer.

My home page organic SERP rankings have dropped from #1 or #2 late last year, to between #8 and #25 a few months ago for a couple of important keywords.

I did not make any on-page changes during this period.

Meanwhile, many inner pages continue to rank well (top 5) for long-tail keywords.

All of my SEO is white hat, no unusual on-page tactics, no purchased or otherwise fishy backlinks. In fact, looking closely at the on-page and backlink tactics of some of the websites that have leapfrogged ahead of me, I'd say some of them have (and continue to) engaged in shady practices.

No errors reported by SEOmoz. I've been judiciously applying canonical tags and other techniques to ensure minimal duplicate content.

I've not received a letter from Google warning me of any unnatural backlink profiles.

The biggest part of the mystery is that it seems like my home page was the only page really affected. As mentioned above, the inner pages continue to rank well for their respective keywords. Thus, I don't think I've been hit by any kind of site-wide penalty.

Also, I continue to rank well on Bing/Yahoo.

Does anybody have an insight into what could have happened? Has anybody else experienced anything similar (home page dropping rank while other pages stay put)?

Good advice, Matt, and thanks for taking a moment to provide an actual data point!

And yes, we definitely take a balanced approach combining on page, off page, PR, social, and more. This just happens to be "on page" day for me.

Hi Matthew,

I agree with all of your points.

Nevertheless, I still think that it's overdramatic for the On Page Report Card to go from an "A" grade to an "F" grade when changing a single letter. It just doesn't seem to reflect the reality of how search engines look at content.

Akira

I have some pages on my website that receive an "A" grade with the SEOmoz On Page Report Card if I use a plural term in a phrase (say, "landscape designers in Dallas") but an "F" grade if I use the singular term (say, "landscape designer in Dallas").

Is this a true and accurate reflection of how search engines see the page?

Am I going to rank well for plural terms but not for singular, given the specific on-page content I'm using in this example?

Or is this an opportunity for SEOmoz to tweak its' algorithm to better reflect the search engines?

Note to self, and to others who see this thread later, the 301 for this situation is:

RewriteCond %{QUERY_STRING} x=([0-9]+)&y=([0-9]+)

RewriteRule ^product.php$ http://www.example.com/product.php?y=%2 [r=301,nc]

Dr. Pete,

Thanks for the very helpful answer. I've gotten rid of the rel-canonical tag for this part of the site and I'll try to figure out what's up with the 301s.

BTW, I listened to the recorded version of your "future proofing" webinar this morning and learned a lot.

Akira

Suppose I have two dynamic URLs that lead to the identical page:

www.example.com/product.php?x=1&y=1

and

www.example.com/product.php?y=1

The x=1 parameter had some historical meaning, but is now unused. All references to the x=1 parameter have been removed from internal links and sitemaps.

I have implemented two solutions:

First, the header of www.example.com/product.php?x=1&y=1 includes:

Second, the .htaccess file includes the following:

Redirect permanent /product.php?x=1&y=1 http://www.example.com/product.php?y=1

Questions:

1. I assume that since canonical is still relatively new, it's best to play it safe and implement both solutions. Is this correct?

2. When I point my browser to www.example.com/product.php?x=1&y=1, it does NOT redirect to www.example.com/product.php?y=1. The address bar continues to show the non-canonical URL. Is this because the canonical tag somehow takes precedence over the 301 redirect?

3. How long will Google Webmaster Tools continue to show these as duplicates, even though I've implemeted BOTH canonical and 301? It's been a few weeks and I thought it would have rolled off by now.

Thanks!

If you have backlinks pointing to www.etraxc.com/default.asp then using the canonical tag to point to www.etraxc.com/ instead should still be helpful. Or you can do a permanent redirect from the former to the latter.

Add rel="canonical" href="http://www.etraxc.com/" /> to the header of default.asp.

But first, learn how the canonical tag works so you don't hurt yourself by mistake: http://www.seomoz.org/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps

Thanks for a very thoughtful reply; I agree that it's important to promote the best content, and I definitely do that.

I misspoke earlier - the excerpts are just the title and 45 words, not 85. Maybe 3 sentences. Do you still think this is a duplicate content risk?

Last, a new question: Can you think of ANY good reason to put blog excerpts in a site-wide footer the way  http://springboard.com/ and http://seoaware.com do? I currently do it randomly; these other sites excerpt the latest posts.

Ben,

Thanks for the reply. I didn't know about the nofollow on the internal links.

Yes, the body of every page has unique content.

My entire site revolves around serving the needs of a specific type of individual, and the content in the body and the content in the blog work together to fill the information needs of the visitor - so hopefully yes, the blog content is relevant.

I see lots of websites incorporating a blog excerpt in the footers or sidebars of their pages (e.g., http://springboard.com/ and http://seoaware.com). The problem I have is that they almost always link to the most recent blog post(s), concentrating attention on a small subset of the blog inventory. That means older, still-relevant, still-useful content is getting neglected. That's the main thing I'm trying to overcome.

I do have good site search so people who know what they are looking for can find it - the problem is that many people don't know what they want until they see it.

Thanks again!

I have roughly 150 non-blog pages and 500 articles in my blog.

The footer of every non-blog page includes excerpts from 3 blog posts selected at random from the inventory of 500.

The posts in the footer of each page change with every page refresh.

So, if you scroll to the bottom of any non-blog page, you'll see about 85 words for each of 3 randomly selected blog posts, with a link to the source article in the blog section of my site. Each page will link to 3 different posts.

One of my objectives is to drive visitors to some older blog content that has become buried deep in the archives over the years.

Question 1: In a post-Panda/Penguin world, is this a good or bad technique?

Question 2: Should the links to the full content in the blog use rel="nofollow"? Without it, the internal link structure for this part of the site looks pretty crazy and random - I assume nofollow would help make things look more orderly (and prevent my main non-blog pages from passing excess link juice to my blog).

Thoughts or comments?