You sir are a genius.  Thank you very much.

Also mad props to the MozBar for picking up on this.

Some observations...

1. We've had these noindex tags on there for almost a year and just now Google found them or decided to honor them.

2. We had a recent situation where Google was basically probing our website with fake URLs to see if anything turned up.  Source? incorrectly parsed URLs from JavaScript on houzz.com.  I think they are making some big changes on how they crawl JavaScript recently.

3. One of these "problem" pages has some links pointing at it with a query string.  Nothing changes on the site because of these.  Google indexed the page with a query string despite the noindex.  Weird.  I guess Google does not always honor the noindex directive.

4. One of these "problem" pages would come up when looking up the cached version in Google but could not be found in the index.  Odd again.

Lesson to me is don't count on the meta robots tag for two reasons ...

1. Google inconsistently implements it

2. It's dangerous as a little bad code can take your whole site out of the index.

Yes, these pages...

Home Page: http://www.produnkhoops.com

Other Pages
http://www.produnkhoops.com/basketball_goals/adjustable_in_ground/
http://www.produnkhoops.com/basketball_goals/adjustable_in_ground/pro_dunk_diamond.php
http://www.produnkhoops.com/basketball_goals/adjustable_in_ground/pro_dunk_platinum.php
http://www.produnkhoops.com/basketball_goals/adjustable_in_ground/pro_dunk_gold.php
http://www.produnkhoops.com/basketball_goals/adjustable_in_ground/pro_dunk_silver.php

Screen shot attached.

I'd ignore this but Google seems to see the same thing.  You'll see a query using the site operator does not return any of these pages.

https://www.google.com/search?q=site%3Ahttp%3A%2F%2Fwww.produnkhoops.com%2Fbasketball_goals%2Fadjustable_in_ground%2Fpro_dunk_gold.php

.htaccess looks okay to me but take a look for yourself...

RewriteEngine on
RewriteBase /

Allows for php caching of images

RewriteRule ^cache_image/([^/.]+).gif?$ img.php?img=$1 [L]

#TEMP Get crawlers to the correct new parts page - Remove in Fall 2015
RewriteRule ^basketball_goals/parts/index.php(.?)$ http://www.produnkhoops.com/parts/ [R=301,L]
RewriteRule ^basketball_goals/parts(.?)$ http://www.produnkhoops.com/parts/ [R=301,L]
RewriteRule ^basketball_goals/parts/pole_padding.php(.?)$ http://www.produnkhoops.com/parts/ [R=301,L]
RewriteRule ^basketball_goals/parts/backboard.php(.?)$ http://www.produnkhoops.com/parts/ [R=301,L]
RewriteRule ^basketball_goals/parts/backboard_padding.php(.?)$ http://www.produnkhoops.com/parts/ [R=301,L]
RewriteRule ^basketball_goals/parts/pier_kit.php(.?)$ http://www.produnkhoops.com/parts/ [R=301,L]
RewriteRule ^basketball_goals/parts/rim.php(.?)$ http://www.produnkhoops.com/parts/ [R=301,L]

Retain Link Juice

RewriteRule ^giveaway(.?)$ http://www.produnkhoops.com [R=301,L]

Force www version of site

RewriteCond %{HTTP_HOST} !^www.
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=301,L]

Capture.JPG

I'm having a hard time figuring out where the noindex value for Meta Robots is coming from so I can fix it.  Can anybody spot the issue or point me to some docs that show were the MozBar finds this information http://www.produnkhoops.com

I was hoping this wasn't going to be the answer.

One other thing I've started to tinker with is AdWords scripts.  Do you know if the scripts have the ability to segment out Network (With Search Partners) effectively?

Okay, I'm now finding a big issue with this.  Even when I segment out "Network (With Search Partners)"  I cannot order by Google search numbers.  So, I get a lot of false positives that I have to manually cull.

This is really a bigger issue.  What's the best way to pull JUST Google search numbers?  I don't think there is a good way within Google's own tool.

Okay, after tinkering, this is what I've got...

I looked at the last few months of CTR.  I found about a week's period that looks good.  I compared this to the most current week that doesn't look so good.

I expanded the CTR column so I could see Change (%) and ordered by this to see where I've been hit the worst.

I filtered this to only include keywords that have over 200 impressions to filter out some noise.

Now I can go through this list and see what might be happening ... whether it be an ad that's not speaking to searchers anymore, increased competition, etc.

Does anybody have a list of metrics to look at when diagnosing a list like this?

At the campaign level I see my CTR has been suffering for the last week or two relative to the past few months.

It could be a few bad actors casing the downward trend.  How can I quickly and easily locate specific keywords that are trending in the wrong direction?

Why filters will not work...

They don't take change into account.  I want to identify Keywords that were once doing well are now doing poorly.  Filters will yield all keywords that are doing poorly which is not what I'm looking for.  Obviously I could be missing something with filters but this is my understanding.

A few other points to note...

1. I have search partners turned on.  I do not want to take this traffic into account in my analysis.

2. I'd also like to determine if it might be an ad that's not performing as well as it once was.  I assume the same method used to find poorly trending keywords can be applied to ads as well but if not, is there a solution for this?

We have ~3,000 photos that have all been tagged.  We have a wonderful AJAXy interface for users where they can toggle all of these tags to find the exact set of photos they're looking for very quickly.

We've also optimized a site structure for Google's benefit that gives each category a page.  Each category page links to applicable album pages.  Each album page links to individual photo pages.  All pages have a good chunk of unique text.

Now, for Google, the domain.com/photos index page should be a directory of sorts that links to each category page.  Alternatively, the user would probably prefer the AJAXy interface.

What is the best way to execute this?

For example, if I sell widgets and I want my ads to appear for "cyber Monday widget deals"

If I add "Cyber Monday" I'll come up for every search related to cyber Monday.  The alternative seems to be putting every possible word combination in such as...

blue widget cyber Monday deals
red widget cyber Monday sale
cyber Monday yellow widget
big widgets cyber Monday for sale
etc.

As you can see there are almost an infinite number of possibilities there making it unfeasible without doing something programmatically such as...

X = Widget Descriptor (empty is a possible value)
ie: big, red, green, blue, fast, slow

Y = Widget Synonym (plural considered possible value)
ie: widget, widgets, wiggy-what

Z = Deal Synonym (empty is a possible value)
ie: deal, sale, promotion, promo

Query Structures...
X Y Cyber Monday Z
Cyber Monday Z X Y
etc.

Is there some kind of query match I'm missing as an alternative to this programmatic brute force method?  How have you guys executed this?

Awesome responses guys.  That's how I was leaning.  Google's fishing expedition isn't going to work on me.

I recently received a unnatural link notice from Google.  The thing is, I haven't done any link building in the last two years.

So, I looked at my latest links report within Google Webmaster Tools and see where one of our dealer's website has been updated recently and now includes a site-wide link to us.

Site note: My brand name as the anchor text ... no keywords nearby.

I think this is what may be causing the problem because nothing else within the report really jumps out at me.  Also, it lines up with the timeline.  It looks like they changed their site to include the link at the beginning of April.  Google was noticing the links throughout April and to the present.  I got the notice May 19th.

Now, I have a good relationship with this dealer and can have him change anything I want at the drop of a hat.

The question is, by having him change it am I admitting to Google wrongdoing and thus hurting myself by inviting further scrutiny.  Should I leave it as is since it truly is a legitimate link?

Google having false positives like this really muddies the water.

Thanks guys!

Not just search volume mind you. Do you think mentions on sites around the web have an impact as well?

Do you think volume of brand / domain mentions and co-occurrence influence organic rankings as well?

Your premise sounds good in a perfect world.

There is a featured post by Rand that asks how to manipulate Google search suggestions.

I believe these same tactics could be used to skew organic results as well.  Some tactics suggested were...

Mechanical Turk

Offline Advertising

Online Advertising

For example, if semoz.org had 20,000 Turkers a month search for Seomoz Toasters, The legitimate toaster site may not stand a chance because CTR and bounce rates have been manipulated.

For example, if 20,000 people searched for "seomoz toasters," do you think a page on seomoz.org that mentioned toasters would begin to rank well for the query "toasters"?

And one of these tools will tell me when each brand is the most active?

I've created a gift-guide with some big brand items on it.  I'd like to reach them when I post the gift-guide.

Is there a graph that will show me when they are active on their twitter accounts?

Thank you for verifying ... I didn't think there was a great answer.

Google should provide a "(Not Provided)" eg. "logged into Google" segment.  This way, at least I could compare apples to apples from a year ago.

Even then though, the number of users logged into Google this year must be higher than last year so it is still very dirty data ... just less dirty.

We have a seasonal business.  It is of no use to compare this month to last month or even this week to last week.

The only real benchmark is to compare the same time period a year ago.

I'd like to know how well I'm doing for a certain keyword.  If I compare this month's performance to the same month's performance a year ago, I get corrupt results as much is now hidden in (Not Provided).

Any solutions?

So, when being linked to ... or even mentioned without a link something like...

"Dell Personal Computers" or "Xerox Copiers"

I tend to agree with you at face value only I have done back-link analysis on the competitors coming up as brands and stores and I have many many more links like this than they do on quality sites (not being discounted)

I think search query volume might play a part in all of this ... anybody have experience with that?

Our company is both a store and brand as we sell manufacture direct.  We are not included in Google's "Related Searches for widgets:"

Picture attached as reference (we are not selling computers ... just an example)

What is Google looking for to pull these brands and stores?

hXSLn.gif

Good point.  It does have links.  That's a tough one.  How do I salvage the links...

I think I'm going to 301 everything to the home page and start the resource over on a sub-domain.

And just to be clear, you do believe having such a prominent irrelevant page on my site is hurting my ability to rank for the terms I want to...

The traffic doesn't convert so it's of very little use to me.

If I don't want to put the effort into trying to convert it, would moving it to a sub-domain solve my domain level topic modeling problem?

80% of my site's organic traffic is coming through a resource that is only somewhat related.

Does Google think the main topic of my site is terms this resource targets thus bumping the terms I care about to a sub-topic level of sorts?

If this is the case, would putting the resource information into a sub-domain help to solve the problem?