Where are the 301s taking Googlebot on those IP addresses? And are they the same IP addresses every time? Have you narrowed those IP addresses down to any particular datacenter/country? It could be possible there is some configuration with your server that treats IP addresses differently depending on the country... it could also be that the IP addresses getting the 301s are known blacklisted spam IP addresses but are masking themselves as Googlebot so your server's blacklist software is keeping them out. It's really hard to say without looking into the data myself but I'm definitely interested in what you find out.

I would just use a sitemap.xml file instead for Google, Bing and Yahoo. Then you can submit the sitemap.xml file within the Google Webmaster Tools and Bing Webmaster Tools (includes Yahoo). You can easily create an XML sitemap at http://www.xml-sitemaps.com/

Great call on both the canonical tags and breaking out of the frame. I imagine the site will be indexed properly within a week using these tactics.

Interesting, I am now seeing what you are talking about...

Can you share your URL so we can take a look? Feel free to PM me if you want to keep it private.

It's hard to say exactly what the cause is without knowing what the site/keywords are, but I'd recommend checking if these are all New Visits and/or Unique Visitors and where they are geographically located. I've seen huge spikes in traffic by the same group of people/computers in foreign countries before who were hitting my site over and over for some unknown reason. Also, check which search engine is sending the traffic, which landing pages they visited first, etc.

And of course, this spike could be legit, especially if your site ranks for keywords related to a particular topic that happened to get a lot of searches yesterday (breaking news, a controversial report debunking commonly held belief, viral video, etc).

I am referring to Web users. If a user or search engine tried to view those directory listing pages, they will get a Forbidden message, which is what you want to happen. The content in those directories will still be accessible by the pages on the site since the files still exist in those directories, but the pages listing the files in those directories won't be accessible in the browser to users/search engines. In other words, turning off the Directory indexes will not affect any of the content on the site.

There's numerous ways Google could have found those pages and added them to the index, but there's really no way to determine exactly what caused it in the first place. All it takes is for one visit by Google for a page to be crawled and indexed.

If you don't want these pages indexed, then blocking those directories/pages in robots.txt would not be the solution because you would prevent Google from accessing those pages at all going forward. But the problem is that these pages are already in Google's index and by simply using the robots.txt file, you are just telling Google not to visit those pages from now on and thus your pages will remain in the index. A better solution would be to add the no-index, no-cache tags to those pages so the next time Google accesses those pages, they will know to remove those pages from the index.

And now that I've read through your post again, I am now realizing you are talking about file directories rather than normal webpages. What I've wrote above mainly still applies, but I think the quick and easy fix would be to turn off Directory Indexes all together (unless you need them for some reason?). All you have to do is add the following code to your .htaccess file -

Options -Indexes

This will turn off these directory listings so users/search engines can't access them and they should eventually fall out of the Google index.

I know this can be done with location insertion, which requires the use of local extensions. I am not sure if you will need an actual physical presence/Google Places listing for each city or not, but this blog post should help you get moving in the right direction - http://adwords.blogspot.com/2012/04/get-local-with-adwords.html

If you want to redirect all index.html(s) to their roots, then try this code -

RewriteEngine On
RewriteRule ^index.html$ / [R=301,L]
RewriteRule ^(.*)/index.html$ /$1/ [R=301,L]

And yes, Google will treat them as 301 redirects so your juice will be transferred and consolidated.

It's definitely a good idea to 301 redirect index.html to your root to consolidate link juice. I am assuming you have an .htaccess file so the following lines should work -

RewriteEngine On
RewriteBase /

redirect html pages to the root domain

RewriteRule ^index.html$ / [NC,R,L]

If you want to redirect all index.html(s) to their roots, then try this code -

RewriteEngine On
RewriteRule ^index.html$ / [R=301,L]
RewriteRule ^(.*)/index.html$ /$1/ [R=301,L]

And yes, Google will treat them as 301 redirects so your juice will be transferred and consolidated.

You can use the Google AdWords Traffic Estimator to get a rough idea of a recommended budget and estimated CPC - https://adwords.google.com/o/TrafficEstimator

(not set) keywords are usually visits from a Google AdWords campaign where the Google AdWords account is not correctly linked to the Google Analytics account. http://support.google.com/analytics/bin/answer.py?hl=en&answer=2820717 and http://support.google.com/analytics/bin/answer.py?hl=en&answer=1714454

(not provided) keywords are organic searches from Google but are being hidden from your website since Google is encrypting searches from users who are logged into Google or otherwise using the secure version of Google.

The good news is that you should be able to fix the (not set) issue but unfortunately you won't be able to do much regarding the (not provided) keywords.

EDIT - (not set) could also mean visits from all other traffic sources, such as Direct or Referral sites, which inherently are not coming from keyword searches and that is why (not set) is showing up in Google Analytics if you try to look up the keyword used by either of those mediums.

I get this question a lot from my clients. If the domains are brand new and they've never resolved to a site before, then there won't be any kind of impact for SEO, positive or negative. Just make sure the domain names 301 redirect to the primary URL instead of pointing them/setting them up as aliases.

The only time there could be a positive impact is if a domain name used to resolve to a website and that site ranked in the search engines and links were pointed to it. Then you could 301 redirect those domain names to a new URL to transfer at least some of the link juice.

While likely effective in the short term, I think buying links from Fiverr is definitely risky and I would advise against it.

Instead, I would suggest the following

1. Add the site to Google Webmaster Tools and submitting a sitemap (if you haven't already done so).

2. Post links to the site/pages on Twitter, Facebook and other social media sites.

3. Try pinging services such as http://pingler.com/ and http://freebacklinktool.com/

Usually this will be all it takes for a new site to start being indexed in Google.

It's hard to say exactly what the cause is without knowing what the site/keywords are, but I'd recommend checking if these are all New Visits and/or Unique Visitors and where they are geographically located. I've seen huge spikes in traffic by the same group of people/computers in foreign countries before who were hitting my site over and over for some unknown reason. Also, check which search engine is sending the traffic, which landing pages they visited first, etc.

And of course, this spike could be legit, especially if your site ranks for keywords related to a particular topic that happened to get a lot of searches yesterday (breaking news, a controversial report debunking commonly held belief, viral video, etc).

I would just use a sitemap.xml file instead for Google, Bing and Yahoo. Then you can submit the sitemap.xml file within the Google Webmaster Tools and Bing Webmaster Tools (includes Yahoo). You can easily create an XML sitemap at http://www.xml-sitemaps.com/

There's numerous ways Google could have found those pages and added them to the index, but there's really no way to determine exactly what caused it in the first place. All it takes is for one visit by Google for a page to be crawled and indexed.

If you don't want these pages indexed, then blocking those directories/pages in robots.txt would not be the solution because you would prevent Google from accessing those pages at all going forward. But the problem is that these pages are already in Google's index and by simply using the robots.txt file, you are just telling Google not to visit those pages from now on and thus your pages will remain in the index. A better solution would be to add the no-index, no-cache tags to those pages so the next time Google accesses those pages, they will know to remove those pages from the index.

And now that I've read through your post again, I am now realizing you are talking about file directories rather than normal webpages. What I've wrote above mainly still applies, but I think the quick and easy fix would be to turn off Directory Indexes all together (unless you need them for some reason?). All you have to do is add the following code to your .htaccess file -

Options -Indexes

This will turn off these directory listings so users/search engines can't access them and they should eventually fall out of the Google index.

As long as the site and URL structure remain unchanged, then a server migration shouldn't make any impact in the rankings.

That being said, there are a few relatively uncommon things that could potentially cause some issues -

1. If you are going to be moving from a PHP server to an IIS server and you need to change your URLs to .asp or something like that, then you make take a temporary hit while the search engines crawl and index the new versions.

2. If the new server is slower or is prone to frequent downtime, then yes, it could negatively affect your rankings.

3. If you are moving to a new shared server, make sure the rest of the sites on that server are all legitimate sites that haven't been blacklisted by the search engines for any reason. For example, if you move your site to a server that has 100 Russian sites or Nigerian spammer sites all on the same IP, then it's possible your site could be negatively impacted.

Again, these are all relatively uncommon scenarios as long as you are simply moving your site over to another server while keeping everything else the same, so you shouldn't notice any difference in your rankings.