It's hard to say exactly what the cause is without knowing what the site/keywords are, but I'd recommend checking if these are all New Visits and/or Unique Visitors and where they are geographically located. I've seen huge spikes in traffic by the same group of people/computers in foreign countries before who were hitting my site over and over for some unknown reason. Also, check which search engine is sending the traffic, which landing pages they visited first, etc.

And of course, this spike could be legit, especially if your site ranks for keywords related to a particular topic that happened to get a lot of searches yesterday (breaking news, a controversial report debunking commonly held belief, viral video, etc).

I am referring to Web users. If a user or search engine tried to view those directory listing pages, they will get a Forbidden message, which is what you want to happen. The content in those directories will still be accessible by the pages on the site since the files still exist in those directories, but the pages listing the files in those directories won't be accessible in the browser to users/search engines. In other words, turning off the Directory indexes will not affect any of the content on the site.

There's numerous ways Google could have found those pages and added them to the index, but there's really no way to determine exactly what caused it in the first place. All it takes is for one visit by Google for a page to be crawled and indexed.

If you don't want these pages indexed, then blocking those directories/pages in robots.txt would not be the solution because you would prevent Google from accessing those pages at all going forward. But the problem is that these pages are already in Google's index and by simply using the robots.txt file, you are just telling Google not to visit those pages from now on and thus your pages will remain in the index. A better solution would be to add the no-index, no-cache tags to those pages so the next time Google accesses those pages, they will know to remove those pages from the index.

And now that I've read through your post again, I am now realizing you are talking about file directories rather than normal webpages. What I've wrote above mainly still applies, but I think the quick and easy fix would be to turn off Directory Indexes all together (unless you need them for some reason?). All you have to do is add the following code to your .htaccess file -

Options -Indexes

This will turn off these directory listings so users/search engines can't access them and they should eventually fall out of the Google index.

I know this can be done with location insertion, which requires the use of local extensions. I am not sure if you will need an actual physical presence/Google Places listing for each city or not, but this blog post should help you get moving in the right direction - http://adwords.blogspot.com/2012/04/get-local-with-adwords.html

If you want to redirect all index.html(s) to their roots, then try this code -

RewriteEngine On
RewriteRule ^index.html$ / [R=301,L]
RewriteRule ^(.*)/index.html$ /$1/ [R=301,L]

And yes, Google will treat them as 301 redirects so your juice will be transferred and consolidated.

It's definitely a good idea to 301 redirect index.html to your root to consolidate link juice. I am assuming you have an .htaccess file so the following lines should work -

RewriteEngine On
RewriteBase /

redirect html pages to the root domain

RewriteRule ^index.html$ / [NC,R,L]

Hi Jenna,

It's not so much the fact you have 404 pages that is the problem for SEO, but rather the fact your site is creating a problem for the search engines to crawl the site correctly and efficiently since they are getting caught in an endless loop. This can be a problem because the crawlers may get caught in the endless loop and just give up on your site and leave, which means the search engines may not be able to access the rest of the pages on your site and may have a negative impact on your rankings as a whole. One of the most important parts of SEO is to make your website as "friendly" to the search engines as possible so if they caught in endless loops then that is definitely not ideal. Hope that helps!

Patrick

It looks like the crawler is being caught in an endless loop, most likely a result of using relative URLs somewhere on your site. Yes, this is a problem for the site as a whole so I highly recommend implementing absolute URLs throughout the entire site.

Edit - I just looked at your site and this is exactly what it is. The links in your navigation are relative, such as "<a <="" span="">href="</a>../development/default.aspx"" so just change it to absolute URLs such as http://www.yoursite.com/development/default.aspx and it should fix the problem.

Well, all they do is switch the names of the accounts. I'm not sure what Tourism Australia's username was before, but Twitter switched their username over to australia and that way Tourism Australia's followers and tweets transferred over automatically. And since Tourism Australia's username is now australia, which was previously your username, then that means your account had to be moved to a new username which in this case turns out to be Australia1_. Unfortunately I don't think there's much you can do about it at this point.

I'm assuming the account was transferred over to Tourism Australia since the account had not tweeted in over a year https://twitter.com/Australia1_. I'm not sure what Twitter's "official" policy is when it comes to handing over usernames, but I've been successful requesting access to inactive accounts in the past. Twitter frowns upon "username squatting" so it's not surprising they turned the account over to someone who would actually use it.

Do you want to capture the parameter value or do you want to just remove it immediately? If you want to remove it without worrying about capturing the value in Google Analytics, then this code may work -

RewriteEngine On
RewriteCond %{QUERY_STRING} !="" #if there is a query string
RewriteCond %{QUERY_STRING} !^p=.* #wordpress users: allow post tempalinks
RewriteCond %{REQUEST_URI} !^/wp-admin.* #wordpress users: don't break admin
RewriteRule ^(.*)$ /$1? [R=301,L] #remove query string

I got this code from http://briancray.com/posts/htaccess-hack-remove-url-query-strings/