Theresa,

I'm going to support the two answers here: creating a new account is not a good use of time.

The main reason, in my mind, is just that you wouldn't have anything to compare the new site with. If you have new code, new content, new everything, it's important to compare the changes with the old site to make sure you made positive changes.

The best example I can give of a scenarios where you'd want that historic data is with any page on the site that already ranks well. After the page changed, did traffic increase? Decrease? Time of page? It's good to know how these were changed.

I'll also add, historic analytic data can help you find problems caused by the move. Disallowed robots.txt from the test site? Structure issues? Many of these can be found out from looking at your analytic data and having a baseline is important.

Hope that helps!

I'm seeing the meta tag that's added for the first option:

<meta name="robots" content="index, follow" />

... but I could actually access a file at domain.com/robots.txt that had the content mentioned above. When I logged in via FTP, it wasn't there. I added an actual file there with the correct information and reloaded it to make sure it was showing the correct information.

I tested it on my local install and I'm not seeing a robots file being generated.

Very odd!

Just make sure you don't set that Privacy setting in a live directory. It takes weeks/months to fully recover.

I should mention that this is a WordPress site and, with that, I may have answered my own question. Perhaps WordPress generates a robots.txt dynamically when the setting is active at Settings > Privacy?

So here's a weird one...

Client comes to me for some simple changes, turns out there are some major issues with the site, one of which is that none of the correct content pages are showing up in Google, just ancillary (outdated) ones. Looks like an issue because even the main homepage isn't showing up with a "site:domain.com"

So, I add to Webmaster Tools and, after an hour or so, I get the red bar of doom, "robots.txt is blocking important pages." I check it out in Webmasters and, sure enough, it's a "User agent: * Disallow /" ACK!

But wait... there's no robots.txt to be found on the server. I can go to domain.com/robots.txt and see it but nothing via FTP. I upload a new one and, thankfully, that is now showing but I've never seen that before.

Question is: can a robots.txt file be stored in a way that can't be seen?

Thanks!