Invisible robots.txt?

[joshcanhelp]

So here's a weird one...

Client comes to me for some simple changes, turns out there are some major issues with the site, one of which is that none of the correct content pages are showing up in Google, just ancillary (outdated) ones. Looks like an issue because even the main homepage isn't showing up with a "site:domain.com"

So, I add to Webmaster Tools and, after an hour or so, I get the red bar of doom, "robots.txt is blocking important pages." I check it out in Webmasters and, sure enough, it's a "User agent: * Disallow /" ACK!

But wait... there's no robots.txt to be found on the server. I can go to domain.com/robots.txt and see it but nothing via FTP. I upload a new one and, thankfully, that is now showing but I've never seen that before.

Question is: can a robots.txt file be stored in a way that can't be seen?

Thanks!

[AjazMozPro]

Hi Josh

Did you ever find out how this was happening? 
I've got the same issue with a wordpress site.. no robots.txt visible in FTP but it is accessible in a browser to view.

[joshcanhelp]

I'm seeing the meta tag that's added for the first option:

<meta name="robots" content="index, follow" />

... but I could actually access a file at domain.com/robots.txt that had the content mentioned above. When I logged in via FTP, it wasn't there. I added an actual file there with the correct information and reloaded it to make sure it was showing the correct information.

I tested it on my local install and I'm not seeing a robots file being generated.

Very odd!

[Desiree-CP]

Yes, you probably answered your own question.  In WordPress, there are two different settings under Settings > Privacy:

1. I would like my site visible to everyone, including search engines and archivers.

2. I would like to block search engines, but allow normal visitors

If option #2 was selected, WordPress doesn't create a robots.txt file for you but instead it automatically generates a tag on every single page.

I hope that helps!

[joshcanhelp]

Just make sure you don't set that Privacy setting in a live directory. It takes weeks/months to fully recover.

[Superb]

This is interesting. I am currently working on the robots.txt and testing it for different purposes. I also thought to do some test with wordpress websites as well so thanks for the update I’ll keep that in mind before actually testing different stuff.

Thanks!

[joshcanhelp]

I should mention that this is a WordPress site and, with that, I may have answered my own question. Perhaps WordPress generates a robots.txt dynamically when the setting is active at Settings > Privacy?