Not possible. Google's not going to index something that is not accessible to everyone.
Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Posts made by Copstead
-
RE: How to allow googlebot past paywallposted in Technical SEO
-
RE: Removing robots.txt on WordPress site problemposted in Technical SEO
yes, the urls being blocked are includes from your Wordpress program.
-
RE: Removing robots.txt on WordPress site problemposted in Technical SEO
Well there is a robots.txt file. You can view it here: http://containerforsale.co.uk/robots.txt
What warnings are you getting in your sitemap submission area? It appears to look alright: http://containerforsale.co.uk/sitemap.xml But I tried to validate it and got a 504 Gateway Time-out error. http://www.xml-sitemaps.com/index.php?op=validate-xml-sitemap&go=1&sitemapurl=http%3A%2F%2Fcontainerforsale.co.uk%2Fsitemap.xml&submit=Validate
-
RE: Removing robots.txt on WordPress site problemposted in Technical SEO
Doesn't appear to be blocked, so maybe it has something to do with your /wp-includes/ directory.
Change the robots.txt file to this:
User-agent *
Disallow:
-
RE: Removing robots.txt on WordPress site problemposted in Technical SEO
copy whatever you have in your robots.txt file here and we will tell you the issue.
SEOmoz has a great article about Robots.txt files here: http://www.seomoz.org/learn-seo/robotstxt