Quick update - by amending the robots text file and switching sitemap plugin over to Yoast I finally got the sitemap to index without robots.txt warnings although the Home page of site was not indexed, 'oh dear'.  5 out of the 7 pages in the sitemap were indexed by Google so It's a start but some more investigating to be done on my side.

Dan,

Cant thank you enough! The sitemap request is still pending in Google - maybe I sent too many requests But it's time to sit back and wait for the good news hopefully. Thanks again.

Hi Dan,

I followed the above advice and switched to the Yoast generated sitemap but after testing on http://www.xml-sitemaps.com/validate-xml-sitemap.html I got the following result - no idea what it means but it looks nasty...

Schema validating with XSV 3.1-1 of 2007/12/11 16:20:05Schema validator crashed

The maintainers of XSV will be notified, you don't need to
send mail about this unless you have extra information to provide.
If there are Schema errors reported below, try correcting
them and re-running the validation.Target: http://containerforsale.co.uk
   (Real name: http://containerforsale.co.uk
    Server: Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4)The target was not assessedLow-level XML well-formedness and/or validity processing output
Warning: Undefined entity raquo
 in unnamed entity at line 16 char 83 of http://containerforsale.co.uk
Warning: Undefined entity nbsp
 in unnamed entity at line 160 char 10 of http://containerforsale.co.uk
Error: Expected ; after entity name, but got =
 in unnamed entity at line 274 char 631 of http://containerforsale.co.u

Indeed, thanks everyone - it's really appreciated!

I have updated the robots.txt as indicated and re submitted site map but looks like Google still has problems with my site since the error warning for robots is there after the processing is done.

Quick question - I am using a plugin called Google XML Sitemaps which has the following tick box option.

'Add sitemap URL to the virtual robots.txt file'.
The virtual robots.txt generated by WordPress is used. A real robots.txt file must NOT exist in the blog directory!'

Should this box be ticked or un-ticked please? Fyi I currently don't have the box ticked.

Ok thanks Brent, I changed to

User-agent: *

Disallow:

Sitemap: http://containerforsale.co.uk/sitemap.xml.gz

Guess I will just have to wait for Google to refresh now...

Thanks for the heads up.

The warning just says 7 Url''s blocked by robots.txt. - have seen this issue posted on the WordPress boards by others but no real insight into solutions.

Perhaps I should try your idea of

Change the robots.txt file to this:

User-agent *

Disallow:

Sitemap: http://containerforsale.co.uk/sitemap.xml.gz

Its weird, the front page warning on Google webmaster for robots has disappeared now, but still got the warnings in the sitemap submission area. My host suggests I just wait a bit longer for Google to update because he said same as you - that there doesn't seem to be any robot.txt file.

Hey Guys,

Thanks for your replies...the domain is http://containerforsale.co.uk ,My host told me to look in the Public HTML file folder for the robots.txt file and just delete it but can't see it in there?

My host said he found a tester site and it doesn't report any issues:

http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php

This is the display I get from http://containerforsale.co.uk/robots.txt

User-agent *

Disallow: /wp-admin/
Disallow: /wp-includes/

Sitemap: http://containerforsale.co.uk/sitemap.xml.gz

Hi..am a little confused since I ticked the box in WordPress to allow search engines to now crawl my site (previously asked for them not to) but Google webmaster tools is telling me I still have robots.txt blocking them so am unable to submit the sitemap.

Checked source code and the robots instruction has gone so a little lost. Any ideas please?

Hey Ben,

Thanks. Am using a Wootheme called Simplicity. I have just left the minify box unchecked in W3 Total Cache since this was causing the problem. The site speed seems to be ok though even with this disabled.

In the interests of making my site faster I have recently come across the suggestion of removing unwanted PHP from my WooThemes WordPress theme. The suggestion is to hard code the choices I have made in the WordPress template to reduce on database calls.

Has anyone actually done this to their WordPress theme before and seen any measurable results?