Robots.txt

[internetsalesdrive]

Hi All

Having a robots.txt looking like the below will this stop Google crawling the site

User-agent: *

[MarieHaynes]

If that is the only line in your robots.txt file then it really shouldn't accomplish anything. It's like saying, "Hey...all search engines...take note of this....oh forget it, there's nothing to see here."

I agree with Dave...try to fetch the page in Webmaster tools (Google Search Console). You can also use the Webmaster Tools robots.txt tester which often will tell you if there are issues.

• Hi this is what we thought but Google has not indexed any pages

How old is the site? It can take weeks for a new site to get indexed and then to get ranked as well. Do you see any pages on a site: search for your domain? (i.e. site:example.com). This might sound silly, but are you sure that there is no noindex tag on the page?

[BeanstalkIM]

Via Search Console try to "Fetch As Google" and assuming that works without errors use the submit function.  You'll know very quickly whether you've got technical issues and get the page into the index very quickly.

[BradsDeals]

Hey, throw us a link to your robots.txt file and we can take a look, probably tell you pretty quickly. Without seeing it, we're all pretty much just taking guesses.

[KevinBudzynski]

David's spot on. The User-agent: * mean this section applies to all robots. If you want Google (or any robot) to index your whole site, no need for a robots.txt file.

[O2C]

From your original post I presumed you had not wanted Google to index your pages?

If you want Google to index your pages it can take some time to happen naturally. You might want to submit a sitemap and ask Google to crawl your site within Webmaster Tools.

Robots.txt is normally only used to block crawlers, so you will not need to put any code in there for it to allow Google to crawl.

[internetsalesdrive]

Hi this is what we thought but Google has not indexed any pages

[O2C]

Wouldn't have thought so, you'd need to include this line of code as well:

Disallow: /

That will stop anything from crawling the site.