Robots.txt error

[Shanidel]

Moz Crawler is not able to access the robots.txt due to server error. Please advice on how to tackle the server error.

[jocameron]

Hello Shanidel,

Jo from the Moz help team here.

I've had a look at your site and I've not been able to access your robot.txt file, this is what I'm seeing in the browser

https://screencast.com/t/JjQI1WTH3ni

I'm also seeing this error when I check your robots.txt file through a third party tool

https://screencast.com/t/pxsP9pL5

So it looks to me like may be some intermittent issues with your robots.txt file. I would advise reaching out to your web developer to see if they can check your robots.txt file and make sure it's accessible.

If you're still having trouble please let us know at help@moz.com

Best of luck!

Jo

[Shanidel]

Hi,

I'm still having this problem. Moz is unable to crawl the site saying there is a problem with the robots.txt file.

Sorry.

[Roman-Delcarmen]

happy to been useful

[Shanidel]

Below is the exact message that i received:

**Moz was unable to crawl your site on Aug 29, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.

[Roman-Delcarmen]

yoursite.com/robot.txt -----> this is how your robot.txt file should be, so first I will recommend you test your robot.txt file to see if everything is ok, if dont there is an explanation about how to create a robot.txt

How to create a /robots.txt file

Where to put it

The short answer: in the top-level directory of your web server.

The longer answer:

When a robot looks for the "/robots.txt" file for URL, it strips the path component from the URL (everything from the first single slash), and puts "/robots.txt" in its place.

For example, for "http://www.example.com/shop/index.html, it will remove the "/shop/index.html", and replace it with "/robots.txt", and will end up with "http://www.example.com/robots.txt".

So, as a web site owner you need to put it in the right place on your web server for that resulting URL to work. Usually that is the same place where you put your web site's main "index.html" welcome page. Where exactly that is, and how to put the file there, depends on your web server software.

Remember to use all lower case for the filename: "robots.txt", not "Robots.TXT.

See also:

• What program should I use to create /robots.txt?
• How do I use /robots.txt on a virtual host?
• How do I use /robots.txt on a shared host?

[manoman88]

Hi,

Can you please share the message you're receiving ? Also, did you check your Google Search Console to see if Google can access to your website ? Knowing the type of errors is the key to advice you.