Crawl error robots.txt
-
Hello, when trying to access the site crawl to be able to analyze our page, the following error appears:
**Moz was unable to crawl your site on Nov 15, 2017. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
Can help us?
Thanks!
-
@Linda-Vassily yes
-
The page is: https://frizzant.com/ And don't have noindex
-
Thanks Lind and Tawny! i 'll check it
-
Hey there!
This is a tricky one — the answer to these questions is almost always specific to the site and the Campaign. For this Campaign, it looks like your robots.txt file returned a 403 forbidden response to our crawler: https://www.screencast.com/t/f42TiSKp
Do you use any kind of DDOS protection software? That can give our tools trouble and cause us to be unable to access the robots.txt file for your site.
I'd recommend checking with your web developer to make sure that your robots.txt file is accessible to our user-agent, rogerbot, and returning a 200 OK status for that user-agent. If you're still having trouble, it'll be easier to assist you if you contact us through help@moz.com, where we can take a closer look at your account and Campaign directly.
-
I just popped that into ScreamingFrog and I don't see a noindex on that page, but I do see it on some other pages. (Though that shouldn't stop other pages from being crawled.)
Maybe it was just a glitch that happened to occur at the time of the crawl. You could try doing another crawl and see if you get the same error.
-
The page is: http://www.yogaenmandiram.com/ And don't have noindex
-
Hmm. How about on the page itself? Is there a noindex?
-
Yes, our robots.txt it's very simple:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
-
That just says that you are blocking the Moz crawler. Take a look at your robots.txt file and see if you have any exclusions in there that might cause that page not to be crawled. (Try going to yoursite.com/robots.txt or you can learn more about this topic here.)
-
Sorry, the image don't appear
Try now -
It looks like the error you are referring to did not come through in your question. Could you try editing it?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Campaign Dashboard Error: "Our Connection to Your Google Account Has Been Lost"
I keep getting the following error in my Campaign Dashboard: “Our connection to your Google account was lost. Don’t worry, you won’t lose any data”. Please reauthorize now. I do reauthorize but by the time I log out and log back in, the same message gets displayed. Any ideas? Thanks, Alan
Product Support | | Kingalan10 -
Moz has stopped crawling my site
Hello, We have been using Moz Pro for over a year on our site and in the last month noticed that our site is not being crawled anymore. I took a look in Google Search Console, and everything seems to be fine there, so I think it is just the Moz tool that is not working. Has anyone else experienced this? Are there any tips for troubleshooting it? Thank you, Adam
Product Support | | cwells0 -
I have suddenly got a lot of duplicate page title errors in Moz, please can you advise what to do ?
my site used to be http and i have now converted to https. my site is www.paulsummerfieldphotography.co.uk, please advise how to solve the duplicate page title errors ?
Product Support | | Paul_Ward0 -
Google plus and 404 error message
So this is strange - i created a bunch of pages on Google Plus - and now that I started to use MOZ I noticed that there were 95 404 errors - When I look at them I see the Google+ there ( see example) Has any one experienced this issue? If yes, anyway to fix these 404 errors? or what's the best way to address this issue?http://www.accu-time.com/solutions-2/google.com/+AccuTimeSystemsIncEllington
Product Support | | carlosbernal100 -
HTTPS (SSL) Error Encountered.
Hi guys, We received the following error from our Moz report, our hosting companies says it's isn't an issue but I wanted to get your feedback as it seems a bit odd. We recently moved eventfull.co.nz to a VPS and setup an EV SSL. We moved another site to the same VPS and added EV SSL and it isn't reporting any issue. Hosting companies feedback below error message | Crawl Error Error Code 804: HTTPS (SSL) Error Encountered Your page requires an SSL security certificate to load (using HTTPS), but the Moz Crawler encountered an error when trying to load the certificate. Our crawler is pretty standard, so it's likely that other browsers and crawlers may also encounter this error. If you have this error on your homepage, it prevents the Moz crawler (and some search engines) from crawling the rest of your site. | | I am just running some tests on the SSL cert installation now, but so far all appears to be fine. I've checked your .htaccess file and there is nothing that should be blocking them, and the logs show nothing unexpected or any SSL failures. SSL Labs test returns an A rating with no errors. One possibility (though, it's a long shot) is that we never increased the HSTS time beyond 3600, so it's possible that the crawler is failing because of this perceived insecurity. However, if there is some other issue going on, increasing the HSTS time would be a bad idea, so I suggest some further monitoring and testing. You may need to contact Moz about the issue and see if they can help. Is there any other evidence of SSL failing to load? Have you experienced any other issues, or is it only the Moz report that is indicating any problem? |
Product Support | | ModowestNZ0 -
What is the difference between the "Crawl Issues" report and the "Crawl Test" report?
I've downloaded the CSV of the Crawl Diagnositcs report (which downloads as the "Crawl Issues" report) and the CSV from the Crawl Test Report, and pulled out the pages for a specific subdomain. The Crawl Test report gave me about 150 pages, where the Crawl Issues report gave 500 pages. Why would there be that difference in results? I've checked for duplicate URLs and there are none within the Crawl Issues report.
Product Support | | SBowen-Jive0 -
I have removed a subdomain from my main domain. We have stopped the subdomain completely. However the crawl still shows the error for that sub-domain. How to remove the same from crawl reports.
Earlier I had a forum as sub-domain and was mentioned in my main domain. However i have now discontinued the forum and have removed all the links and mention of the forum from my main domain. But the crawler still shows error for the sub-domain. How to make the crawler issues clean or delete the irrelevant crawl issues. I dont have the forum now and no links at the main site, bu still shows crawl errors for the forum which doesnt exist.
Product Support | | potterharry0 -
Moz Campaign Error
A lot of links seem to be broken or glitching with the Moz Campaigns dashboard. For whatever reason moz decided to archive my campaigns without permission. It then will not let me activate them again, forcing me to recreate them. When I go to the campaigns page, it will not show me my active campaigns, including ones that I have just made and it is impossible to get to them unless I save the campaign id and manually navigate to it in the browser. This has been tested on every web browser, so it is not some kind of user error or browser thing on my end. How can I get to my active campaigns since Moz will not show them to me?
Product Support | | Atomicx1