Crawl error robots.txt
-
Hello, when trying to access the site crawl to be able to analyze our page, the following error appears:
**Moz was unable to crawl your site on Nov 15, 2017. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
Can help us?
Thanks!
-
@Linda-Vassily yes
-
The page is: https://frizzant.com/ And don't have noindex
-
Thanks Lind and Tawny! i 'll check it
-
Hey there!
This is a tricky one — the answer to these questions is almost always specific to the site and the Campaign. For this Campaign, it looks like your robots.txt file returned a 403 forbidden response to our crawler: https://www.screencast.com/t/f42TiSKp
Do you use any kind of DDOS protection software? That can give our tools trouble and cause us to be unable to access the robots.txt file for your site.
I'd recommend checking with your web developer to make sure that your robots.txt file is accessible to our user-agent, rogerbot, and returning a 200 OK status for that user-agent. If you're still having trouble, it'll be easier to assist you if you contact us through help@moz.com, where we can take a closer look at your account and Campaign directly.
-
I just popped that into ScreamingFrog and I don't see a noindex on that page, but I do see it on some other pages. (Though that shouldn't stop other pages from being crawled.)
Maybe it was just a glitch that happened to occur at the time of the crawl. You could try doing another crawl and see if you get the same error.
-
The page is: http://www.yogaenmandiram.com/ And don't have noindex
-
Hmm. How about on the page itself? Is there a noindex?
-
Yes, our robots.txt it's very simple:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
-
That just says that you are blocking the Moz crawler. Take a look at your robots.txt file and see if you have any exclusions in there that might cause that page not to be crawled. (Try going to yoursite.com/robots.txt or you can learn more about this topic here.)
-
Sorry, the image don't appear
Try now -
It looks like the error you are referring to did not come through in your question. Could you try editing it?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Solved Mozbar Chrome Extension 404 Error
Hello, I am trying to check my website https://whatnumberisiv.com Domain Authority but my Mozbar is not working, When I try to install Mozbar Chrome Extension; it is showing 404 Error. What is the solution of this error? whatnumberisiv.com
Product Support | | Sohail03730 -
Campaign Dashboard Error: "Our Connection to Your Google Account Has Been Lost"
I keep getting the following error in my Campaign Dashboard: “Our connection to your Google account was lost. Don’t worry, you won’t lose any data”. Please reauthorize now. I do reauthorize but by the time I log out and log back in, the same message gets displayed. Any ideas? Thanks, Alan
Product Support | | Kingalan10 -
MOZ not accepting our recent changes it still showing us old Crawl Diagnostics report
Hi, 507 Temporary Redirect We made changes for 302 redirects which are listed in crawl diagnostics report. Now "Compare" and "Wishlist" links are already removed from our source code. All required changes are made but still your report listed Compare and wishlist links. We made changes on Friday (14/8/2015) and waiting for new updated report. Link: http://www.stopwobble.com/ Please let us know what is the exact issue. So that we can fix it.
Product Support | | torbett0 -
I have removed a subdomain from my main domain. We have stopped the subdomain completely. However the crawl still shows the error for that sub-domain. How to remove the same from crawl reports.
Earlier I had a forum as sub-domain and was mentioned in my main domain. However i have now discontinued the forum and have removed all the links and mention of the forum from my main domain. But the crawler still shows error for the sub-domain. How to make the crawler issues clean or delete the irrelevant crawl issues. I dont have the forum now and no links at the main site, bu still shows crawl errors for the forum which doesnt exist.
Product Support | | potterharry0 -
Why is Moz Crawl Diagnostics labelling pages as duplicate when they appear to be different?
Moz Crawl Diagnostics is flagging some pages on the Doorfit website as duplicate, yet the page content is completely different and not identical. Example. Page: http://www.doorfit.co.uk/locks-security/secondary-security Duplicate: http://www.doorfit.co.uk/seals-and-sealants?cat=279 Does anybody have any suggestions as to why this might be the case? Thanks
Product Support | | A_Q0 -
Number of pages crawled = 1; Why?
Since November, we've been trying to figure out why, when I select Crawl Diagnostics, my number of pages crawled is only 1. In mid-november, we changed our URL. That is, we went from www.example.com/home-page/ to www.example.com/new-home-page/. My first assumption was that I needed to re-create my Moz profile. That didn't fix it. The only crawl error we get is the no rel="cannonical" found -- but it's there. We find it on every page, including the home page. Our content shows up in search. Moz bar shows us info for every page. I just don't know what else to check. Everything else in my dashboard seems to look as expected. Specifically, I've turned to Crawl Diagnostics to find 4XX errors on our site. Typically we find one or two per week. Sometimes 0. Sometimes 4 or more. But it's been 0 since November. I highly doubt we've arrived at perfection. Any thoughts?
Product Support | | seo-nicole0 -
Are these MA dashboard issues errors or ccorrect ?
Hi As you know i had spotted some bugs with the new MA dash and now on a more recent report update for a project see the below messages so pls confirm if they are bugs or genuine so i know to inform clients dev to attend to or not ? (PM me for the client/project name if you need it): Crawl Error Moz was unable to crawl your site during the last campaign update!We encountered the following issue:904 Response from Server Timed out Without Completing Page Request Crawl Issue Found: 500 Errors More than 5% of site pages served 500 errors during the last crawl
Product Support | | Dan-Lawrence0 -
MOZ Crawl help
Our MOZ report says it crawled 1800 pages so it reports a lot of errors based on those pages. We don't have that many pages on our site. What is MOZ crawling? I updated the profile to make sure it crawls the filtered page section of Google Analytics.
Product Support | | JessiK0