Moz could not crawl my httpS website
-
Hi,
we have a website with HTTPS, moz could not crawl it and we get "902 : Network errors prevented crawler from contacting server for page" while in logs we see moz robot access but fail after some seconds, what could be the problem, while moz can access site when it is without httpS
| 902 : Network errors prevented crawler from contacting server for page. |
-
I check and seems that your site is OK from mine side - GCE, Amazon EC2 and in Eastern Europe. There are few issues - SSL3, weak DH and files as: https://www.pouyasazan.org/IT-news/25876/css/css/css/css/css/css/css/css/css/css/css/css/css/tooltipster.css (breaking relative and absolute path), also no 404 page because everything return "200 OK" (soft 404). But they isn't critical for your site working and it works. Server works and i just hammering it with many requests.
Please contact with Moz on mail here: help@moz.com or fill form there: https://moz.com/help/contact/pro to investigate your situation.
-
Hi,
website is httpS://www.pouyasazan.org
-
Hi,
I have read those and investigated, those are not reasons, because first server logs shows following:
<code>Line 31915: 2015-12-23 09:23:27 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /robots.txt - 80 - 54.167.7.207 HTTP/1.1 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 422 293 32 Line 31915: 2015-12-23 09:23:27 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /robots.txt - 80 - 54.167.7.207 HTTP/1.1 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 422 293 32 Line 31933: 2015-12-23 09:23:30 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET / - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 302 0 0 362 302 33 Line 31933: 2015-12-23 09:23:30 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET / - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 302 0 0 362 302 33 Line 31953: 2015-12-23 09:23:46 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /sitemap.xml - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 16407 313 74 Line 31953: 2015-12-23 09:23:46 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /sitemap.xml - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 16407 313 74 second crawl test shows following: http://www.pouyasazan.org HTTP Status Code 302 httpS://www.pouyasazan.org HTTP Status Code 0 this shows that your crawler could connect third, when i goto http://analytics.moz.com/search/on-page/4311859.1138194 i can optimiget results for keywork analys that shows your crawler could access my site</code>
-
Did you block AWS IP range or Mozbot? Or did you applied some firewall policy?
PM me your URL so i can take a look on server and network level with different tools.
-
Hello Hamed!
Here's a little more information from our Crawl Errors page:
_ 902 Unable to contact server_
The crawler resolved an IP address from the host name but failed to connect at port 80 for that address. This error may occur when a site blocks Moz's IP address ranges. Please make sure you're not blocking AWS.
This question has come up a few times here in Q&A, so I'd recommend going to the following post for further help in getting this all diagnosed:
https://moz.com/community/q/odd-crawl-test-issuesI hope this helps. Thank you!
Kevin
Help Team
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved What are the top 5 things you recommend you should do when you first subscribe to Moz?
Hi
Getting Started | | djnlowe
Im new to Moz and would like to get the most out of it as quickly as possible. so like the subject say - What are the top 5 things you would recommend I should to get the most out of Moz? #getstarted0 -
How can I identify most relevant websites in Mexico that create content about a specific term?
Hi. I need to define the most relevant sites which are talking about a specific keyword ir order to create an PR strategy based on that term. How can I identify those sites?
Getting Started | | HarolRuiz0 -
Moz Not Crawling Angular SPA
I have a client that just launched a redesigned website using Angular as a single page app. Google appears to be able to crawl the site just fine, but Moz crawl is only finding one page. We have updated the htaccess to allow for Rogerbot and Dotbot, but still unable to crawl any pages other than the home page. Does anyone have experience with this or ideas of why it won't crawl all pages, and how to allow for Moz to crawl all pages? There is a sitemap with approx. 390 pages. Thanks!
Getting Started | | PIN_Celler1 -
Error Selecting Search Engine During Moz Campaign Set-Up
Hi, I'm setting up a Moz campaign for a new client, and when I get to the third step, "Choose the search engines for which you would like to track rankings," there's nothing to select. No drop down, or anything, just a blank field under the heading. If I try to advance, I get an error pop-up that says I must select a search engine. Has anyone else run into this problem? What should I do? GlWPAsc zAmzqFf
Getting Started | | tommyKPseo0 -
Crawling Website
How often does Moz crawl the website? I have a lot of high priority crawl issues but my web development team says that it fixed them. However, Moz still shows me that none of the errors are fixed.
Getting Started | | Leoni0 -
More problems with Moz - Monthly Reporting
Hello All, I may be missing something, but I only want to create monthly reports for my clients from this system. A week in SEO is a terrifying rollercoaster, but any I digress. Currently, i'm adding clients to the system, setting up Analytics, Twitter, Google+, keywords etc. I then go to the custom reports menu as I want to build my report now and schedule it to run at the end of the month, however, because a month has not yet passed, I can only create weekly report segments. This means that I have to add a client to your system and then come back a month later to build a report, this is silly, am I using this wrong or is this not right? I don't care if any data is available, I just want to create the report so I don't have to do it later ideally.
Getting Started | | Paul_Tovey1 -
What is the "other" group in traffic to your site on the moz dashboard ?
What is the "other" group in traffic to your site on the moz analytics dashboard ? "Not provided" is in search - organic search history so it's not that.
Getting Started | | Crocodesign0