Moz could not crawl my httpS website
-
Hi,
we have a website with HTTPS, moz could not crawl it and we get "902 : Network errors prevented crawler from contacting server for page" while in logs we see moz robot access but fail after some seconds, what could be the problem, while moz can access site when it is without httpS
| 902 : Network errors prevented crawler from contacting server for page. |
-
I check and seems that your site is OK from mine side - GCE, Amazon EC2 and in Eastern Europe. There are few issues - SSL3, weak DH and files as: https://www.pouyasazan.org/IT-news/25876/css/css/css/css/css/css/css/css/css/css/css/css/css/tooltipster.css (breaking relative and absolute path), also no 404 page because everything return "200 OK" (soft 404). But they isn't critical for your site working and it works. Server works and i just hammering it with many requests.
Please contact with Moz on mail here: help@moz.com or fill form there: https://moz.com/help/contact/pro to investigate your situation.
-
Hi,
website is httpS://www.pouyasazan.org
-
Hi,
I have read those and investigated, those are not reasons, because first server logs shows following:
<code>Line 31915: 2015-12-23 09:23:27 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /robots.txt - 80 - 54.167.7.207 HTTP/1.1 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 422 293 32 Line 31915: 2015-12-23 09:23:27 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /robots.txt - 80 - 54.167.7.207 HTTP/1.1 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 422 293 32 Line 31933: 2015-12-23 09:23:30 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET / - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 302 0 0 362 302 33 Line 31933: 2015-12-23 09:23:30 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET / - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 302 0 0 362 302 33 Line 31953: 2015-12-23 09:23:46 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /sitemap.xml - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 16407 313 74 Line 31953: 2015-12-23 09:23:46 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /sitemap.xml - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 16407 313 74 second crawl test shows following: http://www.pouyasazan.org HTTP Status Code 302 httpS://www.pouyasazan.org HTTP Status Code 0 this shows that your crawler could connect third, when i goto http://analytics.moz.com/search/on-page/4311859.1138194 i can optimiget results for keywork analys that shows your crawler could access my site</code>
-
Did you block AWS IP range or Mozbot? Or did you applied some firewall policy?
PM me your URL so i can take a look on server and network level with different tools.
-
Hello Hamed!
Here's a little more information from our Crawl Errors page:
_ 902 Unable to contact server_
The crawler resolved an IP address from the host name but failed to connect at port 80 for that address. This error may occur when a site blocks Moz's IP address ranges. Please make sure you're not blocking AWS.
This question has come up a few times here in Q&A, so I'd recommend going to the following post for further help in getting this all diagnosed:
https://moz.com/community/q/odd-crawl-test-issuesI hope this helps. Thank you!
Kevin
Help Team
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I want to audit my website
Hi professionals, I am newbie in SEO field, I want to audit my company blog "https://iowatreeplanting.com/" How to check domain authority, page authority and other matrics of this blog. I want to rank this blog in google and other search engines but no one can help me to do this! Please guide me in depth i am very thankful to you Thanks!
Getting Started | | Smartlanjabdul1 -
Site Crawl - Crawls only homepage?
Hi Moz Comunity! Joined Moz just 2 weeks ago and slowly trying to get used to tools available in here! Great tools and info available on this site! My concern is that Site Crawl of Moz in my Campaign seems to have crawled only my homepage and no other sub-domains, is there any reason for this? FOr some reason it seems that Moz interacts only with my homepage? Even when I tried the Keyword Exlporer set on Keyword to see if any of my pages rank for any keywords, it seems only my homepage was ranking for a few keywords. It's possible my other sub-domains don't rank for any keywords yet but still, seems suspicious... I have added a link to Site Crawl that says it has crawled only 3 pages on my site, and all are my homepage... Thanks for any help! Jacob s!AlxV7sobbcgmhJB_fXcF4EPzbPSovA
Getting Started | | Shotlife_Studio0 -
Page Count per campaign - Crawl Usage 500,000 Pages
How to you find the page crawl count per campaign? I have 3 campaigns and Moz stats I have used 150,000 pages from 500,000. I want to check this. Thanks
Getting Started | | SJMDT0 -
Keyword ranking in google but not in MOZ
Good day, When I type my keyword in google, the website related to this keyword appears on page 1 (7th link from the top). However in Moz report, the keyword position is #13. I don't understand why. Can someone explain the reason? Thank you,
Getting Started | | Wolfpacksweet0 -
Moz Analytics Shows isights, but none when i click
Hi, i have a campaign report that shows 8 Insights, however when i click on that link which says 8 Insights, the page is blank. Any info would be appreciated. Thanks
Getting Started | | andrewlos0 -
Website errors?
Where can I see my domain website errors. Things like how may pages are missing meta description, duplicate title tags or broken links. I use to see it when I signed in. Now I can't find it.
Getting Started | | gsam1 -
How do I update the crawl issues & Notifications?
I have a list of errors, most relating to missing meta descriptions. I have added a meta description to a page, visited the site and viewed the source, and the meta description is now there. When I go to analyze issues, the report it gives back for the link contains the same missing meta description as previously. How do I get it to update and realize the issue has been fixed?
Getting Started | | ETGg0 -
MOZ cannot connect to GA
As I went thru the process of setting up my campaign, MOZ could not connect to my GA account. Any idea why?
Getting Started | | selectlocks0