Moz could not crawl my httpS website
-
Hi,
we have a website with HTTPS, moz could not crawl it and we get "902 : Network errors prevented crawler from contacting server for page" while in logs we see moz robot access but fail after some seconds, what could be the problem, while moz can access site when it is without httpS
| 902 : Network errors prevented crawler from contacting server for page. |
-
I check and seems that your site is OK from mine side - GCE, Amazon EC2 and in Eastern Europe. There are few issues - SSL3, weak DH and files as: https://www.pouyasazan.org/IT-news/25876/css/css/css/css/css/css/css/css/css/css/css/css/css/tooltipster.css (breaking relative and absolute path), also no 404 page because everything return "200 OK" (soft 404). But they isn't critical for your site working and it works. Server works and i just hammering it with many requests.
Please contact with Moz on mail here: help@moz.com or fill form there: https://moz.com/help/contact/pro to investigate your situation.
-
Hi,
website is httpS://www.pouyasazan.org
-
Hi,
I have read those and investigated, those are not reasons, because first server logs shows following:
<code>Line 31915: 2015-12-23 09:23:27 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /robots.txt - 80 - 54.167.7.207 HTTP/1.1 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 422 293 32 Line 31915: 2015-12-23 09:23:27 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /robots.txt - 80 - 54.167.7.207 HTTP/1.1 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 422 293 32 Line 31933: 2015-12-23 09:23:30 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET / - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 302 0 0 362 302 33 Line 31933: 2015-12-23 09:23:30 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET / - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 302 0 0 362 302 33 Line 31953: 2015-12-23 09:23:46 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /sitemap.xml - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 16407 313 74 Line 31953: 2015-12-23 09:23:46 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /sitemap.xml - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 16407 313 74 second crawl test shows following: http://www.pouyasazan.org HTTP Status Code 302 httpS://www.pouyasazan.org HTTP Status Code 0 this shows that your crawler could connect third, when i goto http://analytics.moz.com/search/on-page/4311859.1138194 i can optimiget results for keywork analys that shows your crawler could access my site</code>
-
Did you block AWS IP range or Mozbot? Or did you applied some firewall policy?
PM me your URL so i can take a look on server and network level with different tools.
-
Hello Hamed!
Here's a little more information from our Crawl Errors page:
_ 902 Unable to contact server_
The crawler resolved an IP address from the host name but failed to connect at port 80 for that address. This error may occur when a site blocks Moz's IP address ranges. Please make sure you're not blocking AWS.
This question has come up a few times here in Q&A, so I'd recommend going to the following post for further help in getting this all diagnosed:
https://moz.com/community/q/odd-crawl-test-issuesI hope this helps. Thank you!
Kevin
Help Team
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I access old data/keyword research if I cancel my Moz Pro account?
I'm currently on the free month trial period for Moz Pro and I will probably cancel the account before the free period ends, but if I want to renew my subscription later, what happens to all the previous data? And does all the keyword research I've done disappear when I cancel it, or is it restored when I renew the subscription? Any insight is helpful! Thank you!
Getting Started | | TeamOneRep0 -
Changing the Moz Crawl Date
Hello, I am wondering where I can change the date of Crawl by Moz. I would like to change this crawl period from one week to 2 or even 3 weeks for Moz to crawl my website. Hope to hear from anyone soon. Kind regards, Koen.
Getting Started | | Koenniiee1 -
Can Moz Monitor a JS Site?
We are building a new site that, on the blog landing page, uses JS to populate the individual blog article links on the page. The links are not viewable in the page source, but do appear once the page fully renders. After running an on-demand crawl of the site (in QA...not indexed yet), it appears that Moz isn't indexing these pages, and it also isn't reading other page elements that load later (like an H1 that is rendered in JS but not in page source). Are we going to be able to use Moz to track this site? Is there some setting to help?
Getting Started | | Rodrigo-DC0 -
Why is analytics.moz.com showing up a traffic source in Google Analytics?
Under Traffic Sources, analytics.moz.com is listed as a source for 37 sessions this past month. What might this be? Is it legitimate traffic or should I be filtering this?
Getting Started | | mphdavidson0 -
New to Moz Pro? Join our latest free webinar this Friday!
Hello everyone! We'll be holding a webinar on tomorrow to help new members learn about what all Pro has to offer, show some off our most popular tools, and get you comfortable with the dashboard. Register here: https://www3.gotomeeting.com/register/355765902 Date: Friday, September 26th (this Friday!) Time: 10:00 AM - 11:00 AM PDT Hope to see you all there!
Getting Started | | jennita1 -
Getting Redirect Loops in MOZ using Chrome
Been getting bizarre Redirect Loops from Chrome after I log-in to MOZ. Has anyone had something like this happen? I've tried clearing cache, rebooting, etc. but no luck. Thanks in advance!
Getting Started | | danny.wood1 -
Moz API and license
Hi Support,We are new to SEOMOZ and would appreciate if you can answer the following:1. Can we build our own Hubspot right to our own website where members can access and see how their sites are performing using your API? If that is possible, do you have tips/suggestions on how?2. What is a crawl limit? How does that work?3. How many sites can we work on under the developers - professional license ?Looking forward to your responses.Thanks,Tony
Getting Started | | chickenjoy20130