'Domain Does Not Respond To Web Requests'
-
Hi everyone,
This seems to be a fairly common query on the Q&A section, but I haven't been able to find a solution by reading through previous threads.
When I try to set up an SEOmoz campaign for spryz.co.nz, I get that ol' favourite error message:
'We have detected that the domain spryz.co.nz does not respond to web requests. Using this domain, we will be unable to crawl your site or present accurate SERP information.'
The problem isn't being caused by a Robots.txt file, and the site has experienced 99% uptime since it was launched.Traffic stats show that visits are coming through to the site via the search engines, suggesting that it may not be all crawlers that fail to access the site.
I've tried to set up this campaign several times throughout the day, since I've read that sometimes Roger goes on the blink, but I've still not been successful.
Any suggestions as to why Roger might be unable to crawl my website would be great.
-
Having the same issue - would prefer not to post the domain name though. It's my first time using SEOMoz, new PRO user, and I'm literally blocked from my first campaign. The site is responsive, hosted on a great server, there's a robots.txt that's intact and is only disallowing access to one particular directory, and the site is currently indexed in Google because I can see the pages.
Methinks there's something wrong on your end - this question has 120 views which means it's more than a few isolated cases. Should I just go ahead and set up the campaign anyway, ignoring the error?
-
Hi there,
Thanks for reaching out! That is interesting indeed that you aren't able to create a campaign for that particular domain. I do notice that I had trouble at first, but I tried again immediately and was able to go past the no response screen and create a campaign. http://screencast.com/t/H2apiZb0tvB. I have checked with our team and we are not seeing an issue with the crawler so far, however in the past our crawler have had trouble with pages that is missing a Robots.txt (404 when I checked it last) http://screencast.com/t/oacVSiHM3Nyf. However as I have shown, our crawler will now crawl sites in the absence of a robot.txt.
There's a few reason why it could be not responding to our web request:
1. The uptime was sporadic when we made the attempt.
2. Something on the server level is blocking our web service, which is provided by Amazon.Unfortunately the problem is too sporadic, I will start a ticket for you on my end and let's see how my campaign comes back before we decide anything. Hope that helps!
Best,
Peter SEOmoz Help Team.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site with 2 domains - 1 domain SEO opimised & 1 is not. How best to handle crawlers?
Situation: I have a dual domain site:
Getting Started | | DGAU
Domain 1 - www.domain.com is SEO optimised with product pages and should of course be indexed.
Domain 2 - secure.domain.com is not SEO optimised and simply has checkout and payment gateway pages. I've discovered that Moz automatically crawls Domain 2 - the secure.domain.com site and consequently picks up hundreds of errors.
I have put an end to this by adding a robots.txt to stop rogerbot and dotbot (mozs crawlers) from crawling domain 2. This fixes my errors in Moz reports however after doing more research into 'Crawler Control' I figure this might be the best option. My Question: Instead of using robots.txt to stop moz from crawing all of Domain 2 should I use on each page of domain 2? I believe this would then allow moz and google to crawl Domain 2 but also tell them both not to index it.
My understanding is that this would be best, and might even help my overall SEO by telling google not to give any SEO value to the Domain 2 pages?0 -
Seeing all of our site's rankings
Hi everyone, I've started tracking some speciifc keywords here, but I'd also like to get a global overview of all the different keywords we're currently ranking for and in what positions. When I used Ahrefs it offered this sort of report. Does Moz offer such a report?
Getting Started | | JoolsStoneCTM0 -
Domain added to the url
Hi, I am having a problem with my Wordpress site http://www.food-and-garden. It seems the domain www.food-and-garden.com is added to the url, for example http://www.food-and-garden.com/recipe-items/blini-lumpfish-roe/www.food-and-garden.com Right now I have 249 404 errors. That's a lot! I found a Q&A similar to this, and as far as I understand, it has to do with relative links. I quess somewhere on my site there is href="www.food-and-garden.com" instead of href="http://www.food-and-garden.com" My question is, how do I find the broken link? Thank you!
Getting Started | | Food-Garden0 -
Why can't I Ctrl + click on links on Moz any more?
I'm interested if it's just me that gets frustrated by this? I've just Ctrl + clicked a few links to open them in separate tabs and then realised that none of them had opened. I know it's been like this for a while. It's a usability issue as it goes against expected norms, and now I have to right-click and then click "Open in new tab" on each link, which is more time-consuming and frustrating. More and more websites seem to be losing their Ctrl + click on links ability (JavaScript often breaks it). I don't know if there's a Mac equivalent... Anyway, I hope that doesn't seem like I'm too angry. It just frustrates me a little and I hope it gets fixed. 🙂 Edit - I've just realised these are getting blocked by Chrome's pop-up blocker - but why? It's only an issue on a small number of websites.
Getting Started | | Alex-Harford1 -
Can you help me with setting up a campaign? I get stuck after I put in the domain name and the other information doesn't go through.
Can you help me with setting up a campaign? I get stuck after I put in the domain name and the other information doesn't go through. Regards Roshni M
Getting Started | | harmanseo0 -
Why I can't add more campaign to my moz analytics?
As we know that being a standard pro member in Moz community, we have at least 5 campaigns to add and track & as well in new moz analytics. But when I just made http://www.norwoodgreen.in website campaign an archive to make active or add one of the old himachalpackagetour.com campaign which I was unable to do WHY WHY WHY? Let me know . .? Also do guide or train me How to use Moz analytics as I was not enough familiar with it as I am habitual with in Pro tools. Best,
Getting Started | | Futura
Teginder0 -
How to get moz to crawl a staging domain that is blocked by robots.txt
Is it possible to get Moz to do a crawl report on a domain blocked by robots.txt and actually display all errors instead of only one saying the domain was blocket in robots.txt? Anything i can add to robots.txt to make moz able to do the crawl report but still hinder google from crawling a staging domain?
Getting Started | | classifiedtech0