Moz can't crawl domain due to IP Geo redirect loop
-
Hi,
I'm trying to crawl our domain www.salvationarmy.org.au via my Moz account and it only ever returns results for one page when it should be crawling more than 3,000 pages.
In talking to support, they have said that because of the redirect we have in place it is creating a 302 loop and therefore not delivering results. Usually in this case I would obtain Moz's IP addresses and add them to the redirect settings as an exception, but Moz have said they use cloud-based services for crawling so the IPs change all the time.
Does anyone have any idea how to solve this issue? At this point I've paid for a year's subscription to a product I can't use.
Thanks,
Mel
-
Hi,
If you have manually set up your geo redirect in your htaccess then you could modify your rules to redirect only if not Moz’s crawler (rogerbot) like this:
uk redirect
RewriteCond %{HTTP_USER_AGENT} !=rogerbot
RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^GB$
RewriteRule ^(.*)$ http://uk.abcd.com$1 [L]
Which means both conditions must be satisfied before the redirect happens, the user agent must not be rogerbot, and then it checks the country code. You may have to adjust it a bit depending on your setup but it’s just the same as adding an exception based on IP, so if you could already do that you can set up a user agent condition just as easily.
If you’re using PHP you could use $_SERVER['HTTP_USER_AGENT'] wrap your geoip function with something like:
if($_SERVER['HTTP_USER_AGENT'] != 'rogerbot' ){
You will have to check if it is not empty before you implement it (or work it into your code) as some servers have $_SERVER['HTTP_USER_AGENT'] as not set.
Thanks, hope that gives you a few ideas to try!
Tom
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I copy a list a of keywords that I have manually selected in the KW explorer?
I am analyzing the keywords of a competitors, but only a bunch will be relevant for me, how can I manually select those I want to export?
Link Explorer | | Anagg0 -
MOZ link explorer is not showing any result
Whenever I am searching in link explore, it is only showing the blanks with default zero value. It was showing data for my website 1 hour ago but now not showing. SYphSNw
Link Explorer | | gb.select4you3 -
Historical Domain Authority
Hello, Where can I find historical information on our domain authority? I would like to compare year over year. TIA!
Link Explorer | | jennifer-garcia3 -
Internal Equity-Passing Links not getting crawled in Moz Open Site Explorer?
Internal Equity-Passing Links not getting crawled in Moz Open Site Explorer. What is the cause of this? We've checked the robots.txt and htaccess file, but so far we can't find anything that would be blocking Moz from crawling the internal links. We manage loads of other clients on this platform and this is the first time we've run into this issue. What else can I check?
Link Explorer | | OozleMedia0 -
I'm doing a link audit and I want to download a spreadsheet of all the links to my website in OSE. There has to be a way to do that right?!
I'm doing a link audit and I want to download a spreadsheet of all the links to my website in OSE. There has to be a way to do that right?!
Link Explorer | | jennie.evans0 -
OSE Compare Domains had Gone
So I have not been on Moz for a while but I have always liked the ability to compare multiple domains in Open site explorer. Has Moz removed this feature. if so... why?
Link Explorer | | BryanCasson0 -
How can I get a list of all the links going out from my website
How can I get a list of all the links going out from my website and from which pages of my site. can I get this list using Moz, I also have ahref.com account, can I use that if yes how or do I need to use some other tools. Thank you
Link Explorer | | SinaKashani0 -
Domain Authority still down
It was mentioned several weeks ago (http://moz.com/community/q/domain-authority-drops) that there had been a 2 point drop in the DA, and it was to be fixed at the "next" index. The index refresh happened 17th Jan but the DA seemed to remain the two points lower than expected. Is it just my sites 😞 or has this not been fixed yet... and will it be fixed? Regards, Gordon
Link Explorer | | Gordon_Hall0