Moz can't crawl domain due to IP Geo redirect loop
-
Hi,
I'm trying to crawl our domain www.salvationarmy.org.au via my Moz account and it only ever returns results for one page when it should be crawling more than 3,000 pages.
In talking to support, they have said that because of the redirect we have in place it is creating a 302 loop and therefore not delivering results. Usually in this case I would obtain Moz's IP addresses and add them to the redirect settings as an exception, but Moz have said they use cloud-based services for crawling so the IPs change all the time.
Does anyone have any idea how to solve this issue? At this point I've paid for a year's subscription to a product I can't use.
Thanks,
Mel
-
Hi,
If you have manually set up your geo redirect in your htaccess then you could modify your rules to redirect only if not Moz’s crawler (rogerbot) like this:
uk redirect
RewriteCond %{HTTP_USER_AGENT} !=rogerbot
RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^GB$
RewriteRule ^(.*)$ http://uk.abcd.com$1 [L]
Which means both conditions must be satisfied before the redirect happens, the user agent must not be rogerbot, and then it checks the country code. You may have to adjust it a bit depending on your setup but it’s just the same as adding an exception based on IP, so if you could already do that you can set up a user agent condition just as easily.
If you’re using PHP you could use $_SERVER['HTTP_USER_AGENT'] wrap your geoip function with something like:
if($_SERVER['HTTP_USER_AGENT'] != 'rogerbot' ){
You will have to check if it is not empty before you implement it (or work it into your code) as some servers have $_SERVER['HTTP_USER_AGENT'] as not set.
Thanks, hope that gives you a few ideas to try!
Tom
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Google results linking domains
When I check the linking domains of a root domain, a lot of Google results from other countries appear. Does this mean all of these domains are linking to my root domain? Can anybody explain this and tell me if this can be seen as spam or not? linking-domain-moz.png
Link Explorer | | dirkjandirkjan02 -
Links in link lists not being discovered by Moz
I uploaded a couple of link lists to Moz about a week ago and have been tracking them. Most of the links on the list are from significant 90+ DA domains such as Wikipedia, Medium, etc. So far, Moz only seems to have discovered about 4 the 200 or so links I loaded. Do we know how long it takes for these to be discovered by Moz? Quite new to this, so any help understanding timelines for this would be appreciated.
Link Explorer | | RobertPhillips180 -
If I migrate to a new domain, does my Domain Authority score get migrated also?
For business reasons, I migrated my domain with a DA score of 85 to a new domain. After some time, the DA score on the new domain is only 45. My migration plan and redirects went smooth and all redirected with best practices in mind. I am still ranking well for almost all of the keywords where I've 301 redirected old URL to new. Any feedback welcome!
Link Explorer | | KenSchaefer1 -
Can I change the Link Explorer from check with Google US?
I'm trying to check the ranking keywords for a specific article with the Link Explorer, but when I hover over Ranking Keywords on the Overview it tells me that it's looking on Google US. We're a UK based company and the keywords we're trying to rank for are UK based - is there a way of changing this to match?
Link Explorer | | Kestra0 -
Why are recently deleted pages still appearing in the latest MOZ crawl?
Newbie, so please forgive!! OK, so I'm doing my 1st site optimization. It is reporting errors from pages that were deleted a couple of days ago. And I JUST signed up today. Where is this info coming from? Thanks, Billy
Link Explorer | | NewSEOguy0 -
How to force moz to crawl my backlinks?
I have some good number number of backlinks in my webmaster tools. But, open site explorer is showing very few backlinks. How to force moz to crawl all the backlinks? Or is there any way to submit backlinks to moz?
Link Explorer | | sankar7890 -
Why does Open site explorer only show a fraction of the linked Domains that Google does?
One of my sites ( http://www.georgerossphotography.com ) shows 19 root domains and 155 links on Open Site Explorer. This doesn't pass the sniff test with me and when I look at Google Webmasters they show 120 domains with 1,870 links. I am assuming that Google use it's own data for ranking purposes which makes me question the validity of Open Site Explorer? Am I missing something ? I have been using OSE as my primary tool and now I feel that it has little value. I would appreciate any feedback. George.
Link Explorer | | sirgeorge0