Moz can't crawl domain due to IP Geo redirect loop
-
Hi,
I'm trying to crawl our domain www.salvationarmy.org.au via my Moz account and it only ever returns results for one page when it should be crawling more than 3,000 pages.
In talking to support, they have said that because of the redirect we have in place it is creating a 302 loop and therefore not delivering results. Usually in this case I would obtain Moz's IP addresses and add them to the redirect settings as an exception, but Moz have said they use cloud-based services for crawling so the IPs change all the time.
Does anyone have any idea how to solve this issue? At this point I've paid for a year's subscription to a product I can't use.
Thanks,
Mel
-
Hi,
If you have manually set up your geo redirect in your htaccess then you could modify your rules to redirect only if not Moz’s crawler (rogerbot) like this:
uk redirect
RewriteCond %{HTTP_USER_AGENT} !=rogerbot
RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^GB$
RewriteRule ^(.*)$ http://uk.abcd.com$1 [L]
Which means both conditions must be satisfied before the redirect happens, the user agent must not be rogerbot, and then it checks the country code. You may have to adjust it a bit depending on your setup but it’s just the same as adding an exception based on IP, so if you could already do that you can set up a user agent condition just as easily.
If you’re using PHP you could use $_SERVER['HTTP_USER_AGENT'] wrap your geoip function with something like:
if($_SERVER['HTTP_USER_AGENT'] != 'rogerbot' ){
You will have to check if it is not empty before you implement it (or work it into your code) as some servers have $_SERVER['HTTP_USER_AGENT'] as not set.
Thanks, hope that gives you a few ideas to try!
Tom
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I find http backlinks to my domain?
Hi, We spotted a number of sites linking to us from http sites, not https sites. How can I use backlink tools to find just those from http sites? Right now I'm looking at exporting a spreadsheet with 3.3 million entries and filtering - and there's not one person here who wants to do that! Thanks in advance. Screen Shot 2021-11-03 at 10.49.12.png
Link Explorer | | mark_seomoz10 -
Does Moz Pro provide a list of external (vs internal) links on our website?
Is there a tool (in MOZ?) that lists all the links on our website that point to external links (vs internal links)?
Link Explorer | | analytics-moz0 -
MOZ Domain Authority Change frequency
Hello Team, I just want to know - is there any MOZ DA algorithm update frequency because we have not seen any movement in DA on website from last few months. Also, is there any matrices which affecting DA that might we are missing. Thanks in Advance.
Link Explorer | | adlift0 -
If I have a MOZ PRO account, do I still need Screaming Frog?
Hi, I have a MOZ PRO account which is brilliant for me. I am about to move my site from HTTP to HTTPS and was looking for a smooth way to list every page and its details so I can make sure I do the correct redirects etc.. before the switch to SSL. Do I need to get Screaming Frog, or can the tools with MOZ Pro ease my path just as well? Thanks
Link Explorer | | SeoSheikh0 -
Can i Force opensiteexplorer to crawl a url?
Hello. I check the QA and didnt find an answer to this. I've check one link on my site which is indexed by google but not by mozopensite explorer. If i search the url i find it on google but not i nopen site explorer. This only means that open site explorer havent find it. IF google index that page this means i'm getting the juice? My question is if google index a page for sure it follow one of the links (follow link). This is importan for me since this is my best DA link, probably a game changer for us. Also, Is there a Way to force opensiteexplorer to crawl this page? Thank you.
Link Explorer | | ebest.cl0 -
How can I tell what pages have links TO this page?
Hello, Mozzers! My client has a rather messy situation, with redirects for HTTP vs. HTTPS, and two different home pages, both in subdirectories. I could explain the whole setup, but I don't think it's relevant for my question, which is this: OSE is showing three links to the page in question, but doesn't tell me WHAT those links are. Am I just missing the information somewhere? Please see screenshot for details. Thanks so much! ~ Scott view?usp=sharing
Link Explorer | | measurableROI0 -
What if all the URL's in my website reported by MoZ do not exist on my website? They existed on an old one that has now been rebuilt?
What if all the top URL's report by Moz in my website do not exist on my website? They only existed on an older website? The only exception in the homepage
Link Explorer | | FCB0 -
Is there an option that's more precise over Open Site Explorer?
I've had folks explain to me before that OpenSiteExplorer is just an estimation, etc. but there are some fairly easy statistics that seem to be different and it makes me nervous that either its wrong or I'm doing something wrong. As you can see in the image, moz isn't read social metrics correctly. It actually used to be pretty spot on, but as you can see with Google+'s its off. Maybe not a big deal right? But for my clients new Moz Analytic print outs it makes somewhat of a difference. Any help? UlwQs3w
Link Explorer | | jonnyholt0