Moz can't crawl domain due to IP Geo redirect loop
-
Hi,
I'm trying to crawl our domain www.salvationarmy.org.au via my Moz account and it only ever returns results for one page when it should be crawling more than 3,000 pages.
In talking to support, they have said that because of the redirect we have in place it is creating a 302 loop and therefore not delivering results. Usually in this case I would obtain Moz's IP addresses and add them to the redirect settings as an exception, but Moz have said they use cloud-based services for crawling so the IPs change all the time.
Does anyone have any idea how to solve this issue? At this point I've paid for a year's subscription to a product I can't use.
Thanks,
Mel
-
Hi,
If you have manually set up your geo redirect in your htaccess then you could modify your rules to redirect only if not Moz’s crawler (rogerbot) like this:
uk redirect
RewriteCond %{HTTP_USER_AGENT} !=rogerbot
RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^GB$
RewriteRule ^(.*)$ http://uk.abcd.com$1 [L]
Which means both conditions must be satisfied before the redirect happens, the user agent must not be rogerbot, and then it checks the country code. You may have to adjust it a bit depending on your setup but it’s just the same as adding an exception based on IP, so if you could already do that you can set up a user agent condition just as easily.
If you’re using PHP you could use $_SERVER['HTTP_USER_AGENT'] wrap your geoip function with something like:
if($_SERVER['HTTP_USER_AGENT'] != 'rogerbot' ){
You will have to check if it is not empty before you implement it (or work it into your code) as some servers have $_SERVER['HTTP_USER_AGENT'] as not set.
Thanks, hope that gives you a few ideas to try!
Tom
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need help understanding this Moz Chart comparing link metrics against competitors...
Why is my website at 60% and what does it mean? And more importantly, what needs to be done to fix (or not fix) this situation??? Please help... Image of chart attached kcd5r40
Link Explorer | | SamCitron0 -
301 Permanent Redirects
ISSUE: 301 Permanent Redirect | http://www.ctgmanagedit.com 301 Permanent Redirect Yes Target https://www.ctgmanagedit.com | | https://www.ctgmanagedit.com/ 301 Permanent Redirect No | https://moz.com/blog/301-redirection-rules-for-seo According to the above article you lose 15% page rank on permanent redirects. Our site and domain was launched without any migrations. Is there a way to fix this problem?
Link Explorer | | CTGManagedIT
Please 101 me on this question. Thanks,
Kevin Welch
817-557-40910 -
Question in regards to Comparing Metrics with MOZ tools
Hi all! After using the "Compare Links Metrics" in the research tools, a few questions popped up. Particularly, in regards to what key metrics to look for when determining if a websites outbound links are trustworthy and may pass Link-juice. Basically, "if my link is put on that website, will search engines trust it"? I've copied an example below and typed a few comments next to particular metrics. Please share some insight them: Page Authority: 48 Page MozRank: 4.94 -Is 4.94 considered OK? Page MozTrust: 4.96 -Is 4.96 considered OK? Internal Equity-Passing Links: 2,819 External Equity-Passing Links: 2,326 -Lower = better? Total Equity-Passing Links: 5,145 Total Internal Links: 2,844 Total External Links: 2,383 - Lower = better? Total Links: 5,227 Followed Linking Root Domains: 88 Total Linking Root Domains: 129 Linking C Blocks: 30 Equity-Passing Links vs
Link Explorer | | 90miLLA
Non-Equity-Passing Links:** I am assuming that the higher the Non-equity passing links VS the Equity passing links is typically better? What would be an idea ratio?** Internal Links vs
External Links: Lower external = better?0 -
Hi guys. My site, www.x-mini.com attained more links and got better alexa ranking. However, my DA and PA dropped. How can I explain this?
My site, www.x-mini.com attained more links and got better alexa ranking. However, my DA and PA dropped. How can I explain this? Siz5lkp.png
Link Explorer | | Dineshr840 -
I have a robots.txt error on Moz but not on Google Webmaster tools. Wondering what to do.
For the site www.patrickwerry.com, I'm getting a DA of 1 and a Error Code 612: Error response for robots.txt However, when I check webmaster tools, it's showing no errors and allowing robots.txt for the domain. Is there anything I can do to fix the issue on the Moz side so I can get better data? If you can respond in layman's terms even better. 🙂 Not an SEO. Lisa
Link Explorer | | LisaGerber0 -
Linking Domains - Missing
Hello everyone, I was wondering if someone could help me. I keep reviewing my linking domains as I'm expecting a link back from an awards website that I recently sponsored / partnered. However its just not showing. I can see the link on their site and I can see that the other partners links are being tracked when I look at them in Open Site Explorer. I've got a couple of occurrences of this I'm and confused as to why? Thanks for your help!
Link Explorer | | Naomi_AE0 -
How much Domain Authority score required ?
I am creating private blog network. I am buying expired domain. How much domain authority score required to popular my domain. I mean some of the domain i seen have 10+ DA score and age is also 10+ years. Kindly advice.
Link Explorer | | pnb5670 -
How Can I Check New Backlinks?
I have attempted to use Open Site Explorer to attempt to see if the work I have been doing to earn new links is working and I cannot seem to find any new links. I tried the "just Discovered" beta, but I think that relies on Twitter, which these links would not appear on. How can I effectively monitor for new links (1 week or less in age) either using a Moz tool or an external tool that shows the date each link was discovered?
Link Explorer | | dsinger0