Moz can't crawl domain due to IP Geo redirect loop
-
Hi,
I'm trying to crawl our domain www.salvationarmy.org.au via my Moz account and it only ever returns results for one page when it should be crawling more than 3,000 pages.
In talking to support, they have said that because of the redirect we have in place it is creating a 302 loop and therefore not delivering results. Usually in this case I would obtain Moz's IP addresses and add them to the redirect settings as an exception, but Moz have said they use cloud-based services for crawling so the IPs change all the time.
Does anyone have any idea how to solve this issue? At this point I've paid for a year's subscription to a product I can't use.
Thanks,
Mel
-
Hi,
If you have manually set up your geo redirect in your htaccess then you could modify your rules to redirect only if not Moz’s crawler (rogerbot) like this:
uk redirect
RewriteCond %{HTTP_USER_AGENT} !=rogerbot
RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^GB$
RewriteRule ^(.*)$ http://uk.abcd.com$1 [L]
Which means both conditions must be satisfied before the redirect happens, the user agent must not be rogerbot, and then it checks the country code. You may have to adjust it a bit depending on your setup but it’s just the same as adding an exception based on IP, so if you could already do that you can set up a user agent condition just as easily.
If you’re using PHP you could use $_SERVER['HTTP_USER_AGENT'] wrap your geoip function with something like:
if($_SERVER['HTTP_USER_AGENT'] != 'rogerbot' ){
You will have to check if it is not empty before you implement it (or work it into your code) as some servers have $_SERVER['HTTP_USER_AGENT'] as not set.
Thanks, hope that gives you a few ideas to try!
Tom
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Thousands of links with same IP address found in backlinks
Hey, While researching the backlinks of a client's website I found around 3000 links with the structure: http://54.201.231.138/page. The IP doesn't seem to be of their website and the MOZ Spam score for the domain is 1%. Wondering what these links are and if they're a big problem? Thank you in advance. Warmly,
Link Explorer | | JoePen1010
Joe Screenshot 2022-09-30 at 10.58.12.png0 -
What is difference between Moz data than ahrefs?
hi, On ahrefs it's showing me that my web Fit Food has 1.2k backlinks and 140 referring domain. while on Moz it's totally different stats which making me confuse to understand. So is Domain rating is different than Domain Authority?
Link Explorer | | SophiaRPereira1 -
My site links to my clients not showing in Moz
I've got a web design business with a site that's been out there for 17 years - www.biztoolsone.com. I've got links to my clients' sites in my portfolio page. The backlinks list for my clients, when I check them in Moz, don't show the links from this site. They always showed in the list until a few months ago. The site has a DA of 59 & PA (for home page) of 50. The spam score is 0/17. I'm having a tough time telling my clients that the Moz backlinks list is accurate when that site doesn't show in the list. The same is true for www.fayettevillenc.biz - this is one I created years ago to generate backlinks. It's got a spam score of 4/17, so I'm a little more understanding of that one. Thanks for any guidance!
Link Explorer | | wcksmith10 -
My domain is almost 3 years old with a lot of backlinks, but my PA=1 DA=1
Hi there! I have .photography domain. The domain is 3 years old. We know we have a lot of backlinks. However, the DA and PA are still 1. Also the number of linking root domains shows as zero. Can someone help me with it?
Link Explorer | | Tatiana30000 -
Open Site Explorer Messge - Mysite.com redirects to www.mysite.com
Hello All, When I plug in my site's URL http://mysite.com ( Not my actual URL of-course) into the OpenSiteExplorer tool I get the following message: "You entered the URL http://Mysite.com which redirects to http://www.Mysite.com/.
Link Explorer | | jjimen03
Because it's likely to have more accurate metrics, we're showing data for the redirected URL instead.
Click here to analyze http://Mysite.com instead?" Not sure why I am getting this message. On my hosting server, I have defined my preferred domain to be the non-www. version. I have also used a Canonical Tag on Mysite.com/index.html file ( ). In addition, on Google Search Console ( Webmaster Tools ) , I have added both the www. and non-www. properties, where on both property settings, I have defined the non-www. version as my preferred one. Also, on my server, I have not implemented any 301 redirects at all. So my questions are: Why am I receiving this message when plugging in my non-www. URL version of my website into OpenSiteExplorer? Is my DA or PA possibly being split up between the two versions as a result? Should I be concerned? How can I fix this? Thanks in advance.0 -
Why don't links from Reddit show up in Site Explorer?
Some are follow and some are no follow. They show up in webmaster tools and not in site explorer. In addition why do some other links show up in webmaster tools but not site explorer?
Link Explorer | | RafeTLouis0 -
Question in regards to Comparing Metrics with MOZ tools
Hi all! After using the "Compare Links Metrics" in the research tools, a few questions popped up. Particularly, in regards to what key metrics to look for when determining if a websites outbound links are trustworthy and may pass Link-juice. Basically, "if my link is put on that website, will search engines trust it"? I've copied an example below and typed a few comments next to particular metrics. Please share some insight them: Page Authority: 48 Page MozRank: 4.94 -Is 4.94 considered OK? Page MozTrust: 4.96 -Is 4.96 considered OK? Internal Equity-Passing Links: 2,819 External Equity-Passing Links: 2,326 -Lower = better? Total Equity-Passing Links: 5,145 Total Internal Links: 2,844 Total External Links: 2,383 - Lower = better? Total Links: 5,227 Followed Linking Root Domains: 88 Total Linking Root Domains: 129 Linking C Blocks: 30 Equity-Passing Links vs
Link Explorer | | 90miLLA
Non-Equity-Passing Links:** I am assuming that the higher the Non-equity passing links VS the Equity passing links is typically better? What would be an idea ratio?** Internal Links vs
External Links: Lower external = better?0 -
Does Moz allow you to upload your disavow to take into account when analysing links?
I was wondering if moz allows you to upload your disavow file and if it has a consideration in the data supplied? Otherwise how can we accurately use moz to understand the true analytics of our site?
Link Explorer | | MoneySite0