How to get rogerbot whitelisted for application firewalls.
-
we have recently installed an application firewall that is blocking rogerbot from crawling our site. Our IT department has asked for an IP address or range of IP addresses to add to the acceptable crawlers. If rogerbot has a dynamic IP address how to we get him added to our whitelist? The product IT is using is from F5 called Application Security Manager.
-
Hi Joel,
Just wondering if you guys came up with a fix for this one? I have the same issue myself...
Thanks!
-
We have a similar issue where we block the Amazon cloudspace but want to allow rogerbot. Our firewall finds it much easier to block by IP range rather than user-agent (rogerbot).
Can you please tell us more about the solution you suggested?
-
Hi, our IP is changing all the time. We do have an option that may resolve your issue though. I'm going to create a ticket as I will need some personal information regarding your account.
Thanks,
Joel
-
Hi kgoss -
I'm not an SEOmoz employee - but just fyi, here are the IP addresses rogerbot used when accessing my site since the start of 2012:
50.17.63.219
184.72.211.58
50.19.77.29
174.129.177.94
184.73.93.243
107.20.64.186
23.22.26.208
107.22.19.89
50.19.188.211
50.19.195.34
107.22.107.114
107.22.137.134
216.244.72.3
75.101.205.127
67.202.42.32
216.176.191.232
23.20.247.207
216.244.72.9
23.22.48.203
216.176.191.234
184.73.13.203
107.20.11.69
50.17.96.67
184.73.137.220
184.73.22.42
184.72.175.148
67.202.13.192
23.21.33.19
107.22.0.173
107.22.157.76
50.16.164.143
50.16.43.91
107.21.143.222
216.176.191.201
107.20.91.74
50.16.88.111
204.236.213.120
184.73.118.141
50.17.38.180
107.21.150.218
216.244.72.10
216.244.72.11
216.244.72.12
23.22.53.56Hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Rogerbot blocked by cloudflare and not display full user agent string.
Hi, We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
Moz Pro | | BB_NPG
Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help. robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool (https://technicalseo.com/tools/robots-txt/) with rogerbot as the user agent this get through fine and we can see our rule has allowed it. When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ? Thanks
Ben Cloudflare activity log, showing differences in user agent strings
2022-07-01_13-05-59.png0 -
Competitor getting External Links from search.aol.com
Recently, I noticed that one of the competitors I track within my Moz campaign received about 12 new inbound links. As a result, there DA jumped about 10 points. I reviewed these new external links and was surprised to see that they are all "search.aol.com/aol/search?query= ..." with Link Anchor Text that is good for the industry we compete in. Can anyone tell me why these are being counted as "Inbound Links". It just doesn't seem right. Is this some sort of black hat seo tactic?
Moz Pro | | itvisionsinc0 -
I'm sure you get this all the time but....
Hi New to Moz, I'm finding it all very helpful and daunting at the same time. I've read heaps of Seo the Moz content and watched every Video I can find. It's all going well and I've managed to get A's on some of my pages, I was hoping some one cold tell me why this site (Link to screan grab http://tinypic.com/r/dnj9di/8 ) is way above me in the Google search results? It appears to be under performing me in Moz but in actual fact it is greatly out performing me in Google. Any insights are more than gratefully received! Thanks Ben
Moz Pro | | SussexChef830 -
Ajax4SEO and rogerbot crawling
Has anyone had any experience with seo4ajax.com and moz? The idea is that it points a bot to a html version of an ajax page (sounds good) without the need for ugly urls. However, I don't know how this will work with rogerbot and whether moz can crawl this. There's a section to add in specific user agents and I've added "rogerbot". Does anyone know if this will work or not? Otherwise, it's going to create some complications. I can't currently check as the site is in development and the dev version is noindexed currently. Thanks!
Moz Pro | | LeahHutcheon0 -
Getting odd results with MOZbar. (Some pages are 0,0,0)
I'm trying to review the Domain aurhotiry, Page Authority, and MozRank & Moztrust for some news websites and I found it odd that may sites will have excellent DA,MT,MR & PA on most of the pages but then when I view one of their blog posts the PA,MR & MT are 0. Here's are two examples Site
Moz Pro | | SheffieldMarketing
http://www.washingtonpost.com/lifestyle Individual Post
http://www.washingtonpost.com/lifestyle/home/checking-in-with-thomas-pheasant/2012/10/30/a2920ed4-1df5-11e2-ba31-3083ca97c314_story.html Site
http://www.philly.com/philly/living/ Individual Post
http://www.philly.com/philly/home/Home_Style_Silver_makes_holiday_decorations_really_shine_.html Does that mean that links from blog posts would not be very benificial? The domain authoity is still crazy high but everything elese is 0. Anyone know why? I'm new to using the Moz bar. Thanks0 -
How I plan to go about getting business - what do you think?
I prepared a real professional looking website analysis packet. The emphasis is on it being VERY professional and classy, nothing cheesy about it. (If you want a copy send me a p.m. and I will email you one). It will take me about 8 minutes to look at their site and run it through a couple tools and and have a quick list of things that are missing. Another 10 minutes for my office manager to edit the template and print a good looking copy. My plan is to take out and deliver about 10 of these a day and call and follow up on them over the phone. My hope is that when they see the quality and professionalism of the report the chances are low it gets thrown in the trash and when I call I have a REASON to call and not some random idiot cold calling. **What do you think? Any suggestions? ** I have a lot of sales background and don't mind talking to people and I know the key will be in FOLLOWING UP which is what MOST people are NOT willing or able to bring themselves to do. I realize that giving away a 'free' report may devalue what I have to offer in some peoples minds...... thought about that. I am just getting started and only have a few clients but I HAVE produced great things for those few but I need to get the ball rolling a little quicker. Thanks for weighing in with your two cents! 🙂 Have an AWESOME week! Matthew
Moz Pro | | Mrupp440 -
Anyone know how/where to get keyword yield ratio?
I'm trying to incorporate keyword yield ratio (the ratio of keywords to pages yielding search traffic) in my seo report. I want to be able to compare on a monthly basis the percentage ratio of keywords to a landing page yielding search traffic. So for example, in July we might have 5 keywords that earned a click to landing page A. But in September, the number of keywords grew to 8 keywords that earned a click to landing A. I would like to be able to report monthly on the % ratio growth of number of keywords to landing pages that yield search traffic. Aim is to grow # of keywords per page... Is there a way I can get this data through seomoz tools or google analytics? Thanks, Gemma
Moz Pro | | RBA0 -
Idea for the Best Method of getting REAL Keyword Search Data
I've been using googles keyword tools to get keyword search data, but I'm pretty sceptical of some of the results it throws out on number of searches. On a number of keywords - particularly when you get to longtail, I've seen results which cannot possibly be right. So - I am trying to find out a way of getting REAL data I can trust. Here are my thoughts on a possible method of getting this key data:
Moz Pro | | James77
1/. Firstly I am taking on the assumption that the adwords campaign monitoring statistics are more reliable and accurate than the keyword research tools - correct me if I am wrong! 2/. Then take a load of keywords you wish to monitor and create didfferent ads for them based of EXACT match. 3/. Make sure non of your adverts conflict with each other, so you may have to turn off any other campaigns you have running. 4/. Put you bids and available spend as high as possible, so that ideally you will your ads will always be shown for every search match, and ideally always come in No1 spot on the ads. IE your aim is to create a neutral environment where your ads are shown for every search match and appear in the exact same position for each search match. Run the campaign for a period of time long enough to be confident that you have enough search data. From this you should have the vital data on keyword searches done on your exact match keywords. 5/. Repeat the test, but this time use phrase matches and again make sure there are no conflicts. 6/. Repeat again with broad match - this would require very careful implimentation not to have any conflicts and you would likely need to make heavy use of the "Not Include" keywords. What are your thoughts on the above process? - Any flaws, or other better solutions? Obviously one key thing with doing this is you need to have to be prepared to have a decent budget to get this data - but it won't be wasted as you will also be getting the adwords traffic. Thanks0