SEO Web Crawler IP addresses
-
What are the IP addresses for the SEO Web Crawler?
There is a firewall on my clients website before it goes live, I would like to crawl the site before it goes live, but need to provide the web crawlers IP addreses.
Thank you for your time
-
Hi Jason,
This is Megan from the SEOmoz Help Team. Unfortunately, since we crawl from the cloud, RogerBot's IP address is changing all of the time so we cannot specify a particular IP address. Also, we do need the site to be live before we can crawl it, but you could remove the firewall and block all other bots except rogerbot so it won't get indexed anywhere else before it's ready.
I hope this helps! Please let me know if you have any other questions.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO impact of redirecting high ranking mirror site to the main website
During SEO audit for a client I noticed that they had over a dozen duplicate websites that are carbon copies of the main website. This was done via CMS platform and DNS. One of the mirror sites has about 400 indexed pages and has Moz DA of 42 and 137k External Equity-Passing Links. Full metrics comparison is attached. I originally planned on doing rel="canonical" on the mirror site but the CMS vendor never even heard of it and is refusing to implement it in the header. My only other option is doing one to one 301 redirects. Since the mirror site ranks well, even competes with main domain for some positions on the 1st page of SERP, what will be the impact after the redirects? Is doing 301's still the best option? Thanks! PrUpN3q
Moz Pro | | dasickle0 -
SEO for e-commerce, linking, brand mentions, insights diagnostics
Hi, I run the seo for www.bilthamber.com It is an e-commerce site selling car care products, and we also manufacture our own products. I was wondering how the seo for an e-commerce site might compare to the seo for a blog or information based site, or other types of site. And also how the seo for a particular product page on my site might compare to a information based page on my site. Are the tactics the same? Should you try and rank for similar terms, the same generic terms, or complelty unique terms for the best seo? Or should every page try and rank for different terms, to ensure that the engines dont rank 2 or more different pages for the same search term? And how many keywords or search terms should I try and rank a particular product page for? Or an information page, or blog page? Is there a minimum or maximum that is recommomended? Does the more terms you try and rank for on a page take the weight out of each of them? So having 4 really good search terms might be better than having 10 ok generic ones? How do I find out what are my best, most highly converting, most profitable keywords overtime for Google? I can find some information like this, but it is showing the stats for Bing UK, not Google, even though Google UK is my main engine in Moz. My domain authority has decreased by 3 this week, this is the first time it has gone down since I started with Moz a few months back, can anyone explain why? And how this will affect the rankings ect, and how I can get it back up higher agian? And for linking and brand mentions, what are the goals of this feature? Should I find my mentions and ask them to talk about us more and link to us properly? I'm not sure if this is right. Should I be asking other websites to make links to our pages, or should it be organic? And for the insights section, I am recieveing tons of issues a lot to do with duplicate content, will having the rel=carnocial tag fix these issues? I know thats a lot for 1 question, but its been building up in my head! Thanks in advance! Max
Moz Pro | | BiltHamber10 -
Crawlers reporting upper case letter url versions although these have been 301'd to lower case !?
Hi I have a client e-com site who's dev platform is on a windows server Their product pages have been auto-named after the product title, with the first letter in each word being upper case, which has hence translated to the URL having upper cases instances too. I asked them to set up 301 redirects for all url's that had upper case instances to lower case versions, which they say they have done. However I'm still seeing url's with upper case instances showing up in webmaster tools and moz crawl reports but when I copy & paste them into a browser they do redirect to, & resolve in, the lower case version. Its also upper case versions reported in the Google cache! So how come webmaster tools & Moz etc are reporting the upper case versions, surely if redirected it should be the lower case versions All Best Dan
Moz Pro | | Dan-Lawrence0 -
Fresh Web Explorer explorer doesnt seem to work properly
How do I get Fresh Web Explorer explorer to work properly. The feedback I’m getting shows that my brand, Health 2000 had over 4000 mentions on Monday the 13th of January. However when I check out the links that supposedly mention our brand, they bring up general terms related to Health and make no reference to our brand. How can I refine this?
Moz Pro | | Hardley1110 -
Can not start a SEO MOZ campaign?
I keep getting this message, I have a 301 from my domain to my domain with the www prefix. What's the deal? We have detected that the domain www.photosbykristopher.com and the domain photosbykristopher.com both respond to web requests and do not redirect. Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here.
Moz Pro | | KrisPhoto0 -
SEO Yoast data export
Just thought I would give something back. (Is this the right place!) I use Wordpress with the excellent SEO Yoast plugin. I needed a way of extracting the focus keywords that I have entered onto my pages along with the url for use on the SEOmoz On-page Optimisation tool. So I created GetYoastData which outputs to the browser the required data (and a bit more) that can be saved into an csv (Excel) file. Hope you find it useful - Yes it's not polished and yes it might output a blank line now and again but it's fairly useful. http://deanandrews.uk/get-yoast-seo-data/
Moz Pro | | DeanAndrews0 -
Where is the best SEO Glossary of Terms?
Where do you feel the best existing Glossary of SEO / SEM Terms explained currently resides. (and is there a better way to ask this question?) 🙂
Moz Pro | | iansears0 -
Can't find email address or contact form on website I want link from
Hi, How do you use whois or other resources to find an email address for a site that doesn't list their email address or have a contact form? Our competitors look like they have contacted the site but I don't know how to contact the site myself. They have a facebook fan page and a twitter account. Thanks.
Moz Pro | | BobGW0