Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
SEO Web Crawler IP addresses
-
What are the IP addresses for the SEO Web Crawler?
There is a firewall on my clients website before it goes live, I would like to crawl the site before it goes live, but need to provide the web crawlers IP addreses.
Thank you for your time
-
Hi Jason,
This is Megan from the SEOmoz Help Team. Unfortunately, since we crawl from the cloud, RogerBot's IP address is changing all of the time so we cannot specify a particular IP address. Also, we do need the site to be live before we can crawl it, but you could remove the firewall and block all other bots except rogerbot so it won't get indexed anywhere else before it's ready.
I hope this helps! Please let me know if you have any other questions.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Ooops. Our crawlers are unable to access that URL
hello
Moz Pro | | ssblawton2533
i have enter my site faroush.com but i got an error
Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct
what is problem ?0 -
Url-delimiter vs. SEO
Hi all, Our customer is building a new homepage. Therefore, they use pages, which are generated out of a special module. Like a blog-page out of the blog-module (not only for blogs, also for lightboxes). For that, the programmer is using an url-delimiter for his url-parsing. The url-delimiter is for example a /b/ or /s/. The url would look like this: www.test.ch/de/blog/b/an-article www.test.ch/de/s/management-coaching Does the url-delimiter (/b/ or /s/ in the url) have a negative influence on SEO? Should we remove the /b/ or /s/ for a better seo-performance Thank you in advance for your feedback. Greetings. Samuel
Moz Pro | | brunoe10 -
Canonical URLs all show trailing slash on main site pages - using Yoast SEO for Wordpress - how to correct
We are using Yoast for a number of our sites. We use naked domain as the canonical. I have noticed in the header tags that all our sites show the canonical URLs as having a trailing slash: Example: http;//foxspizzajc.com, when I look at the source code, it shows the canonical as http;//foxspizzajc.com/ Of course, it is much more likely that all sites that link to us will not use the trailing slash - so preferably we do not want that to be the canonical - among other reasons. Does this need to be fixed so the trailing slash is removed? I cannot see how to do this in Yoast SEO or in Permalinks structure for Wordpress. Sorry for my ignorance. Thanks for any help.
Moz Pro | | Adam_RushHour_Marketing1 -
Will Moving Categories Affect SEO
I recently designed a new website to replace an old site. We managed to hold 90% ranking and traffic by keeping the same URS's and content. Now that we have completed that we are now updating the whole site. It is an commerce website. Some of the items we were selling we are getting from a new vendor. If I move these products from one one vendor to another will this affect SEO? Here is an example. I have a product called "Green Zipper Sweater". This product is Anchored via Manufacturer and Category. The URL and Title Tag are green-zipper-sweater. If the Sweater was made by "Nike Green Sweaters" and our new supplier is "Gap Yellow and Green Sweaters". If I change the the Manufacturer and now put it under "Gap Green and Yellow Sweaters" will this affect my ranking. We are continuing to stock products from the original supplier "Nike Green Sweaters" and we have an aggressive SEO plan and are ranking very well for "Nike Green Sweaters". We also have good product ranking so "Green Zipper Sweater" brings us a lot of traffic. I want to be sure I do not loose ranking for the product page "Green Zipper Sweater" and the Brand "Nike Green Sweaters" Any advice would be appreciated.
Moz Pro | | robbieire0 -
Does anyone have an SEO strategic plan template for a beginning SEOmoz'r?
I'm fairly new to the SEO world and am going to be creating an SEO strategic plan for my boss (with his help). I was wondering if anyone had any recommendations on a template to use, outline to follow, must have's/don't need, etc. Anything you're willing to share would be greatly appreciated!
Moz Pro | | kyle21ct0 -
In alt tag of a image can we use #hashtag or domain.com ? Is that good SEO or not allowed ?
Some of the Google Search shows a title has a hashtag of an article, which contain keyword and while tweeting them, the title which has a hashtag automatically very good used for getting traffic to the blog. And other one, can we use the hash tag inside the alt attribute ? Or our domain name with .com in it. Like Google.com or #Google ?
Moz Pro | | Esaky0 -
Is there a whitelist of the RogerBot IP Addresses?
I'm all for letting Roger crawl my site, but it's not uncommon for malicious spiders to spoof the User-Agent string. Having a whitelist of Roger's IP addresses would be immensely useful!
Moz Pro | | EricCholis1 -
Is there a recommended format when placing a business address on a webpage?
Hi All, I ask the question as I was trying to GeoTarget tool which happened to not recognise a business address I place in the footer on one of my sites. The tool states that including the address on page helps the search engines identify your location, so I'm curious whether a specific format works best when optimizing for local search? Thanks.
Moz Pro | | davebrown19750