We are using Hotlink Protection on our server for jpg mostly. What is moz.com address to allow crawl access?
-
We are using Hotlink Protection on our server for jpg mostly. What is moz.com crawl url address so we can allow it in the list of allowed domains? The reason is that the crawl statistics gives our a ton of 403 Forbidden errors.
Thanks.
-
Hi there!
Thanks for reaching out to us! I can certainly understand your need to have our crawler be accepted into your link protection program. Unfortunately our crawler doesn't operate using a url to crawl your site, we use a collections of IP addresses behind the scenes to mimic a search engine crawler to provide the best diagnoses around. With that said, a lot of our customer has had some success allowing our crawler on their server level through either their HTTP access file or other methods. Unfortunately I am not a Web developer or a server admin, so I couldn't exactly let you know how to implement it. I would recommend you to perhaps change up your question and post a new question regarding some work around for your software.
Thanks for your time, I hope that helps.
Peter
Moz Help Team.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unable to site crawl
Hi there, our website was revamped last year and Moz is unable to crawl the site since then. Could you please check what is the issue? @siteaudits @Crawlinfo gleneagles.com.my
Technical SEO | | helensohdg380 -
Google crawling but not indexing for no apparent reason
Client's site went secure about two months ago and chose root domain as rel canonical (so site redirects to https://rootdomain.com (no "www"). Client is seeing the site recognized and indexed by Google about every 3-5 days and then not indexed until they request a "Fetch". They've been going through this annoying process for about 3 weeks now. Not sure if it's a server issue or a domain issue. They've done work to enhance .htaccess (i.e., the redirects) and robots.txt. If you've encountered this issue and have a recommendation or have a tech site or person resource to recommend, please let me know. Google search engine results are respectable. One option would be to do nothing but then would SERPs start to fall without requesting a new Fetch? Thanks in advance, Alan
Technical SEO | | alankoen1230 -
IP address geolocation
I would classify our site as large (www.covers.com), with our primary target being traffic within the US (about 70%) but I have a feeling we are limited by our IP being located within Canada. I have changed all the targeting to the US in WMT, but could undertake a project to virtually 'move' the site to a US IP, if there would be benefit. I have found articles coming down on both sides of the issue. Thoughts?
Technical SEO | | evansluke1 -
Htaccess mod rewrite from server address
Hi there, I'm not massively experienced with creating mod rewrite rules and I'm worried I've got this wrong as its slightly different to what I'm used to. The web dev and content creators were working on http://5.10.105.45/~isea/ to create content. I want to redirect all URL's to www.iseasurfwear.co.uk. This is what I've written //Rewrite to www
Technical SEO | | BlueTree_Sean
Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} ^5.10.105.45/~isea/[nc]
RewriteRule ^(.*)$ http://www.iseasurfwear.co.uk/$1 [r=301,nc] Can anyone tell me if this is correct?0 -
Using both .co.uk and .com
Hello a client has launched a website with both the .com and .co.uk The content is identical. I understand that you should add rel="alternate" hreflang="x" to the code. However, will there be a problem with the identical content? It would be hard to localise the content to one country. I understand why the client has got both domains, particularly the UK one but the actual content is not specific to one country. It is written for English speaking customers really. Also what about links? In this case do you need to build two sets of links to make them both rank? Thanks for any help.
Technical SEO | | AL123al0 -
Crawl Diagnostics Report 500 erorr
How can I know what is causing my website to have 500 errors and how I locate it and fix it?
Technical SEO | | Joseph-Green-SEO0 -
Fixing Crawl Errors
Hi! I moved my Wordpress blog back in August, and lost much of my site traffic. I recently found over 1000 crawl errors in Webmaster Tools because some of my redirects weren't transferred, so we are working on fixing the errors and letting Google know. I'm wondering how long I should expect for Google to recognize that the errors have been fixed and for the traffic to start returning? Thanks! Jodi - momsfavoritestuff.com
Technical SEO | | JodiFTM0 -
Why the number of crawled pages is so low¿?
Hi, my website is www.theprinterdepo.com and I have been in seomoz pro for 2 months. When it started it crawled 10000 pages, then I modified robots.txt to disallow some specific parameters in the pages to be crawled. We have about 3500 products, so thhe number of crawled pages should be close to that number In the last crawl, it shows only 1700, What should I do?
Technical SEO | | levalencia10