Ask moz staff
-
Was trying to crawl a page and the moz on page grader tells that URL is not accessible. The URL is accessible thorugh my mobile and desktop. Any idea what is the issue ? The URL is https://www.practo.com/singapore/dentist
-
Hi there!
Generally speaking, it's much easier to block crawlers unilaterally than to allow just one specific crawler and no others onto your website. (It's also worth mentioning that if Rogerbot is the only crawler with access to your site, you risk falling off the radar on Google's index! Yikes!) Here's a great post about when, why and how to block crawlers from your site: http://searchenginewatch.com/sew/news/2067357/bye-bye-crawler-blocking-parasites
I'm by no means an SEO expert, but that article pointed to http://www.robotstxt.org/ which looks like a great resource for learning more about web robots, when to block them, and how to do it. I hope they help!
-
Thanks for the response. Is it possible to open our site only for RogerBot ? Would be glad if you can share details of that
-
Hi there!
Tawny from the Help Team here! It looks like in this case Rogerbot was blocked from crawling your site! When he tried, the server returned a 403 Forbidden error. If Rogerbot can't crawl your site, that might be an indication that other crawlers might be having trouble, too.This is probably something you should ask your webmaster to help you out with. They should be able to unblock Rogerbot and let the crawl begin!
I hope this helps! If you have any more questions, feel free to write in to us on the Help Team!
https://moz.com/help/contact -
There is a similar question here - https://moz.com/community/q/why-rogerbot-can-t-crawl-site-https-unplag-com. The advice is similar - check your logfiles for Rogerbot / Moz and see what the response is.
When I checked the other question I noticed that there was an issue with the configuration of the name servers - it is possible that it's a coincidence but you seem to have issues with your name servers as well: http://dnscheck.pingdom.com/?domain=www.practo.com×tamp=1445520589&view=1
You might want to correct that.Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need help fixing a duplicate content issue for my website. The moz crawl is show OMG my website with https:// and https://www. But I have never used the url https:// so I don’t understand why moz is showing this
Moz is showing my url with two different starts. Https:// and then the one I use https://www. The problem is I don’t think I have ever used the url without the www. at the start. How do I fix this?
Moz Bar | | jdp_uk0 -
I use json-ld in our site but schema not found in markup tool bar moz
i use json-ld in our site. structure data in test without any error but in test markup moz in our site schema is not found! Is it time for Google to identify the code to verify the schema?
Moz Bar | | shokoufezand0 -
I have too many tittle tag issue for my site on moz site crawl error
I have too many tittle tag issue in site crawl error but when I checked manually for the error there is no title in source code. Please Help me to understand
Moz Bar | | Nileshaggarwal0 -
Are we actually getting accurate data on keyword volumes from Moz (or other sources)?
I have a client who does patio furniture repair and restoration. When performing keyword research in Moz for terms like "patio furniture repair" I see that only 11-50 people in the entire US are searching for this term according to the Moz data. However, running an Adwords campaign currently and our top keyword is the phrase match for "patio furniture repair" which has generated over 100 clicks in just a couple of months in ONE county. Is there a better way to research more accurate results on search volume estimates? This makes organic SEO and keyword targeting hard! Thanks, Ricky
Moz Bar | | RickyShockley1 -
The Page Optimization tool keeps asking for several changes that are already in place! How can I get it to recognize them?
Hi there...the Page Optimization tool shows a 71 score for one of my pages, but the most critical needs it noted have already been in there for some time. What's the deal with this? Thanks...
Moz Bar | | adirondack0 -
I can't seem to get Moz Crawl to run? Re-bootyourbody.com. Told its a subdomain...What do I do?
I can't seem to get Moz Crawl to run? Re-bootyourbody.com. Told its a subdomain...What do I do?
Moz Bar | | Joseph.Lusso0 -
The Moz Spam Score tells me my site has too few backlinks for such a large site. How many links per page would I need to not trigger this filter and stop appearing spammy?
Hello! One of my sites is triggering the 'too few backlinks for large site' filter. I am wondering how many backlinks I need so as not to trigger this. Many thanks for your help. Toby
Moz Bar | | T0BY0 -
Chrome moz toolbar page analysis not loading
Often the chrome moz toolbar page analysis doesn't load just says Please wait until the page finishes loading.
Moz Bar | | genkee0