Ask moz staff
-
Was trying to crawl a page and the moz on page grader tells that URL is not accessible. The URL is accessible thorugh my mobile and desktop. Any idea what is the issue ? The URL is https://www.practo.com/singapore/dentist
-
Hi there!
Generally speaking, it's much easier to block crawlers unilaterally than to allow just one specific crawler and no others onto your website. (It's also worth mentioning that if Rogerbot is the only crawler with access to your site, you risk falling off the radar on Google's index! Yikes!) Here's a great post about when, why and how to block crawlers from your site: http://searchenginewatch.com/sew/news/2067357/bye-bye-crawler-blocking-parasites
I'm by no means an SEO expert, but that article pointed to http://www.robotstxt.org/ which looks like a great resource for learning more about web robots, when to block them, and how to do it. I hope they help!
-
Thanks for the response. Is it possible to open our site only for RogerBot ? Would be glad if you can share details of that
-
Hi there!
Tawny from the Help Team here! It looks like in this case Rogerbot was blocked from crawling your site! When he tried, the server returned a 403 Forbidden error. If Rogerbot can't crawl your site, that might be an indication that other crawlers might be having trouble, too.This is probably something you should ask your webmaster to help you out with. They should be able to unblock Rogerbot and let the crawl begin!
I hope this helps! If you have any more questions, feel free to write in to us on the Help Team!
https://moz.com/help/contact -
There is a similar question here - https://moz.com/community/q/why-rogerbot-can-t-crawl-site-https-unplag-com. The advice is similar - check your logfiles for Rogerbot / Moz and see what the response is.
When I checked the other question I noticed that there was an issue with the configuration of the name servers - it is possible that it's a coincidence but you seem to have issues with your name servers as well: http://dnscheck.pingdom.com/?domain=www.practo.com×tamp=1445520589&view=1
You might want to correct that.Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Temperature Widget
I've just tried to add the Moz temperature widget to my site (https) and it is getting blocked by Google Chrome for an incorrectly configured SSL certificate (apparently!) The widget embeds this page - https://mozcast.com/embed/yesterday/horizontal Which comes up with a security issue when visited in Chrome Do I need to wait for Moz to fix? Kind Regards Liam
Moz Bar | | ZaddleMarketing0 -
How do I go about fixing my High Priority issues that SEO moz says I have on a PHP site?
I am been trying to deal with this problem for some time now. I have talked to several IT people and SEO moz. None seem to know how to fix these issues on the type of site our company is. Our biggest issue with is Duplicate Page Content. We also have some title issues. Our site is built with PHP coding and variable, meaning the site is not a typical static website. We have a handful of pages that are dynamic depending on what the users chooses to see and do. So, my problem is I can't just go to a specific page and put the canonical or the redirect. It isn't multiple pages for our category pages, for example, it is just one that builds the page depending on the search. Please help!
Moz Bar | | JoshMaxAmps0 -
Using Moz to find blogger with good PR.
Hi, My manager has recently informed me that you can use Moz in order to find blogger by category and their page rank. Is this possible? The reason for this is because we are looking for blogger with a good page rank and who are relevant to our sector, which is days out, outdoor activities, zipwires, adventure etc. We then want to contact them to see if we can invite them to write about our Tree Top Adventure and zip trekking activities. Any help would be much appreciated.
Moz Bar | | GoApe_20140 -
Moz ranking report shows massive loss in Bing SERP's. But it is not the case?
My weekly ranking report shows massive loss of SERP's on Bing. For most keywords I was no longer in the top 50 which where first top 5. But when I checked it out for myself, I'm still ranked top 5 for these keywords. I then used the Moz Rank Tracker and there it also says I'm still ranking top 5 for these keywords. What should I believe now?
Moz Bar | | wellnesswooz0 -
Can we export branded reports from Moz Analytics Beta yet?
Hey guys, Just looking around the new Moz-alytics (droooool) and it is superb, but are we able to export and build reports from it yet? Or is this something that will be in the final release? Apologies if this has already been asked. Regards! Joe Sturgess
Moz Bar | | Sturge
Twitter: @Sturge871 -
Can I delete a SEO campaign in Moz, and start a new one for a different website?
Would be nice to know, as I'm limited to 5 campaigns, and the most important work is done for 2 site's, so I would like to switch that to 2 other website's. Regards,
Moz Bar | | mrblue910 -
Screaming Frog, Moz and other crawlers
Hi Ignorant question, but is it possible to use Screaming Frog or the Moz crawler or any other reputable crawler for a site still in development i.e. it is yet to be indexed? If so, could someone provide some quick instructions on how this can be done. Thanks in advance for any support. Neil
Moz Bar | | mccormackmorrison0