Ask moz staff
-
Was trying to crawl a page and the moz on page grader tells that URL is not accessible. The URL is accessible thorugh my mobile and desktop. Any idea what is the issue ? The URL is https://www.practo.com/singapore/dentist
-
Hi there!
Generally speaking, it's much easier to block crawlers unilaterally than to allow just one specific crawler and no others onto your website. (It's also worth mentioning that if Rogerbot is the only crawler with access to your site, you risk falling off the radar on Google's index! Yikes!) Here's a great post about when, why and how to block crawlers from your site: http://searchenginewatch.com/sew/news/2067357/bye-bye-crawler-blocking-parasites
I'm by no means an SEO expert, but that article pointed to http://www.robotstxt.org/ which looks like a great resource for learning more about web robots, when to block them, and how to do it. I hope they help!
-
Thanks for the response. Is it possible to open our site only for RogerBot ? Would be glad if you can share details of that
-
Hi there!
Tawny from the Help Team here! It looks like in this case Rogerbot was blocked from crawling your site! When he tried, the server returned a 403 Forbidden error. If Rogerbot can't crawl your site, that might be an indication that other crawlers might be having trouble, too.This is probably something you should ask your webmaster to help you out with. They should be able to unblock Rogerbot and let the crawl begin!
I hope this helps! If you have any more questions, feel free to write in to us on the Help Team!
https://moz.com/help/contact -
There is a similar question here - https://moz.com/community/q/why-rogerbot-can-t-crawl-site-https-unplag-com. The advice is similar - check your logfiles for Rogerbot / Moz and see what the response is.
When I checked the other question I noticed that there was an issue with the configuration of the name servers - it is possible that it's a coincidence but you seem to have issues with your name servers as well: http://dnscheck.pingdom.com/?domain=www.practo.com×tamp=1445520589&view=1
You might want to correct that.Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz claims we have meta noindex but we don't
Hi, I'm encountering an issue where moz scan says we have meta noindex, but I have confirmed across several of our pages that this simply isn't true. I have confirmation that the below tag is present: name="robots" content="index, follow" /> I also verified our https header through https://www.webconfs.com/http-header-check.php and see nothing indicating that we are sending any no index headers. Why would the crawler report this when it doesn't seem to be the case? Let me know if I need to provide more information.
Moz Bar | | charper_floqast0 -
Links and images in custom css are not recognised by Moz Toolbar or Yoast plugin
Hi, I am building a website in Wordpress and when checking the pages through Moz toolbar or Yoast plugin, I notice that these don't recognise the internal links in custom CSS and the images neither. Here some code to clarify how it is built: [vc_row][vc_column width="1/2"][ultimate_heading main_heading="Veldrijden" alignment="left" main_heading_margin="margin-bottom:20px;"][/ultimate_heading][vc_single_image image="6497" img_size="full" onclick="custom_link" link="http://k488239.hosting.kinamo.be/activiteiten/veldrijden/"][vc_btn title="Meer info" color="mulled-wine" align="left" link="url:http%3A%2F%2Fk488239.hosting.kinamo.be%2Factiviteiten%2Fveldrijden%2F|||"][/vc_column] The links are embedded in images and in 'More info' buttons. Also the internal links in the main website navigation are not recognised, but the normal a href links are recognised. For the images it is a little different: the alt text from the images gets recognised by Moz, but Yoast says there are no images on the page. The only thing I can think of that there is no 'a href' in the links and for the images that it has something to do with the page builder. Can this effect my SEO results? Thanks.
Moz Bar | | Mat_C0 -
Moz tool bar shows "No markup schema" in webpage despites having proper schema code in page.
We have proper markup schema installed in webpage (validated using data structure tool by google) but moz tool bar says "Schema.org not found on this page." What should be the problem ?
Moz Bar | | NortonSupportSEO1 -
Does Moz's keyword tool pull data from your IP address?
Does anyone know how Moz's keyword tool pulls their keyword ranks? Do they take it based off of the IP (history and cookies) that is being used? I am trying to find a way to collect keyword data that is neutral and not based off of my previous searches, etc. TIA
Moz Bar | | ReviveMedia0 -
Moz Tool Bar Annoyance - How do I make the green keyword difficulty box go away?
A highly annoying light green keyword difficulty box just started showing up to the right of the search box. How do I make this go away? ulBzP9W
Moz Bar | | JenKeller0 -
Checking Do-follow using Moz Bar
Does anyone know how to check if a link is a do-follow or no-follow using the Moz bar? I believe there's a function that highlights and color codes the link so as to tell? Can't seem to turn this function on in my Moz Bar settings. Am I missing something completely obvious?
Moz Bar | | Gavo0 -
Monthly Data Option not appearing in Moz AGAIN
Hello Again, It's a real shame that I seem to come to this forum to only complain about your product again and again. I remember a time when I used to really respect this company, but your tool is absolutely abysmal. Anyway, my latest issue (to add to the pile) is that campaigns that have been in my account for almost 2 months STILL do have options for me to make monthly reports on. What's wrong now? Edit: I just checked the date I added these profiles, it was around the end of last month, so one and a half months ago.
Moz Bar | | Paul_Tovey0