Ask moz staff
-
Was trying to crawl a page and the moz on page grader tells that URL is not accessible. The URL is accessible thorugh my mobile and desktop. Any idea what is the issue ? The URL is https://www.practo.com/singapore/dentist
-
Hi there!
Generally speaking, it's much easier to block crawlers unilaterally than to allow just one specific crawler and no others onto your website. (It's also worth mentioning that if Rogerbot is the only crawler with access to your site, you risk falling off the radar on Google's index! Yikes!) Here's a great post about when, why and how to block crawlers from your site: http://searchenginewatch.com/sew/news/2067357/bye-bye-crawler-blocking-parasites
I'm by no means an SEO expert, but that article pointed to http://www.robotstxt.org/ which looks like a great resource for learning more about web robots, when to block them, and how to do it. I hope they help!
-
Thanks for the response. Is it possible to open our site only for RogerBot ? Would be glad if you can share details of that
-
Hi there!
Tawny from the Help Team here! It looks like in this case Rogerbot was blocked from crawling your site! When he tried, the server returned a 403 Forbidden error. If Rogerbot can't crawl your site, that might be an indication that other crawlers might be having trouble, too.This is probably something you should ask your webmaster to help you out with. They should be able to unblock Rogerbot and let the crawl begin!
I hope this helps! If you have any more questions, feel free to write in to us on the Help Team!
https://moz.com/help/contact -
There is a similar question here - https://moz.com/community/q/why-rogerbot-can-t-crawl-site-https-unplag-com. The advice is similar - check your logfiles for Rogerbot / Moz and see what the response is.
When I checked the other question I noticed that there was an issue with the configuration of the name servers - it is possible that it's a coincidence but you seem to have issues with your name servers as well: http://dnscheck.pingdom.com/?domain=www.practo.com×tamp=1445520589&view=1
You might want to correct that.Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I added a privacy policy link to my footer and now Moz is showing thousands of 4xx errors
My website didn't have a privacy policy so I added one and put the link in the footer menu. When I did this, Moz came back telling me that there are a lot of new errors on the site. Is this a bad thing? Do I need to address it? HY59Iks sYyAHCB
Moz Bar | | elisa175910 -
Links and images in custom css are not recognised by Moz Toolbar or Yoast plugin
Hi, I am building a website in Wordpress and when checking the pages through Moz toolbar or Yoast plugin, I notice that these don't recognise the internal links in custom CSS and the images neither. Here some code to clarify how it is built: [vc_row][vc_column width="1/2"][ultimate_heading main_heading="Veldrijden" alignment="left" main_heading_margin="margin-bottom:20px;"][/ultimate_heading][vc_single_image image="6497" img_size="full" onclick="custom_link" link="http://k488239.hosting.kinamo.be/activiteiten/veldrijden/"][vc_btn title="Meer info" color="mulled-wine" align="left" link="url:http%3A%2F%2Fk488239.hosting.kinamo.be%2Factiviteiten%2Fveldrijden%2F|||"][/vc_column] The links are embedded in images and in 'More info' buttons. Also the internal links in the main website navigation are not recognised, but the normal a href links are recognised. For the images it is a little different: the alt text from the images gets recognised by Moz, but Yoast says there are no images on the page. The only thing I can think of that there is no 'a href' in the links and for the images that it has something to do with the page builder. Can this effect my SEO results? Thanks.
Moz Bar | | Mat_C0 -
Moz is not showing traffic stats..;(
I have connected my analytics account in moz. But it is not showing my traffic stats.
Moz Bar | | daniyal_ahmed0 -
Moz keyword mention on-page counting errors
Hi. Moz is showing 18 mentions of the keyword 'street furniture' on this landing page https://www.broxap.com/street-furniture.html But I can only count 6 in total in the body copy and 13 if you include navigation links. This is the same on other pages too for that keyword. Does anyone know where it's counting these extra keywords from? I don't want to fall foul of keyword stuffing but as far as I can see we're not! Could Moz be miscalculating? Any help appreciated! Thanks Joe
Moz Bar | | iweb_agency0 -
Moz Content --- for SEO or simply user engagement?
What is the primary function of Moz Content? It looks like it is most useful for managing content as a user engagement tool. Our content strategy is centered on boosting organic placement - with user engagement as a nice but unessential side product. Besides providing general descriptive details of a site's content / authorship - how can Moz Content help with SEO?
Moz Bar | | cvonhassell0 -
Using Moz to find blogger with good PR.
Hi, My manager has recently informed me that you can use Moz in order to find blogger by category and their page rank. Is this possible? The reason for this is because we are looking for blogger with a good page rank and who are relevant to our sector, which is days out, outdoor activities, zipwires, adventure etc. We then want to contact them to see if we can invite them to write about our Tree Top Adventure and zip trekking activities. Any help would be much appreciated.
Moz Bar | | GoApe_20140 -
Have any insight into why our Moz Rank dropped?
I'm working on a site with a very low domain authority to start and in viewing our historical MozRank comparison to competitors I see that we had a MozRank between 2 and 3 two months ago, but now have a MozRank of 0. What could have triggered this dropoff? It's clear we need to boost domain authority, but we have never had any so we're no worse in that department now than we were two months ago. Any insight here would be useful. Thanks! W2A1u2D.png
Moz Bar | | bshanahan0 -
Moz "Crawl Diagnostics" doesn't respect robots.txt
Hello, I've just had a new website crawled by the Moz bot. It's come back with thousands of errors saying things like: Duplicate content Overly dynamic URLs Duplicate Page Titles The duplicate content & URLs it's found are all blocked in the robots.txt so why am I seeing these errors?
Moz Bar | | Vitalized
Here's an example of some of the robots.txt that blocks things like dynamic URLs and directories (which Moz bot ignored): Disallow: /?mode=
Disallow: /?limit=
Disallow: /?dir=
Disallow: /?p=*&
Disallow: /?SID=
Disallow: /reviews/
Disallow: /home/ Many thanks for any info on this issue.0