Ask moz staff
-
Was trying to crawl a page and the moz on page grader tells that URL is not accessible. The URL is accessible thorugh my mobile and desktop. Any idea what is the issue ? The URL is https://www.practo.com/singapore/dentist
-
Hi there!
Generally speaking, it's much easier to block crawlers unilaterally than to allow just one specific crawler and no others onto your website. (It's also worth mentioning that if Rogerbot is the only crawler with access to your site, you risk falling off the radar on Google's index! Yikes!) Here's a great post about when, why and how to block crawlers from your site: http://searchenginewatch.com/sew/news/2067357/bye-bye-crawler-blocking-parasites
I'm by no means an SEO expert, but that article pointed to http://www.robotstxt.org/ which looks like a great resource for learning more about web robots, when to block them, and how to do it. I hope they help!
-
Thanks for the response. Is it possible to open our site only for RogerBot ? Would be glad if you can share details of that
-
Hi there!
Tawny from the Help Team here! It looks like in this case Rogerbot was blocked from crawling your site! When he tried, the server returned a 403 Forbidden error. If Rogerbot can't crawl your site, that might be an indication that other crawlers might be having trouble, too.This is probably something you should ask your webmaster to help you out with. They should be able to unblock Rogerbot and let the crawl begin!
I hope this helps! If you have any more questions, feel free to write in to us on the Help Team!
https://moz.com/help/contact -
There is a similar question here - https://moz.com/community/q/why-rogerbot-can-t-crawl-site-https-unplag-com. The advice is similar - check your logfiles for Rogerbot / Moz and see what the response is.
When I checked the other question I noticed that there was an issue with the configuration of the name servers - it is possible that it's a coincidence but you seem to have issues with your name servers as well: http://dnscheck.pingdom.com/?domain=www.practo.com×tamp=1445520589&view=1
You might want to correct that.Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why do my Moz duplicate content results show me pages with no noticeably similar content?
Sometimes the "Pages with Duplicate Content" results under Content Issues show pages that, from what I'm able to see or otherwise test, have no duplicate content, save for the same navigation that exists on all of my pages. For example, a recent issue said that the following pages had duplicate content:
Moz Bar | | rickmic
https://freezerworks.com/index.php/html/slider-overlay
https://freezerworks.com/index.php/ufaqs/what-do-i-get-with-my-purchase-of-freezerworks
https://freezerworks.com/index.php/videos/fda-and-freezerworks-2
https://freezerworks.com/index.php/lims-testing-module Even a side-by-side of the page source in a text comparison tool shows nothing but navigation and scripts used in every page. Am I not seeing something?2 -
Any Moz staff out there with Magento experience?
Would be great to hook up with Moz staff that are Magento-integration savvy. Or maybe qualified users . . . Need to go through Magento Admin in order to correct errors found in Site Crawl. Thanks, Steve
Moz Bar | | spiderx.ing0 -
How accurate is Moz Rank Tracker tool? It's showing different results than a Google incognito search.
I have a keyword/url combo with Moz Rank Tracker showing 3 spots above what a Google Incognito search showed. I performed my Google Incognito search based on these suggestions: https://moz.com/community/q/best-and-easiest-google-depersonalization-method Is the Moz Rank Tracker tool off?
Moz Bar | | chiefmoz1 -
Significant difference with DA scores on Moz Chrome app VS Website Tool
I saw a big difference between DA scores on Moz Chrome app VS Website Tool. Which DA score is the correct one? I personally believe the scores on Moz's site is most accurate. Do you happen to have issues syncing the scored to your Chrome app? Chrome App VS. Moz Research Tool (https://moz.com/researchtools/ose/)
Moz Bar | | iPrice_Marketing
Moz.com: 92 / 88 Here's a screenshot of the difference: https://ipricegroup-my.sharepoint.com/:i:/p/jeremy_chew/ESy9lzUTC3lOl8o93Sx6LsUB8mYXo8LmYuUj2aa0xLXi1A?e=A6OrXs This was evident in other websites too:
Priceza (priceza.com.my😞 38 / 24
Shopback (shopback.my😞 43 / 41
Cuponation (https://www.cuponation.com.my/😞 27 / 250 -
MOZ Freshweb reported mentions that really does not contain any mention - anyone else experiencing this?
MOZ Freshweb reported links from "traffic.fullcontentrss.com" that doesn't really contain any mention. Is this a MOZ error or is it maybe since fullcontentrss.com has a javascript that pulls data into their own RSS, MOZ ended up crawling partially and as a result not reporting the correct URL? Anyone else experiencing this anomaly?
Moz Bar | | CPR_PTANTONO0 -
Moz Page Analysis Country different to Who.is?
If I analyse a domain with Moz Page Analysis tool, it says that the domain is hosted in the United States but if look up the same domain on who.is, the hosting location is Italy?
Moz Bar | | Marketing_Today0 -
Moz reporting for C-Blocking
Hey Mozers, I see Moz has a reporting tool for C-blocking and for november I had 330. Does this mean 330 Ip addresses came from the same location in the month of november?
Moz Bar | | rpaiva1 -
Moz Conversion Tracking
Is there a place here on Moz where goals and conversions can be set up or viewed? If the latter, does Moz just take the data from Analytics? Any information on this would be appreciated. Thanks,
Moz Bar | | xvpn9020