On Page Grader can't access my URLs
-
HI-
I am trying to grade some specific pages for keywords with the on page grader but it keeps telling me "Sorry, but that URL is inaccessible. "
I can reach them via the browser and they are not https. Any thoughts?
Here is a sample:
www.bulkcandystore.com/kosher-candy
Any help is appreciated.
Ken
-
Hey Ken,
Thanks for following up. You can add a crawl delay for rogerbot in your robots.txt file to limit how quickly we make requests to the site. I would just recommend that you don't add a crawl delay higher than 10 because that can cause us to be unable to crawl the site in a reasonable amount of time to complete a full crawl.
I hope that helps!
Chiaryn
-
Thanks for the info. I checked with my ISP and it turns out that they automatically blocked Rogerbot because of excessive requests (this happened Jan 15). Is there a way to crawl dely between requests longer or was that maybe a one time fluke?
Thanks
Ken
-
Hey Ken,
Thanks for writing in and sorry for the confusion. I'm afraid the server for www.bulkcandystore.com/kosher-candy is returning a 403 forbidden error for our crawler, so we are not allowed to access that page with our tools. The server would respond differently to the browser vs. our tools so I would recommend contacting the webmaster for the site and white-labelling the user-agent rogerbot so that we can access the site in the future.
I hope this helps. Please let me know if you have any other questions.
Chiaryn
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Crawl 1-page 301 status error but httpstatus.io says its 403
I am trying to run a site crawl for my website and MOZ is only resulting in 1 page crawled with the home page URL Status Code of 301. However when I run it in httpstatus.io it is giving me a 403 status error. Im curious as to why MOZ is saying its a 301 and httpstatus.io is saying 403. Is there anything I can do in MOZ first to get the site crawled before asking my developers to look into the 403 error?
Moz Bar | | JohnConover0 -
How can i fix? some information is wrong on the report.
https://moz.com/researchtools/ose/spam-analysis/flags?subdomain=bassinotary.com hi please advice. please see the link to my spam report. some information is wrong. like no email address or social profile link found I have already had links on that page and email and address and schema.org is in place. please help thanks.
Moz Bar | | grbassi0 -
Moz is only crawling 2 pages
Hi, I found a similar thread, but it did not provide a clear-cut answer. We have had this campaign running for over a year, and we are always adding content to the website, but Moz is only ever able to crawl 2 pages, Screaming Frog only picks up 12, but I know there is a lot more than that. None of our pages are set to no-index, so I do not know what is causing this. Welcoming any ideas/solutions. Thanks
Moz Bar | | GavinAdv0 -
Moz Bar doesn't show any data and keeps asking me to log in when actually I'm logged in.
Hi all, I've been using Moz Bar for years. It ran well until about three weeks ago. It suddenly failed to show the DA and PA of sites that I open after I log in. And it keeps asking me to log in when I did. I tried to uninstall the Mozbar extension and reinstalled it several times. Nothing worked. I also tried to uninstall Chrome and clear the cookies, still, nothing changed. Did anyone experience this? How do you solve it and make it run on the track? Any information will be appreciated. [admin edited support category]
Moz Bar | | Bennie22339 -
Find all the back links to all the posts/pages within the blog subdirectory only.
Hi, I am new to Moz. I using the open site explorer to find backlinks for a website's blog. The website itself is huge. I want to find all the backlinks to all the posts/pages within the blog subdirectory only. Not the regular website. I ran a few reports, but it is giving me links to that page, not all the sub pages.Fi
Moz Bar | | DarrenD0 -
Moz reporting appropriate Canonical tag usage but no canonical tag on page !?
I take it this means that the page in question has been referenced via a different pages canonical tag but that the page in question itself does not have a self referencing canonical tag (and that it should do) cheers dan
Moz Bar | | Dan-Lawrence0 -
Moz Rank Tracker doesn't work with "PHRASE" Keywords!?
Hello, If at http://ranktracker.moz.com trying to track phrase KW - the system wont accept it. But adding [EXACT] match with [] - works well. Have I missed something? Cheers.
Moz Bar | | SEOisSEO0