On Page Grader can't access my URLs
-
HI-
I am trying to grade some specific pages for keywords with the on page grader but it keeps telling me "Sorry, but that URL is inaccessible. "
I can reach them via the browser and they are not https. Any thoughts?
Here is a sample:
www.bulkcandystore.com/kosher-candy
Any help is appreciated.
Ken
-
Hey Ken,
Thanks for following up. You can add a crawl delay for rogerbot in your robots.txt file to limit how quickly we make requests to the site. I would just recommend that you don't add a crawl delay higher than 10 because that can cause us to be unable to crawl the site in a reasonable amount of time to complete a full crawl.
I hope that helps!
Chiaryn
-
Thanks for the info. I checked with my ISP and it turns out that they automatically blocked Rogerbot because of excessive requests (this happened Jan 15). Is there a way to crawl dely between requests longer or was that maybe a one time fluke?
Thanks
Ken
-
Hey Ken,
Thanks for writing in and sorry for the confusion. I'm afraid the server for www.bulkcandystore.com/kosher-candy is returning a 403 forbidden error for our crawler, so we are not allowed to access that page with our tools. The server would respond differently to the browser vs. our tools so I would recommend contacting the webmaster for the site and white-labelling the user-agent rogerbot so that we can access the site in the future.
I hope this helps. Please let me know if you have any other questions.
Chiaryn
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Monthly Keyword Volume Differences Keyword Research V's Campaign Rankings
Why are there differences in the keyword volume data, is this a UK only issue? Am I missing something? Within Keyword Research;
Moz Bar | | GrouchyKids
suspended ceiling systems swindon (0-10) Within Campaign Rankings:
suspended ceiling systems swindon (no data) It's not just one instance either its across multiple keywords.0 -
How do I see amount of traffic to a particular page on a website?
I am trying to figure out how much traffic goes to a particular page on a website. How would I go about finding this?
Moz Bar | | jbcorcoran6 -
Moz on-page grader can't see keyword in H1
Hi, Trying to do grade for whatiswhere.com, keyword 'poi search'. The tool can't see keyword in h1, but It is there. Could you please check what is the problem? Thank you, Andrei.
Moz Bar | | Anazar_20010 -
Moz Bar doesn't Show Anything
Hi, I use the MozBar chrome extension, and it has worked fine in the past. But lately it doesn't show any metrics. Am I missing an update or something? Thanks.
Moz Bar | | TMI.com0 -
On-Page Grader Url is inaccessible
Hi everybody. I'm trying to use on -page grader for https://www.upscaledinnerclub.com and get "Sorry, but that URL is inaccessible." Robots.txt are empty, another thread on MOZ was talking about DNS check - it's all good. So, I can't figure out why this is happening. Also I am trying the same for another website https://www.regexseo.com - the same story. Common thing is that they both are on Google App Engine. And at first i thought that was the problem. Bu then i checked this one : https://www.logitinc.com/ and it's working, even though this website is on GAE as well. None of these website have robots.txt or any differences in setup or settings. Any thoughts?
Moz Bar | | DmitriiK0 -
On-Page Grader not working?
I've been using the On-Page Grader tool for a while now, and have always had great success using it to better optimize my pages. But just recently I've noticed that it's not detecting any changes I make to any of my pages.
Moz Bar | | CMinc
For example: One of my pages was coming up with Exact Keyword is Used in Page Title. So I changed it and uploaded it to the site and then clicked the re-grade page button. The only problem is that it's not seeing any of the changes that I made.1 -
Way has the number of pages crawled plummeted?
Why has the number of pages crawled for our campaign plummeted in Moz Analytics – down to 729 from over 10k? Don't see any issues in Google Analytics with crawling our site.
Moz Bar | | EyeglassesGuy0