Why wont rogerbot crawl my page?
-
How can I find out why rogerbot won't crawl an individual page I give it to crawl for page-grader? Google, bing, yahoo all crawl pages just fine, but I put in one of the internal pages fo page-grader to check for keywords and it gave me an F -- it isn't crawling the page because the keyword IS in the title and it says it isn't. How do I diagnose the problem?
-
Very glad to see you got it working!
You can mark the question as answered to let others know it is fixed.
-
Thanks. The robots.txt file was the problem. It originally (yesterday) excluded rogerbot (by default) and then I remembered that and put it in as rogerbot but that didn't work. So I changed it to RogerBot and that didn't work. Today I removed the robots.txt file completely and it worked. Then I put it back with rogerbot and it is working.
It APPEARS that maybe it read the robots.txt yesterday before i put in rogerbot and for some reason didn't read it after I put it in. Will never know but it is now working.
Thanks for the help!
-
I know in robots.txt any URL's are case sensitive, I am not sure about user agents (bots/crawlers) but you do have RogerBot spelled with a capitol "B", changing it to lower case (Rogerbot) may fix the issue.
Another thing to test would be to simply remove the mass exclusion just to see if Rogerbot somehow is being blocked by it. Let me know how it goes.
User-agent: * Disallow: /
-
Hi sure, thanks. This page shouldn't have a speed issue but maybe you can see what the issue is:
www.qjamba.com/local-coupons/wentzville/mo/all
Thanks.
-
Hi Theodore,
Last time I looked at this issue for another community member they had a site that had huge images and slow script. This decreased the load time of the page and Roger just got frustrated. Rogerbot is not as sophisticated as the huge Search Engines crawlers and can easily be put off.
As Martijn asked, for us to help we really would have to look at the site to pick out possible issues.
-
Hi Theodore, could you share the specific URL with us so we could help you diagnose what the issue could be?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
On page Grader is not working on blogvali.com
Hope you people are doing well. I am getting error while using MOZ onpage grader. Page Optimization Error There was a problem loading this page. Please make sure the page is loading properly and that our user-agent, rogerbot, is not blocked from accessing this page. ScreenShot of error: http://prntscr.com/v2wv3f My Site link: https://blogvali.com/ v2wv3f
Getting Started | | ismailvela0330 -
Moz Not Crawling Angular SPA
I have a client that just launched a redesigned website using Angular as a single page app. Google appears to be able to crawl the site just fine, but Moz crawl is only finding one page. We have updated the htaccess to allow for Rogerbot and Dotbot, but still unable to crawl any pages other than the home page. Does anyone have experience with this or ideas of why it won't crawl all pages, and how to allow for Moz to crawl all pages? There is a sitemap with approx. 390 pages. Thanks!
Getting Started | | PIN_Celler1 -
Moz campaign for a subdomain with no home page
Hello all, I'm trying to setup a campaign in Moz for one of our subdomains that we use for marketing landing pages. Unfortunately there really is no home/landing page for the subdomain so when I enter it into the campaign settings it says the site isn't accessible. Is there anyway for me to add this? I tried entering the subdomain with a star, e.g. pages.mydomain.com but it would not accept it. Any ideas? Thanks!
Getting Started | | Brando160 -
What is the full user-agent for rogerbot?
IT is blocking AWS via a proxy in front of our server. We've tried allowing the "roberbot" user-agent but crawling functionality still isn't working in my Moz Pro account. Is there a more specific user-agent we can allow in our proxy software? Thank you.
Getting Started | | uShip1 -
Duplicate Page Content and Title
Hi. New to Moz here. I really like all of the tools. Moz recently sent me a Crawl Report and flagged every page as a duplicate content and duplicate title. The only reason/difference between the pages it flagged is the beginning of the URL. The crawl report shows the http://websitename.com and then http://www.websitename.com/ Is there something I'm missing about setting up my website? You can use either form to browse to my website, with the www and without. Why would Moz flag that as a duplicate? Scott
Getting Started | | David-Sulak0 -
All Pages are "F Grade"
All of my pages are "F Grade" upon last week there's no problem but now all pages are Grade F Is there any problem with crawl my page? Could you please check? Thanks
Getting Started | | nopsts0 -
Getting errors -- Pages with Title Missing or Empty
I am getting a high priority warning that I have pages with title missing or empty, however, when I check the page, there is a title tag <title>Long Term Care Planning</title> . Anyone have any idea what could be doing on? I check the CMS (using AEM - adobe experience manager CQ5) and the title field is filled out as well.
Getting Started | | Laura-Genworth0 -
Why has my website gone from 2 duplicate pages to 5000+ duplicate pages n 1 week?
Hi Can anyone please help. Using the weekly moz reports I realised that my website has gone from 2 duplicate pages to over 5000. As well as this pages with "too long URL" has jumped to over 5000 as well as missing meta tags. Any help would be greatly appreciated! Cheers
Getting Started | | Stubs0