Why wont rogerbot crawl my page?
-
How can I find out why rogerbot won't crawl an individual page I give it to crawl for page-grader? Google, bing, yahoo all crawl pages just fine, but I put in one of the internal pages fo page-grader to check for keywords and it gave me an F -- it isn't crawling the page because the keyword IS in the title and it says it isn't. How do I diagnose the problem?
-
Very glad to see you got it working!
You can mark the question as answered to let others know it is fixed.
-
Thanks. The robots.txt file was the problem. It originally (yesterday) excluded rogerbot (by default) and then I remembered that and put it in as rogerbot but that didn't work. So I changed it to RogerBot and that didn't work. Today I removed the robots.txt file completely and it worked. Then I put it back with rogerbot and it is working.
It APPEARS that maybe it read the robots.txt yesterday before i put in rogerbot and for some reason didn't read it after I put it in. Will never know but it is now working.
Thanks for the help!
-
I know in robots.txt any URL's are case sensitive, I am not sure about user agents (bots/crawlers) but you do have RogerBot spelled with a capitol "B", changing it to lower case (Rogerbot) may fix the issue.
Another thing to test would be to simply remove the mass exclusion just to see if Rogerbot somehow is being blocked by it. Let me know how it goes.
User-agent: * Disallow: / -
Hi sure, thanks. This page shouldn't have a speed issue but maybe you can see what the issue is:
www.qjamba.com/local-coupons/wentzville/mo/all
Thanks.
-
Hi Theodore,
Last time I looked at this issue for another community member they had a site that had huge images and slow script. This decreased the load time of the page and Roger just got frustrated. Rogerbot is not as sophisticated as the huge Search Engines crawlers and can easily be put off.
As Martijn asked, for us to help we really would have to look at the site to pick out possible issues.
-
Hi Theodore, could you share the specific URL with us so we could help you diagnose what the issue could be?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz not able to crawl our site - any advice?
When I try and crawl our site through Moz it gives this message: Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt. Anyone got any advice on where I could go to get this sorted?
Getting Started | | MyFamilClubLtd1 -
Why does the moz bar link count differ from the moz link analysis page
Hi all, Why does the Moz Bar show a different link count from the Moz link analysis page? For example, when I check the SERP below, for the first result, the bar shows 936 page links from 4 RDs. But when I check out the link analysis page, it tells me there are just 141 page links from 4 RDs. What gives? For the second entry, the bar shows 6 page links from 0 RDs. Not sure how that's possible. Can anyone explain these things. Thanks! Andy Reviewed SERP: https://www.google.com/search?source=hp&ei=hhRVW5yyH5C60PEP-_-isAw&q=mountain+bike+trails+near+me&oq=mountain+bike+trails+near+me&gs_l=psy-ab.3..0l7j0i22i30k1l3.645.4654.0.5322.28.19.0.8.8.0.243.2920.3j11j5.19.0....0...1c.1.64.psy-ab..2.26.2961...0i131k1.0.C4lAxLkGgH0
Getting Started | | AndyKubrin0 -
Scheduled update - Re-Crawl - recrawl
Can I not perform a manual update? I setup a campaign without GA as I did not have access, I got access, added the GA account to the campaign but no data is showing as I think I require an update, but have to wait 7 days? Is that right? Thanks
Getting Started | | SJMDT0 -
Crawl test
Can anyone give me an idea how to use the MOZ crawl test results...I'm a little confused on how to read it? I have a lot of "no's"...I think this is good?
Getting Started | | sdwellers0 -
Missing Meta Description Tag and duplicate page title
Hi , I have a problems that I am trying to fix for a few days but with no luck My site is www.pc-varna.com Moz page issues is saying that I have Missing Meta Description Tag and duplicate page title also. I use meta description and I cant see the duplicate title. Can you please tell me what I do wrong? Thx in advance
Getting Started | | cyrixr0 -
How do I interpret Duplicate Content in a Crawl Report, when it only gives me a URL? How do I know what is duplicated on that page somewhere else?
I need help interpreting the Crawl Report for Duplicate Content. It gives me the URLs of pages that have duplicate content, but how do I know what content exactly is duplicated elsewhere? And how do I figure out where it is duplicated? Also, are there Moz Analytics articles or videos teaching you how to use each component of the analytics programs? Thanks!
Getting Started | | NancyBryan0 -
Duplicate page titles reported on same links
Moz is reporting duplicate titles on my clients website, citing different versions of the same link, one without dubs as domain.com/link and the other with dubs as www.domain.com/link. for example ... duplicate title = Warranty Requests For Custom New Home Construction in Jacksonville
Getting Started | | ElykInnovation
http://myhjhome.com/new-home-construction-warranty-request.html
http://www.myhjhome.com/new-home-construction-warranty-request.htmlthe links are the same, one is just with www and the other is without. in another example ... Moz reports 5 duplicate titles on the Home page, citing 5 different URLs that all load the same home page ...
duplicate title = Jacksonville Custom Home Builder, Quality Construction On Your Lot
http://myhjhome.com
http://myhjhome.com/
http://myhjhome.com/index.php
http://www.myhjhome.com/
http://www.myhjhome.com/index.php is this just something that's always going to be reported as a duplicate title or is there a way for me to change the website so Moz doesnt report a duplicate title on the same page (since there really isnt a duplicate title in these examples)?0 -
High Number of Crawl Errors for Blog
Hello All, We have been having an issue with very high crawl errors on websites that contain blogs. Here is a screenshot of one of the sites we are dealing with: http://cl.ly/image/0i2Q2O100p2v . Looking through the links that are turning up in the crawl errors, the majority of them (roughly 90%) are auto-generated by the blog's system. This includes category/tag links, archived links, etc. A few examples being: http://www.mysite.com/2004/10/ http://www.mysite.com/2004/10/17/ http://www.mysite.com/tagname As far as I know (please correct me if I'm wrong!), search engines will not penalize you for things like this that appear on auto-generated pages. Also, even if search engines did penalize you, I do not believe we can make a unique meta tag for auto-generate pages. Regardless, our client is very concerned seeing these high number of errors in the reports, even though we have explained the situation to him. Would anyone have any suggestions on how to either 1) tell Moz to ignore these types of errors or 2) adjust the website so that these errors now longer appear in the reports? Thanks so much! Rebecca
Getting Started | | Level2Designs0