Robots review

[sprynewmedia]

Anything in this that would have caused Rogerbot to stop indexing my site?  It only saw 34 of 5000+ pages on the last pass.  It had no problems seeing the whole site before.

User-agent: Rogerbot

Disallow: /default.aspx?*
//Keep from crawling the CMS urls default.aspx?Tabid=234.  Real home page is home.aspx

Disallow: /ctl/
// Keep from indexing the admin controls

Disallow: ArticleAdmin
// Keep from indexing article admin page

Disallow: articleadmin
// same in lower case

Disallow: /images/
// Keep from indexing CMS images

Disallow: captcha
// keep from indexing the captcha image which appears to be a page to crawls.

general rules lacking wildcards

User-agent: * Disallow: /default.aspx Disallow: /images/ Disallow: /DesktopModules/DnnForge - NewsArticles/Controls/ImageChallenge.captcha.aspx

[AaronWheeler]

Well, our crawler is supposed to respect all standard robots.txt rules, so you should be good just adding them all back in as you normally would and seeing what happens. If it doesn't go through properly, I'll ask our engineers to take a look and find out what's happening!

[sprynewmedia]

Thanks Aaron.

I will add the rules back as I want Roger to have nearly the same experience to Google and Bing.

Is it best to add one at a time?  That could take over a month to figure out what's happening.  Is there an easier way to test?  Perhaps something like the Google Webmaster Tools Crawler Access tool?

[AaronWheeler]

Hey! Sorry you didn't have a good experience with your help ticket. I talked with Chiaryn and it sounds like there was some confusion over what you wanted removed from your crawl; it had mentioned that you wanted only one particular page blocked. I think she found something different in your robots.txt - the rules you outline above - so she tried to help you with that situation. Roger does honor all robots.txt parameters so the crawl should only be limited in the way you define, though the wildcards do open you up to a lot of blockage.

It looks like you've since removed your restrictions from roger. Chiaryn and I spoke about it and we'll try to help with your specific site over your ticket. Hope this helps explain! If you want to re-add those parameters and then see what pages are wrongly blocked, I'd love to do that with you - just let us know when you've changed the robots.txt.

[sprynewmedia]

All urls are rewritten to default.aspx?Tabid=123&Key=Var.  None of these are publicly visible once the re-writer is active.  I added the rule just to make sure the page is never accidentally exposed and indexed

[Vahe.Arabian]

Could you clarify the URL structure for the default.aspx and the true home page. It's only because if you add Disallow: /default.aspx?* (with the wild card) then it will disallow all pages within the /default.aspx folder structure. Just use the same rule for rogerbot as you did for the general rule, this being Disallow: /default.aspx Hope this helps, Vahe

[sprynewmedia]

Actually, I asked help this question (essentially) first then the lady said she wasn't a web developer and I should ask the community.  I was a little taken back frankly.

[sprynewmedia]

Can't. Default.aspx is the root of the CMS and the redirect will take down the entire website.  Rule exists for only a small period where Google indexed the page incorrectly.

[Vahe.Arabian]

Hi,

If I was you, I would 301 redirect the default.aspx to the real home page. Once you do that simply remove it from the robots.txt file.

Not only would you strengthen the true home page, but prevent from crawling errors to occur.

There would be a concern that people might even still link to default.aspx which might be causing search engines to index the page. This might be the reason  to which rogerbot has stopped crawling your site.

If that's an issue just put a canonical tag for that URL, but still remove that reference.

Hope this helps,

Vahe

[KeriMorgret]

Hi! If you don't get an answer from the community by Monday, send an email to help@seomoz.org and they'll look at it to see what might be the problem (they're not in on the weekends, otherwise I'd have you send them an email right away).

Thanks!

Keri