Moz not crawling opencart product pages

[marlvass1]

hi,

i have waited for over 2 weeks now and the crawler only got 8 pages, and is not getting all the open cart pages and products.

any idea of what can be wrong?

im using joomla 2.5.11 and mijoshop 2.0.5 (which uses opencart 1.5.5.1).

thanks

[RuthBurrReedy]

Hi Marlon,

There are a lot of different issues that may cause Rogerbot to be unable to crawl your site. For troubleshooting tips, I suggest reading http://moz.com/help/pro/why-can-t-rogerbot-crawl-my-site. Hope that helps!

[marlvass1]

ok thanks

[bjs2010]

Screaming Frog has an option in the menu -

Configuration > Spider > "Ignore Robots.txt"

Tick this box and re-scan and see if you get different results.

And also, may be worth checking this out:
http://moz.com/learn/seo/robotstxt

Good luck!

[marlvass1]

hi

thanks for the answer, tried screaming frog now as well and seems it has the same problem,

is it possible to allow only certain spiders/robots to crawl all files, with the robots.txt?

thanks

[bjs2010]

It's hard to say without knowing the domain name -

Have you tried Screaming Frog software to see if it also has issues crawling?

Double check your robots.txt to see if some directories and pages are blocked.