Moz not crawling opencart product pages
-
hi,
i have waited for over 2 weeks now and the crawler only got 8 pages, and is not getting all the open cart pages and products.
any idea of what can be wrong?
im using joomla 2.5.11 and mijoshop 2.0.5 (which uses opencart 1.5.5.1).
thanks
-
Hi Marlon,
There are a lot of different issues that may cause Rogerbot to be unable to crawl your site. For troubleshooting tips, I suggest reading http://moz.com/help/pro/why-can-t-rogerbot-crawl-my-site. Hope that helps!
-
ok thanks
-
Screaming Frog has an option in the menu -
Configuration > Spider > "Ignore Robots.txt"
Tick this box and re-scan and see if you get different results.
And also, may be worth checking this out:
http://moz.com/learn/seo/robotstxtGood luck!
-
hi
thanks for the answer, tried screaming frog now as well and seems it has the same problem,
is it possible to allow only certain spiders/robots to crawl all files, with the robots.txt?
thanks
-
It's hard to say without knowing the domain name -
Have you tried Screaming Frog software to see if it also has issues crawling?
Double check your robots.txt to see if some directories and pages are blocked.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site crawl warning - concatenated urls from Wordpress
I could use some help on how to fix this. I asked at the walkthrough but was told it was a Wordpress issue but so far I can't find anything to point me in the right direction. There are no errors in the files on server side and I have asked my hosting company too. I am hoping someone here may be able to shed some light on it. One of my websites it giving 404 errors on links that are formed as below and there are over 12.7K of them! Example: <mydomainurl>/www.instagram.com/www.instagram.com/<instagram username=""></instagram></mydomainurl> The link that relates to my website is valid and working, but I don't understand the rest. I am totally stumped on how to move forward with this. Any advice, suggestions, tips on how to fix these errors and stop these types of links getting generated. Thanks.
Moz Bar | | emercarr0 -
MozBar is displaying different numbers than Moz Pro
We have over 15 accounts with Moz Pro, and I just noticed that almost every single one has a different DA when comparing the MozBar to their campaign. Any idea why this is? The DA of most of our sites seems to have dropped across the board, in some cases somewhat severely.
Moz Bar | | anchour0 -
Crawling password protected sites such as dev or staging areas to look at sites b4 going live ?
Hi Ive instructed clients to password protect dev areas so dont get crawled and indexed but how do we set up Moz crawl software so we can crawl theses sites for final check of any issues before going live ? Is there an option i havnt seen to add logins/passwords for crawl software to access ? cheers dan
Moz Bar | | Dan-Lawrence0 -
Have there been recent changes to the On-Page Grader tool?
I use this page all the time to grade my primary and secondary keywords and for four years now have consistently managed to achieve A-C grades for all. Today however, I noticed I am flagging F grades for 50% of the keywords. I do make a note of the initial grades for all pages (as the majority of the keywords I grade using the tool are NOT added/tracked in MOZ 'campaigns'). To make sure I wasn't going mad I went back and graded some keywords I had achieved A-C grades for just 6 days ago. The changes are as follows: keyword 1 initially graded an A - 6 days later is still an A keyword 2 initially graded C and is now graded as an F keyword 3 initially graded C and now an F keyword 4 initially a C grade and now an F keyword 5 initially a B grade and has remained a B. Out of interest I then went and added one of the keywords that is now graded an F (but last week was a C) into the 'Add and manage Page Grades' section and graded once again - it's now grading this as a C. I then removed this keyword from my tracked keywords and graded again using the 'On-Page Grader' tool and ... it's back down to an 'F'. Please could you shed some light on what's going on? I don't want to track each keyword I grade over time but I do want the parameters for grading to remain the same if I use the 'On-Page Grader' tool - what's changed? I haven't received any alerts about any changes to the tool so if anyone could shed some light on any changes I would appreciate it.
Moz Bar | | em_h_uk0 -
On Page Grader can't access my URLs
HI- I am trying to grade some specific pages for keywords with the on page grader but it keeps telling me "Sorry, but that URL is inaccessible. " I can reach them via the browser and they are not https. Any thoughts? Here is a sample: www.bulkcandystore.com/kosher-candy Any help is appreciated. Ken
Moz Bar | | CandymanKen0 -
Moz Report: Number of Domains linking to Domain
In the Moz report, what does the "number of domains linking to domain" indicate? I'm analyzing our competitors link profile and noticed some numbers as high as 2369232? Is there a form that explains the reports a little more in detail? Thanks,
Moz Bar | | WebRiverGroup0 -
Duplicate content - Which is the other duplicate page?
Hi I just ran a campaign, and I got a duplicate content warning for some of my pages. When I go into the diagnostic report, I am unable to find the page detected by google as 'duplicate' to the main page. Unless I know which 2 pages are being detected as duplicate, it'll be really difficult to actually solve the problem. Would be great to have any kind of help here. Thanks in advance!
Moz Bar | | rjchugh0