Moz not crawling opencart product pages
-
hi,
i have waited for over 2 weeks now and the crawler only got 8 pages, and is not getting all the open cart pages and products.
any idea of what can be wrong?
im using joomla 2.5.11 and mijoshop 2.0.5 (which uses opencart 1.5.5.1).
thanks
-
Hi Marlon,
There are a lot of different issues that may cause Rogerbot to be unable to crawl your site. For troubleshooting tips, I suggest reading http://moz.com/help/pro/why-can-t-rogerbot-crawl-my-site. Hope that helps!
-
ok thanks
-
Screaming Frog has an option in the menu -
Configuration > Spider > "Ignore Robots.txt"
Tick this box and re-scan and see if you get different results.
And also, may be worth checking this out:
http://moz.com/learn/seo/robotstxtGood luck!
-
hi
thanks for the answer, tried screaming frog now as well and seems it has the same problem,
is it possible to allow only certain spiders/robots to crawl all files, with the robots.txt?
thanks
-
It's hard to say without knowing the domain name -
Have you tried Screaming Frog software to see if it also has issues crawling?
Double check your robots.txt to see if some directories and pages are blocked.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Optimization Error
Hi, I am trying to track a page optimization feature for one of my project, https://www.writemyessay247.com for keyword: write my essay but i keep getting this below error: "Page Optimization Error There was a problem loading this page. Please make sure the page is loading properly and that our user-agent, rogerbot, is not blocked from accessing this page." I checked robots.txt file, it all looks fine. Not sure what is the problem? Is it a problem with Moz or the website?
Moz Bar | | j.rahimi19900 -
4 days waiting for a Moz Crawl - How quick are yours?
Hi there Please could anyone say how long they have been waiting for crawl results. I requested a crawl on a 20 page website and I have been waiting 4 days since last weekend. I checked Moz Health and there have been no related issues there: http://health.moz.com/ Your response would be welcome. Thanks
Moz Bar | | SEOguy10 -
How do I cancel a crawl request?
I was farting around and exploring the Crawl Test tool, and accidentally sent out a crawl for a competitor's site (I wanted to see if the tool would decline to crawl without verification). I do NOT want to actually crawl that site, nor do I want the competitor to see that we requested it (for obvious reasons) - how do I cancel it?
Moz Bar | | mkbeesto0 -
Has using Moz got me banned from Google search?
Dear fellow Mozzers, For the last couple of weeks when ever I do a google search I have to enter a captcha code. Now I hardly use google anymore which is annoying ! Then I went onto support.google.com and saw this: https://support.google.com/websearch/answer/86640?hl=en&ref_topic=3378866 "Automated traffic includes: Sending searches from a robot, computer program, automated service, or search scraper Using software that sends searches to Google to see how a website or webpage ranks on Google" Is Moz alone doing this? Thanks in advance for your response. Ash
Moz Bar | | -Ash-0 -
Duplicate page content/page titles on tages
Hello everyone. New to the community and loving it already. Question, I am receiving an error of 6 pages with duplicate content and page titles. A majority of these are tag pages. Should I be worried about these? IN the column listed duplicate urls it is listing 0 ( screen shot: http://screencast.com/t/azvuVk0ucWt) Are these tags a problem? Will SEO be hurt because of this? What are TAG pages? Actually pages, categories, should I eliminate these?
Moz Bar | | Jasonalanmagic0 -
Understanding Onsite Elements and Moz's Onpage Grader
I'm currently helping a friend with his website. One of the pages I am targeting for his business is: http://stalbansdentist.com.au/dentist-cairnlea/ The main keyword this page is targeting will be "dentist cairnlea" (Cairnlea is a suburb near Melbourne, Australia) To perfect onsite optimization, I've used the Moz Onpage Grader. Generally, its straightforward, but I have encountered some issues... Appropriate Use of Rel Canonical The page does have the following tag: <link rel="<a class="attribute-value">canonical</a>" href="http://stalbansdentist.com.au/dentist-cairnlea/" />. I thought this would suffice. What exactly do I need to do to fix this critical problem? Appropriate Characters in the URL I don't understand this one. The URL has completely appropriate characters. Why does Moz Onpage Grader insist that URLs here need to be appropriated? No More Than One H1 Tag There appears to be only one H1 tag on this page. Is Moz just wrong on this one or perhaps a little delayed? There were 2 H1 tags before one was deleted and a recheck done with this problem remaining on Moz's onsite check. Any assistance here on these 3 points and just understanding the Moz Onpage grader would be appreciated!
Moz Bar | | Gavo0 -
Why moz does not update actual keyword possition
I managed to get first place on google by keyword: vervo.lv kravu pārvadājumi in google.lv (it is in first place all this week) But in moz reports it shows just second position? I added so far about 100 keywords to monitor so if it shows incorrect this one how can i trust to other keyword results. Report says it updated on 3. september and it already was in first place. I really need to monitor keywords but if it does not show actual positions it is useless.
Moz Bar | | vervo0 -
Moz just did my weekly "crawl" and claims my meta titles and descriptions are "missing," but they aren't. What's up with this?
I can check for my titles and descriptions in the source. They are there, always have been. what in the hell?
Moz Bar | | LifeisRosie0