Help Me With These Horrible Crawl Errors
-
I am currently using Opencart to run a fairly popular ecommerce website. I have done quite well in terms of rankings so far. However, the Opencart platform only goes so far.
There are endless amounts of crawl errors, from duplicate content down to missing meta tags. There is no easy way to add canonical tags. The pages that are being listed in the crawl errors seem to be mainly search URL's that i have no idea how Moz found them.
An example of the page is:
.....com/index.php?route=product/search&tag=home+office+flooring
Which has ten duplicate pages all missing meta tags.
I have used Google webmaster tools and htaccess to deny access to these pages with the slug "route=product/search" but it doesn't seem to work.
So my question to you guys is as follows:
Is it worth trying to fix these errors and do they have an effect on SEO?
If so, how do I prevent these errors that seem to grow at every crawl?
Cheers,
Danny
-
Hi Danny! How'd that crawl end up? If Andy answered your question, mind marking his response as a "Good Answer?" It'll get him some bonus MozPoints, and it helps us keep track of things in the forum.
-
I have tested with the robots tool too. Does seem to be blocked but running a crawl to see what moz brings up.
-
You don't need to do another crawl - go into WMT and go to Robots.txt tester.
Get the live robots file (if it doesn't pull it in, copy and paste it).
Find a URL you wanted blocking and see if Google crawls it and you should get a warning message
-
Great, thanks Andy, Just doing a crawl now.
-
Nope you need to add the wildcard onto the end of the product line so it looks like this
User-agent: *
Disallow: /index.php?route=checkout/cart
Disallow: /index.php?route=account/return/insert
Disallow: /index.php?route=product/*Otherwise it will just exclude that specific url.
Let me know if this works.
Thanks
Andy
-
Andy,
My robots file currently looks like this
User-agent: *
Disallow: /index.php?route=checkout/cart
Disallow: /index.php?route=account/return/insert
Disallow: /index.php?route=product/Is this correctly set up?
-
Hi
Yes its worth fixing.
Could you not add Disallow: ?route=product/* to your robots file?
Can you edit the robots file.
Thanks
Andy
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does anyone know of a predictive demographics software that helps a website predict its audience based on cookies, or whatever info it has?
Hi guys, I'm looking for a predictive analytics software for better understanding our audience online. Has anyone heard of or used a software that takes the visitors currently coming to your site and uses that data to 'predict' more information about them? Such as age, location, purchasing power, etc? Please let me know if you have!
Search Behavior | | Raconteur
Thanks,
Sabilah0 -
Help with robots.txt on Magento
Hi everybody, I need your help in order to fix some problems with HTML errors and Crawling errors generated by Magento on my client's website www.casabiancheria.it I have some problems with duplicate meta informations due to the fact that there are a lot of links such as /stampe-romagnole/tovaglie-con-tovaglioli**/colore/**beige,marrone,giallo,lilla/show/all.html /stampe-romagnole/tovaglie-con-tovaglioli**/colore/**beige,marrone,lilla/show/all.html that are generated by the filter /colore/ and so they have duplicate content and meta information on them. I activated the canonicals on Magento but this hasn't fixed the problem yet. On the sitemap there are only 1 link for each product, so it seems that the canonicals are working, but bot Google Webmaster Tools and SEO Moz are giving me errors on duplicate content and meta informations. I would like to solve these problems by excluding from robots.txt all the urls that contain the filter parameters, such as /colore/, /price/, /dimensions/, etc. (take a look to the attachment). I tried different solutions in order to exclude these links from robots, but I wasn't able to succeed. Below you can find my current robots.txt... can someone help me in order to write the correct form of this file and finally exclude all these urls generated by filters on Magento? Finally, is it worth it to exclude also the images from Magento? (take a look to the final lines of the robots below). Thank you very much for your help! Alberto User-agent: *
Search Behavior | | OptimizedGroup
Disallow: /CVS
Disallow: /.svn$
Disallow: /.idea$
Disallow: /.sql$
Disallow: /.tgz$
Disallow: /w1nL1f3L0g1c/
Disallow: /app/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /shell/
Disallow: /var/
Disallow: /404/
Disallow: /cgi-bin/
Disallow: /magento/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /api.php
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /get.php
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /README.txt
Disallow: /RELEASE_NOTES.txt
Disallow: /?dir
Disallow: /?dir=desc
Disallow: /?dir=asc
Disallow: /?limit=all
Disallow: /?mode*
Disallow: /index.php/
Disallow: /?SID=
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /cgi-bin/
Disallow: /cleanup.php
Disallow: /apc.php
Disallow: /memcache.php
Disallow: /phpinfo.php
Disallow: /control/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/
Disallow: /?*
Disallow: //colore/
Disallow: //price/
Disallow: //misura/
Disallow: //marca/
Disallow: //sort-by/
Disallow: //combinazione/
Disallow: /*/seleziona-colore/
Disallow: /colore/
Disallow: /price/
Disallow: /misura/
Disallow: /marca/
Disallow: /sort-by/
Disallow: /combinazione/
Disallow: /seleziona-colore/
Disallow: /*colore/
Disallow: /*price/
Disallow: /*misura/
Disallow: /*marca/
Disallow: /*sort-by/
Disallow: /*combinazione/
Disallow: /*seleziona-colore/ UmuEX4z0 -
Blocking links from being crawled
For my blog, should I block category, "read more" and author links from being crawled by search engine robots? Is it good for SEO? Thanks
Search Behavior | | uesat0 -
Is putting/removing Adsense ads on the site affects crawling for seo?
Because I noticed when we totally remove Adsense ads on our site, the pages crawled per day on the google webmastertools suddenly dropped into noticable amount and we tested again to turn it on for a singe day, it jumped up again. So if Adsense affects crawling then, if we have adsense ad on all pages of the site the more chances it gets crawled by the bot? WS76R8w
Search Behavior | | CruiseControl0 -
404 errors
I am seeing alot of 404 erros recently but all seem to be pages like http://www.finalduties.co.uk/2011/06/21/legal-services-board-to-investigate-will-writers/feed/ (FEED) at the end of the URL also http://www.finalduties.co.uk/2011/06/23/rules-are-complex-when-it-comes-to-dying-intestate/?wpmp_switcher=desktop ( I know this wpmp_switcher is from a mobile ready plugin we used a while back which has since been removed after duplicating many pages. also ?wpmp_switcher=desktop is in my robots txt file to help block the robots from crawling these pages and displaying 404's but its still happening, Ihave 700+ 404 errors most ending in feed /tag and ?wpmp_switcher=desktop Any Ideas I know that 404's aren't that bad but seeing so many this morning I need to figure out why these are coming up all of a sudden. We have been getting these since our site was accidently taken down a while back, trying to figure out why we have lost so many rankings. we seem to have SERPS like YOYO's from 1 day to the next one keyword goes up but 150 and then next day drops same the next day???any ideas? Seems SEO posr penguin is one big contadiction, some seo experts give you one bit of advice another gives you another very confsuing. Thanks Elissa 🙂
Search Behavior | | Chris__Chris0 -
I need Help with Google!!!!
I am trying to have my picture on the first page just like SEOmoz when someone search just the name, I know have something to do with google plus, but I am so new doing that no luck or probably I am doing wrong, I have been looking in the internet but I haven't found anything. Is there anyone who can write a tutorial and post here. Maybe is already done and I don't know where. Please see the picture attached so you understand better what I want to do Again I want appear just like this but with my company http://www.sombras.co.uk/images/pic.jpg pic.jpg
Search Behavior | | teksyte0 -
HELP! 75% Drop in Traffic with no Explaination
Hello Everyone, After looking at one of my websites, http://pokeronamac.com/ I noticed that there is a significant drop in all traffic to my site. I cannot figure out what the exact source is, as we received no warnings in WMT, and as far as I know there are no major updates released in the last week or so by Google. I took a look at the content section of GA and I saw that virtually all of my content was seeing a decrease in traffic, meaning that it isn't specific to a few pages, which leads me to believe that it is an algorithmic penalty. My confusion was that there was no changes made to the website in the past few months, although, on September 10th (5 days after traffic dropped) I updated the meta descriptions and titles site wide. I don't believe that the penalty is from Panda or Penguin because we actually experienced a positive influx in traffic during the week of April 25th. I'd appreciate any insights into helping me recover from this so I can get the visits again that I had before. If any other metrics are required, i'd gladly provide them. Thanks Zach Russell 8NBwn.png
Search Behavior | | Zachary_Russell0 -
Google Penalisation - Any help would be appreciated!
Hi,
Search Behavior | | ChrisHolgate
We’ve recently received a Google notification of unnatural linking along with a confirmation that we're being penalised. There were a few other sites that we owned that perhaps had too many links pointing to our main domain so we trimmed them down and submitted a reconsideration request and got the following back: "Dear site owner or webmaster of http://www.refreshcartridges.co.uk/,
We received a request from a site owner to reconsider http://www.refreshcartridges.co.uk/ for compliance with Google's Webmaster Guidelines.
We've reviewed your site and we still see links to your site that violate our quality guidelines.
Specifically, look for possibly artificial or unnatural links pointing to your site that could be intended to manipulate PageRank. Examples of unnatural linking could include buying links to pass PageRank or participating in link schemes.
We encourage you to make changes to comply with our quality guidelines. Once you've made these changes, please submit your site for reconsideration in Google's search results.
If you find unnatural links to your site that you are unable to control or remove, please provide the details in your reconsideration request.
If you have additional questions about how to resolve this issue, please see our Webmaster Help Forum for support.
Sincerely,
Google Search Quality Team" I want to stress that we have never in the past and do not currently buy any backlinks. The problem that we face now is that our site has been online for best part of a decade, there are thousands of people linking to us and I have absolutely no idea where to start. We don’t use an SEO Company but in the past few months have been using SEOmoz to improve our on-page optimisation. I know it’s a massive ask but if could a member of the SEOmoz community or a staff member quickly take a gander and let us know if anything in particular sticks out like a sore thumb it would mean a great deal to me. Of course, if needed we’ll employ the services of an SEO company but I’m hoping one of you guys will see something immediately obvious that could really help us out! Thanks in advance. Kind regards Chris0
