Hi Alan,
Thaks very much for offering your thoughts on this. Really useful comments.
You're right in that I had very much over simplified my analysis of the issue. I just wanted to throw a few factors out there in order to encourage a response like yours - packed full of useful next steps for me to evalaute this further.
On of the points you raised was 'crawl efficeincy; and on further investigation I noticed that we have an awful lot of pagination on our website which users use to browse through our articles (we have 10's of thousands). However the competitor site tends to have all results on one page along with a 'show more' button. Might this be a good place to start? Looking at the cache dates of some of the results in Google, it certainly looks like the competitor pages are being crawled more often.
I also noticed from Google webmaster tools that our internal links report shows that we are giving great prominence to many of our category landing pages, rather than the article pages themselves. Does this sound like an area worh investigating?
5000 or so average crawl errors probably also isn't helping. Particularly given that we seem to only get around 6000 or so page crawled per day. Again, I;m guessing this is worth a lot of attention.
Duplicate page titles also seem to be an issue, so there is certainly lots for me to look at here.
Thanks again
Lou