Crawl Diagnostics - unexpected results
-
I received my first Crawl Diagnostics report last night on my dynamic ecommerce site.
It showed errors on generated URLs which simply are not produced anywhere when running on my live site. Only when running on my local development server.
It appears that the Crawler doesn't think that it's running on the live site.
For example
http://www.nordichouse.co.uk/candlestick-centrepiece-p-1140.html
will go to a Product Not Found page, and therefore Duplicate Content errors are produced.
Running
http://www.nhlocal.co.uk/candlestick-centrepiece-p-1140.html
produces the correct product page and not a Product Not Found page
Any thoughts?
-
Hi Nordichouse,
Sorry it took awhile for me to get back to you on this.
I agree with the SEOmoz techs, it doesn't matter if it is a crawler or a actual person, if you go to an invalid url you should be redirected as 301 to the actual page. If the product doesn't exist it should not allow for superfluous urls.
So basically what you should have is if the product exist then the site redirects to the correct URL. If it doesn't exist then send any query for that product to the same page and display the oscommerce product not found message. By doing this you prevent the system from creating upteenthousand urls for each product.
If you want to test what I mean you can visit our store a www.rubberstore.com/catalog and try a few urls like:
catalog/nipple-clips-p-1000.html
we don't have a product with the id of 1000 so you'll get redirected to the not found message and the root page
-p-1000.htmlhowever if you try:
catalog/a-fake-url-p-29.html
you'll get redirected to our actual product page matching this product id.Hope that makes since. All this is done with the .htaccess url re-writter I posted above.
-
Don
Yes, that is how it is done and there is no problem with that. The above is just how inbound URLs get processed.
The issue here is how the crawler works. The only possible way for this particular URL to be generated is for a certain parameter to be appended to the URL - and that would be unusual (unless SEOmoz techies tell me different)
Alan
-
Did you ever have a product with the id of 1140? If you look at your products table just check the auto number in the product_id column..
If you did and it was live at some point it could be finding the old product based on the old url it used to have.
If you never made that product live then I don't know how a crawler could of found a product that doesn't exist unless they starting using some technology that I'm unaware of.
Since you said you use OSC this what we use to deal with the problem I outlined above..
Begin Ultimate SEO V2.2d
Options +FollowSymLinks
RewriteEngine On# RewriteBase instructions
# Change RewriteBase dependent on how your shop is accessed as below.
# http://www.mysite.com = RewriteBase /
# http://www.mysite.com/catalog/ = RewriteBase /catalog/
# http://www.mysite.com/catalog/shop/ = RewriteBase /catalog/shop/# Change the following line using the instructions above
RewriteBase /catalog/RewriteRule ^(.)-p-(.).html$ product_info.php?products_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-c-(.).html$ index.php?cPath=$2&%{QUERY_STRING}
RewriteRule ^(.)-m-(.).html$ index.php?manufacturers_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-pi-(.).html$ popup_image.php?pID=$2&%{QUERY_STRING}
RewriteRule ^(.)-by-(.).html$ all-products.php?fl=$2&%{QUERY_STRING}
RewriteRule ^(.)-t-(.).html$ articles.php?tPath=$2&%{QUERY_STRING}
RewriteRule ^(.)-a-(.).html$ article_info.php?articles_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-au-(.).html$ articles.php?authors_id=$2&%{QUERY_STRING}
#RewriteRule ^(.)-pr-(.).html$ product_reviews.php?products_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-pri-(.).html$ product_reviews_info.php?products_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-f-(.).html$ faqdesk_info.php?faqdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-fc-(.).html$ faqdesk_index.php?faqPath=$2&%{QUERY_STRING}
RewriteRule ^(.)-fri-(.).html$ faqdesk_reviews_info.php?faqdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-fra-(.).html$ faqdesk_reviews_article.php?faqdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-i-(.).html$ information.php?info_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-links-(.).html$ links.php?lPath=$2&%{QUERY_STRING}
RewriteRule ^(.)-pm-([0-9]+).html$ info_pages.php?pages_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-n-(.).html$ newsdesk_info.php?newsdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-nc-(.).html$ newsdesk_index.php?newsPath=$2&%{QUERY_STRING}
RewriteRule ^(.)-nri-(.).html$ newsdesk_reviews_info.php?newsdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-nra-(.).html$ newsdesk_reviews_article.php?newsdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-po-([0-9]+).html$ pollbooth.php?pollid=$2&%{QUERY_STRING}End Ultimate SEO V2.2d
You may try it to see if it helps fix your issue.
-
Thanks, Don
You are right in your analysis - it is osC, but highly modified by myself. Yes, it does redirect.
That, however, is not the point. On the live site, the URL containing 1140 (for example) is never generated.
The mystery is how the Crawler can find something that isn't there! Magic.
Alan
-
Hi nordichouse,
You may want to check with your CMS provider. The urls are similar to Oscommerce which I'm experienced with, but I can see that isn't an Oscommerce setup. The system should have some sort of URL re-writer to deal with this problem.
The issue that I see is the system actually doesn't care what you type in between .co.uk/ and -p-1140.html
For example try this url to get a valid product..
http://www.nordichouse.co.uk/nipple-clips-p-1000.html
which is the same as
http://www.nordichouse.co.uk/-p-1000.html
But should 301 redirect to: http://www.nordichouse.co.uk/linen-style-collection-p-1000.htmlOscommerce has a URL 301 re-writer that prevents the system for using incorrect URL's I would hope your system does as well.
I'm not trying to avoid helping you, but the without an exact knowledge of how the system handles URL's it generates it is hard to troubleshoot, however since it is a CMS somebody who works on it should already have this knowledge.
My best,
Don
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I got an 803 error yesterday on the Moz crawl for most of my pages. The page loads normally in the browser. We are hosted on shopify
I got an 803 error yesterday on the Moz crawl for most of my pages. The page loads normally in the browser. We are hosted on shopify, the url is www.solester.com please help us out
Moz Pro | | vasishta0 -
Using Keyword Tool Results
I love the keyword tool for giving me direction and helping to prioritize. My question: Once I have the prioritized keywords, do you recommend having a page for each of the top keywords, using the keyword as the page name? Or is it better to sprinkle the keywords into existing pages? Thank you.
Moz Pro | | bhsiao0 -
Crawl report - duplicate page title/content issue
When the crawl report is finished, it is saying that there are duplicate content/page titles issues. However there is a canonical tag that is formatted correctly so just wondered if this was a bug or if anyone else was having the same issues? For example, I'm getting a error warning for this page http://www.thegreatgiftcompany.com/categories/categories_travel?sort=name_asc&searchterm=&page=1&layout=table
Moz Pro | | KarlBantleman0 -
Why does Crawl Diagnostics report this as duplicate content?
Hi guys, we've been addressing a duplicate content problem on our site over the past few weeks. Lately, we've implemented rel canonical tags in various parts of our ecommerce store, over time, and observing the effects by both tracking changes in SEOMoz and Websmater tools. Although our duplicate content errors are definitely decreasing, I can't help but wonder why some URLs are still being flagged with duplicate content by our SEOmoz crawler. Here's an example, taken directly from our Crawl Diagnostics Report: URL with 4 Duplicate Content errors:
Moz Pro | | yacpro13
/safety-lights.html Duplicate content URLs:
/safety-lights.html ?cat=78&price=-100
/safety-lights.html?cat=78&dir=desc&order=position /safety-lights.html?cat=78 /safety-lights.html?manufacturer=514 What I don't understand, is all of the URLS with URL parameters have a rel canonical tag pointing to the 'real' URL
/safety-lights.html So why is SEOMoz crawler still flagging this as duplicate content?0 -
Crawl Diagnostics Shows thousands of 302's from a single url. I'm confused
Hi guys I just ran my first campaign and the crawl diagnostics are showing some results I'm unfamiliar with.. In the warnings section it shows 2,838 redirects.. this is where I want to focus. When I click here it shows 5 redirects per page. When I go to click on page 2, or next page, or any other page than page 1 for that matter... this is where things get confusing. Nothing shows. Downloading the csv reveals that 2,834 of these are all showing: URL: http://www.mydomain.com/401/login.php url: http://www.mydomain.com/401/login.php referrer: http://www.mydomain.com/401/login.php location_header: http://www.mydomain.com/401/login.php I guess I'm just looking for an explanation as to why it's showing so many to the same page and what possible actions can be taken on my part to correct it (if needed). Thanks in advance
Moz Pro | | sethwb0 -
Wild fluctuation in number of pages crawled
I am seeing huge fluctuations in the number of pages discovered the crawl each week. Some weeks the crawl discovers > 10,000 pages and other weeks I am seeing 4-500. So, this week for example I was hoping to see some changes reflected for warnings from last weeks report (which discovered > 10,000 pages). However, the entire crawl this week was 448 pages. The number of pages discovered each week seems to go back and forth between these two extremes. The more accurate count would be nearer the 10,000 mark than the 400 range. Thanks. Mark
Moz Pro | | MarkWill0 -
What causes Crawl Diagnostics Processing Errors in seomoz campaign?
I'm getting the following error when seomoz tries to spider my site: First Crawl in Progress! Processing Issues for 671 pages Started: Apr. 23rd, 2011 Here is the robots.txt data from the site: Disallow ALL BOTS for image directories and JPEG files. User-agent: * Disallow: /stats/ Disallow: /images/ Disallow: /newspictures/ Disallow: /pdfs/ Disallow: /propbig/ Disallow: /propsmall/ Disallow: /*.jpg$ Any ideas on how to get around this would be appreciated 🙂
Moz Pro | | cmaddison0