Why is 4XX (Client Error) shown for valid pages?
-
My Crawl Diagnostics Summary says I have 5,141 errors of the 4XX (Client Error) variety. Yet when I view the list of URLs they all resolve to valid pages.
Here is an example.
http://www.ryderfleetproducts.com/ryder/af/ryder/core/content/product/srm/key/ACO 3018/pn/Wiper-Blade-Winter-18-Each/erm/productDetail.doThese pages are all dynamically created from search or browse using a database where we offer 36,000 products.
Can someone help me understand why these are errors.
-
We had spaces in the URL that browsers handled well but not spiders. We replaced the spaces with dashes with dynamic code and...it's off to the next problem.
Thanks
-
I think something is going on with a space vs a %20
- Copy and paste in the url that you listed above to Brent's recommended tool, you get a 404 response.
- If you copy and paste that url to a browser, however, your page comes up.
- Now take THAT url and paste into the tool Brent recommends, and you get a 200 (good) response.
The only difference that I see, is that when I copy and paste that url to a browser (Chrome in my case), it adds a %20 where you have a space.
Since this is the thing that makes these other url checkers work, I am guessing that the crawl diagnostics tool is having a similar problem. See the comparison below (much abbreviated to the area in question)
ACO 3018 (from your post, and gives you the error)
ACO%203018 (when it resolves in the browser, and shows a good response in the tools)
I am just smart enough to tell you that these are different, but not smart enough to know why it causes problems for crawlers, but not for browsers.
The good news is that your pages work for users. The bad news is that Google probably never sees them.
-
You need to look at the pages like a spider would. Here is a great tool to check and view the server response. In this case, you will need to go to your developers and allow them to look at this as well.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Subdomain 403 error
Hi Everyone, A crawler from our SEO tool detects a 403 error from a link from our main domain to a a couple of subdomains. However, these subdomains are perfect accessibly. What could be the problem? Is this error caused by the server, the crawlbot or something else? I would love to hear your thoughts.
Technical SEO | | WeAreDigital_BE
Jens0 -
Page content not being recognised?
I moved my website from Wix to Wordpress in May 2018. Since then, it's disappeared from Google searches. The site and pages are indexed, but no longer ranking. I've just started a Moz campaign, and most pages are being flagged as having "thin content" (50 words or less), when I know that there are 300+ words on most of the pages. Looking at the page source I find this bit of code: page contents Does this mean that Google is finding this and thinks that I have only two words (page contents) on the page? Or is this code to grab the page contents from somewhere else in the code? I'm completely lost with this and would appreciate any insight.
Technical SEO | | Photowife1 -
Joomla creating duplicate pages, then the duplicate page's canonical points to itself - help!
Using Joomla, every time I create an article a subsequent duplicate page is create, such as: /latest-news/218-image-stabilization-task-used-to-develop-robot-brain-interface and /component/content/article?id=218:image-stabilization-task-used-to-develop-robot-brain-interface The latter being the duplicate. This wouldn't be too much of a problem, but the canonical tag on the duplicate is pointing to itself.. creating mayhem in Moz and Webmaster tools. We have hundreds of duplicates across our website and I'm very concerned with the impact this is having on our SEO! I've tried plugins such as sh404SEF and Styleware extensions, however to no avail. Can anyone help or know of any plugins to fix the canonicals?
Technical SEO | | JamesPearce0 -
My website pages are not crawled, what to do?
Hi all. I have made some changes on the website so i like to crawled them by the search engines Google especially. I have made these changes around 2 weeks ago. I have submitted my website on good bookmarking websites. Also i used a tool available in Google webmasters "Fetch as Google", Resubmitted a sitemap.xml. Still my pages are not crawled your opinion please. Thanks
Technical SEO | | lucidsoftech0 -
Page for Link Building
Hello Guys, My question is about a link building process. We all know that some directories/sites do require a reciprocal link. Does it make any sense to creat a page in website exclusively to reciprocal links? And what we do with this webpage in terms of indexing, do folow, crawling...etc. Any sugestions are more then welcome 🙂 Tks in advance! PP
Technical SEO | | PedroM0 -
Noindex Pages indexed
I'm having problem that gogole is index my search results pages even though i have added the "noindex" metatag. Is the best thing to block the robot from crawling that file using robots.txt?
Technical SEO | | Tedred0 -
Have a client that migrated their site; went live with noindex/nofollow and for last two SEOMoz crawls only getting one page crawled. In contrast, G.A. is crawling all pages. Just wait?
Client site is 15 + pages. New site had noindex/nofollow removed prior to last two crawls.
Technical SEO | | alankoen1230 -
Canonical on ecommerce pages
I have seen some competitors using the nofollow tag as well as canonical on all refinements and sorts on their ecommerce pages. Example being if you went to their hard drive category page and refined by 500gb hard drives then that page would have a canonical element to send it back to hard drives page without the refinement. I see how this could be good for control indexation and the amount pages Google crawls, but do you see problems in using the canonical tag this way? Also I have seen competitors have category page descriptions (describing what that type of product is) on all pagenation and refinements (the exact same block of text on all of the pages). Would this be a duplicate content problem or is it not that big of a deal since the content is only on their site so they are only competiting with themselves. Thanks for your help
Technical SEO | | Gordian0