20 x '400' errors in site but URLs work fine in browser...
-
Hi, I have a new client set-up in SEOmoz and the crawl completed this morning... I am picking up 20 x '400' errors, but the pages listed in the crawl report load fine... any ideas?
example -
-
Most major robots obey crawl delays. You could check your errors in Google Webmaster Tools to see if your site is serving a lot of error pages when Google crawls.
I suspect Google is pretty smart about slowing down its crawl rate when it encounters too many errors, so it's probably safe to not include a crawl delay for Google.
-
Sorry, one last question.
Do I need to add a similar delay for Google Bots, or is this issue specifically a Roger Bot problem?
Thanks
-
Fantastic, thanks, Cyrus and Tampa, prevented many more hours of scratching head!!!
-
Hi Justin,
Sometimes when rogerbot crawls a site, the servers and/or the content management system can get overwhelmed if roger is going to fast, and this causes your site to deliver error pages as roger crawls.
If the problem persists, you might consider installing a crawl delay for roger in your robots.txt file. It would look something like this:
User-agent: rogerbot
Crawl-delay: 5This would cause the SEOmoz crawlers to wait 5 seconds before fetching each page. Then, if the problem still persists, feel free to contact the help team at help@seomoz.org
Hope this helps! Best of luck with your SEO!
-
Thanks Tampa SEO, good advice.
Interestingly, the URL listed in SEOmoz is as follows:
www.morethansport.co.uk/brand/adidas?sortDirection=ascending&sortField=Price&category=sport and leisure
But when I look at the link in the referring page it is as follows:
/brand/adidas?sortDirection=ascending&sortField=Price&category=sport%20and%20leisure
notice the "%" symbol instead of the spaces.
The actual URL is the one listed in SEOmoz but even if I copy and paste the % version, the browser removed the '%' and the page loads fine.
I still can't get the site to throw-up a 400.
-
Just ran the example link that you provided through two independent HTTP response code checkers, and both are giving me a 200 response, i.e. the site is OK.
This question has been asked before on here; you're definitely not the first person to run into the issue.
One way to diagnose what's going on is to dig a little deeper into the crawling report that SEOmoz generated. Download the CSV file and look at the referring link, i.e. on which page Roger found the link. Then go to that page and look if your CMS is doing anything weird with the way it outputs the links that you create. I recall someone back in December having the same issue and eventually resolved it by noticing that his CMS put all sort of weird slashes (i.e. /.../...) into the link.
Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google search: 'define:____'
See: http://screencast.com/t/oFSzIt5rRm Thrilled that Google is pulling our content over wikipedia (in this instance). Wondering how we can assure more success like this. Mike Corso
On-Page Optimization | | Mike_c
Gartner.com1 -
How to Fix Google Webmasters Soft 404 Errors for Wordpress Site?
I am getting Soft 404 errors in my Google Webmasters Tools don't know how to fix it the site in on WordPress CMS the site is "http://appdictions.com/" i am getting errors in the http://appdictions.com/members section...suggestion to fix the issue will be appreciated..
On-Page Optimization | | preferati0 -
URL best practices
Hi, I have a problem here, I used http://www.vietnamvisacorp.com/faqs.html instead of http://www.vietnamvisacorp.com/faqs. Hence, http://www.vietnamvisacorp.com/faqs will be caused 404 page. My question is should I change from faqs.html to faqs (no .html)? Thanks in advance any advice?
On-Page Optimization | | JohnHuynh0 -
How to avoid product's lists from making your site's content duplicated?
Hi there! We at Outitude, recently launched an outdoor activities marketplace and to make it easy for users to compare activities we show a list of available activities in each activity view. The problem is that though the content is different, the first half is practically identical. Example:
On-Page Optimization | | alexmc
Sailing for a full day: http://outitude.com/en/sailing/world/sailing-full-day and sailing for half a day: http://outitude.com/en/sailing/world/sailing-half-day both URL's are different, their content is different but most of it is not (first half of the page), so that the user can compare the activity it is currently seing with others. Questions: How can we show the activities list without it ruining the page rank? Do you advise the use of "", "" surrounding the duplicated content aka activities lists? Thanks in advance.0 -
Site Structure
I'm confused about the best way for seo to set up the site structure . i understand the examples of the pyramid diagrams and how link juice flows, however does this mean that global navigation is not good? It appears the pyramid structure leads to the designated number of category pages (we'll use five) and they lead to the 5 content pages etc and some "superman pages" can be linked to from the home page but is this is global navigation or anchor text navigation and is gloval navigation acdeptable for content pages? Any input would be greatly appreciated. Thank you.
On-Page Optimization | | JulB0 -
What is wrong with this site
Hello, I need advice regarding design ( current design as well as any changes ) and seo advice as to what we should do to promote the site. The site in question is - Allkindofessays.com Thanks
On-Page Optimization | | seoug_20050 -
Site architecture
Hi guys, I have a new website in the recycling bins sector called recyclingbins (dot) co (dot) uk. We are building it as we speak and just wondered if the below would be a good choice for our category pages? office recycling bins kitchen recycling bins outdoor recycling bins school recycling bins home recycling bins What i am particularly interested in is whether having the keyword recycling bins at the end of every category is too much? Thank you Jon
On-Page Optimization | | imrubbish0 -
Best site structure for SEO
Hi, I'm currently in the process of redesigning/rebuilding a well ranking but a dated looking and structured website. Using analytics info I'm trying to put togerther an optimied site map plan for the site based on keywords. Currently the site is structured like this (a few examples) for some of its best ranking keywords / landing pages www.companyname.co.uk/frames/software/companyname-software/keyword/overview.php www.companyname.co.uk/frames/software/companyname-software/keyword/keyword.php I'd like to simplfy this as part of the re build so url's look like this www.companyname.co.uk/companyname-software/softwarecatogry/keyword Obviously I would 201 the old urls. My question is : A. is this a good idea? (From what I've read it is?) B. is there any benifit from having the company name repeated in the url (ie www.companyname.co.uk/companyname-software). My thinking before this is that companyname-software currently ranks well and brings a good amount of traffic. Or should I just go with www.companyname.co.uk/software/softwarecatogry/keyword as opposed to www.companyname.co.uk/companyname-software/softwarecatogry/keyword? Many thanks in advance!
On-Page Optimization | | JamesJacobs0