20 x '400' errors in site but URLs work fine in browser...
-
Hi, I have a new client set-up in SEOmoz and the crawl completed this morning... I am picking up 20 x '400' errors, but the pages listed in the crawl report load fine... any ideas?
example -
-
Most major robots obey crawl delays. You could check your errors in Google Webmaster Tools to see if your site is serving a lot of error pages when Google crawls.
I suspect Google is pretty smart about slowing down its crawl rate when it encounters too many errors, so it's probably safe to not include a crawl delay for Google.
-
Sorry, one last question.
Do I need to add a similar delay for Google Bots, or is this issue specifically a Roger Bot problem?
Thanks
-
Fantastic, thanks, Cyrus and Tampa, prevented many more hours of scratching head!!!
-
Hi Justin,
Sometimes when rogerbot crawls a site, the servers and/or the content management system can get overwhelmed if roger is going to fast, and this causes your site to deliver error pages as roger crawls.
If the problem persists, you might consider installing a crawl delay for roger in your robots.txt file. It would look something like this:
User-agent: rogerbot
Crawl-delay: 5This would cause the SEOmoz crawlers to wait 5 seconds before fetching each page. Then, if the problem still persists, feel free to contact the help team at help@seomoz.org
Hope this helps! Best of luck with your SEO!
-
Thanks Tampa SEO, good advice.
Interestingly, the URL listed in SEOmoz is as follows:
www.morethansport.co.uk/brand/adidas?sortDirection=ascending&sortField=Price&category=sport and leisure
But when I look at the link in the referring page it is as follows:
/brand/adidas?sortDirection=ascending&sortField=Price&category=sport%20and%20leisure
notice the "%" symbol instead of the spaces.
The actual URL is the one listed in SEOmoz but even if I copy and paste the % version, the browser removed the '%' and the page loads fine.
I still can't get the site to throw-up a 400.
-
Just ran the example link that you provided through two independent HTTP response code checkers, and both are giving me a 200 response, i.e. the site is OK.
This question has been asked before on here; you're definitely not the first person to run into the issue.
One way to diagnose what's going on is to dig a little deeper into the crawling report that SEOmoz generated. Download the CSV file and look at the referring link, i.e. on which page Roger found the link. Then go to that page and look if your CMS is doing anything weird with the way it outputs the links that you create. I recall someone back in December having the same issue and eventually resolved it by noticing that his CMS put all sort of weird slashes (i.e. /.../...) into the link.
Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ecommerce URLs with numbers
Hi everybody! I have to optimize an ecommerce where somebody has previously done the SEO optimization, although the URLs have numbers before the product's name They have told me that these numbers are useful to find the products, so I think it shouldn't be really bad if I don't redirect them to "clear" ones. For example: /colesterol-sobrepeso/2217-hc-grass-capsulas-duras-15-capsulas.html > /colesterol-sobrepeso/hc-grass-capsulas-duras-15-capsulas.html Am I right? After all, they contain the keywords and the subfolders are also ok. Or it would be better if I redirect the whole site? Thanks!
On-Page Optimization | | Estherpuntu0 -
Does CloudFlare Benefit my site?
I am using the $20 / month CloudFlare service. My site is not running faster, so my question is: is this service adding real value? I do not have a lot of highly sensitive customer data (some email addresses and customer names and nothing more - no credit cards). I am already using Amazon Cloud Server and it appears to work fine. Any thoughts appreciated.
On-Page Optimization | | khi50 -
Acquired Old, Bad Content Site That Ranks Great. Redirect to Content on My Site?
Hello. my company acquired another website. This website is very old, the content within is decent at best, but still manages to rank very well for valuable phrases. Currently, we're leaving the entire site active on its own for its brand, but i'd like to at least redirect some of the content back to our main website. I can't justify spending the time to create improved content on that site and not our main site though. What would be the best practice here? 1. Cross-domain canonical - and build the new content on our main website? 2. 301 Redirect Old Article to New Location containing better article 3. Leave the content where it is - you won't be able to transfer the ranking across domain. Thanks for your input.
On-Page Optimization | | Blenny0 -
URL Keyword Variations?
I'm aware that keywords in the url aren't as effective as they used to be, but I'm still convinced that they do have a significant impact (based on results in one of the niches I'm in). My question is, will variations of keywords and "hidden" keywords have as much value as an exact keyword? For example, let's say that I'm trying to target the keyword "day." Will including variations like "daily" in the url work just as well? What about a brand name that includes the keyword hidden in its name, like "Dayest"? And, as a followup question, does including "stop" words have any effect? For example, if I'm trying to target the keyword "Day of the Month", would including "day" and "month" in the url be just as effective as including "day of the month"?
On-Page Optimization | | JABacchetta0 -
On page link question, creating an additional 'county' layer between states and zips/cities
Question We have a large site that has a page for all 50 states. Each of these pages has unique content, but following the content has a MASSIVE amount of links for each zip AND city in that state. I am also in the process of creating unique content for each of these cities and zips HOWEVER, I was wondering would it make sense to create an additional 'county' layer between the states and the zips/cities. Would the additional 'depth' of the links bring down the overall rank of the long tail city and zip pages, or would the fact that the counties would knock the on page link count down from a thousand or so, to a management 50-100 substantially improve the overall quality and ranking of the site? To illustrate, currently I have State -> city and zip pages (1200+ links on each state page) what i want to do is do state -> county (5-300 counties on each state page) -> city + zip (maybe 50-100 links on each county page). What do you guys think? Am I incurring some kind of automatic penalty for having 1000+ links on a page?
On-Page Optimization | | ilyaelbert0 -
Number ID's in URL's
If you have to place a number ID into a URL. Does it matter from an seo perspective whether it is placed in the middle or at the end? Thanks
On-Page Optimization | | simmo2350 -
Checking for content originality in a site
two part question on original content How would you go about checking if a site holds original content accept the long search quary within Google? ans also if I find many sites carrying my content and I am the original source should I replace the content? thanks
On-Page Optimization | | ciznerguy0 -
URL question
Hi guys, the pro campaign thing you got going is wicked, love it. I'm recieving good results with my keywords and have noticed that categories that go beyond sub/sub/sub don't do to well. So I wanna move those that do one step up which makes it go from: http://spytunes.com/practice-guitar/advanced-routine/scales/aeolian to here http://spytunes.com/practice-guitar/advanced-routine/aeolian The existing menu system that follow all these categories across the site will soon go so it won't be a user friendly problem, I will have other type of menus. But, and here is the question: Would I greatly benefit from taking the non existent menu away and just go for: http://spytunes.com/practice-guitar/aeolian while i'm at it? Or do I stick with my current structure? I guess my real question is; how much is there to flat URLs? Cheers -dan lundholm spytunes.com
On-Page Optimization | | spytunes0