20 x '400' errors in site but URLs work fine in browser...
-
Hi, I have a new client set-up in SEOmoz and the crawl completed this morning... I am picking up 20 x '400' errors, but the pages listed in the crawl report load fine... any ideas?
example -
-
Most major robots obey crawl delays. You could check your errors in Google Webmaster Tools to see if your site is serving a lot of error pages when Google crawls.
I suspect Google is pretty smart about slowing down its crawl rate when it encounters too many errors, so it's probably safe to not include a crawl delay for Google.
-
Sorry, one last question.
Do I need to add a similar delay for Google Bots, or is this issue specifically a Roger Bot problem?
Thanks
-
Fantastic, thanks, Cyrus and Tampa, prevented many more hours of scratching head!!!
-
Hi Justin,
Sometimes when rogerbot crawls a site, the servers and/or the content management system can get overwhelmed if roger is going to fast, and this causes your site to deliver error pages as roger crawls.
If the problem persists, you might consider installing a crawl delay for roger in your robots.txt file. It would look something like this:
User-agent: rogerbot
Crawl-delay: 5This would cause the SEOmoz crawlers to wait 5 seconds before fetching each page. Then, if the problem still persists, feel free to contact the help team at help@seomoz.org
Hope this helps! Best of luck with your SEO!
-
Thanks Tampa SEO, good advice.
Interestingly, the URL listed in SEOmoz is as follows:
www.morethansport.co.uk/brand/adidas?sortDirection=ascending&sortField=Price&category=sport and leisure
But when I look at the link in the referring page it is as follows:
/brand/adidas?sortDirection=ascending&sortField=Price&category=sport%20and%20leisure
notice the "%" symbol instead of the spaces.
The actual URL is the one listed in SEOmoz but even if I copy and paste the % version, the browser removed the '%' and the page loads fine.
I still can't get the site to throw-up a 400.
-
Just ran the example link that you provided through two independent HTTP response code checkers, and both are giving me a 200 response, i.e. the site is OK.
This question has been asked before on here; you're definitely not the first person to run into the issue.
One way to diagnose what's going on is to dig a little deeper into the crawling report that SEOmoz generated. Download the CSV file and look at the referring link, i.e. on which page Roger found the link. Then go to that page and look if your CMS is doing anything weird with the way it outputs the links that you create. I recall someone back in December having the same issue and eventually resolved it by noticing that his CMS put all sort of weird slashes (i.e. /.../...) into the link.
Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help for 404 error
Is it ok to use hash(#) in URL as right now it is showing 404 error while checking broken links using check my link chrome extension. E.g. http://instafrsh.com/index.php#aboutUs If not, Please suggest how to resolve this issue
On-Page Optimization | | Obbserv0 -
Same titles in Webmasters' tools
Hello, In webmasters' tools I get a message saying I have 505 pages html improvements that are possible because 505 of my titles are used on two pages. Actually, webmasters' tools is tripping since the doubles they find are: www.example.com and www.example.com/ Anyways, I have indexation problems and I was wondering if this could be the case and how to solve that. Thanks for your help;
On-Page Optimization | | EndeR-0 -
Acquired Old, Bad Content Site That Ranks Great. Redirect to Content on My Site?
Hello. my company acquired another website. This website is very old, the content within is decent at best, but still manages to rank very well for valuable phrases. Currently, we're leaving the entire site active on its own for its brand, but i'd like to at least redirect some of the content back to our main website. I can't justify spending the time to create improved content on that site and not our main site though. What would be the best practice here? 1. Cross-domain canonical - and build the new content on our main website? 2. 301 Redirect Old Article to New Location containing better article 3. Leave the content where it is - you won't be able to transfer the ranking across domain. Thanks for your input.
On-Page Optimization | | Blenny0 -
Number ID's in URL's
If you have to place a number ID into a URL. Does it matter from an seo perspective whether it is placed in the middle or at the end? Thanks
On-Page Optimization | | simmo2350 -
Temp redirects of homepage URLs
Working on a site for a client and the CMS provider has all variants of their domain (.co.uk & .com and / and http:// etc) temp redirecting to the /default.aspx homepage. I know in theory that this means no Google Juice is being passed to the final location page, does this mean that all backlinks that point to their domain are not actually passing Juice onto the homepage? Thanks
On-Page Optimization | | Switch_Digital0 -
Getting 403 error in forum
Hi all, I am getting 403 error for my site where it is throwing error for the following url http://www.topuniversityforum.in/members/member id/ignore and it is showing 7 similar url for 7 user ids. I want to know how can i resolve it and if it is going to have any negative effect on its ranking.
On-Page Optimization | | akhilendra0 -
Seeking URL Advice
Hey Moz Community, I'm looking for some URL structure advice for a new directory of a website. We're trying to rank for the term 'internships abroad in <country>'</country> We have roughly 100 pages targeting specific countries. Right now the URL structure is www.gooverseas.com/internships-abroad/china, but some of my colleagues believe this structure would be better: www.gooverseas.com/internships-abroad/intern-in-china. I personally prefer the shorter structure, but we couldn't come to any agreement so we thought we'd pose the question to the community. Any thoughts? Thanks!
On-Page Optimization | | dunklea0