20 x '400' errors in site but URLs work fine in browser...
-
Hi, I have a new client set-up in SEOmoz and the crawl completed this morning... I am picking up 20 x '400' errors, but the pages listed in the crawl report load fine... any ideas?
example -
-
Most major robots obey crawl delays. You could check your errors in Google Webmaster Tools to see if your site is serving a lot of error pages when Google crawls.
I suspect Google is pretty smart about slowing down its crawl rate when it encounters too many errors, so it's probably safe to not include a crawl delay for Google.
-
Sorry, one last question.
Do I need to add a similar delay for Google Bots, or is this issue specifically a Roger Bot problem?
Thanks
-
Fantastic, thanks, Cyrus and Tampa, prevented many more hours of scratching head!!!
-
Hi Justin,
Sometimes when rogerbot crawls a site, the servers and/or the content management system can get overwhelmed if roger is going to fast, and this causes your site to deliver error pages as roger crawls.
If the problem persists, you might consider installing a crawl delay for roger in your robots.txt file. It would look something like this:
User-agent: rogerbot
Crawl-delay: 5This would cause the SEOmoz crawlers to wait 5 seconds before fetching each page. Then, if the problem still persists, feel free to contact the help team at help@seomoz.org
Hope this helps! Best of luck with your SEO!
-
Thanks Tampa SEO, good advice.
Interestingly, the URL listed in SEOmoz is as follows:
www.morethansport.co.uk/brand/adidas?sortDirection=ascending&sortField=Price&category=sport and leisure
But when I look at the link in the referring page it is as follows:
/brand/adidas?sortDirection=ascending&sortField=Price&category=sport%20and%20leisure
notice the "%" symbol instead of the spaces.
The actual URL is the one listed in SEOmoz but even if I copy and paste the % version, the browser removed the '%' and the page loads fine.
I still can't get the site to throw-up a 400.
-
Just ran the example link that you provided through two independent HTTP response code checkers, and both are giving me a 200 response, i.e. the site is OK.
This question has been asked before on here; you're definitely not the first person to run into the issue.
One way to diagnose what's going on is to dig a little deeper into the crawling report that SEOmoz generated. Download the CSV file and look at the referring link, i.e. on which page Roger found the link. Then go to that page and look if your CMS is doing anything weird with the way it outputs the links that you create. I recall someone back in December having the same issue and eventually resolved it by noticing that his CMS put all sort of weird slashes (i.e. /.../...) into the link.
Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hello, I've heard that the outbound links I provide in my content should have a high degree of relevancy to the topic I'm writing about or they aren't really worth including. Is this true?
Hello, I've heard that relevancy of the content between the source page and the target page of outbound links in my content matters greatly. The outbound links I provide in my content should have a high degree of relevancy to the topic I'm writing about, or they aren't really worth including. Example: Don't just link to the homepage of an organization mentioned in the article, link to a page on their site that is related to the topic you are writing about. Is this true? Would including less relevant links negatively impact SEO in any way?
On-Page Optimization | | DJBKBU0 -
To update or not to update news URLs ?
We manage a huge daily news website in my small country - keeping this a bit mysterious in case competitors are reading 🙂 Our URL structure is www.companyname.com/news/categoryofnews/title-of-article?id=articleid In this hyperreactive news world, title of articles change frequently (may be ten times a day for the main stories). The question we debate is : should we reflect the modification of the title in the URL or not ? Example : "Trump says he wants to ban search engines" would have URL http://www.companyname.com/news/entertainment/Trump-says-he-wants-to-ban-search-engines?id=12345678 Later in the day the title becomes "Trump denies he suggested banning search engines". Should the URL be modified to http://www.companyname.com/news/entertainment/Trump-denies-he-suggested-banning-search-engines?id=12345678 (option A) or not (option B) ? In Google News it makes no difference because of the sitemap, but in Google organic things are different. At present (option B in place), Google apparently doesn't see that the article has been updated, and shows the initial timestamp which is visually (and presumably SEOwise) not good : our new news looks like old news. Modifiying the URL would solve that issue, but could, may be, create another one : the new URL, being considered a new article, would lose, the acquired weight of the previous one in terms of referrals, social trafic and so on. Or not ? What do you think is the best option ? Thanks for your expertise, Yves
On-Page Optimization | | yves678901 -
Two sites into one
I have two sites owned by one client, he wants to merge them into one keeping one website, but which one? I've been using the Moz Pro to look at the stats for both sites; page authority, inbound links etc, but they're both fairly close in results. The client wants to know what would be the best course to take with these two sites, what site should he keep and which should he merge? Any advice?
On-Page Optimization | | barrowr0 -
Opencart category urls
Hi, I have a problem with the category urls in Opencart. I have duplicate page content because of this: www.mydomain.com/category and www.mydomain.com/category?page=1 are with same content. There is also a very new problem, there are new urls - autogenerated like this. www.mydomain.com/category/category?page1 These three urls are with same content and title. I tried with 301 redirect like this: RewriteRule ^category/category?page1$ www.mydomain.com/category [L,R=301] but it doesnt work. Pls help me.
On-Page Optimization | | ankali0 -
Canonical URL Tag
Hi, I have two pages that are identical on my site: http://www.absolutepower.nl/creatine-monohydraat and http://www.absolutepower.nl/CREATINE/creatine-monohydraat Should I use the canonical URL tag in this case? Thanks, Jasper
On-Page Optimization | | Japking0 -
Getting 403 error in forum
Hi all, I am getting 403 error for my site where it is throwing error for the following url http://www.topuniversityforum.in/members/member id/ignore and it is showing 7 similar url for 7 user ids. I want to know how can i resolve it and if it is going to have any negative effect on its ranking.
On-Page Optimization | | akhilendra0 -
SEOmoz crawl error
Hi, I'm getting a crawl error and it complains about there being missing meta description... But, the errors are all for non existent index files in directories that only contain pdf files and some thumbs of the front page... Just started trying to learn this stuff...! Cheers Rod
On-Page Optimization | | DrWho0 -
Best site structure for SEO
Hi, I'm currently in the process of redesigning/rebuilding a well ranking but a dated looking and structured website. Using analytics info I'm trying to put togerther an optimied site map plan for the site based on keywords. Currently the site is structured like this (a few examples) for some of its best ranking keywords / landing pages www.companyname.co.uk/frames/software/companyname-software/keyword/overview.php www.companyname.co.uk/frames/software/companyname-software/keyword/keyword.php I'd like to simplfy this as part of the re build so url's look like this www.companyname.co.uk/companyname-software/softwarecatogry/keyword Obviously I would 201 the old urls. My question is : A. is this a good idea? (From what I've read it is?) B. is there any benifit from having the company name repeated in the url (ie www.companyname.co.uk/companyname-software). My thinking before this is that companyname-software currently ranks well and brings a good amount of traffic. Or should I just go with www.companyname.co.uk/software/softwarecatogry/keyword as opposed to www.companyname.co.uk/companyname-software/softwarecatogry/keyword? Many thanks in advance!
On-Page Optimization | | JamesJacobs0