20 x '400' errors in site but URLs work fine in browser...
-
Hi, I have a new client set-up in SEOmoz and the crawl completed this morning... I am picking up 20 x '400' errors, but the pages listed in the crawl report load fine... any ideas?
example -
-
Most major robots obey crawl delays. You could check your errors in Google Webmaster Tools to see if your site is serving a lot of error pages when Google crawls.
I suspect Google is pretty smart about slowing down its crawl rate when it encounters too many errors, so it's probably safe to not include a crawl delay for Google.
-
Sorry, one last question.
Do I need to add a similar delay for Google Bots, or is this issue specifically a Roger Bot problem?
Thanks
-
Fantastic, thanks, Cyrus and Tampa, prevented many more hours of scratching head!!!
-
Hi Justin,
Sometimes when rogerbot crawls a site, the servers and/or the content management system can get overwhelmed if roger is going to fast, and this causes your site to deliver error pages as roger crawls.
If the problem persists, you might consider installing a crawl delay for roger in your robots.txt file. It would look something like this:
User-agent: rogerbot
Crawl-delay: 5This would cause the SEOmoz crawlers to wait 5 seconds before fetching each page. Then, if the problem still persists, feel free to contact the help team at help@seomoz.org
Hope this helps! Best of luck with your SEO!
-
Thanks Tampa SEO, good advice.
Interestingly, the URL listed in SEOmoz is as follows:
www.morethansport.co.uk/brand/adidas?sortDirection=ascending&sortField=Price&category=sport and leisure
But when I look at the link in the referring page it is as follows:
/brand/adidas?sortDirection=ascending&sortField=Price&category=sport%20and%20leisure
notice the "%" symbol instead of the spaces.
The actual URL is the one listed in SEOmoz but even if I copy and paste the % version, the browser removed the '%' and the page loads fine.
I still can't get the site to throw-up a 400.
-
Just ran the example link that you provided through two independent HTTP response code checkers, and both are giving me a 200 response, i.e. the site is OK.
This question has been asked before on here; you're definitely not the first person to run into the issue.
One way to diagnose what's going on is to dig a little deeper into the crawling report that SEOmoz generated. Download the CSV file and look at the referring link, i.e. on which page Roger found the link. Then go to that page and look if your CMS is doing anything weird with the way it outputs the links that you create. I recall someone back in December having the same issue and eventually resolved it by noticing that his CMS put all sort of weird slashes (i.e. /.../...) into the link.
Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Star snippet not work
Hi I write a json+ld script for star snippet in my website but not work in my first page. you can see it in this URL https://www.tehran-borj.ir
On-Page Optimization | | pifpaffarm
when I googled my keyword my competitor show with star snippet in SERP but my site doesn't show.0 -
Should I change my website urls?
We're translating our website in a few languages (FR / DE / JP) using subdirectories. So our website will have the following urls www.brand.com/en
On-Page Optimization | | dcalexandra
www.brand.com/fr
www.brand.com/de
www.brand.com/jp I would like to change the url structure of a few pages from www.brand.com/section/feature-name to www.brand.com/feature-name Is it a good idea to do this now since we're adding the subfolders and these are anyway new urls in google's eyes?0 -
Help finding someone to handle crashing site/site optimization.
I need someone who can handle website/WordPress issues as they come up.For example, my site has gone down 4 times tonight, and my host can't figure it out. They also keep recommending that I optimize my site, but I don't know how. I need a go-to web person for this sort of thing. Any recommendations?
On-Page Optimization | | cbrant7770 -
Best URL structure for my page
Hey everyone, I am wondering what is the best URL structure for my activity booking page in Bali. http://www.thingstodoinbali.com/bali/sports/water-sports/rafting/ or http://www.thingstodoinbali.com/bali/sports/water-sports/rafting-bali/ or would you recommend something else? Most people would google for "activity bali" in this case "rafting bali". Please advice. Thanks for your time and help in advance!
On-Page Optimization | | c.eiermann0 -
I've just manually edited all the page titles and meta descriptions on a site, when will this show in Google results?
I've just manually edited all of the page titles, meta descriptions and optimised the copy on a client's site. I submitted this for a new crawl on Google via Webmaster Tools but when I do a Google search the old versions are still showing. Will it still take a few weeks for the new versions to show even though Google has crawled it via Webmaster?
On-Page Optimization | | aoifep0 -
Optimize URL
Hello, My website have been running over five years. I have just reviewed and seen some URLs had not good. It is http://www.vietnamvisacorp.com/faqs/who-need-visa-to-vietnam---1.html, containing characters "---1". Should I remove unnecessary characters "---"?. Thanks for any advice!
On-Page Optimization | | JohnHuynh0 -
If you were working on a wine site would you include the wine year in the URL?
I've come across a case where I'm asking myself what the best direction would be to go and while there is no right direction I would like to here some feedback from others. I'm working with some great content pages all about wine. As you probably know the difference between a 07 wine and a 95 is vastly different and up to this point I'm using the full year in the url much like this: grapesinyourtoesexample.com/2007-cellar-pod-viognier-adelaide-hills/. What I'm worried about is my use of the year in the URL. I feel it's very important for it to be used in the page title and on page but I'm concerned that it might be setting me back with my use of it in the url. My concern is that search engines might be interpretting it as a datestamp rather than as a informational piece of data describing the asset. Looking at my competitors, my content is one of the only sites using the year and in most searches for various wines my content is in the second half of the SERPs. If you were creating this content would you use the year? If you were working with current content would you drop the year across all of the site and implement to necessary redirects? Just to be clear this is a client related project so my use of "my site|my content" refers to the client's content.
On-Page Optimization | | DotCar0 -
URL Rewrite
(By Google Traductor) Hello, I wanted to ask about some changes that we are evaluating for the issue of passing the url with variables to be more descriptive, for example: http://www.agroads.com.ar/detalle.asp?clasi=139592 tohttp://www.agroads.com.ar/humedimetro-para-cereales-draminski-gmm-139592.html In this case corresponds to the breakdown of a product if you have long published andcan be well positioned to change the title of this position would be lost unless youmanage it with a 301, as one would manage when you have more than 30000 products and title may change several times? There are tools to manage this? Finally, we must apply this to all listed with their respective filters, recommends doingtheir part with 301 redirects and analyze what funciene well to continue with the rest or implement a complete change? I hope I can bring a little light to implement this. Greetings and thanks! Roberto
On-Page Optimization | | romaro0