Significant Google crawl errors
-
We've got a site that continuously like clockwork encounters server errors with when Google crawls it. Since the end of last year it will go a week fine, then it will have two straight weeks of 70%-100% error rate when Google tries to crawl it. During this time you can still put the URL in and go to the site, but spider simulators return a 404 error. Just this morning we had another error message, I did a fetch and resubmit, and magically now it's back. We changed servers on it in Jan to Go Daddy because the previous server (Tronics) kept getting hacked. IIt's built in html so I'm wondering if it's something in the code maybe?
-
This is the URL error list in Webmaster Tools
| Forms/Camp.pdf | 404 | 7/9/13 |
| | 2 | sportsinsurance.php | 404 | 5/2/13 |
| | 3 | Forms/Waiver.pdf | 404 | 7/2/13 |
| | 4 | metro/index.htm | 404 | 6/21/13 |
| | 5 | Forms/Camp_Tournament_Application.pdf | 404 | 7/9/13 |
| | 6 | Forms/Spectator.pdf | 404 | 7/9/13 |
| | 7 | Forms/Boxing.pdf | 404 | 5/6/13 |
| | 8 | sports-camp-insurance.html | 404 | 6/16/13 |
| | 9 | forms/T.C.S._ | 404 | 7/3/13 |
| | 10 | Camp | 404 | 6/14/13 |
| | 11 | Forms/Sports.pdf | 404 | 4/21/13 |
| | 12 | pages/clients.html | 404 | 4/15/13 || |
http://www.campteam.com/: Googlebot can't access your site****July 10, 2013
Over the last 24 hours, Googlebot encountered 13 errors while attempting to connect to your site. Your site's overall connection failure rate is 72.2%.
I've got 23 of these messages going back to 11/12
It tells me that no Robots.txt Fetch issues were encountered, or DNS issues. All errors are related to server connectivity according to Google.
-
I see that your site is dealing fine with 404 errors. Hrmm. Could you copy and paste the crawl error URLs you are getting from webmaster tools? Thanks!
BTW I noticed that you have a duplicate content issue in that you haven't removed the www from your URL. You should add the following code to your .htaccess file.
<code class="htaccess" title="in your .htaccess file">RewriteEngine On RewriteCond %{HTTP_HOST} !^my-domain\.com$ [NC] RewriteRule ^(.*)$ http://my-domain.com/$1 [R=301,L]</code>
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl Test Question
Good Morning, I am just looking for a little bit of advice, I ran a crawl report on our website www.swiftcomm.co.uk. I have resolved most of the issues myself, however I have two questions;- Screenshot image http://imgur.com/VlFEiZ2 Highlighted blue, we have two homepages www.swiftcomm.co.uk and www.swiftcomm.co.uk/ both are set with a Rel-Canonical Target of www.swiftcomm.co.uk/. Will this cause me any SEO issues and or other potential issue? If this may cause an issue how would I go about resolving? Highlighted yellow, Our contact and referral-form are showing as duplicate title and meta description. Both of these pages have separate title and meta desc which it does seem to be detecting. If I search the page in google it returns the correct title and meta desc. The only common denominator behind these pages is that both have php pages behind them for the contact form. Do you think that the moz crawl may be detecting the php page over the html? Could this be cause any issues when search engines crawl the site? Kind Regards Jonathan Mack VlFEiZ2
Intermediate & Advanced SEO | | JMack9860 -
Search engine blocked by robots-crawl error by moz & GWT
Hello Everyone,. For My Site I am Getting Error Code 605: Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag, Also google Webmaster Also not able to fetch my site, tajsigma.com is my site Any expert Can Help please, Thanx
Intermediate & Advanced SEO | | falguniinnovative0 -
Google Not Displaying Rich Snippets
We implemented rich snippets for products some time ago. When viewing our site through a site:xxxx.com on Google, they don't show for every product, despite the fact that they should. I've taken some of the URLs that don't show rich snippets in the SERPs, ran them through Google's testing tool, and they display fine. Not sure what's going wrong here. Any thoughts?
Intermediate & Advanced SEO | | Kingof50 -
Google Places - Advanced Question
Hi All, Ive found multiple threads about previous issues but I haven't found any tailored to my specific question.I know there are a large amount of factors so I wanted to see if any other individuals had ran into this previously. We are currently in a centralized position in a major city. We are discussing moving the main office about 15 miles away into another city, moving us out of the main city where we have been for the past 3 years. The city where we are currently located has a lot more GEO search volume compared to the new city search terms and variants of. If we move will our local rankings drop when someone searches in the city where we were previously? How long would it take for this ranking to fall? Or would we still rank because we are moving a short distance away and have a large amount of citations there? I know we would need to change over all our online directories, on page etc..Any other suggestions on a smooth transition? I know there are many factors that go into this and any past experience, guidance and/or assistance is greatly appreciated. Cheers!
Intermediate & Advanced SEO | | PRKEL0 -
Are links that are disavowed with Google Webmaster Tools removed from the Google Webmaster Profile for the domain?
Hi, Two part question - First, are links that you disavow using google webmaster tools ever removed from the webmaster tools account profile ? Second, when you upload a file to disavow links they ask if you'd like to replace the previously uploaded file. Does that mean if you don't replace the file with a new file that contains the previously uploaded urls those urls are no longer considered disavowed? So, should we download the previous disavow file first then append the new disavow urls to the file before uploading or should we just upload a new file that contains only the new disavow urls? Thanks
Intermediate & Advanced SEO | | bgs0 -
Google and private networks?
I have one or two competitors (in the UK) in my field who buy expired 1 - 8 year old domains on random subjects (SEO, travel, health you name it) and they are in the printing business and they stick 1 - 2 articles (unrelated to what was on there before) on these and that's it. I think they stick with PA and DA above 30 and most have 10 – 100 links so well used expired domains, hosted in the USA and most have different Ip’s although they now have that many (over 70% of their backlink profile) that some have the same ip. On further investigation none of the blogs have any contact details but it does look like they have been a little smart here and added content to the about us (similar to I use to run xxx but now do xxx) also they have one or two tabs with content on (article length) that is on the same subject they use to do and the titles are all the same content. So basically they are finding expired 1 – 10 year old domains that have only been expired (from what I can see) 6 months max and putting 1 – 2 articles on the home page in relation with print (maybe adding a third on the subject the blog use to cover), add 1 – 3 articles via tabs at the top on subjects the sites use to cover, registering the details via xbybssgcf@whoisprivacyprotect.com and that’s it. They have been ranking via this method for the last couple of years (through all the Google updates) and still do extremely well. Does Google not have any way to combat link networks other than the stupid stuff such as public link networks, it just seems that if you know what you are doing you get away, if your big enough you get away with it but the middle of the ground (mum and pop sites) get F*** over with spam pointing to there site that no spammer would dream of doing anyway?
Intermediate & Advanced SEO | | BobAnderson0 -
How to Block Google Preview?
Hi, Our site is very good for Javascript-On users, however many pages are loaded via AJAX and are inaccessible with JS-off. I'm looking to make this content available with JS-off so Search Engines can access them, however we don't have the Dev time to make them 'pretty' for JS-off users. The idea is to make them accessible with JS-off, but when requested by a user with JS-on the user is forwarded to the 'pretty' AJAX version. The content (text, images, links, videos etc) is exactly the same but it's an enormous amount of effort to make the JS-off version 'pretty' and I can't justify the development time to do this. The problem is that Googlebot will index this page and show a preview of the ugly JS-off page in the preview on their results - which isn't good for the brand. Is there a way or meta code that can be used to stop the preview but still have it cached? My current options are to use the meta noarchive or "Cache-Control" content="no-cache" to ask Google to stop caching the page completely, but wanted to know if there was a better way of doing this? Any ideas guys and girls? Thanks FashionLux
Intermediate & Advanced SEO | | FashionLux0 -
How to stop Google crawling after 301 redirect?
I have removed all pages from my old website and set 301 redirect to new website. But, I have verified old website with Google webmaster tools' HTML verification file which enable me to track all data and existence of pages in Google search for my old website. I was assumed that, Google will stop crawling and DE-indexed all pages after 301 redirect. Because, I have set 301 redirect before 3 months. Now, I'm able to see Google bot activity on my website with help of Google webmaster tools. You can find out attachment to know more about it. How can it possible & How Google can crawl removed pages? You can see following image to know more about it. First & Second
Intermediate & Advanced SEO | | CommercePundit0