Website Server Issue?
-
I'm getting error messages that a website cannot be crawled and it might be due to the following issues:
Couldn't access the webpage because the server either timed out or refused/closed the connection before our crawler could receive a response. How to fix: Please contact your web hosting technical support team and ask them to fix the issue
Could Possibly Be:
1. DDoS protection system.
OR
2. Overloaded or misconfigured serverThey asked me to talk to my hosting company about this issue and he's at a loss (I don't think he knows everything he needs to know potentially). Have you seen these issues before? Where is the best spot to start troubleshooting this issue?
-
Hi,
Try using any other crawler to see if they provide you with the same message (ScreamingFrog), in some cases it could mean that the site is overloaded and will start blocking the crawler. We're doing something similar for ours as we have a lot of scrapers but exclude the 'good' sites.
Martijn.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ability to Transition Completed Wordpress Website to New Coder/Developer
We have worked with the same Wordpress developer since 2012. They recently redesigned our Wordpress site. We purchased a real estate theme and they performed major modifications to it. The project took 8 months. There are many customized widgets and multiple plugins. We hired a new SEO. The SEO is very comfortable coding. The SEO performed certain modifications and the code broke. The original developer stepped in and and helped restore the code. The SEO stated that the site should not be so delicate; that too many plugins and widgets are used making it inherently unstable. The original developer is claiming that the SEO did not follow best practices (they did not use a dev server to test). For a non technical business owner this is very disturbing. We finally agreed that the new SEO would make changes on a dev server and the original developer will check these changes to ensure they do not break the code. My question is, shouldn't a Wordpress site be simple enough to hand over to a decent coder with little risk of breaking the code? Are there any standards regarding the hand over of a site? I am comfortable with my developers, but what if they change professions or close their company? How would I transition the site? There must be standards and protocols that allow a third party, such as an SEO to change code without causing havoc. Any one have some insight?
Web Design | | Kingalan11 -
Free websites that are good with SEO?
Dear members, I am looking for a free/ almost free website which is good for SEO. For example i am looking at WIX right now but i keeping reading that they aren't optimal for SEO. Does anybody has some tips which website is can use, example weebly, strato, etc?? Many thanks!
Web Design | | rijwielcashencarry0400 -
Geo Tagging Your Website?
Is it worth it to do this to your site if it has a local focus? What are the advantages and disadvantages? Thanks! ~Ricky
Web Design | | RickyShockley0 -
Redesigning a really old Website with old-fashioned permalinks
Hey SEO-Pros, I'm currently redesigning a quite old website, with the following URL structure: TLD/category/category.php?interview_id=819 The new Version will be a little more SEO-friendly: TLD/interviews/name-of-interview/ I know I have to do a 301-Redirect for all the old URLs to the new ones in order to keep the (until now pretty good) Google Ranking. If the rankings drop after the redesign has been done, I'll surely get killed 😉 So, Is there any easy way of creating these 301-Redirects (must be thousands of URLs...)? Best Regards guys and thanks for your help!
Web Design | | dominator0 -
Recommended Website Monitoring Tools
Hi, I was wondering what people would recommend for website monitoring (IE is my website working as it should!). I need something that will:
Web Design | | James77
1/. Allow multiple page monitoring not just homepage
2/. Do header status checking
3/. Do page content checking (ie if the page changes massively, or include the word "error") then we have an issue!
4/. Multiple alert possibilities. We currently use www.websitepulse.com and it is a good service that does all the above, however it just seems so overly complex that its hard to understand what is going on, and its complex functionality and features are really a negative in our case. Thanks0 -
How to work with US Website and UK Website?
We currently have two websites. Our headquarters in the US and our other in the UK. We currently rank very well for both websites in both countries. Currently, US is hosted in the states and the UK is hosted in the UK. I would like to keep it this way. However, I am going to be converting them to a CMS and redsign both of them. We need our main website to have a feature that they can choose which location they prefer. We also offer other locations through the US. Which we would like our customers to know that we have a few different locations. Also, we will be expanding to Austraillia and maybe others as well. We need to make the websites to look almost identical, yet, different in a way that they can tell one is US, one is UK and so on. Some will only have some of our products, so they will be smaller websites. What is the best way to go about doing this? I know some CMS offer a copy feature that allows you to make the same website. However, how can you do this if they need to be hosted in the different countries? Do I need to do them all individually on each of there servers, or can this be done differently? Also, if they are hosted in there own countries, can the content be the same, or is that still considered duplicate content?
Web Design | | hfranz0 -
Old links in Google, new website affecting SEO?
Hi Guys, I have launched my website in october and it has already been indexed by google. Now I'm going to launch my redesign which comes with a new structure, content, links, etc. So the question is, do I have to resubmit my website to google to get rid of old links? Onsite Explorer shows links to my forum which has been spammed with p* stuff which has been already indexed as well. The forum is off now. I want to use SEOmoz to track my new website but I guess this could be a hard thing as old links etc will be shown as well. Is there any tool to let Google know about my changes? Does it affect my SEO in any way? Thank you for your help. Nick
Web Design | | NickITW0