Website Server Issue?
-
I'm getting error messages that a website cannot be crawled and it might be due to the following issues:
Couldn't access the webpage because the server either timed out or refused/closed the connection before our crawler could receive a response. How to fix: Please contact your web hosting technical support team and ask them to fix the issue
Could Possibly Be:
1. DDoS protection system.
OR
2. Overloaded or misconfigured serverThey asked me to talk to my hosting company about this issue and he's at a loss (I don't think he knows everything he needs to know potentially). Have you seen these issues before? Where is the best spot to start troubleshooting this issue?
-
Hi,
Try using any other crawler to see if they provide you with the same message (ScreamingFrog), in some cases it could mean that the site is overloaded and will start blocking the crawler. We're doing something similar for ours as we have a lot of scrapers but exclude the 'good' sites.
Martijn.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What type of website is best for seo.
I need a new website for my health insurance business. What type is best for SEO? Many thanks
Web Design | | laurentjb0 -
New Re-design will my website rankings drop?
Hi guys, I have had to re-design my site although we are only 4 months into the seo game we have seen some good progress with our rankings. My question is there anything I need to consider before implementing the new designs so it doesn't effect my current rankings or any of our SEO work. Our current designs are content thing and so we have had to create more content to better optimize our site, however if doing so will this loose our current ranking position? Apperciate any advice around this Thanks
Web Design | | edward-may0 -
Any advice for best practice for mobile versions of websites?
Hi Mozzers, I am looking to develop a mobile version of one of my sites and was wondering what was considered best practice for the following: Where to host it: m.domain.com or domain.com/mobile ? Is rel='canonical' enough to avoid mobile version being indexed or should I use "noindex" on mobile version? How to handle tablets? Which screen size sees mobile version? Which sees full site version? Thanks in advance Geoff
Web Design | | SEM-Freak1 -
What are the top 3 things you evaluate when contracted to SEO a website?
As a freelancer, what are the top 3 things you evaluate when contracted to SEO a website for a client?
Web Design | | StreetwiseReports0 -
I need to buy a website template - anyone knows a good place to buy it? where you should buy one template?
I used the www.templatemonster.com or http://themeforest.net/, but is difficult to find a template with a focus on SEO. Also with the necessity to include text in the beginning of the website, is difficulty to found something really good.
Web Design | | Naghirniac0 -
Free website templates without spam
Where is a good site for joomla and wordpress website templates that are free? I want to learn how to do everything on a zero budget (except for hosting and domain name) and I'd like to find some decent free templates.
Web Design | | ilyaelbert0 -
Multilingual Website. Best SEO Strategy
Hello.
Web Design | | teconsite.com
We have a client that will extend his business to severak countries in Europe.
So far its main market was the Spanish one. In this market it works with his own domain that is the same that its brand name. The domain does not contain any important keyword but it has been working for 11 years: www.efutgol.com. In Spain has a good ranking. It ranks number 1 or number 2, it fluctuates. But it also sells to France and Portugal. In the France market they used a domain with exact match with the French keyword that they are interested in: www.equipementsfootball.com/. The same strategy www.equipamentosfutebol.com in Portugal. Both have number 1 for the main keyword in those Countries for that keyword Now they are going to sell in more countries.
The problem is that each website have a different CMS what makes necessary double maintenance. We are going to unificate in only one domain, one single design, one single CMS all websites. But were have doubts about SEO consequences of doing that What is the best thing for URLs?
www.efutgol.com/fr/ .....
www.efutgol.com/es/
www.efutgol.com / pt What should we do with the domains that are so well positioned in france and portugal? 301 redirect will work? What should we expect? What would you do? What things we should consider to keep the current position?0 -
How to work with US Website and UK Website?
We currently have two websites. Our headquarters in the US and our other in the UK. We currently rank very well for both websites in both countries. Currently, US is hosted in the states and the UK is hosted in the UK. I would like to keep it this way. However, I am going to be converting them to a CMS and redsign both of them. We need our main website to have a feature that they can choose which location they prefer. We also offer other locations through the US. Which we would like our customers to know that we have a few different locations. Also, we will be expanding to Austraillia and maybe others as well. We need to make the websites to look almost identical, yet, different in a way that they can tell one is US, one is UK and so on. Some will only have some of our products, so they will be smaller websites. What is the best way to go about doing this? I know some CMS offer a copy feature that allows you to make the same website. However, how can you do this if they need to be hosted in the different countries? Do I need to do them all individually on each of there servers, or can this be done differently? Also, if they are hosted in there own countries, can the content be the same, or is that still considered duplicate content?
Web Design | | hfranz0