Website Server Issue?
-
I'm getting error messages that a website cannot be crawled and it might be due to the following issues:
Couldn't access the webpage because the server either timed out or refused/closed the connection before our crawler could receive a response. How to fix: Please contact your web hosting technical support team and ask them to fix the issue
Could Possibly Be:
1. DDoS protection system.
OR
2. Overloaded or misconfigured serverThey asked me to talk to my hosting company about this issue and he's at a loss (I don't think he knows everything he needs to know potentially). Have you seen these issues before? Where is the best spot to start troubleshooting this issue?
-
Hi,
Try using any other crawler to see if they provide you with the same message (ScreamingFrog), in some cases it could mean that the site is overloaded and will start blocking the crawler. We're doing something similar for ours as we have a lot of scrapers but exclude the 'good' sites.
Martijn.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Changing website to Mobile site
Hi All, We are building a new website and would take it live in next few weeks. I want help to understand what all should we consider.. The site structure would be same with additional features but the URL path and site name is same...Should i take any precausions before shutting down the old site and getting the new one live? Thanks
Web Design | | jomin740 -
SEO strategy for UK / US websites
Hi, We currently have a UK-focused site on www.palmatin.com ; We're now targeting the North American market as well, but the contents of the site need to be different from UK. One option was to create another domain for the NA market but I assume it would be easier to rank with palmatin.com though. What would you suggest to do, if a company is targeting two different countries in the same language? thanks, jaan
Web Design | | JaanMSonberg0 -
Website organic traffic unchanged, impressions took a 98% drop in the last week.
Hi all, I have a very curious predicament and I'd be grateful if someone could shed some light on the situation. As mentioned in the title, organic traffic to our website has remained unchanged, but organic impressions have taken a 98% drop in the last week. This happened suddenly over one day; on October 22, impressions were 700, on October 23, they were 500, and on October 24 they drastically dropped to 50. The next two days they were at 22 and then up to 35. Organic traffic, however, showed the normal "weekend drop" as of October 24, and is still showing normal level (even increased a bit) continuing into this week. These are organic impressions according to Google Analytics and Google Webmaster tools. We did perform a complete site redesign a month ago. Could this be an effect from the redesign? We also noticed drop in Domain Authority, but our competitors suffered a similar (if not greater) drop as well, so we wondered if it could be due in part to the algorithm update. If anyone could shed some light on the situation I would be so appreciative! Thanks!
Web Design | | Joanne_Pendon0 -
How can a Pincode finder website be SEO optimised?
Guys, I wanted to build a simple Pincode finder website for India. The targeted visitors as is obvious will be from India. Alike other Pincode finder websites, the users in this case too will have to key in the location / area of whose pincode he is looking for and they will get Pincode from that very location / area. Other than this, users will also come to this website when they search for something like " <location name="">pincode</location>" on Google (for instance, users will search for something like "Hiranandani Gardens Powai Pincode") Along with data fethced from our sources via Indian postal departments and other data available in public domain, we shall be using data from Google Maps API too. My question in regards to the same is as follows: What should the page-structure / structure of the website be for ranking well on Google? What should be the URL structure? Other suggestions to rank well on Google in this regards? Competition: (You can search for the term "Hiranandani Gardens Powai Pincode" to know how these sites show data) http://www.getpincode.info http://www.pincode.net.in Pls. help...
Web Design | | ShalinTJ0 -
Website Updates, will this affect my ranking/ DA or PA?
Hello All! My website is due for a big update soon, my current site is doing ok, however I want to update all of the content, structure, style and look of my whole site. I feel that with the way it's been going lately that it's time for an upgrade, my question is, will this affect my ranking? Will my page ranking's be affected? Will my DA be affected? I intend to optimize this new website perfectly too with all the things I've learnt since joining this forum. All the best, Paul
Web Design | | Paul_Tovey0 -
What do YOU look for when reviewing technical website criteria?
Wanted to get everyone's thoughts on this. One of my clients was asking me what they thought of a certain website design company. They wanted to use this company because their designs were very beautiful but wanted to be sure that contracting them would be the right decision. After I made my formal review and recommendation I thought this might be a great topic of discussion for the amazing SEOMOZ community. So, what factors do you look at when determining which design firms are worth their weight?
Web Design | | JusinDuff0 -
Changing URL's for a website redesign
Hi folks, We're redesigning our website and looking for some advice on how changing our URL's would affected our rankings. If the page URLs are changing how can we carry out redirects to avoid losing any SEO rank? Thanks, Ross
Web Design | | Will_Craig0