Website Server Issue?
-
I'm getting error messages that a website cannot be crawled and it might be due to the following issues:
Couldn't access the webpage because the server either timed out or refused/closed the connection before our crawler could receive a response. How to fix: Please contact your web hosting technical support team and ask them to fix the issue
Could Possibly Be:
1. DDoS protection system.
OR
2. Overloaded or misconfigured serverThey asked me to talk to my hosting company about this issue and he's at a loss (I don't think he knows everything he needs to know potentially). Have you seen these issues before? Where is the best spot to start troubleshooting this issue?
-
Hi,
Try using any other crawler to see if they provide you with the same message (ScreamingFrog), in some cases it could mean that the site is overloaded and will start blocking the crawler. We're doing something similar for ours as we have a lot of scrapers but exclude the 'good' sites.
Martijn.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does a similar CMS fabric/theme used by 2 colleagues, cause SEO issues?
Does using the same CMS fabric/theme with substantially different content for each website but the same business address (on google places and the websites) cause SEO issues? For example: 2 colleagues with somewhat similar services located at the same business addresses desire to use the same CMS fabric /theme but will have different content on the sites. Will this hurt their SEO / should they use a different website skin/theme?
Web Design | | toti5880 -
Setting up analytics for a website redesign
Hey all, so in the past when I make changes to a site, I make the changes, review the analytics in the wake of the changes, analyze and go from there. Little things here and there, no biggie. With my new company, we're doing a full website redesign from scratch (Currently on Wordpress, moving to custom). They are asking me about analytics and reporting and I was hoping to get some insight here. When the new site is ready, they are launching it at www2.ourdomain.com and sending 25% of traffic to ourdomain.com to that with the other 75% going to www.ourdomain.com (current site). So two questions- how would you go about setting up analytics for that? And how do you ensure the www2 version doesn't get indexed but stay in Google's good graces? If you de-index your "home page" that 25% are seeing I can't imagine that's helpful for SEO. Hopefully that makes sense! Trying to look at how to A/B test to ensure the new site is working and converting before pushing all traffic to it.
Web Design | | DanDeceuster0 -
How to properly setup a website for the first time
Good morning, I am fairly proficient at SEO and understand a lot of the required elements - but am in no way an expert, which is why I am here now. We are in the process of setting up a brand new website (true PHP -Laravel) This will be for a nationwide website and will need to attract from the north, south, east and west. Although I have no worries about putting my domain out here, the site is not live and I don't want to possibly have a broken link pointing to me before I even launch - so I will give you an example URL and work from there. I will tell you it is a DOT Rentals website and we will call the business Widgets. So the URL is Widgets.Rentals There is no .com, .net - nothing. It is an exact match domain. The site operates as peer to peer - meaning customers will upload their widgets, photos, etc. of their widgets for rent. Now, having said that, let me ask this. If you were setting up this website - how would you recommend we do this right - right from the beginning? Since the customers (not the vendors) will search for widget rentals in a specific location - would you recommend that we maintain Geo Location with Google with a URL like this? Widget.Rentals/Orlando-Florida-beautiful-widgets-for-rent-at-a-great-price Or is it better to not Geo Locate? Widget.Rentals/beautiful-widgets-for-rent-at-a-great-price
Web Design | | Blitzburgh
I have a lot more in regards to setting up the site like Title bars, headers 1, 2, 3 - Alt text images, etc. But for now would appreciate the collective best advice from the group. Mike0 -
Breadcrumbs show for some section and dont come for some. What could be the issue
Hello, We have implemented breadcrumbs - which comes up nicely for Blog and Forum in Google Search However, when we look at our 2 other section - New Car Product and Latest News - it does not comes up, is there any issue which can foresee Sample URL of New Car - http://www.mycarhelpline.com/index.php?option=com_newcar&view=product&Itemid=2&id=8&vid=188 Sample URL of News - http://www.mycarhelpline.com/index.php?option=com_latestnews&view=detail&n_id=474&Itemid=10 Its been over 1 month - since breadcrumbs been introduced in both New Car and News. is there any issue like in them as why is is not showing in search. Any recommendations Many Thanks !!
Web Design | | Modi0 -
Im having duplicate content issues in wordpress
all of my pages are working fine. but i added my sitemap to my footer in my website and when i click on my blog from my footer it takes me to the homepage. so now im having duplicate content for two diff urls. ive tried adding a rel=canonical and a 301 redirect to the blog page but it doesnt resolve the problem. also, when i go to my footer and click blog. after it brings me to the homepage ill try to click on my pages from the original bar at the top of my screen and it will bring me to the right pages. but it will have the same blog url in the search bar even when im on other pages. other than that all of my pages in my footer and in my homepage toolbar work fine. its just that one particular problem with the blog page in the footer and how it stays with the same blog url on every page after i click the blog in the footer. can someone please help. im using yoast and idk if i should disable it or what.
Web Design | | ClearVisionDesign0 -
Website Blog causes duplicate pages
Hello, I added a blog to my website, which is hosted at weebly. I was told this would drive traffic but I have actually fallen way, way down in Alexa rankings. When I ran a campaign here, the results show over a 100 errors, all to do with the website blog. It states they are duplicate pages and titles. I dont see a way to rename the pages. Am I better off getting rid of the blog? Thanks
Web Design | | Gardengirl0 -
I need to buy a website template with seo focus - any tip? what technology is better?
Due to lack of resources, my company usually buys website templates and we adjust it internally. So we can always be with new designs, spending less money. With all this changes of google panda recently, I will change my website again. Usually we buy then from templatemonster, but some templates are far away from seo standards. Do you have tips where to buy than? Should I search for a specific technology or layout? Is this JS Animated website a goog choice?
Web Design | | Naghirniac0 -
Where is the best place to put reciprocal links on our website?
Where should reciprocal links be placed on our website? Should we create a "Resources" page? Should the page be "hidden" from the public? I know there is a right answer out there! Thank you for your help! Jay
Web Design | | theideapeople0