How can I fix this home page crawl error ?
-
My website shows this crawl error =>
612 : Home page banned by error response for robots.txt.
I also did not get any page data in my account for this website ...
I did get keyword rankings and traffic data, I am guessing from the analytics account.
url = www.mississaugakids.com
Not sure really what to do with this !
Any help is greatly appreciated.
-
Hi there,
Is this still happening, or does it seem to have been taken care of?
Cheers,
Jane
-
Or just noindex it for now? Seems worth a try unless someone more technical has a better suggestion.
Are you seeing the error only when Moz crawls the site? Google Webmaster Tools isn't signaling any problems?
D
-
Thank you Donna,
That's pretty much what I had found, but did not think that would cause the home page error I am seeing. I know that events calendar is a problem for speed. Maybe I will move the calendar off the domain and re scan. If the crawl is fine maybe I will move the calendar to a separate domain permanently.
-
Hi.
https://publib.boulder.ibm.com/infocenter/discover/v8r4/index.jsp?topic=/com.ibm.discovery.es.ad.doc/monitoring/iiysawhttp.htm says it's an error that occurred when the crawler attempted to connect to your Web server. It says a slow site or network might be the cause of the problem.
Your robots.txt is set correctly and your site is accessible (I just tried), perhaps your best bet is to wait and see if the problem recurs.
I also ran a scan of your site using screaming frog and got close to 2,000 internal server errors (response code 500) primarily from pages in this directory - http://mississaugakids.com/mississauga-events-calendar/. The pages are loading very slowly. That might be contributing to your problem.
Maybe start there and then circle back to see if the 612 error is recurring? I'm not very technical, but perhaps they're somehow related?
-
Yeah, your robot.txt seems fine, but the answer sounded like the error code could be misleading, so maybe you're looking in the wrong area for the root of the problem due to that reason. Wish I could be of more help.
-
Hello William, Thnaks for the heads up on that thread. I did see it, however my robot.txt file should be correct, so the answer in that thread did not help in my case. Looking for other options that could be the problem here. Cheers !
-
This was brought up a little while ago, hopefully Chiaryn's answer here can help: http://moz.com/community/q/without-robots-txt-no-crawling
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
404 Errors for Form Generated Pages - No index, no follow or 301 redirect
Hi there I wonder if someone can help me out and provide the best solution for a problem with form generated pages. I have blocked the search results pages from being indexed by using the 'no index' tag, and I wondered if I should take this approach for the following pages. I have seen a huge increase in 404 errors since the new site structure and forms being filled in. This is because every time a form is filled in, this generates a new page, which only Google Search Console is reporting as a 404. Whilst some 404's can be explained and resolved, I wondered what is best to prevent Google from crawling these pages, like this: mydomain.com/webapp/wcs/stores/servlet/TopCategoriesDisplay?langId=-1&storeId=90&catalogId=1008&homePage=Y Implement 301 redirect using rules, which will mean that all these pages will redirect to the homepage. Whilst in theory this will protect any linked to pages, it does not resolve this issue of why GSC is recording as 404's in the first place. Also could come across to Google as 100,000+ redirected links, which might look spammy. Place No index tag on these pages too, so they will not get picked up, in the same way the search result pages are not being indexed. Block in robots - this will prevent any 'result' pages being crawled, which will improve the crawl time currently being taken up. However, I'm not entirely sure if the block will be possible? I would need to block anything after the domain/webapp/wcs/stores/servlet/TopCategoriesDisplay?. Hopefully this is possible? The no index tag will take time to set up, as needs to be scheduled in with development team, but the robots.txt will be an quicker fix as this can be done in GSC. I really appreciate any feedback on this one. Many thanks
Technical SEO | | Ric_McHale0 -
Home page canonical issues
Hi, I've noticed I can access/view a client's site's home page using the following URL variations - http://example.com/
Technical SEO | | simon-145328
http://example/index.html
http://www.example.com/
http://www.example.com/index.html There's been no preference set in Google WMT but Google has indexed and features this URL - http://example.com/ However, just to complicate matters, the vast majority of external links point to the 'www' version. Obviously i would like to tidy this up and have asked the client's web development company if they can place 301 redirects on the domains we no longer want to work - I received this reply but I'm not sure whether this does take care of the duplicate issue - Understand what you're saying, but this shouldn't be an issue regarding SEO. Essentially all the domains listed are linking to the same index.html page hosted at 1 location My question is, do i need to place 301 redirects on the domains we don't want to work and do i stick with the 'non www' version Google has indexed and try to change the external links so they point to the 'non www' version or go with the 'www' version and set this as the preferred domain in Google WMT? My technical knowledge in this area is limited so any help would be most appreciated. Regards,
Simon.0 -
During my last crawl suddenly no errors or warnings were found, only one, a 403 error on my homepage.
There were no changes made and all my old errors dissapeard, i think something went wrong. Is it possible to start another crawl earlyer then scheduled?
Technical SEO | | KnowHowww0 -
Pages Linking to Sites that Return 404 Error
We have just a few 404 errors on our site. Is there any way to figure out which pages are linking to the pages that create 404 errors? I would rather fix the links than create new 301 redirects. Thanks!
Technical SEO | | jsillay0 -
X-cart page crawling question.
I have an x-cart site and it is showing only 1 page being crawled. I'm a newbie, is this common? Can it be changed? If so, how? Thanks.
Technical SEO | | SteveLMCG0 -
Why is it that in the exported CSV there are no refrerring pages shown for 404 errors?
Within some of my campaigns i can see issues regarding 404 pages. Then when i export the data to a csv, sometimes the referring pages that lead tot the 404 are not shown. Am i missing something here?
Technical SEO | | 5MMedia0 -
Can URL re writes fix the problem of critical content too deep in a sites structure?
Good morning from Wetherby UK 🙂 Ok imagine this scenario. You ask the developers to design a site where "offices to let" is on level two of a sites hierachy and so the URL would look like this: http://www.sandersonweatherall.co.uk/office-to-let. But Yikes when it goes live it ends up like this: http://www.sandersonweatherall.co.uk...s/residential/office-to-let Is a fix to this a URL re - write? Or is the only fix relocating the office to let content further up the site structure? Any insights welcome 🙂
Technical SEO | | Nightwing0 -
Discrepency between # of pages and # of pages indexed
Here is some background: The site in question has approximately 10,000 pages and Google Webmaster shows that 10,000 urls(pages were submitted) 2) Only 5,500 pages appear in the Google index 3) Webmaster shows that approximately 200 pages could not be crawled for various reasons 4) SEOMOZ shows about 1,000 pages that have long URL's or Page Titles (which we are correcting) 5) No other errors are being reported in either Webmaster or SEO MOZ 6) This is a new site launched six weeks ago. Within two weeks of launching, Google had indexed all 10,000 pages and showed 9,800 in the index but over the last few weeks, the number of pages in the index kept dropping until it reached 5,500 where it has been stable for two weeks. Any ideas of what the issue might be? Also, is there a way to download all of the pages that are being included in that index as this might help troubleshoot?
Technical SEO | | Mont0