Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Multiple Sites for an Attorney Practicing Different Areas of Law
Hello, I'm trying to make sure I give a potential client the correct advice. This person is an attorney whose current site ranks well. The site deals solely with his traffic ticket defense practice. He's considering building a new site to highlight his personal injury practice but is unsure whether to build an altogether new site or redesign his current site in such a way that it includes his personal injury practice along side his traffic ticket defense practice. Obviously he doesn't want to lose his current rankings, but my concern is that he'll actually dilute his rankings somewhat with the multiple sites. Both practices have pretty different sets of keywords they would need to rank for, with pretty different difficulty levels. Any advice? Thanks.
Web Design | | lawfather2 -
Is switching from a very old HTML table site to HTML5 going to make a big difference
Hello, My site owner has been having calls that our old HTML site needs to switch to HTML5. Is it really worth the ubgrade from an archaic HTML site? Please explain. Bob
Web Design | | BobGW0 -
Site Doing Horrible After Redesign
Hello Fellow Forum Members: Thank you all for taking the time to read this. This is in follow up to one of my previous questions, but I now have more information. I will try to be as concise as possible and want to sincerely thank anybody who invests time in answering this. Around February 9, 2013, we launched our new site on the Bigcommerce platform. We moved from Volusion after 6 years. We had paid the Bigcommerce partner for an upgraded 301 redirect package as I was thoroughly concerned about losing rankings. By the end of February our rankings were diminishing. We expected a slight dip due to the new site. As of May, our organic traffic had dropped by 82%. Google WMT is showing 1500+ 404 errors. Many have to do with review page type URLs and some were just plain never redirected apparently. In May, we hired a wonderful SEO company that is a heavy contributor to the Moz community. They have been generous and wonderful to work with. By the end of this last week it was determined that most of the coding suggestions our SEO was making could NOT be implemented in Bigcommerce because Bigcommerce will not allow access to the PHP files by our developer, thus hindering the execution of these suggestions. Some of these were move the blog to the root, use canonical on the home page, use canonical for pagination, stop the indexing of https URLs and a few more. Today, June 25 we are at a complete loss and trying to just keep our business alive. The opinion of both the SEO and the developer is that my choice of Bigcommerce as a platform was not the best. So my main question is what are the odds our rankings have decreased due to the lack of 301 redirects during our migration to Bigcommerce versus the rankings decreasing do to Bigcommerce being a bad choice as a platform? We are being advised to redevelop our entire site on an Open Source platorm such as Wordpress or Magento, but if that's not needed I certainly don't want to have to do that. I hope I have provided a decent amount of history and information. Thank you for any help/advice you are willing to offer.
Web Design | | josh3300 -
On site SEO opinions
Hi all, I have been testing different configurations for my on-site SEO for a while now and I think I am pretty much there. However it is always nice to know what other SEO's think about my keyword density and usage. My site is http://www.tomlondonmagic.com I am curious as to what you think regarding landing page content, whether you need lots or text or little text? I have just removed links in the text as I feel I want to keep as much juice on my landing page as possible. Thanks all!
Web Design | | TomLondon0 -
Does Google have problem crawling ssl sites?
We have a site that was ranking well and recently dropped in traffic and ranking. The whole site is https and and not just the shopping pages. Thats the way the server is setup, they make whole site https. My manager thinks the drop in ranking is due to google not crawling https. I think contrary, but would like some feedback on this. Site is here
Web Design | | anthonytjm0 -
Unable to set preferred domain, can I verify a site that's already redirected?
I'm in the process of trying to set a preferred domain in webmaster tools -- to set our www version as preferred vs. the non www. version. IT is already redirecting non-www to www, but I get this message when trying to change settings "Part of the process of setting a preferred domain is to verify that you own http://mnn.com/. Please verify http://mnn.com/." While we own the domain, I am not sure how we can have Google access a file at [http://mnn.com/some_file when we are forwarding all requests for non-www to our www site.
Web Design | | Aggie
Note: The apache rewrite predates me and I'm not sure how / why we have two domains set up, but I'm trying to fix the preferred domain now.Am I able to verify the non version once the redirect is in place.Any ideas??? Help???Thanks!Lisa0 -
Can google crawl text in jquery sliders?
We are redesigning our website and want to present a fair amount of text within jquery sliders. Will google crawl this text or is it treated the same way as actual script? Perhaps there is a way to just have the text as plain html but use jquery to display it?
Web Design | | Netboost0
