Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Specifying image dimensions for site speed vs. responsive
I'm working on improving site speed from an SEO perspective, and one bit of advice I see often is to specify the dimensions of the images you're using so the browser knows the size of the image it needs to download. However, I am wondering what impact this may have if the site is responsive? If you specify the large dimensions suited for a desktop browser, would you be forcing a mobile browser to use that sized image? Has anyone seen dramatic improvements in site speed using the <picture>tag for responsive images?</picture> Thanks! Jannette
Web Design | | JannetteP1 -
Site with no ads hit by Page Layout update?
Hi there! Can a site that has no ads on it be hit by Google's latest Page Layout update? Can it be hit for just one or two keywords? My site (www.ink2paper.com) has a decline in Google organic traffic in early Feb so my suspicion is the Page Layout update. However I have no ads on the site. Digging into GWMT I find that it is only one or 2 keywords that seems to have taken a dive, mainly [photo paper]. I used to get around 80 imps a day for this term. Then on 6 Feb it was down to 50; 7 Feb = 34; 8 Feb just 4 impressions! I got a spike back at usual levels on 10 & 11 Feb, but since then it has been back down to only 5 or so impressions a day. [photographic paper] took a small hit at the start of February, but has nose dived since the start of April. The homepage performs well for Google organic traffic - low bounce (22%) and good ecom conversion rate (14%) - although this is likely to be largely branded traffic. I feel my site is a 'good' result for the search term [photo paper], although there is always room for improvement of course! Any suggestions as to why Google has stopped showing my site for these keywords? All help is greatly appreciated. Cheers,
Web Design | | SimonHogg
Simon0 -
/index.php/ What is its purpose and does it hurt SEO?
Hello Moz Forum, I am still in the process of cleaning up the lack of attention to detail and betrayal set by our soon to be ex-SEO company. You can see a previous question I ask regarding betrayal SEO. I am analyzing every page on our website and i am noticing this /index.php/ in most of our URLs. We want to leave our expression engine cms and convert to wordpress. I have been reading about index.php but most of it is over my head for now. What does concern me is the "layman's" findings i am seeing through analytics. Our main domain has two URLs. one that ends in .com and the other ends in .com/index.php/ The one that ends in .com has a higher page rank than the ladder. And there are other internal pages with the same two variations. Can someone please explain to me what is /index.php/ ? what are the benefits of it? what are the cons? What will happen to my site once we move to wordpress? As always, your comments and suggestions are greatly appreciated.
Web Design | | CamiloSC0 -
Best Way to Re-Direct traffic from existing site to new site?
Good Morning, I have an existing site (http://goo.gl/QKkpi) running on a slow server. We decided to create a new site (http://goo.gl/XUH3f) with the intention of de-commissioning the first one. Both sites are on the same WMT account. What is the best way to permanently redirect any backlinks/traffic (all levels, from home page to product pages) from the old site to the new site (prior to shutting down the first one). Thanks a lot!
Web Design | | Prime850 -
How SEO friendly (or unfriendly) this site structure is
We have a client who wants a site structure like this http://thethomasoliverband.com/home - try to scroll down on the content and see how the url of the site changes. Would there be any problems on trying to SEO this type of structure?
Web Design | | paulct0 -
Site Ranks on Page 1 - Would launching new site hurt that
Hello, I currently have a website ranking in the top 7 for my main keyword. The website was built in 2004 and is definitely outdated, yet still ranks very high and brings in business. If i launched a new site on this domain, what would happen to my rankings? Would they drop? would they rise? If i don't launch the new site, will this site eventually drop due to being old and outdated? Any advice would be helpful...
Web Design | | Prime850 -
Custom 404 Page Indexing
Hi - We created a custom 404 page based on SEOMoz recommendations. But.... the page seems to be receiving traffic via organic search. Does it make more sense to set this page as "noindex" by its metatag?
Web Design | | sftravel0 -
Does changing nameservers and a new site design affect SEO dramatically
We are about to change nameservers and upload a new website design design, completely rebuilt website to that new hosting, will this effect our seo efforts previously and have an effect on our SEO rankings?
Web Design | | CompleteOffice0