Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website removed from Bing and Yahoo index
Hi, are website servicemanualrepairs.com was removed from Bing and Yahoo index, I emailed Bing via Webmaster tools they first said it was Backlinks I did look via at the inbound links tool to analyze the site's Backlinks I did find 20 links and used the Disavow tool, they said "I'm afraid but after careful and thorough investigation, your site still did not meet the Bing and Microsoft guidelines You may also refer to the things to avoid section of the Webmaster Guidelines for additional information. As an effect, the site is still blocked and it cannot be lifted" the website was in Bing and Yahoo index for 3 Years and only after the 20 Backlinks were added to the site it was de-index any help would be greatly appreciated Thanks
Web Design | | vista5211 -
Is switching from a very old HTML table site to HTML5 going to make a big difference
Hello, My site owner has been having calls that our old HTML site needs to switch to HTML5. Is it really worth the ubgrade from an archaic HTML site? Please explain. Bob
Web Design | | BobGW0 -
Google Translate with "no-follow" just for users' use, no ranking
Hello, I tried to search but here is a little unique situation. I would like to translate my website in 2-3 different languages not for ranking purpose, but only for some minority users within Italy to understand the content in their native language. Using "no-follow" with a Google Translate link would damage SEO? if not I would like to use it. Here are few points: Give users the ability to switch the content to their language with a link Tell Google not to follow the translated pages, because I don't want them to be used for ranking or searches I would start simply with Google Translate to see if people actually are interested, then later translate by human but still don't want google to follow I could also start with human translation instead of Google Translate if really needed, I know it is a no no. What I'm very interested is to make sure that those pages under "no-follow" won't affect my SEO in good or bad right now, because we would like to keep as it is. Thanks a lot
Web Design | | angelowei0 -
Sitemap Question - Very Old Ecommerce Site, Never Used A Map
I help manage a family website, that has about 10,000 products... It was top ranked since 1996, then got smacked by Penguin and recovered but its still receiving only a fraction of the natural traffic it used to get. Something we have never used... Is a sitemap. I'm curious if anyone knows reliable software that will generate a sitemap? My cart is custom built, website uses html pages across the board. Dynamic content and parameters are set up properly, onsite seo is in the excellent range. The only thing that I haven't been utilizing is a sitemap. Because the cart was hand built, it would a huge convenience to use a lightweight program thats compatible with any website, has parameter settings, exclusions and anything else useful to negate any duplicate content. I have a few highly dynamic pages as well... If anyone knows a product or a possible solution, it would be much appreciated. Working it up myself would be very time consuming. Thx
Web Design | | Southbay_Carnivorous_Plants0 -
On site SEO opinions
Hi all, I have been testing different configurations for my on-site SEO for a while now and I think I am pretty much there. However it is always nice to know what other SEO's think about my keyword density and usage. My site is http://www.tomlondonmagic.com I am curious as to what you think regarding landing page content, whether you need lots or text or little text? I have just removed links in the text as I feel I want to keep as much juice on my landing page as possible. Thanks all!
Web Design | | TomLondon0 -
Responsive Vs Mobile Sites
I know this is some cutting edge technology, but I think that this will be a very important topic in the coming months, as html5/css3 becomses more and more the standard, or at least standardized, I think the topic of this in relation to SEO will also arise much more. My question is simple, is it better to code a responsive site, or a completely mobile site for a small company with no special needs (mobile ordering, ecommerce, etc...) I obviously know the visuall differences, and, personally, I think respomsive websites look better. From an seo perspective, my big thing is for the resizing, for example, with WordPress, when you reach the tablet size you can set the sidebar to basically display:none, can that impact your website? I would really appreciate any feedback
Web Design | | ZacharyRussell0 -
Google Bot cannot see the content of my pages
When I go to Google Webmaster tools and I type in any URL from the site http://www.ccisolutions.com in the "Fetch as Google Bot" feature, and then I click the link that says "success," Google bot is seeing my pages like this: <code>HTTP/1.1 200 OK Date: Tue, 26 Apr 2011 19:11:50 GMT Server: Apache/2.2.6 (Unix) mod_ssl/2.2.6 OpenSSL/0.9.7a DAV/2 PHP/5.2.4 mod_jk/1.2.25 Set-Cookie: CCISolutions-UT-Status=66.249.72.55.1303845110495128; path=/; expires=Thu, 25-Apr-13 19:11:50 GMT; domain=.ccisolutions.com Last-Modified: Tue, 28 Oct 2008 14:36:45 GMT ETag: "314b26-5a-2d421940" Accept-Ranges: bytes Content-Length: 90 Keep-Alive: timeout=15, max=99 Connection: Keep-Alive Content-Type: text/html Any clue as to why this could be happening?</code>
Web Design | | danatanseo0 -
Why is this page removed from Google & Bing indices?
This page has been removed from indices at Bing and Google, and I can't figure out why. http://www.pingg.com/occasion/weddings This page used to be in those indices There are plenty of internal links to it The rest of the site is fine It's not blocked by meta robots, robots.txt or canonical URL There's nothing else to suggest that the page is being penalized
Web Design | | Ehren0
