Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will There Be Much Impact When Moving Site To New Root Folder?
Hi, ok so I have a pretty big site that is located on my sever /root/current-folder/. I want to rebuild the site completely as it's using software that is out of date and not our main focus anymore (OpenCart). We want to move to a Wordpress platform, but want to have as little impact on the SEO as possible. Our current strategy is: List all URLs/Titles/Meta indexed with Google on current site Create new folder on the server /root/new-folder/ My question is... if I move to a new folder on the server (same TLD) and then re-route the TLD to go to this new folder, will there be more of an impact on SEO that if I start a fresh in the current folder? Thanks
Web Design | | Easigrass0 -
Site structure and Visual Sitemaps
Aside from mind mapping software are there any tools ( recommended) to build a visual sitemap of the internal linking structure of a URL? I've been trying to 'show' clients the structure of a website as it pertains to internal and external links. Here is one I've tried it's "Close" - http://site-visualizer.com/ . I've been using the excel export function, import into mind meister and building it. It's a teeny bit time consuming for large websites. Site structure I feel is a valuable portion of SEO and a down and dirty visual explanation would be great. Don't get me wrong, it offers other benefits as well- it's just I'd like to free up the time it takes. Thank you in advance. Screen shots are available on the website of the organization.
Web Design | | TammyWood0 -
SSL, SEO, and Site Migration question
When migrating a site to a new url and one where the old url had no https and the new url will be full https does it matter if the 301 redirect points at http://thisisthenewsite.com ? Meaning, should the new site have the ssl / https up prior to redirecting the old site? Does it matter if you redirect the old site to http://thisisthenewsite.com or https://thisisthenewsite.com? Since the site will force to https anyway?
Web Design | | Atlanta-SMO0 -
Does stock art photo attribution negatively impact SEO by leaking Google Page Rank?
Greetings: Companies such as Shutterstock often require that buyers place credit attribution on their web pages when photos you buy from them appear on these pages.. Shutterstock requests that credit attribution links such as these be added: Songquan Deng / Shutterstock.com Do these links negatively impact SEO? Or do search engines view them as a positive? Thanks,
Web Design | | Kingalan1
Alan0 -
3 Products & 50 Options each, How does Google handle product variant or options?
We are selling furnace filter and we might move our existing store host by BigCommerce to Americommerce or Corecommerce. Before moving the store, I have a questions about our online store structure. We are selling 3 different furnace filters, GOLD, SILVER and BRONZE Series. Each furnace filter come in about 50 different sizes, for a total of about 150 different products. The way our store is setup now, it is 150 different product, 150 different URL, 150 different page name... The way it is setup now, might look like duplicate content. All the product page are the same, all the pictures are the same, the only thing that change, is the furnace filter size in the product description. Look at those pages for example: http://www.furnacefilterscanada.com/20x20x4-Furnace-Filters/ http://www.furnacefilterscanada.com/categories/2-Inches-Thick-Filters/10x20x2-inches/ http://www.furnacefilterscanada.com/categories/2-Inches-Thick-Filters/16x25x2-inches/ Would it be better to only have 3 products and 50 variables or size options? What would be the best structure in a SEO point of view? One thing we have to keep in mind, when searching for a furnace filter, shooper will use keywords like: 16x25x4 furnace filter filter 20x20x1 air furnace filter 10x20x1 furnace filter 24x24x4 canada furnace filter Most of the Google search will included the filter size_._ How does Google handle product variant or options_?_ If I have 3 products, I will have only 3 URL and 3 different page name. I know for the shoppers, 3 products with sizes options might provide a better experience, but what about Google ranking the products? What is opinion the best online store structure in our case? Thank you for your help, preciouse time and support. BigBlaze www.furnacefilterscanada.com/
Web Design | | BigBlaze2050 -
Google Penalizing Websites that Have Contact Forms at Top of Website Page?
Has anyone else heard of Google penalizing websites for having their contact forms located at the top of the website? For example http://www.austintenantadvisors.com/ Look forward to hearing other thoughts on this.
Web Design | | webestate1 -
Empirical Data on the effect of embedded Google Maps on Search Ranking
Does anyone have any empiric data on the effect of an embedded map on SERP's? Please understand that I already have anecdotal info and a personal opinion. I am looking for data. Thanks
Web Design | | RobertFisher0 -
Old links in Google, new website affecting SEO?
Hi Guys, I have launched my website in october and it has already been indexed by google. Now I'm going to launch my redesign which comes with a new structure, content, links, etc. So the question is, do I have to resubmit my website to google to get rid of old links? Onsite Explorer shows links to my forum which has been spammed with p* stuff which has been already indexed as well. The forum is off now. I want to use SEOmoz to track my new website but I guess this could be a hard thing as old links etc will be shown as well. Is there any tool to let Google know about my changes? Does it affect my SEO in any way? Thank you for your help. Nick
Web Design | | NickITW0