Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google is a definition engine / Question answering engine
But what do you do to rank when you sell a product and people don't have questions and none of the companies that rank on your keywords answers questions... Let me give you a few example. Title tag for example : People want to know what is it and then have question about it What is the maximum number of characters ? How do you write a good one ? Etc... How to remove stains is another example : Please want to know how to remove all the different types of stains, (ketchup, grease etc..) But what about when you are a online business and want to sell usb keys , rent bicycles https://www.spinlister.com or even sell a software to do A/B testing on your website https://bit.ly/2a6cBuF Can someone explain me how those people mentioned rank without giving definition or answering questions... because according to me they don't do any of that on the pages that I mentioned. I look forward to your reply. Thank you,
Web Design | | seoanalytics0 -
Problems preventing Wordpress attachment pages from being indexed and from being seen as duplicate content.
Hi According to a Moz Crawl, it looks like the Wordpress attachment pages from all image uploads are being indexed and seen as duplicate content..or..is it the Yoast sitemap causing it? I see 2 options in SEO Yoast: Redirect attachment URLs to parent post URL. Media...Meta Robots: noindex, follow I set it to (1) initially which didn't resolve the problem. Then I set it to option (2) so that all images won't be indexed but search engines would still associate those images with their relevant posts and pages. However, I understand what both of these options (1) and (2) mean, but because I chose option 2, will that mean all of the images on the website won't stand a chance of being indexed in search engines and Google Images etc? As far as duplicate content goes, search engines can get confused and there are 2 ways for search engines
Web Design | | SEOguy1
to reach the correct page content destination. But when eg Google makes the wrong choice a portion of traffic drops off (is lost hence errors) which then leaves the searcher frustrated, and this affects the seo and ranking of the site which worsens with time. My goal here is - I would like all of the web images to be indexed by Google, and for all of the image attachment pages to not be indexed at all (Moz shows the image attachment pages as duplicates and the referring site causing this is the sitemap url which Yoast creates) ; that sitemap url has been submitted to the search engines already and I will resubmit once I can resolve the attachment pages issues.. Please can you advise. Thanks.0 -
Does google prefer expanded text to text that you have to mouse over to show?
Does a long scrolling page of text perform better than a page that has the content in sections that have to be moused over to be seen? Are there any articles or research on this?
Web Design | | SirSud0 -
How do I optimize a site designed to be one scrolling page of content?
Our website uses section ID's as its navigation so all the content is on one page. When you click About Us, the page scrolls down to About Us. Products, the page scrolls to Products section, and etc. I am getting crawl errors for meta descriptions but will this go away once the main domain has this info? We just added the meta keywords and description to the header and since the navigation sections use the same page, I assume it will correct the errors. Any other advice on optimizing for site designs like ours would be great. www.theicecubekit.com is the site. Thanks,
Web Design | | bangbang
Chris0 -
How does a Responsive Site kill SEO?
How does a Responsive Site poentially kill SEO? I've seen a few feeds on twitter how a website took a rankings dive after implementing a Responsive theme; yet, it's not clear to me what is actually going on within a Responsive site that would cause the SEO rank to tank? I can only speculate that it introduces a bunch of 404 errors, or that it changes all of the URLs into gibberish, so you loose all of the links coming into your website if not 301'ed? Can someone clarify, what are the actual mechanical issues on a Responsive website that becomes a concern to SEO? Thanks.
Web Design | | ExploreConsulting1 -
Site Re-Design - Running old XML site map for 301's
Hi all, We are going to launch a new site design for our current e-commerce site. I have taken this opportunity to change some categories due to keyword research and all old categories will be 301ed to best fitting new category. So I have 2 questions about moving stuff over; 1. I read that leaving the old xml site map running for the first week, would help, because this would give crawlers the chance to run through the site and follow the 301s, which would help pass the juice. How true does this sound? 2. I was thinking of re-writing all category and sub category titles, meta descriptions and on page content. The positive of this is loads of fresh content - but doing this all at the same time with the new site launch might see some major dropping in search ranking. I've identified our top traffic keyword terms/pages, would it be more wise to leave these pages, and change the others, or would the total new fresh burst have a better impact? Cheers
Web Design | | ToxicFox0 -
Duplicate content on mobile sites
Hi Guys We are launching a mobile webshop later this year and have decided to use a subdomain for this. (m.domainname.xx). The content will be more or less identical with the one on the standard desktop site (domainname.xx), but im struggeling to find out if this will create dipplicate content between the mobile and desktop site. Does anyone have a solid answer for this one?
Web Design | | AndersDK0 -
Considering site navigation options
I am working on a site redesign and re evaluating concepts I haven't thought about for a few years. I generally see site navigation that is either "top-down" or "left bar". Top down navigation normally uses the left nav. for search refinements. The benefit of top nav. is that it clears up the center of the page for non navigation content. The drawback is that you can't fit as many categories in a top nav. Left side nav. can hold a long list of categories, but subcategories are often in the center of the page. In the past, I have preferred to use left nav. with a multi level scroll over search refinement. I believe this allowed users to get to their destination page with fewer clicks. (I have always believed that every required additional click causes lost customers). I also believe that this has caused me to get more juice flowing to deeper pages on sites and better long-tail conversion. This means I have had pages with a LOT of links. With this method, I have tightly controlled my categories. What on other sites are often dynamic search refinements, are on my sites additional categories. I am considering making a site with a top down navigation system. I like the additional screen space in the center I get to work with. Is my assumption about pages created by search refinement wrong? Is it ok for SEO to have a left nav that has a bunch of search refinements that are dynamically created?
Web Design | | EugeneF0