Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does interlinking on mobile site helps in seo & improvement in rankings
Hi, Does interlinking on mobile site helps in seo & improvement in rankings. Our desktop site & mobile site has same urls. Regards
Web Design | | vivekrathore0 -
How to make my site title H1?
Hi In my Header.php I have the following php code for my title: <title><br /><?php<br />// Generate Page Title dynamically<br />if (is_home()) {<br /> bloginfo('name'); ?> - <?php bloginfo('description');<br />} elseif (is_category()) {<br /> single_cat_title(); ?> - <?php bloginfo('name');<br />} elseif (is_single()) {<br /> single_post_title();<br />} elseif (is_page()) {<br /> bloginfo('name'); ?>: <?php single_post_title();<br />} elseif (is_404()) {<br /> bloginfo('name'); ?> - <?php _e("Page not found", "fungames");<br />} elseif (is_search()) {<br /> bloginfo('name'); ?> - <?php _e("Search results for", "fungames"); echo esc_html($s, 1);<br />}<br />?><br /></title> This generates a good title different for every page/post I have on my site. But is now H1. I want the same code if, but with H1 tag somewhere in it. Cant figure it out how to do it! Can u help please?
Web Design | | Catinas970 -
What causes rankings to drop while moving a site.
Hi, we recently moved a PHP based site from one web developer to another (switched hosting providers as well). Amidst the move our rankings drastically dropped and our citation and trust flow were literally cut in half as per Majestic SEO. What could have caused this sudden drop?
Web Design | | Syed_Raza0 -
Does Google take email server IP blacklists into account?
This is just a hypothetical, but would Google use information from email server blacklists to determine the quality of a website? The reason is that we're planning to code in an e-mail queuing system for our next CMS, and we would put SPF and DKIM in place. We wouldn't be sending any bulk e-mails (we use Constant Contact for this), but we might be sending personalised follow up e-mails, unpaid order emails and that sort of thing. There's no reason to think we'll be blacklisted, but from experience I know that these email blacklist directories quite often give false positives when an e-mail server is incorrectly configured. So the risk is that we might get blacklisted by mistake when we start using this new feature. Would Google take this into account as part of the algorithm? And if so, would the damage be permanent? (I.e. does getting removed from the blacklist mean Google will stop thinking we're a low quality / spammy site)
Web Design | | OptiBacUK0 -
How can I reduce my warnings for excesive links on our site?
Our campaign overview shows well over 100 warnings that could be hurting our google ranking based on excessive links on pages. Each page listed, however, is simply due to listing the brands we carry, and linking to the products. Is there a way to do this without hurting our ranking? A better way than linking, perhaps? Thanks in advance!
Web Design | | guycochran0 -
Mobile Sitemap for Site with Media Queries
I'm doing SEO for a site. It uses Media Queries and the CSS to automatically resize the site for the screen size in use. I.e. the site detects the screen size of say an iPhone and the CSS knows which elements to hide for that screen size and still make it look good. This is great because it will automatically cut down the content to display nicely on small screens - obviating the need for a separate mobile site. What kind of sitemap should be generated since the urls are for desktop and mobile use? Yoast (sweet SEO) said it should have both regular and mobile style sitemap to get both the regular and mobile bots to visit, but didn't elaborate on how that sitemap should look. Do you have a recommendation for how exactly the sitemap should look? Should the sitemap have the urls all twice, i.e. once regular and once with the mobile indicator?
Web Design | | GregoryHaze1 -
Suggestions For My Ecommerce Site
I am starting to work on an ecommerce site that I am part owner of. My partner who is the other owner started the site a while back and because he has no internet marketing experience the site didn't come out very well. I am currently overhauling the site and here is a list of things first on my list so I can at least get started on some seo and even ppc. I would really appreciate it if you could take a look at our store and see if you think I am missing anything or you could suggest anything else that really should be done immediately or something that is wrong. The site is www.clubfitnesswarehouse.com Update magento to newest version Fix url structure to be SEO friendly for homepage, category pages, product pages etc. For an example of a site that has great seo friendly urls please refer to examples on http://www.bigfitness.com/ site. For example these pages. http://www.bigfitness.com/treadmillstore.html , http://www.bigfitness.com/bosplincy.html , Remove top navigation menu and instead create left sidebar navigation menu created to navigate site and products. For example of these types of sidebars please refer to sidebar on http://www.bigfitness.com/ or http://www.americanfitness.net/ We will NOT be using same structure as what is in top navigation currently. Categories and keywords of categories will be changed and some will stay the same structure as is currently in navigation For header we would like to feature our shipping policy, returns, privacy policy, no sales tax etc. For example I have seen that I like refer to http://www.spiderofficechairs.com/ Obviously we do not want to copy but something similar for our own site. All images on homepage must be changed to include clickable txt so google can read text not just pictures. We would also like product pages modified to make more user friendly and conversion increased. Add to cart button needs to be changed, text needs to be brighter instead of dull grey color. Add to cart button must also be moved ABOVE THE FOLD! Also on product pages we would like to add sections in addition to Product Description, of Specs, and About section for each product page. We also need the ability to change this information as needed when we need to.
Web Design | | PEnterprises0 -
How long does Google take to re-cache a site?
Specifically, I just redesigned my site. I'm reading Danny Dovers book, and learned about checking the cache version of the site to see what google is REALLY seeing . . . . . . which evidently is my old site. Obviously, my sites not going to make any real progress with SEO as long as the site is out of date. It says it last checked the site on 5/5 and I launched the site on 5/9. Obviously, it does not do these things immediately, but anyone have any ideas on how long it should take before google starts to show me some love?
Web Design | | damon12120