Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What things i should do to add more attractiveness in my site?
I have 4 years of experience in feedback and survey of different companies for the people who want to learn some from online education about how to do something? and people want to get ease in daily life skills Kroger feedback provides them with a platform to learn everything about surveys and how to drive some to its highest peaks. It all works for People out there through Education SO I have now my own website ready to go but there is a problem in its on-page ranking factors I shall be thankful to you if some from you answer my question What things I should do to add more attractiveness to my site? take a look at my recent view of my site take a look at this Kroger feedback if somebody from you can answer my request??
Web Design | | SEOTOOLS021 -
Best way to Load Responsive Images for Responsive Site?
Hello All, Can anyone suggest be best technique to load responsive images? We are developing responsive site so looking for good ideas from your side so that it load very fast. Thanks!
Web Design | | micey1230 -
I want to create a 301 redirect on a WordPress site, nothing's working...
Hello all, I'm hoping someone out there can give me a hand with this. I'm trying to modify my .htaccess file so that the site will go from maxcarnagemusic.com to www.maxcarnagemusic.com and also, so viewers will be redirected to www.maxcarnagemusic.com/home when they try to access the site. I've tried a few different things, including adding the 301 redirect plugin for Wordpress, but nothing seems to work. Can someone out there show/tell me how to create an htaccess file that will execute as much. I apologize in advance, my Apache experience is very, very limited. Thank you all in advance!
Web Design | | maxcarnage0 -
Google HTML, CSS and javascript styleguides ?
Who's following the Google style guides especially in HTML, CSS and javascript? What are the benefits of following the style guides? I am thinking of sending the style guides to our web development team before we launch our new site but I think there might be some conflicts. I'm an SEO and not programmer or web developer and I'm sure there are some "rules" that these web dev guys should follow and break as well. Thanks in advance! 🙂
Web Design | | esiow20130 -
Internal linking for small site
I have a site with 13 pages, 6 are product pages, 5 are free tips pages (the other 2 are the home page and contact form). Currently I have the navbar at top of site with a "products" dropdown menu for the 6 product pages and a "Tips" dropdown menu for the 5 tip pages. All categories except the contact page are at the bottom as breadcrumbs, the homepage is "home" and the rest are relevant user friendly keyword anchor text. So I have 2 more pages to ad to "Tips" and am wondering whether to have a new 2nd level tips page that links to a 3rd level of 7 different tips pages, or keep it shallow as it is, with only 2 levels from the homepage to the other (now 13) pages, with a potential of 22 pages in the foreseable few years? (and some graphics work to make it user friendly like how Zappo's has categories to the side on each of its drop down navbar menu's and non-link text categories for its bottom of page breadcrumb links) Can those aforementioned pages linking to each other in the footer dilute link equity? (I think that's one of the primary reasons I'm curious). What do you think of this: http://www.dbswebsite.com/blog/2012/08/08/internal-linking-101-5-best-practices/ (I guess I should no follow my contact page), could it be better to have a 2nd level page for "Tips" to get more equity to that page rather than across all 7 tips pages? I have read around about this on here (hence how I found out about Zappo's) and elsewhere and wanted ask to make sure.
Web Design | | Zoolander0 -
How to link to a site without passing ANY linkjuice (other than simply nofollowing)
I have heard that there are other ways of linking to a site, to completely avoid passing any seo value I think it was even in a whiteboard friday video where I saw Rand say something about doing a 307 "temporary" redirect, or something like that? Basically, I want to let my customers compare our prices with ebay, but I don't want to have ebay outrank us (for obvious reasons) Any help?
Web Design | | TylerAbernethy0 -
Time On Site and SEO?
Does time on site impact rankings? If a person visits your site from the serps or directly visits it by typing in your name in the search field and then leaves within a minute, will that impact your serps? What is the best way to increase time on site?
Web Design | | bronxpad0 -
Google also indexed trailing slash version - PLEASE HELP
Hi Guys, We redesigned the website and somehow our canonical extension decided to add a trailing slash to all URLs. Previously our canonical URLs didn't have a trailing slash. During the redesign we haven't changed the URLs. They remained same but we have now two versions indexed. One with trailing slash one without. I've now fixed the issue and removed the the trailing slash from canonical URLs. Is this the correct way of fixing it? Will our rankings be effected in a negative way? Is there anything else I need to do. The website went live last Tuesday. Thanks
Web Design | | Jvalops0