Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What’s the best tool to visualize internal link structure and relationships between pages on a single site?
I‘d like to review the internal linking structure on my site. Is there a tool that can visualize the relationships between all of the pages within my site?
Web Design | | QBSEO0 -
Website Form Doesn't Work for Visitors from Google Search App
I've got a strange issue where if visitor is viewing a website from within the Google Search App, the app doesn't seem to be able to handle online forms. The form relies users on entering data, which is then POSTed to an external booking engine site. Normally this works fine, except when Google Search App is acting as the browser, the post payload is empty and the URL breaks. We're a bit stumped as to how to move forward. So far the only lead is that Android users can override having the app behave as a browser - but doesn't seem like iOS users can. Any additional ideas/tips are welcome here - thanks. 🙂
Web Design | | mirabile0 -
Multiple Sites for an Attorney Practicing Different Areas of Law
Hello, I'm trying to make sure I give a potential client the correct advice. This person is an attorney whose current site ranks well. The site deals solely with his traffic ticket defense practice. He's considering building a new site to highlight his personal injury practice but is unsure whether to build an altogether new site or redesign his current site in such a way that it includes his personal injury practice along side his traffic ticket defense practice. Obviously he doesn't want to lose his current rankings, but my concern is that he'll actually dilute his rankings somewhat with the multiple sites. Both practices have pretty different sets of keywords they would need to rank for, with pretty different difficulty levels. Any advice? Thanks.
Web Design | | lawfather2 -
Responsive Site has "Not Found" Errors for mobile/ and m/ in Google Search Console
We have recently launched a new responsive website for a client and have noticed 2 "Not Found" errors within Google Search Console for /mobile and /m Both these URLs are not linked from anywhere within the site. However Google is reporting them as being linked from the homepage. This is not the first site we have seen in which Google has reported this error, however the other site was not a mobile friendly site. My thoughts are to 301 them back to the Homepage. Anybody else have any thoughts on this? or have recently received the same errors?
Web Design | | JustinTaylor881 -
Google text-only vs rendered (index and ranking)
Hello, can someone please help answer a question about missing elements from Google's text-only cached version.
Web Design | | cpawsgo
When using JavaScript to display an element which is initially styled with display:none, does Google index (and most importantly properly rank) the elements contents? Using Google's "cache:" prefix followed by our pages url we can see the rendered cached page. The contents of the element in question are viewable and you can read the information inside. However, if you click the "Text-only version" link on the top-right of Google’s cached page, the element is missing and cannot be seen. The reason for this is because the element is initially styled with display:none and then JavaScript is used to display the text once some logic is applied. Doing a long-tail Google search for a few sentences from inside the element does find the page in the results, but I am not certain that is it being cached and ranked optimally... would updating the logic so that all the contents are not made visible by JavaScript improve our ranking or can we assume that since Google does return the page in its results that everything is proper? Thank you!0 -
How to bounce back after a new url & new site design?
About a month ago, my company changed domains (from the long-established www.imageworksstudio.com to the new www.imageworkscreative.com) and also did a complete overhaul of our site. We tried to do everything necessary to keep Google happy as we went through this change, but we've suffered a drastic loss of both rankings and traffic. I know that can happen as a result of a redesign AND as a result of a new domain, but I'm wondering how long you would expect it to take before we bounced back and also, what can we do in the meantime to improve?
Web Design | | ScottImageWorks0 -
Will google penalize a website for using a table layout?
I just got a new client today and his entire website layout and structure is using tables instead of divs. This client is on a tight budget and wants to avoid unnecessary hours for re-coding the website, but at the same time he wants me to improve his SEO organically. This is the first time I've been asked to do work on an existing website that uses pure tables for the entire layout and I'm wondering if this effects the SEO in any way. So my question is, will tables effect rankings and SEO in any way?
Web Design | | ScottMcPherson0 -
Google changed the title?
I've been reading that Google will sometimes change the home page title, and I have finally seen this in action. Search for and on the second page you'll find but the title says Academy Dog Training. Which is not what's in the title tag for this page. Did Google change the title tag on us and if so why? How can we get them to use our title?
Web Design | | CFSSEO0