Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When rel canonical tag used, which page does Google considers for ranking and indexing? A/B test scenario!
Hi Moz community, We have redesigned our website and launched for A/B testing using canonical tags from old website to new website pages, so there will be no duplicate content issues and new website will be shown to the half of the website visitors successfully to calculate the metrics. However I wonder how actually Google considers it? Which pages Google will crawl and index to consider for ranking? Please share your views on this for better optimisation. Thanks
Web Design | | vtmoz0 -
Is switching from a very old HTML table site to HTML5 going to make a big difference
Hello, My site owner has been having calls that our old HTML site needs to switch to HTML5. Is it really worth the ubgrade from an archaic HTML site? Please explain. Bob
Web Design | | BobGW0 -
Do you know any tool(s) to check if Google can crawl a URL?
Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool. In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage. Thanks in advance!
Web Design | | esiow20130 -
New Mobile Site Traffic Drop
With all the talk about how much mobile is important and how it is going to return its own search results, we finally decided to make a mobile site for one of our smaller websites to test the water. We put it up about two weeks ago and did Vary HTTP header method to serve the site. Before the change, on the average week we would get 270-300 mobile visitors from organic search results and we converted 0.78% to sales. Since the change, we are now getting about 70 mobile organic visitors per week but converting 2.47% So what can I say but WOW. We are converting way way better but our organic mobile search traffic has dropped off a ton. Luckily our desktop and tablet traffic(we serve the desktop version of the site to tablets) has stayed the same and has not dipped. Do any of you guys have experience or gone through launching a mobile site before? Did you see the immediate drop in organic mobile traffic and did you recover your traffic back to previous levels? If so, do you know how long it takes to recover? I am thinking it is a big change and will take time for Google to adjust but I am not sure since the mobile version has so much less text now on the home page and on category or product list pages or whatever you guys want to call them.
Web Design | | KurtL0 -
How to verify http://bizdetox.com for google webmaster tools
Hey guys i tried to to make a Preferred Domain choice in webmaster tools, but it is not allowing me to save my choice bec its asking me to verify that i own http://bizdetox.com How do i go about doing that and what are the steps I have already verified www.bizdetox.com
Web Design | | BizDetox0 -
For a varied product type or keywords group is it best to have several sites?
Hello everyone... Question: I have 7-8 generic keywords that I would like to rank for, is it possible for one site to rank highly for all these different keywords, or would this be best achieved by making 2 or 3 websites in total targeting different keywords (product sectors)? More info: We are in a niche industry & would like to know if it would be beneficial to have several websites made for specific product types rather than one main site? Although these sub classifications of products are nice, they are competitive as they have a high search volume Would it be better to build specific websites that only do that one type of product and have related keyword in domain, content & blogs on the site to that effect to increase relevance and positions as a result? Thanks
Web Design | | Ray_UK0 -
What can this charity site do to improve SEO?
Hello wise ones, We have been working with the charity Volunteers of America to create a new site for their car donation program at carshelpingpeople.org They are a national charity with extensive local programs run by regional affiliates, so the site is divided into a small national section linked to Regional Affiliate sections. You get to the regional sections either by entering your zip code or clicking on your state in the bottom nav of the national pages. Right now we have developed regional sections for Michigan, Nevada, Maryland, Washington D.C., New Jersey, Delaware and the Philadelphia area. The site is about 2 1/2 months old, and while our conversion rate is pretty good (7%) our organic search ranking isn't improving as quickly as we'd like to see. Car donation is a very competitive space, and we would appreciate any advice on how to improve the SEO of the site. Thanks so much.
Web Design | | Phibnax0 -
Are tables OK in e-commerce sites?
I know that for best results with search engines, tables should not be used. However as I look to redesign our older e-commerce site, I find that not only our website but many other e-commerce sites use tables. Is it acceptable to use tables in e-commerce or am I just viewing outdated e-commerce sites. Could you recommend a good shopping cart solution? We currently use volusion. Thanks for your time and see you in Boston next week!
Web Design | | Lael0