Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is this is Wow HIT ME IN THE Face Google bug or am I missing something?
We have a page on our site https://www.spurshelving.co.uk/shop/bigimage.aspx?m=353&i=3436 which enders happily on all browsers as far as I am aware and is reasonably well optimised. So when google sent me a link to a new test tool I just had to check it out. https://testmysite.withgoogle.com/intl/en-gb/?utm_source=awareness&utm_medium=email&utm_campaign=tmsv1awareness&utm_content=header Well the result was shocking...... The page that renders in the results is a default missing product page and not the page that the link renders on a web page. I played a little and simply used the I=3436 attribute and the page appeared no problem I then reversed the attributes so that they were i=3436&m=353 and the page again resolved totally as expected. This indicates to me that Google have an issue with aspx attributes. Now I know what to do but is this same issue an issue in spidering and indexing pages. If is is wow that is a big smack in the face. Does it also harm search results in other engines. Keen for comments here
Web Design | | Eff-Commerce0 -
DNS Prefetching for wordpress site
We have given below as DNS prefetch in our website. CMS is wordpress. Are these okay? I wonder why fonts.googleapis.com is not working. One of our competitors is using youtube. Can we improve it any how?
Web Design | | vtmoz0 -
What is the Estimated Time for SERP Rankings to Replenish after a Site Redesign?
Hello Fellow Moz'ers, My company's website, www.1099pro.com, is currently OLD and not mobile-friendly! However, we rank #1 for out most important keywords and don't want to lose that ranking. I've recently redesigned our site, currently in testing, to use the same standard desktop pages but to also have responsive, mobile friendly, pages for different view ports. My question is if anyone knows an estimated time frame that search engines (mainly Google) takes to re-crawl the site and restore SERP rankings to their previous levels? The reason is because we are HIGHLY seasonal and if we are not back at our top rankings by early December, at latest (November would be better), then we stand the chance to lose a considerable amount of traffic/revenue. -The Unenlightened One
Web Design | | Stew2220 -
Changing website to Mobile site
Hi All, We are building a new website and would take it live in next few weeks. I want help to understand what all should we consider.. The site structure would be same with additional features but the URL path and site name is same...Should i take any precausions before shutting down the old site and getting the new one live? Thanks
Web Design | | jomin740 -
Does interlinking on mobile site helps in seo & improvement in rankings
Hi, Does interlinking on mobile site helps in seo & improvement in rankings. Our desktop site & mobile site has same urls. Regards
Web Design | | vivekrathore0 -
Accordion Fold Ups Bad For Google
http://fandicoach.com/products Right now I have these accordion things on the website. Are they bad for google in terms of being an SEO best practice? I want to avoid doing anything black hat. Thanks!
Web Design | | OOMDODigital0 -
Sudden disappearance from visibility on Google
Our Google position had a sudden and precipitous drop from #3 to #15 in October for our two main keywords: "web design Santa Barbara" and "Santa Barbara web design". I have steadily gotten it back to #7 by putting a lot of effort to our on-page SEO. As of my December 24 SEOMoz report, it was #7, and as of my check yesterday, Dec. 28 it was #7. We were #3 for "web designers Santa Barbara" as of yesterday. Today, i cannot even find our listing on the first 5 pages. Recently, our partner company was engaging in some link building for us that may have just occurred and I'm worried that this may be the cause. I have not yet had a report and actually don't know for sure yet, it has even been accomplished. We had also done some link building in October by a different company we subcontract to and there was a Google type hack of our site. Those were the two major changes that occurred at that time. I thought it would be safe to have some more link building done now, since we'd already survived that first "sudden" increase of links in October. Does anyone have any idea what may have happened and how I can fix this? I am trying not to panic. Thank you in advance for any ideas to understand and/or remedy the situation!
Web Design | | gfiedel0 -
Google fails to pick out the correct URL of the story
Hi , I have a page with many news storeys on it. Google craws the page but it picks up a more general url even though I've embedded the direct URL within anchor tags around the headline . The snippet below got linked by Google to http://www.irishnews.com/ Any idea how i can get Google to pick-up http://www.irishnews.com/news.aspx?storyId=1180708 would be very welcome Peter Quinn: Family made scapegoats of financial crisis News Peter Quinn: Family made scapegoats of financial crisis THE Quinn family have been made scapegoats of the financial crisis surrounding the former Anglo Irish Bank, tycoon Sean Quinn's brother Peter claimed yesterday.Peter Quinn, a former president of the GAA, said hi read more»
Web Design | | Liammcmullen0