Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Have Your Thoughts Changed Regarding Canonical Tag Best Practice for Pagination? - Google Ignoring rel= Next/Prev Tagging
Hi there, We have a good-sized eCommerce client that is gearing up for a relaunch. At this point, the staging site follows the previous best practice for pagination (self-referencing canonical tags on each page; rel=next & prev tags referencing the last and next page within the category). Knowing that Google does not support rel=next/prev tags, does that change your thoughts for how to set up canonical tags within a paginated product category? We have some categories that have 500-600 products so creating and canonicalizing to a 'view all' page is not ideal for us. That leaves us with the following options (feel it is worth noting that we are leaving rel=next / prev tags in place): Leave canonical tags as-is, page 2 of the product category will have a canonical tag referencing ?page=2 URL Reference Page 1 of product category on all pages within the category series, page 2 of product category would have canonical tag referencing page 1 (/category/) - this is admittedly what I am leaning toward. Any and all thoughts are appreciated! If this were in relation to an existing website that is not experiencing indexing issues, I wouldn't worry about these. Given we are launching a new site, now is the time to make such a change. Thank you! Joe
Web Design | | Joe_Stoffel1 -
Do Google Fonts Slow Down Your Site?
Hi Guys,
Web Design | | jeeyer
I just did a webpage speed test on http://www.webpagetest.org to see how our site is performing.
I noticed that an exteral URL called fonts.gstatic.com has a "huge" impact on our sites loading time. See a screen here: http://monosnap.com/image/z6drzC2ELoJ48d1rM0Tmtuszl3pFpH#
An overview can be seen here: http://monosnap.com/image/9hofUpr5Ld8D7mi7zyaJmGFIGhpBsY# All our scores are green and A (finally!) but I was a bit concerned when I saw the outcome of the pagespeedtest regarding the fonts.
When I load a page on my pc I indeed notice that the text content is usally quite slow in showing up, pops up afer a few seconds. Is this a know problem and Is this something I need to fix? If so what is the best approach? Looking forward on your thoughts!
Joost1 -
Footer links on my site... bad for passing page rank?
i've been told that it is possible that google discounts the weight or page rank passed in footer links of websites and my website has the navigation to many of my pages in the footer of each page. My whole website is about 20 pages so each page has links to the 5 most popular pages at the top and the rest of the links are in the footer of each page. Am i losing page rank by having these links in the footer? Should i make my navigation different? I have lots of articles on my site so i thought it might be not only helpful to my readers but give my pages an seo boost if i placed in context links in the body of my articles to other pages of my site. Does this sound like a good idea? Thanks mozzers! Thanks mozzers!
Web Design | | Ron100 -
What To Do When Improved Site Speed & Layout Result In Higher Bounce Rates & Lower Time On Site
We launched a new Bootstrap 3.0 site template 2 weeks ago. The site loads 5x faster and has a much improved layout (utilizing most common above the fold recommendations ). It's only been two weeks, but our bounce rate has increased 5-10% and our avg time on site decreased by 10-18%. Here is the page for one of our most common products so you can see the general experience: <a>http://www.jwsuretybonds.com/surety-bonds/commercial-bonds/auto_dealer_bond.htm</a> (here is the old version: <a>http://199.119.123.134/surety-bonds/commercial-bonds/auto_dealer_bond.htm</a>) We spent two months implementing the new design and working on a speedy load time. We had anticipated a drastic improvement, not mild downturn in user behavior. I'm hopeful that the Analytics metrics aren't showing the true picture on the keywords we care about (can't see anymore due to "Not Provided" listed as most keywords now. Argh!) and perhaps some of the more important/accurate user behavior metrics that we can't see are improving. We know our industry and our clients needs VERY well. We THOUGHT our new content/layout was perfect so it will be tough for us to try to make improvements at this point. We believe our best plan of action now is to add more content on each page and A/B test it along with other subtle changes. The problem is that our new content is very concise and hits on all of the primary visitor intentions, so additions of content could be redundant and making concise answers more "fluffy", which is what we tried to get away from. What do you think? Is there reason for panic? What would your plan of attack be if your "sure shot" new design didn't provide the improvements you "knew" it would? 🙂
Web Design | | TheDude0 -
Is this a good WP template for this site?
I need some advice, please. My cousin wants to take his site; http://www.gaport.com/ and move it to this wordpress theme http://themeforest.net/item/elvyre-retina-ready-html5-template/6639095. I have two reservations about this, but before I say anything I'd like the community's feedback, please. 1. It's a brand new theme. It's not even out for two weeks. That strikes me as a giant red flag. ** Just to clarify, i's not a new theme; it's new for WP. 2. It only costs $15. I can't tell what's wrong with this theme, but "you get what you paid for" is a cliche for a reason, and that price seems way too low for something quality. Are these legitimate concerns? Do you all have any recommendations for a theme that might suit him better? I'd appreciate any and all feedback. Thanks, Ruben
Web Design | | KempRugeLawGroup0 -
Google changed the title?
I've been reading that Google will sometimes change the home page title, and I have finally seen this in action. Search for and on the second page you'll find but the title says Academy Dog Training. Which is not what's in the title tag for this page. Did Google change the title tag on us and if so why? How can we get them to use our title?
Web Design | | CFSSEO0 -
Alternatives to Wordpress for updating content of a static html site
I have a static html site which I cannot update myself. What solutions/ programs would you recommend for gaining the ability to update it myself? I'm reluctant to switch to WordPress because the sites that use any CMS that are hosted by my web hosting company get routinely hacked. Thank you!
Web Design | | translate0 -
Need help setting up google analytics goals / tracking
I don't use Google analytics to see much more than how many visits I'm getting and what sort of keywords people are using to find our site. I'd like to step up my GA skills a bit. I'm wondring if you guys could give me some advice. I've never really set up any GA goals, or used it to track specific things, but I'd like to. Here are a few things off the top of my head that I would like to track. I'm wondring if these are posable in GA, and if someone could give me some feedback on how to track it / set up goals. Thanks.1 1. I'd like to know how many people click play on a video when they are on a page that has a video. 2. I'd like to know how many people are clicking "like" " google plus, etc.) 3. I'd like to know the path people are taking on our site. For instance, if they click a link from Facebook, and go to a landing page, what page are they visiting next..... 4. How long people are staying on the page I would really like to break this down further by people that visit a link I posted on Facebook, or twitter, or from the link on my twitter profile page, etc... Also if there are any other valuable goals / reports that would be useful for a blogger to track I'd appreciate your feedback. Thanks.
Web Design | | NoahsDad0