Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it Bad to Break Up A Site into Multiple Sites?
I have a big cluttered website with endless pages. It's a non-profit that has content for patients, researchers, therapists, etc.. Would it be a bad idea to turn this cluttered site into 3 or more completely different sites, each focused on their specific demographic? Or should I just figure out how to organize the one site better? Thanks for your help!!!
Web Design | | bosleypalmer0 -
Is WP okay for E commerce sites?
Do any of you out there use wordpress for an ecommerce site? I'm getting some mixed reviews on it (but it's the internet, so that's bound to happen). Is there any sort of site traffic or page limit that would make using wordpress a bad idea? Thanks, Ruben
Web Design | | KempRugeLawGroup1 -
Does Using Magento With Multi Sites Affect SEO
We have a client who has 3 separate websites targeting the US, Australia, and the UK. Each of them has relevant ccTLD's such as: .com .com.au and .co.uk. Our client wants to use the Magento multi-site function so it combines all the stores (which are the exact same products) and merge it into one through Magento. These sites are all hosted in the US and had nothing to do with me haha! I understand Rand has mentioned on a video it would be best having the websites with ccTLD's hosted in that country (if budget permits), however in this case the budget doesn't permit us to go down that road. Has anyone any advice on this matter, has anyone did this before and had a lot of success with the SEO? At present there doesn't seem to be a lot of information about it and opinions are varied and sometimes divided. Any help would be very much appreciated guys Thanks, Matt
Web Design | | HigherthanSEO0 -
Bar Codes for Event Sites
A client is requesting bar codes for simple tracking of event attendees. Their need is to be able to verify at the door someone has registered and then have that data link to the registrant to show who did/did not attend, etc. We looked at QR code solutions (seems bar code makes more sense at this point) and now are at a point where we need input from any who have experience with either system and could potentially make a recommendation. A person would sign up for this once a year event (no charge, informational) on the site and be able to print off their registration ticket to bring to the event. If it could also be downloaded to iPhone/Android that would be a plus as well. When they come through the door the code is scanned and registrant is in. Simple. They currently keep a database of past attendees and only use it for general area the attendees are from and how many show over time, etc. Since we do not want to reinvent the wheel, we were hoping a mozzer might have insight. thanks, Robert
Web Design | | RobertFisher0 -
Google also indexed trailing slash version - PLEASE HELP
Hi Guys, We redesigned the website and somehow our canonical extension decided to add a trailing slash to all URLs. Previously our canonical URLs didn't have a trailing slash. During the redesign we haven't changed the URLs. They remained same but we have now two versions indexed. One with trailing slash one without. I've now fixed the issue and removed the the trailing slash from canonical URLs. Is this the correct way of fixing it? Will our rankings be effected in a negative way? Is there anything else I need to do. The website went live last Tuesday. Thanks
Web Design | | Jvalops0 -
Whats happening with Google UK?
Within the last week we have had a handful of our rankings drop dramatically down the SERPS. About 15% but this an estimate and has not been fully investigated yet. Whilst looking into possible scenarios that could be causing this i wanted to check what the SERPS looked like for the terms that we are still holding position on. Typing "extending dining tables" into Google UK today i was amazed at what i found... Ranking in position 1 and 2 is a massive UK furniture store.
Web Design | | Silkstream
But isnt that the same landing page being returned for both positions?? It appears to be a navigation problem within the site category tags causing duplicate content. However they have been rewarded with the top two positons subsequently pushing our website onto page two. I find it so frustrating that we listen to Googles best practices when it comes to pagination issues yet this is how our hard work is rewarded! Anyone else have any thoughts about this? SERPS.jpg0 -
Turning my Design Business site into a site to promote SEO
I need advice on retooling my website for my SEO biz. I have shifted my business model from graphic designer who does websites, to "internet marketing consultant who does graphics too". My main website and domain name is over 10 years old, so I've made the decision to keep it, even though it has no keywords in the name. The name works well for the new business, otherwise. The site has a PR3 and I rank well for small business advertising terms, which gets me graphic design business. I intend to keep doing graphic design, but that is a smaller part of my income. I had considered making 3 satellite sites with keyword domain names to cover my offerings of graphic design SEO, website development, and internet marketing. But am leaning against it for several reasons (that all of us SEO's know) but mainly the fact that I cannot keep up with both working for my clients and blogging on multiple sites and link building for multiple sites. So my question is (you knew there was one coming, right?), what is the best approach to building categories of web development, internet marketing, and SEO into my existing graphic design/advertising oriented website? This is slightly embarrassing to ask as an SEO, but given the multiple approaches possible, and knowing the importance of doing it right the first time, it's best to get an consensus perspective on the BEST approach. My main concerns are the navigation system and the links from the homepage into the site. I have too many pages I've identified as essential to link off of the home page and navigation menus? (Website development, social media marketing, link building, keyword research, pay per click, online advertising, graphic design, brochures, catalogs, Logos, Branding, SEO, keyword research etc.) I've always tried for the ratio of one link off of any page for every 100 words of content. Do I create a home page that is of monster proportions? Do I just have the 4 basic areas linking off the home page then create a "landing zone" of 4 folders and create down from that? I am concerned about URL length as I go deeper with that approach. Or, does it make more sense to have a dozen second-level pages, and not link them all off the home page, and build from beneath (and relying on external juice). Next issue is the nav system. It will be huge. Am I best off just keeping it to 4-6, and creating subnavigation on everypage within the site according to section (PITA)? I've read dozens of blog opinions on how much nav systems do or do not hurt link juice. I've always thought footer links were right next to worthless to pass any juice, but given this situation, does it make sense to make a footer link for each major page (about 20)? Thanks for your opinions.
Web Design | | JCDenver0 -
What's the best was to structure Product page information on my site?
Hi - I run a hobby related niche new / article / resource site (http://tinyurl.com/4eavaj4). One of the most critical components of the site is our product database. We don't actually sell anything directly - instead we monetize them by displaying relevant affiliate product feeds and price comparisons. However since the Panda update was implemented in February my traffic (particularly my long tail, product related traffic) has dropped off considerably. I had about a 20% drop in overall traffic, but have made up some of the ground in the past week. However I want to know once and for all how I should structure my product related information as I have a ton of great content that is ready to be published in this section but want to be sure I structure it the best possible way from a SEO standpoint. Here are a few different options I've come up with for displaying information about products on my site. For the purpose of these examples I am going to refer to all of the information that makes up my product pages collectively as "product profiles". Please let me know which is the best SEO wise (or if you have a better way of doing it let me know): - Option 1 - Current Method - Divide Content Sections into different pages / urls Example: http://tinyurl.com/4tpdlbl This is how the majority of my product profiles are currently structured. I did this to improve load times and to keep the total number of links per page down. In addition to the core product profile subpages: "Product Details","Compare Prices", **"**Product Review", "Hot Auctions", and "Checklists", I have the Checklists area further segmented by subset, each of which is on its own page that is only accessible through the main Checklists tab of the profile. - Option 2 - Everything on one url / page the old fashioned way, with everything available by scrolling vertically. This would make the page go on forever though. - Option 3 - Everything on one url / page, but visually segmented using css / javascript tabs. Example: http://tinyurl.com/4kqhauh I looked at the source code and all the page text is there, so it looks like it would be spider-able but you tell me. Or would another method of tabbing be better? My site is wordpress based so the functionality comes from a plugin. - Option 4 - Use post tabs that are technically all on the same page, but make each individual tab be accessible through its own suburl, all of which share the same core canonical url. Example: http://tinyurl.com/4bs9pjs Clicking on any of the individual tabs will result in something like ?postTabs=2 being appended to the core url. Example: http://tinyurl.com/4gvgufc Any input would be greatly appreciated asap! Thanks Mike
Web Design | | MikeATL0