Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Have Your Thoughts Changed Regarding Canonical Tag Best Practice for Pagination? - Google Ignoring rel= Next/Prev Tagging
Hi there, We have a good-sized eCommerce client that is gearing up for a relaunch. At this point, the staging site follows the previous best practice for pagination (self-referencing canonical tags on each page; rel=next & prev tags referencing the last and next page within the category). Knowing that Google does not support rel=next/prev tags, does that change your thoughts for how to set up canonical tags within a paginated product category? We have some categories that have 500-600 products so creating and canonicalizing to a 'view all' page is not ideal for us. That leaves us with the following options (feel it is worth noting that we are leaving rel=next / prev tags in place): Leave canonical tags as-is, page 2 of the product category will have a canonical tag referencing ?page=2 URL Reference Page 1 of product category on all pages within the category series, page 2 of product category would have canonical tag referencing page 1 (/category/) - this is admittedly what I am leaning toward. Any and all thoughts are appreciated! If this were in relation to an existing website that is not experiencing indexing issues, I wouldn't worry about these. Given we are launching a new site, now is the time to make such a change. Thank you! Joe
Web Design | | Joe_Stoffel1 -
How long should an old site redirecting to a new site remain activated on a server?
Once I switch a site to a new domain (with links to corresponding/relative pages), will I have to keep the old site live forever for those links to work, or how long should I wait before I inactivate the old site on our server?
Web Design | | jwanner0 -
What do you use for test rendering your dev site?
I'm redesigning our company ecommerce site and need to test render an infinite scroller to ensure that it is as SEO friendly as possible. My problem is that I cannot view it in Webmaster Tools since I am blocking the site from crawlers using robots.txt. I know I could simply unblock Google temporarily but I really would rather not make my dev site available to search engine crawlers.
Web Design | | bearpaw0 -
Is it bad to have /index.php at the end of a uri?
Is it bad for SEO if traffic is directed to "http://www.example.com/someuri/index.php" instead of "http://www.example.com/someuri/" and would it be works setting up a redirect rule at htaccess level?
Web Design | | NoisyLittleMonkey1 -
How to correct error in customized posttype WP site
Hi folks Can anybody help me. I foolishly, dogedly followed a Lynda.com tutorial for developing an 'online portfolio in WP'. Little did I know that my initial assumption - to use the 'twenty twelve' rather than the 'twenty eleven' theme would land me in such deep water. I was attempting to learn php on my own. All went well, until, --- the index page for the customized post type. Now I have two beautiful customized posttypes, 'companies' 'coverage' and no idea how to create an index page for either. I can't do the next step! I have tried every permutation - changing the permalink settings, changing them back, desperately searching for any handle to the nebulous links within the menu section. The only thing I can do (and have done for now) is to link the menu item 'company' and the menu item 'coverage' to a single post. Then the poor visitor has to scroll through the posts individually. I tried contacting the tutor and Lynda.com, to no avail! I have searched forums and found this is a common problem, but because I am so confused and novice to php they might as well be speaking Chinese. To compound my problems, looking through 'Wordpress SEO' for Yoast, I am painfully aware I can't go to the first basic step and fix the peramilinks to 'Postname' as that just makes my flakey menu collapse like a pack of cards. Help!
Web Design | | catherine-2793880 -
Best Way To Have HD Videos On Site That Will Work On Mobile Devices
Hi, I hope someone can help me with this. I am working on a site for a client who works at a video production company. They want to have a fair few HD videos on there site but also for the site and videos to be viewable on mobile devices. I have got a responsive wordpress theme and the site is beginning to take shape. I am wondering however how I can best get the videos to display on mobile devices while maintaining a good load speed. Until now I have been using amazon S3 which stores and feeds the videos and I use Easyvideoplayer to embed the videos. The problem is they do not appear to show up from mobile devices when using wordpress. can anyone suggest the best way for me to still feed the videos from S3 but get them to display on mobile devices. oh, they are private videos so they cannot be placed on youtube.
Web Design | | jensonseo0 -
How can I reduce my warnings for excesive links on our site?
Our campaign overview shows well over 100 warnings that could be hurting our google ranking based on excessive links on pages. Each page listed, however, is simply due to listing the brands we carry, and linking to the products. Is there a way to do this without hurting our ranking? A better way than linking, perhaps? Thanks in advance!
Web Design | | guycochran0 -
Best way to set up a site with multiple brick and mortor locations across Canada
I have a client who is expanding his business locations from 2 cities to 3, and working towards having 10+ locations across Canada. Right now we're building location based landing pages for each city, as well as keyword targeted landing pages for each city. For example, landing pages for "Vancouver whatever clinic" and "Calgary whatever clinic" as well as for "Vancouver specific service", and "Calgary specific service". This means a lot of landing pages will need to be created to target each of 10 or so desirable "service" keywords for each city's location. I've no issue with this, however I was wondering how other companies go about this? What's the best way to be relevant for certain "service" based keyword searches in each city? Many of the "service" keywords are 'localized' meaning they will show Google Places results for local brick and mortar businesses for each location. I'm quite good at optimizing locally for this type of thing. However, many of the "service" keywords are not yet 'localized' by Google, I'd want to have my client webpages show well in the SERP's. for these 'non-localized' "service keywords" as well. the new site will be built in WordPress
Web Design | | AndyKuiper0