Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Footer links on my site... bad for passing page rank?
i've been told that it is possible that google discounts the weight or page rank passed in footer links of websites and my website has the navigation to many of my pages in the footer of each page. My whole website is about 20 pages so each page has links to the 5 most popular pages at the top and the rest of the links are in the footer of each page. Am i losing page rank by having these links in the footer? Should i make my navigation different? I have lots of articles on my site so i thought it might be not only helpful to my readers but give my pages an seo boost if i placed in context links in the body of my articles to other pages of my site. Does this sound like a good idea? Thanks mozzers! Thanks mozzers!
Web Design | | Ron100 -
Duplicate Content? Designing new site, but all content got indexed on developer's sandbox
An ecommerce I'm helping is getting a complete redesign. Their developer had a sandbox version of their new site for design & testing. Several thousand products were loaded into the sandbox site. Then Google/Bing crawled and indexed the site (because developer didn't have a robots.txt), picking up and caching about 7,200 pages. There were even 2-3 orders placed on the sandbox site, so people were finding it. So what happens now?
Web Design | | trafficmotion
When the sandbox site is transferred to the final version on the proper domain, is there a duplicate content issue?
How can the developer fix this?0 -
Site as one page - SEO implications
We may be inheriting a site and will be asked to do SEO for it. We will have control over the development of the site, so this structure is what it is. My question is - how significant of an impact do you think this is going to have and can you think of any workarounds that may help? Basically, the user experience of the site will feel similar to multiple pages. However, this site will, in essence be one page and pull various content through javascript from different locations. I have not seen the site yet (and believe it is still in development), but this is how it has been explained to me. Any thoughts? My first thought was to add a blog to add page depth to the site and expand the content. Any other thoughts are welcome and appreciated. Thanks. (I know this is limited information, I'm sorry. It's just about all I have to work with right now, and I was a little concerned and was hoping for a second opinion)
Web Design | | AdamWormann0 -
What happens if I 301 Redirect my homepage to a different page on site
If i were to 301 redirect the index page of my website to a page in a different subdirectory of my site would that adversely affect SEO? Does your home page need to be in the root of your site? I'm asking because a developer has told me that it would be best to do that since he needs to install OpenCart on the root of our domain...
Web Design | | SheffieldMarketing0 -
So apparently SEO moz will get us de-indexed according to a SEO company!
Each and every day i get called up from an SEO company who promises to get me top spots in Google rankings if i quickly get on their special offer they have today normally i would say "no thanks and put the phone down" but i had a bit of spare time so i indulged the guy and we got talking. After the introductions and speal about his company he was showing me what his company does and how they go about it to get me top ranks (they don't get me ranks but create a website they own which then passes leads to me- kinda clever since they could then start charging me per lead or my competitors) We continued to talk and i mentioned i used SEOmoz to check my rankings and back links etc and he told me that Google are cracking down and anyone using these types of software/websites will get their websites de indexed. This struck me as BS but i wanted to get your thoughts on the matter, i personally don't believe Google would ever do such a thing as this since it would be so easy to get your competitors websites taken down (i.e. negative seo) but its certainly a talking point.
Web Design | | GarethEJones0 -
Sudden disappearance from visibility on Google
Our Google position had a sudden and precipitous drop from #3 to #15 in October for our two main keywords: "web design Santa Barbara" and "Santa Barbara web design". I have steadily gotten it back to #7 by putting a lot of effort to our on-page SEO. As of my December 24 SEOMoz report, it was #7, and as of my check yesterday, Dec. 28 it was #7. We were #3 for "web designers Santa Barbara" as of yesterday. Today, i cannot even find our listing on the first 5 pages. Recently, our partner company was engaging in some link building for us that may have just occurred and I'm worried that this may be the cause. I have not yet had a report and actually don't know for sure yet, it has even been accomplished. We had also done some link building in October by a different company we subcontract to and there was a Google type hack of our site. Those were the two major changes that occurred at that time. I thought it would be safe to have some more link building done now, since we'd already survived that first "sudden" increase of links in October. Does anyone have any idea what may have happened and how I can fix this? I am trying not to panic. Thank you in advance for any ideas to understand and/or remedy the situation!
Web Design | | gfiedel0 -
Google Tag Manager
I recently discovered the Google Tag Manager and I am in the process of updating many of my websites with this feature. I am using Tag Manager to mange Google Analytics, Google Remarketing, Alive Chat, Woopra, etc. I have one question about how Tag Manager actually works. As best I can tell, the Tag Manager code snippet that I insert into my web pages is the same for all my websites and does not include a unique ID. If that is the case, then Tag Manager must search all the URLs in the TM database to find a match. What is to stop someone else from adding some rules for my URLs to their containers? I expect Google has a method to ensure proper matching, but I'm not clear on how that is enforced. Best,
Web Design | | ChristopherGlaeser
Christopher0 -
Duplicate home page /index.asp /index.php etc
We recently moved www.devoted2vntage.co.uk to shopify but seem to have multiple home page variants still in google index. I am concerned that these will be causing duplicate content. I have redirected the offending URLs below to www.devoted2vintage.co.uk/ and have set up a canonical URL but need an expect to tell me if I have taken the current steps and if not, exactly what I need to do. www.devoted2vintage.co.uk/index.php www.devoted2vintage.co.uk/index.htm www.devoted2vintage.co.uk/index.html www.devoted2vintage.co.uk/index.shtml www.devoted2vintage.co.uk/index.aspx www.devoted2vintage.co.uk/index.cfm www.devoted2vintage.co.uk/index.pl www.devoted2vintage.co.uk/index.asp
Web Design | | devoted2vintage0
