Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Manufacturer, New Direct-to-Consumer Site (Separate Site, or Sub-Domain?)
Hi All! Working with an established manufacturer, been around for many years, it's an internationally known brand, and their products are sold by thousands on distributors. They recently started a new website (separate from their old established B2B manufacturer site) which will be used to sell direct to customer. The new site is great, with a nice responsive design, clean look, flexible, etc. The problem is, it's a new site with low Domain Authority. The manufacturer's B2B site has been around a while, very high Domain Authority. So, I'd like to be able to harness all the link equity they've build instead of trying to optimize a brand new site. The problem with this old established site is that it IS in fact old. The design is terrible, it's not responsive, old code, bad look and feel, etc. We could incorporate the new B2C site (which has its own CMS) into a sub-domain, like store.site.com. But, I'd worry that site.com's crapiness will limit growth potential for the new pages at store.site.com. Same issue were we to add the new site into a sub-folder, like site.com/store/. On the other side, we could just keep the new site, with it's own domain, sitestore.com, and have product pages and/or category pages from the manufacturer's B2B site link to the relevant pages on the new B2C site. Thanks!
Web Design | | fiberglass0 -
CMS dynamicly created pages indexed?
Hey Moz'erz, Looking at the indexed pages of my clients eCommerce website I noticed that dynamically created pages are being indexed. For example this page does not "exist" but is created by a drop down filter menu that sorts by product tag: /collections/tools/TAG I can only conclude that this page got indexed either through a backlink or once upon a time there was an internal link pointing to this URL and got indexed (currently there is not). Are either of these cases possibilities? In either case before considering removal or any action I would of-course reference analytics to check for conversions, traffic and any backlinks for those "pages". I believe at the end of the day is recommend a drop down filer that doesn't create new pages as the best solution. Thoughts, comments and experience is greatly welcomed 🙂
Web Design | | paul-bold0 -
Can anyone recommend a tool that will identify unused and duplicate CSS across an entire site?
Hi all, So far I have found this one: http://unused-css.com/ It looks like it identifies unused, but perhaps not duplicates? It also has a 5,000 page limit and our site is 8,000+ pages....so we really need something that can handle a site larger than their limit. I do have Screaming Frog. Is there a way to use Screaming Frog to locate unused and duplicate CSS? Any recommendations and/or tips would be great. I am also aware of the Firefix extensions, but to my knowledge they will only do one page at a time? Thanks!
Web Design | | danatanseo0 -
Will google penalize a website for using a table layout?
I just got a new client today and his entire website layout and structure is using tables instead of divs. This client is on a tight budget and wants to avoid unnecessary hours for re-coding the website, but at the same time he wants me to improve his SEO organically. This is the first time I've been asked to do work on an existing website that uses pure tables for the entire layout and I'm wondering if this effects the SEO in any way. So my question is, will tables effect rankings and SEO in any way?
Web Design | | ScottMcPherson0 -
Site is losing traffic after relaunch
Hello, We've just relaunched this site in the last several days, and we're seeing some small (but stead) traffic decreases, as well as engagement decreases. We're aware that page speed (about 4 seconds from a non-cached browser) and some 404s are an issue, our team is currently working on both. But we're really looking for some constructive criticism here as to what we need to improve. Other issues to be aware of: lots of our social counts went back to 0s, as lots of URLs changed, and it wasn't possible to migrate comments from the old system, so those have gone back to 0 as well. We wonder if this might be affecting both users and search engines perception of the site. Your input would be greatly appreciated. Thanks,
Web Design | | FishAcct
Paul0 -
How will a sites ranking be affected??
Hey guys, My MD has just put this to me, we have a site the is currently ranked in top 3 for all there chosen search terms. The company is undergoing an overhaul of their design and want a new site to match this. They have asked how changing the site design will affect their rankings. More content is going to be added with this there will be new pages links etc Any information would be greatly appreciated.. Thanks Anthony
Web Design | | Anthonykal-group0 -
Need help setting up google analytics goals / tracking
I don't use Google analytics to see much more than how many visits I'm getting and what sort of keywords people are using to find our site. I'd like to step up my GA skills a bit. I'm wondring if you guys could give me some advice. I've never really set up any GA goals, or used it to track specific things, but I'd like to. Here are a few things off the top of my head that I would like to track. I'm wondring if these are posable in GA, and if someone could give me some feedback on how to track it / set up goals. Thanks.1 1. I'd like to know how many people click play on a video when they are on a page that has a video. 2. I'd like to know how many people are clicking "like" " google plus, etc.) 3. I'd like to know the path people are taking on our site. For instance, if they click a link from Facebook, and go to a landing page, what page are they visiting next..... 4. How long people are staying on the page I would really like to break this down further by people that visit a link I posted on Facebook, or twitter, or from the link on my twitter profile page, etc... Also if there are any other valuable goals / reports that would be useful for a blogger to track I'd appreciate your feedback. Thanks.
Web Design | | NoahsDad0 -
Do Pages That Rearrange Set Off Any Red Flags for Google?
We have a broad content site that includes crowdsourced lists of items. A lot of the pages allow voting, which causes the content on the pages (sometimes the content is up to 10 pages deep) to completely rearrange, and therefore spread out and switch pages often among the (up to 10) pages of content. Now, could this be causing any kind of duplicate content or any other kind of red flags for Google? I know that the more the page changes the better, but if it's all the same content that is being moved up and down constantly, could Google think we're pulling some kind of "making it look like we have new content" scheme and ding us for these pages? If so, what would anyone recommend we do? Let's take an example of a list of companies with bad customer service. We let the internet vote them up and down all the time, the order changes depending on the votes in real time. Is that page doomed, or does Google see it and love it?
Web Design | | BG19850