A site is not being indexed by Google Yahoo or Bing
-
This site - http://adoptionconnection.org/ is not being indexed by any of the search engines. I checked the easy stuff - robots text is:
<meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">all, index, follow</a>" /> <meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">noodp</a>" /> <meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">noydir</a>" /> I have checked what I can determine would cause the issue but have found nothing to prevent it from being indexed. I'm thinking it may be re-directs etc. Any answer would be great. Thanks in advance,
-
There lies the problem , best to manual add the admin folder and other folder you dont want spiders accessing in there
This should help : http://www.robotstxt.org/robotstxt.html
-
I think this is it John! I will have them make the change tomorrow & get back on this thread to give the results.
Thanks in advance!!
-
Great answer thanks I will make sure they have submitted to webmaster tools.
-
The robots.txt (at http://adoptionconnection.org/robots.txt) is disallowing all bots from reading the site. Change it from:
User-agent: * Disallow: /
to:
User-agent: * Disallow:
-
You could make sure the relevant content is listed in your xml site map for starters : http://adoptionconnection.org/sitemap.xml ( its currently blank, assuming of course you have submitted it to Google via Webmasters tools )
Jump in to Web Masters Tools and see the crawl stats and see if there are any errors
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Not Being Indexed
Hey Everyone - I have a site that is being treated strangely by google (at least strange to me) The site has 24 pages in the sitemap - submitted to WMT'S over 30 days ago I've manually triggered google to crawl the homepage and all connecting links as well and submitted a couple individually. Google has been parked the indexing at 14 of the 24 pages. None of the unindexed URL's have Noindex or follow tags on them - they are clearly and easily linked to from other places on the site. The site is a brand new domain, has no manual penalty history and in my research has no reason to be considered spammy. 100% unique handwritten content I cannot figure out why google isn't indexing these pages. Has anyone encountered this before? Know any solutions? Thanks in advance.
Technical SEO | | CRO_first0 -
Yahoo Local SERPs Index Issue
I recently updated all my website page Titles and was checking to see how many have been crawled so far. On Yahoo/Bing I noticed something very strange when entering site:bandpages.ie in the search field. Selection Buttons (top of SERPs): 'Web' search shows all my pages indexed 'UK' has has most pages 'Only In Ireland' has just 1 page indexed - which is the site RSS Feed and nothing else! The site has been live for 2 years now. Considering we don't trade with the UK and our main focus is here in Ireland - what is going wrong? Why doesn't Yahoo/Bing index list the site pages in the Ireland index? Any insights or solutions appreciated...
Technical SEO | | Ubique0 -
Site being indexed by Google before it has launched
We are currently coming towards the end of a site migration, and are at the final stage of testing redirects etc. However, to our horror we've just discovered Google has started indexing the new site. Any ideas on how this could have happened? I have most recently asked for robots.txt to exclude anything with a certain parameter in URL. Is there a chance this, wrongly implemented, could have caused this?
Technical SEO | | Sayers0 -
301 redirects tanked our site on google - what now?
We had several hundred old pages on the site with duplicate content and new pages with fresh info on the same topics. So I redirected the old pages to the new pages. Next day, plop, we're dumped off google for almost every keyword. Dang I thought they didn't want duplicate content and old funky pages. What did I do wrong and what can I do to fix it? Thanks so much for anyone who can share their expertise. Jean
Technical SEO | | JeanYates0 -
Removing a site from Google's index
We have a site we'd like to have pulled from Google's index. Back in late June, we disallowed robot access to the site through the robots.txt file and added a robots meta tag with "no index,no follow" commands. The expectation was that Google would eventually crawl the site and remove it from the index in response to those tags. The problem is that Google hasn't come back to crawl the site since late May. Is there a way to speed up this process and communicate to Google that we want the entire site out of the index, or do we just have to wait until it's eventually crawled again?
Technical SEO | | issuebasedmedia0 -
We changed the URL structure 10 weeks ago and Google hasn't indexed it yet...
We recently modified the whole URL structure on our website, which resulted in huge amount of 404 pages changing them to nice human readable urls. We did this in the middle of March - about 10 weeks ago... We used to have around 5000 404 pages in the beginning, but this number is decreasing slowly. (We have around 3000 now). On some parts of the website we have also set up a 301 redirect from the old URLs to the new ones, to avoid showing a 404 page thus making the “indexing transmission”, but it doesn’t seem to have made any difference. We've lost a significant amount of traffic, because of the URL changes, as Google removed the old URLs, but hasn’t indexed our new URLs yet. Is there anything else we can do to get our website indexed with the new URL structure quicker? It might also be useful to know that we are a page rank 4 and have over 30,000 unique users a month so I am sure Google often comes to the site quite often and pages we have made since then that only have the new url structure are indexed within hours sometimes they appear in search the next day!
Technical SEO | | jack860 -
Why do I rank so differently in Bing and Yahoo?
Howdy SEOmozers, I've noticed that I rank very differently in Bing and Yahoo compared to Google (and it’s the bad kind of different) for many of the terms that I'm targeting. Can anyone give some advice on how I can put this right or link me to some good resources about the differences in their ranking algorithms? Thanks Paul
Technical SEO | | pauldoffman1