Help! Pages not being indexed
-
Hi Mozzers, I need your help.
Our website (www.barnettcapitaladvisors.com) stopped being indexed in search engines following a round of major changes to URLs and content. There were a number of dead links for a few days before 301 redirects were properly put in place. And now, only 3 pages show up in bing when I do the search "site:barnettcapitaladvisors.com". A bunch of pages show up in Google for that search, but they're not any of the pages we want to show up.Our home page and most important services pages are nowhere in search results.
What's going on here?
Our sitemap is at http://www.barnettcapitaladvisors.com/sites/default/files/users/AndrewCarrillo/sitemap/sitemap.xml
Robots.txt is at: http://www.barnettcapitaladvisors.com/robots.txtThanks!
-
Thanks for the help! The Screaming Frog site crawler really makes it much easier to understand the robots.txt file. It looks like it was set to "disallow" robots.
-
Nothing really to "resolve" this.
those tools could simply provide you with cues and signs to figure out what went wrong and when... traffic and referral data on those pages could also indicate when they exactly stopped appearing in search and you can the correlate that with your dev actions if you do have some memory or log of those steps taken and backtrack and find out what was set wrong.
So you see absolutely no errors no messages nothing in GWT per this old post?
if you are using drupal, check this thread about it, it may well be your problem...
also see the very bottom of this page that may help you diagnose if you have a setting wrong in your robots file.
-
Hi Brien
The X-Robots tag has set many of your pages to noindex,nofollow. X-Robots is delivered as part of the HTTP response header so you won't find it in your HTML code.
In addition to that, your canonical tags are a little confused: the homepage canonical tag is set to
http://barnettcapitaladvisors.com/miami-financial-advisor-south-florida-financial-planner
when it should be barnettcapitaladvisors.com
I found these things by using Screaming Frog site crawler.
In your Robots file you're not allowing Yandex or Baidu to crawl the site - are you aware of this?
Have your developer correct the X-Robots and canonical issues.
-
Yes, I've searched Google Webmaster Tools and see a big dropoff in impressions on 5/4/13.
There are 14 pages in the sitemap that should be showing up. But, instead, what's getting indexed (9 pages in Google and 2 of the 3 pages in Bing) are pages that aren't in the sitemap -- ones that were created and then taken down or that we don't care to be indexed.
Yes, some of the URLs are new (three weeks ago) and that may be why they haven't appeared yet, but what about the home page? That isn't a new URL and it isn't appearing. Also, one new URL (which is also one of the important services pages that we want to appear) is appearing in bing: www.barnettcapitaladvisors.com/wealth-management but the other ones are not.I use analytics as well, but what particularly in Google Analytics will help me resolve this?
-
have you checked your google/bing webmasters tool yet? that often will give you the best answer in these situations?
Do you use analytics? Have you checked that as well?
if any pages do show up when you site search your domain on google, then you are being indexed, if pages you made are too new, it may take some time for them to become available in google's SERPs
it all just depends, how long has it been?
I see only 2 links in bing results for your site, but for google we have 9 results, how many should it be? 14?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sitemap.gz is being indexed and is showing up in SERP instead of actual pages.
Sitemap.gz is being indexed and is showing up in SERP instead of actual pages. I recently uploaded my sitemap file - https://psglearning.com/sitemapcustom/sitemap-index.xml - via Search Console. The only record within the XML file is sitemaps.gz. When I searched for some content on my site - here is the search https://goo.gl/mqxBeq - I was shown the following search result, indicating that our GZ file is getting indexed instead of our pages. http://www.psglearning.com/catalog 1 http://www.psglearning.com ...www.psglearning.com/sitemapcustom/sitemap.gz... 1 https://www.psglearning.com/catalog/productdetails/9781284059656/ 1 https://www.psglearning.com/catalog/productdetails/9781284060454/ 1 ... My sitemap is listed at https://psglearning.com/sitemapcustom/sitemap-index.xml inside the sitemap the only reference is to sitemap.gz. Should we remove the link the the sitemap.gz within the xml file and just serve the actual page paths? <sitemapindex< span=""> xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"></sitemapindex<><sitemap></sitemap>https://www.psglearning.com/sitemapcustom/sitemap.gz<lastmod></lastmod>2017-06-12T09:41-04:00
Technical SEO | | pdowling0 -
Titling Category Pages Like You Would a Blog Page?
So, with our 600 or so category pages, I was curious... on each of these category pages we show the top 12 products for that category. In trying to increase click through rate, I wonder if it would be prudent to use some of the strategies I see used for Blog posts with thee category pages. i.e. Instead of Category Name - Website Name How about: Top 12 Kitty Litters We Carry - View the Best and the Rest! Or something like that. And then in the description, I could put, "Number 8 made my jaw drop!!!" (Ok, kidding about that one...) But serious about the initial question... Thanks! Craig
Technical SEO | | TheCraig0 -
Drastic increase of indexed pages correlated to rankings loss?
Our ecommerce website has had a drastic increase in indexed pages, and equal loss of Google organic traffic. After 10/1 the number of indexed pages jumped from 240k to 5.7 million by the end of the year, according to GWT. Coincidentally, the sitemap tops at 14,192 pages, with 13,324 indexed. Organic traffic on some top keyphrases began declining by half after 10/26 and ranking (previously placing in the top 5 spots) has dropped to the fifth page of results. This website does produce session id's (/c=) so we been blocking /c=/ in the robots.txt file. We also have a rel=canonical on all pages pointing at the correct url. With all of this in place, traffic hasn't recovered. Is there a correlation between this spike of indexed pages and the lost keyword ranking? Any advice to investigate and correct this further would be greatly appreciated. Thanks.
Technical SEO | | marketing_zoovy.com0 -
.co.uk/index.html or just .co.uk - my on-page reports are different for both - why?
It looks like the same thing, yet it has a different on-page report for each version - why is this. Please share your ideas with me on this. The original url is http://bath.waspkilluk.co.uk/index.html. Many Thanks - Simon.
Technical SEO | | simonberenyi0 -
Page Not Found Help!
Hi, I recently (about 2 months ago) moved a blog from a separate domain name over to my eCommerce site to help with marketing. http://www.moondoggieinc.com/blog. I seem to have gotten it all to work right, but I'm getting tons of 404 errors and they all have " in them for example: http://www.moondoggieinc.com/blog/”http://www.moondoggieinc.com/custom_dog_tanks_and_tees.php” I'm not sure how this happened of how to fix it, but there are about 250 pages like this. I know how to redirect them all with a 301 in htaccess, but Im not sure if that's the appropriate course to fix this or if that's just putting a patch on something that's causing a more major issue. Or do i just need to write 250 301 redirects? Thanks! Kristy O
Technical SEO | | KristyO0 -
Help changing category and page titles on established Magento site.
Hi All, This is my first post, I've been a Pro member for a while now, read many posts but have never asked a question here. I have an established Magento site that I need to do a lot of SEO work on. I am a newbie at SEO, so I wanted to ask for advice here. I just changed one of the category names, the url, meta title, and meta description. When making the change I selected the checkbox in the Magento backend that says "Create Permanent Redirect for Old URL" Example:
Technical SEO | | TrulyLuna
Old Category: http://www.domain.com/old-category-name.html
New Category: http://www.domain.com/new-category-name.html Everything worked fine with the exception of one alarming thing... I lost all of the page authority for the category itself and all of the sub categories and pages below it. Now the category and all pages below it show (on the moz toolbar) a PA of 1 and 0 links and 0 domains. I updated the sitemap. I did not do a 301 redirect in the .htaccess, I only chose the option to redirect on the Magento backend when changing the name and url of the category. I need to change a lot of other category and page names and urls but now I'm a bit gun shy, as I do not want to do something that will damage what little page rank I have at the moment. I'm looking for some advice from one of you guys who might be able to help me do this the correct way. I did some research on Google, but I'm still not sure of the correct method to accomplish this. Thanks in advance for any help that you can offer.0 -
Properly Moving Blog from Index to its Own Page
Right now I have a website that is exclusively a blog. I want to create pages outside of the blog and move the blog to a page other than the index file e.g.) from domain.com to domain.com/blog I will have the blog post pages stay in the root directory. e.g.) domain.com/blog-post Any suggestions how to properly tell SE's and other websites that the blog has moved?
Technical SEO | | Bartell0 -
GWT indexing wrong pages
Hi SEOMoz I have a listings site. In a part of the page, I have 3 comboboxes, for state, county and city. On the change event, the javascript redirects the user to the page of the selected location. Parameters are passed via GET, and my URL is rewrited via htaccess. Example: http:///www.site.com/state/county/city.html The problem is, there is A LOT(more than 10k) of 404 errors. It is happenning because the crawler is trying to index the pages, sometimes WITHOUT a parameter, like http:///www.site.com/state//city.html I don't know how to stop it, and I don't wanna remove it, once it's very clicked by the users. What should I do?
Technical SEO | | elias990