E-Commerce Site Collection Pages Not Being Indexed
-
Hello Everyone,
So this is not really my strong suit but I’m going to do my best to explain the full scope of the issue and really hope someone has any insight. We have an e-commerce client (can't really share the domain) that uses Shopify; they have a large number of products categorized by Collections. The issue is when we do a site:search of our Collection Pages (site:Domain.com/Collections/) they don’t seem to be indexed.
Also, not sure if it’s relevant but we also recently did an over-hall of our design. Because we haven’t been able to identify the issue here’s everything we know/have done so far:
- Moz Crawl Check and the Collection Pages came up.
- Checked Organic Landing Page Analytics (source/medium: Google) and the pages are getting traffic.
- Submitted the pages to Google Search Console.
- The URLs are listed on the sitemap.xml but when we tried to submit the Collections sitemap.xml to Google Search Console 99 were submitted but nothing came back as being indexed (like our other pages and products).
- We tested the URL in GSC’s robots.txt tester and it came up as being “allowed” but just in case below is the language used in our robots:
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkout
Disallow: /9545580/checkouts
Disallow: /carts
Disallow: /account
Disallow: /collections/+
Disallow: /collections/%2B
Disallow: /collections/%2b
Disallow: /blogs/+
Disallow: /blogs/%2B
Disallow: /blogs/%2b
Disallow: /design_theme_id
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /apple-app-site-association
Sitemap: https://domain.com/sitemap.xml - A Google Cache:Search currently shows a collections/all page we have up that lists all of our products.
Please let us know if there’s any other details we could provide that might help.
Any insight or suggestions would be very much appreciated.
Looking forward to hearing all of your thoughts!
Thank you in advance.
Best,
-
It seems odd to deal with filtering here. I'd normally do that in Search Console under URL parameters but you have to be extremely careful altering stuff in there. If you email me the site I'll run a check on why Collections aren't showing.
-
Hello Nigel,
Thank you very much for your help. The reason we were a little hesitant to remove those lines from the robots.txt file is we were not sure if that would cause the ‘filtered’ pages we have for these collections to be indexed (Each collection can be sorted and filtered). We didn’t write the robots.txt originally and don’t really know what the syntax we would be removing is doing. While we try to find out and look for other options do you have any idea of whether it could have that type of impact?
Also, we checked and there’s not a ‘noindex’ tag present.
We currently only have one blog post up but oddly enough it does seem to be ‘indexed’ (even with the /blogs/ URL).
We’re not so experienced with backend Shopify elements but is it possible there’s an issue that related to Google only caching the collections/all page we have up that lists all of our products?
Any help or suggestions you have would be very much appreciated and thank you again!
Please let us know if you have any questions, concerns or we could provide any further details in the meantime.
Looking forward to hearing your thoughts!
Best,
-
Try deleting these from the Robots txt.
Disallow: /collections/+
Disallow: /collections/%2B
Disallow: /collections/%2bSubmit again and see what comes up. I'd hazard that this is the most logical explanation apart from having a noindex tag on the collections pages.
Are the blogs pages coming up by the way?
Right click 'view source' and have a look if there is one there.
Regards
Nigel
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can a duplicate page referencing the original page on another domain in another country using the 'canonical link' still get indexed locally?
Hi I wonder if anyone could help me on a canonical link query/indexing issue. I have given an overview, intended solution and question below. Any advice on this query will be much appreciated. Overview: I have a client who has a .com domain that includes blog content intended for the US market using the correct lang tags. The client also has a .co.uk site without a blog but looking at creating one. As the target keywords and content are relevant across both UK and US markets and not to duplicate work the client has asked would it be worthwhile centralising the blog or provide any other efficient blog site structure recommendations. Suggested solution: As the domain authority (DA) on the .com/.co.uk sites are in the 60+ it would risky moving domains/subdomain at this stage and would be a waste not to utilise the DAs that have built up on both sites. I have suggested they keep both sites and share the same content between them using a content curated WP plugin and using the 'canonical link' to reference the original source (US or UK) - so not to get duplicate content issues. My question: Let's say I'm a potential customer in the UK and i'm searching using a keyword phrase that the content that answers my query is on both the UK and US site although the US content is the original source.
Intermediate & Advanced SEO | | JonRayner
Will the US or UK version blog appear in UK SERPs? My gut is the UK blog will as Google will try and serve me the most appropriate version of the content and as I'm in the UK it will be this version, even though I have identified the US source using the canonical link?2 -
Fetch as Google -- Does not result in pages getting indexed
I run a exotic pet website which currently has several types of species of reptiles. It has done well in SERP for the first couple of types of reptiles, but I am continuing to add new species and for each of these comes the task of getting ranked and I need to figure out the best process. We just released our 4th species, "reticulated pythons", about 2 weeks ago, and I made these pages public and in Webmaster tools did a "Fetch as Google" and index page and child pages for this page: http://www.morphmarket.com/c/reptiles/pythons/reticulated-pythons/index While Google immediately indexed the index page, it did not really index the couple of dozen pages linked from this page despite me checking the option to crawl child pages. I know this by two ways: first, in Google Webmaster Tools, if I look at Search Analytics and Pages filtered by "retic", there are only 2 listed. This at least tells me it's not showing these pages to users. More directly though, if I look at Google search for "site:morphmarket.com/c/reptiles/pythons/reticulated-pythons" there are only 7 pages indexed. More details -- I've tested at least one of these URLs with the robot checker and they are not blocked. The canonical values look right. I have not monkeyed really with Crawl URL Parameters. I do NOT have these pages listed in my sitemap, but in my experience Google didn't care a lot about that -- I previously had about 100 pages there and google didn't index some of them for more than 1 year. Google has indexed "105k" pages from my site so it is very happy to do so, apparently just not the ones I want (this large value is due to permutations of search parameters, something I think I've since improved with canonical, robots, etc). I may have some nofollow links to the same URLs but NOT on this page, so assuming nofollow has only local effects, this shouldn't matter. Any advice on what could be going wrong here. I really want Google to index the top couple of links on this page (home, index, stores, calculator) as well as the couple dozen gene/tag links below.
Intermediate & Advanced SEO | | jplehmann0 -
Best format for E-Commerce Pages in Title Text / Link Text & Markup
Hello Please comment on which you think is best SEO practice for each & any comments on link juice following through. Title text ( on Product Page ) <title>Brandname ProductName</title>
Intermediate & Advanced SEO | | s_EOgi_Bear
OR
<title>ProductName by Brandname</title> on category page <a <span="" class="html-attribute-name">itemprop="name" href="[producturl]">ProductName</a>
<a <span="" class="html-attribute-name">itemprop="brand" href="[brandurl]>BrandName</a> OR <a <span class="html-attribute-name">itemprop="name" href="[producturl]">BrandName ProductName
( Leave Brand Link Out)</a <span> Product Page <a itemprop="name" href="[producturl]">ProductName
<a itemprop="brand" href="[brandurl]>BrandName</a itemprop="brand" href="[brandurl]></a itemprop="name" href="[producturl]"> OR <a itemprop="name" href="[producturl]">BrandName ProductName
( Leave Brand Link Out)</a itemprop="name" href="[producturl]"> Thoughts?0 -
Google Is Indexing The Wrong Page For My Keyword
For a long time (almost 3 mounth) google indexing the wrong page for my main keyword.
Intermediate & Advanced SEO | | Tiedemann_Anselm
The problem is that each time google indexed another page each time for a period of 4-7 days, Sometimes i see the home page, sometimes a category page and sometimes a product page.
It seems though Google has not yet decided what his favorite / better page for this keyword. This is the pages google index: (In most cases you can find the site on the second or third page) Main Page: http://bit.ly/19fOqDh Category Page: http://bit.ly/1ebpiRn Another Category: http://bit.ly/K3MZl4 Product Page: http://bit.ly/1c73B1s All links I get to the website are natural links, therefore in most cases the anchor we got is the website name. In addition I have many links I get from bloggers that asked to do a review on one of my products, I'm very careful about that and so I'm always checking the blogger and their website only if it is something good, I allowed it. also i never ask for a link back (must of the time i receive without asking), and as I said, most of their links are anchor with my website name. Here some example of links that i received from bloggers: http://bit.ly/1hF0pQb http://bit.ly/1a8ogT1 http://bit.ly/1bqqRr8 http://bit.ly/1c5QeC7 http://bit.ly/1gXgzXJ Please Can I get a recommendation what should you do?
Should I try to change the anchor of the link?
Do I need to not allow bloggers to make a review on my products? I'd love to hear what you recommend,
Thanks for the help0 -
Will thousands of redirected pages have a negative impact on the site?
A client site has thousands of pages with unoptimized urls. I want to change the url structure to make them a little more search friendly. Many of the pages I want to update have backlinks to them and good PR so I don't want to delete them entirely. If I change the urls on thousands of pages, that means a lot of 301 redirects. Will thousands of redirected pages have a negative impact on the site? Thanks, Dino
Intermediate & Advanced SEO | | Dino641 -
Adding Orphaned Pages to the Google Index
Hey folks, How do you think Google will treat adding 300K orphaned pages to a 4.5 million page site. The URLs would resolve but there would be no on site navigation to those pages, Google would only know about them through sitemap.xmls. These pages are super low competition. The plot thickens, what we are really after is to get 150k real pages back on the site, these pages do have crawlable paths on the site but in order to do that (for technical reasons) we need to push these other 300k orphaned pages live (it's an all or nothing deal) a) Do you think Google will have a problem with this or just decide to not index some or most these pages since they are orphaned. b) If these pages will just fall out of the index or not get included, and have no chance of ever accumulating PR anyway since they are not linked to, would it make sense to just noindex them? c) Should we not submit sitemap.xml files at all, and take our 150k and just ignore these 300k and hope Google ignores them as well since they are orhpaned? d) If Google is OK with this maybe we should submit the sitemap.xmls and keep an eye on the pages, maybe they will rank and bring us a bit of traffic, but we don't want to do that if it could be an issue with Google. Thanks for your opinions and if you have any hard evidence either way especially thanks for that info. 😉
Intermediate & Advanced SEO | | irvingw0 -
Another E-commerce Canonical Question
Hi guys, Quick question: one of our clients has an e-commerce site with a very poor canonical tag setup and thousands of pages of duplicate content. Let's use this as an example: BRAND > Category > Type > Color
Intermediate & Advanced SEO | | elcrazyhorse
Four separate pages/URLs. The BRAND page lists all products.
The Category page lists all BRAND products for that category.
The Type page lists all BRAND products of a specific type in that category.
The Color page lists all BRAND products of a specific type in that category of a specific color. Anyway, these generate four separate URLs: /BRAND
/BRAND/Category
/BRAND/Category-Type
/BRAND/Category-Type-Color Avoiding duplicate content and product listings, I would appreciate your proposed canonicalization strategy/feedback.0 -
Recently created site indexed; no backlinks showing?
I launched a website for a client in mid-March. The site is already indexed, I have built quite a few links to it (links are also indexed), and ranks well for some targeted keywords. However, when I try to check backlinks to the site with Open Site Explorer, it comes back with "No Data Available For This URL". Is this something I should be worried about or merely a case of 'recency' of page creation'? I know it says that it can take 45-60 days for a site to be included in Linkscape but I'm approaching the 60 days mark and still nothing.
Intermediate & Advanced SEO | | Igor-Avidon0