Robots.txt file in Shopify - Collection and Product Page Crawling Issue
-
Hi, I am working on one big eCommerce store which have more then 1000 Product. we just moved platform WP to Shopify getting noindex issue. when i check robots.txt i found below code which is very confusing for me. **I am not getting meaning of below tags.**
- Disallow: /collections/+
- Disallow: /collections/%2B
- Disallow: /collections/%2b
- Disallow: /blogs/+
- Disallow: /blogs/%2B
- Disallow: /blogs/%2b
I can understand that my robots.txt disallows SEs to crawling and indexing my all product pages. ( collection/*+* ) Is this the query which is affecting the indexing product pages?
Please explain me how this robots.txt work in shopify and once my page crawl and index by google.com then what is use of Disallow:
Thanks.
-
Make sure products are in your sitemap and it has been re-submitted. You can also submit your products to request indexing for them in Google Search Console.
-
Thank you for replying,
But, our main issue is that we have already crawled all collection pages but the product pages haven't crawled yet. Now we don't figure out that whether it's robots.txt issue or other crawling issue?
For example: "www.abc.com/collection/" page is crawled but "www.abc.com/collection/product1/" page hasn't crawled.
Please reply me some tips here.
-
While you may not want context indexed, it's still valuable to be crawled and access your most important content like products.
If you are blocking your /collections pages, Google will not be able to see that page's meta robots set to noindex, causing an issue for you. You may consider allowing robots to crawl your /collections pages but noindex them if they are low value or duplicative.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why to add a product id in the url
Hello ! shop.com/en/2628-buy-key-origin-the-sims-4-seasons/ Why will people use a product id in the link? Is there any advantage like better ranking or else?
White Hat / Black Hat SEO | | kh-priyam0 -
[SEO] Star Ratings -> Review -> Category Page
Hello there, Basically, if you put non-natural star ratings on the category page, like in the attached images, you will get manual ban from google right?
White Hat / Black Hat SEO | | Shanaki
(i know it for sure, cause I had clients with this situation) The real question is:
If I put a form that allows users to write a review about the category products on the category page, for REAL, will google still ban? Any advice? Any example? With respect,
Andrei Irh0O kto4o0 -
Ecommerce sites we own have similar products, is this OK?
Hello, In one of our niches, we have a big site with all products and a couple more sites that are smaller niches of the same niche. The product descriptions are different with different product names. Is this OK. We've got one big site and 2 smaller subsides in different niches that cross over with the big site. Let me know if Google is OK with this. We will have a separate blog for each with completely different content. There's not really duplicate content issues and although only the big site has a blog right now, the small ones eventually will have their own unique blog. Is this OK in Google's eyes now and in the future? What can we do to ensure we are OK? Thank you.
White Hat / Black Hat SEO | | BobGW1 -
One page with multiple sections - unique URL for each section
Hi All, This is my first time posting to the Moz community, so forgive me if I make any silly mistakes. A little background: I run a website that for a company that makes custom parts out of specialty materials. One of my strategies is to make high quality content about all areas of these specialty materials to attract potential customers - pretty strait-forward stuff. I have always struggled with how to structure my content; from a usability point of view, I like just having one page for each material, with different subsections covering covering different topical areas. Example: for a special metal material I would have one page with subsections about the mechanical properties, thermal properties, available types, common applications, etc. Basically how Wikipedia organizes its content. I do not have a large amount of content for each section, but as a whole it makes one nice cohesive page for each material. I do use H tags to show the specific sections on the page, but I am wondering if it may be better to have one page dedicated to the specific material properties, one page dedicated to specific applications, and one page dedicated to available types. What are the communities thoughts on this? As a user of the website, I would rather have all of the information on a single, well organized page for each material. But what do SEO best practices have to say about this? My last thought would be to create a hybrid website (I don't know the proper term). Have a look at these examples from Time and Quartz. When you are viewing a article, the URL is unique to that page. However, when you scroll to the bottom of the article, you can keep on scrolling into the next article, with a new unique URL - all without clicking through to another page. I could see this technique being ideal for a good web experience while still allowing me to optimize my content for more specific topics/keywords. If I used this technique with the Canonical tag would I then get the best of both worlds? Let me know your thoughts! Thank you for the help!
White Hat / Black Hat SEO | | jaspercurry0 -
My site www.sriindustries.com dropped to back page penguin 2.1
My website penguin 2.1 dropped back to page 5 and beyond, can you help me to come out from this ? My head is breaking, also I would like to know how to be on top for local business search (maps)
White Hat / Black Hat SEO | | synchronyinfo0 -
How to Solve Mysteries for Disabled Products?
I want to solve mysteries regarding disabled products on my eCommerce website. I want to give one example for my one product to know more about it. Product URL: http://www.vistastores.com/indoorlighting-patiolivingconcepts-20947.html Product Name: Floor Lamp in Monterey Bronze Finish Before 3 Months, This product was live on my website with In Stock status. Google have crawled that product, added in XML sitemap, added in Google merchant center, added in too many external website during link building campaign. Before 15 Days, This product was live on my website with Out of Stock status. Now, visitor can visit this page but, can not add in shopping cart. Now, This product is disabled from website and not available for sell. I have done lot of work to compile content, image, page rank and many other SEO stuffs to get rank with specific long trail keyword. This product is suddenly disabled from website so, it's shows 404 error and redirect to custom 404 error page. But, I am not satisfy with 301 redirect to set 302 redirect. But, is it really good? Is it require to set 301 or 302 redirect on disable products? I will never sell this product again on website. But, what about my indexing, external links, page authority? This is creating too many up an down in webmaster tools data, merchant center data, xml sitemap data and impression data. What is best solution for it? Can any one share good example for eCommerce website.
White Hat / Black Hat SEO | | CommercePundit0 -
Is domain name or page title "safe" as anchor text?
I am aware of the dangers of excessively optimized anchor text I have seen some suggestions that as long as your anchor text is either the URL or the page title that this will be OK, no matter how many links come in with that anchor text. Does anyone have an opinion, or even any hard data on this? Thx Paul
White Hat / Black Hat SEO | | diogenes0 -
Multiple links to different pages from same page
Hey, I have an opportunity to get listed in a themed directory page, that has a high mozRank of 4+ and a high mozTrust of 5+. Would it be better to just have one link from that page going to one of my internal product category pages, or take advantage of the 'sitelinks' they offer, that allows me to have an additional 5 anchor text links to 5 other pages? I've attached an example. sitelinks.jpg
White Hat / Black Hat SEO | | JerDoggMckoy0