The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Forum
    2. Categories
    3. Research & Trends
    4. White Hat / Black Hat SEO
    5. Robots.txt file in Shopify - Collection and Product Page Crawling Issue

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Robots.txt file in Shopify - Collection and Product Page Crawling Issue

    White Hat / Black Hat SEO
    4 2 4.6k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • HuptechWebseo
      HuptechWebseo last edited by

      Hi, 
      I am working on one big eCommerce store which have more then 1000 Product. we just moved platform WP to Shopify getting noindex issue. 
      when i check robots.txt i found below code which is very confusing for me.
      
      **I am not getting meaning of below tags.**
      
      • Disallow: /collections/+
      • Disallow: /collections/%2B
      • Disallow: /collections/%2b
      • Disallow: /blogs/+
      • Disallow: /blogs/%2B
      • Disallow: /blogs/%2b
      I can understand that my robots.txt disallows SEs to crawling and indexing my all product pages. ( collection/*+* ) Is this the query which is affecting the indexing product pages?
      

      Please explain me how this robots.txt work in shopify and once my page crawl and index by google.com then what is use of Disallow:

      Thanks.

      1 Reply Last reply Reply Quote 0
      • slatronica
        slatronica last edited by

        Make sure products are in your sitemap and it has been re-submitted. You can also submit your products to request indexing for them in Google Search Console.

        1 Reply Last reply Reply Quote 0
        • HuptechWebseo
          HuptechWebseo last edited by

          Thank you for replying,

          But, our main issue is that we have already crawled all collection pages but the product pages haven't crawled yet. Now we don't figure out that whether it's robots.txt issue or other crawling issue?

          For example: "www.abc.com/collection/" page is crawled but "www.abc.com/collection/product1/" page hasn't crawled.

          Please reply me some tips here.

          1 Reply Last reply Reply Quote 0
          • slatronica
            slatronica last edited by

            While you may not want context indexed, it's still valuable to be crawled and access your most important content like products.

            If you are blocking your /collections pages, Google will not be able to see that page's meta robots set to noindex, causing an issue for you. You may consider allowing robots to crawl your /collections pages but noindex them if they are low value or duplicative.

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post

            Got a burning SEO question?

            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


            Start my free trial


            Explore more categories

            • Moz Tools

              Chat with the community about the Moz tools.

              Getting Started
              Moz Pro
              Moz Local
              Moz Bar
              API
              What's New

            • SEO Tactics

              Discuss the SEO process with fellow marketers

              Content Development
              Competitive Research
              Keyword Research
              Link Building
              On-Page Optimization
              Technical SEO
              Reporting & Analytics
              Intermediate & Advanced SEO
              Image & Video Optimization
              International SEO
              Local SEO

            • Community

              Discuss industry events, jobs, and news!

              Moz Blog
              Moz News
              Industry News
              Jobs and Opportunities
              SEO Learn Center
              Whiteboard Friday

            • Digital Marketing

              Chat about tactics outside of SEO

              Affiliate Marketing
              Branding
              Conversion Rate Optimization
              Web Design
              Paid Search Marketing
              Social Media

            • Research & Trends

              Dive into research and trends in the search industry.

              SERP Trends
              Search Behavior
              Algorithm Updates
              White Hat / Black Hat SEO
              Other SEO Tools

            • Support

              Connect on product support and feature requests.

              Product Support
              Feature Requests
              Participate in User Research

            • See all categories

            • Does Google crawl and index dynamic pages?
              esiow2013
              esiow2013
              0
              3
              10.1k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy

            Looks like your connection to Moz was lost, please wait while we try to reconnect.