Robots.txt file in Shopify - Collection and Product Page Crawling Issue
-
Hi, I am working on one big eCommerce store which have more then 1000 Product. we just moved platform WP to Shopify getting noindex issue. when i check robots.txt i found below code which is very confusing for me. **I am not getting meaning of below tags.**
- Disallow: /collections/+
- Disallow: /collections/%2B
- Disallow: /collections/%2b
- Disallow: /blogs/+
- Disallow: /blogs/%2B
- Disallow: /blogs/%2b
I can understand that my robots.txt disallows SEs to crawling and indexing my all product pages. ( collection/*+* ) Is this the query which is affecting the indexing product pages?
Please explain me how this robots.txt work in shopify and once my page crawl and index by google.com then what is use of Disallow:
Thanks.
-
Make sure products are in your sitemap and it has been re-submitted. You can also submit your products to request indexing for them in Google Search Console.
-
Thank you for replying,
But, our main issue is that we have already crawled all collection pages but the product pages haven't crawled yet. Now we don't figure out that whether it's robots.txt issue or other crawling issue?
For example: "www.abc.com/collection/" page is crawled but "www.abc.com/collection/product1/" page hasn't crawled.
Please reply me some tips here.
-
While you may not want context indexed, it's still valuable to be crawled and access your most important content like products.
If you are blocking your /collections pages, Google will not be able to see that page's meta robots set to noindex, causing an issue for you. You may consider allowing robots to crawl your /collections pages but noindex them if they are low value or duplicative.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unlisted (hidden) pages
I just had a client say they were advised by a friend to use 'a bunch of unlisted (hidden) pages'. Isn't this seriously black hat?
White Hat / Black Hat SEO | | muzzmoz0 -
Why would a blank page rank? What am I missing about this page?
In terms of content, this page is blank. Yes, there's a sidebar and footer, but no content. I've seen a page like this rank before. I'm curious if they're implementing something on the back-end I don't realize or if this is just a fluke? Etc. Also, the DA of the site is only a 15, so I don't think that's the reason. http://www.thenurselawyer.com/component/tags/tag/20-pasco-county-personal-injury-lawyers.html Thanks, Ruben
White Hat / Black Hat SEO | | KempRugeLawGroup1 -
Local Doorway Pages
Based on what I've read, setting up localized landing pages ie: /web-design-atlanta, web-design-nyc, /web-design-chicago, etc especially with duplicate content is a big no-no. Remarkably, 2 of our competitors are doing it, (they are just swapping out the locations), and it's working. They don't even have office addresses or local phone numbers listed. They are on the first page for multiple location based searches ("web design nyc", "web design atlanta", etc.). I thought Google penalized for this, or at least didn't index the content. What gives? Am I misinterpreting Google's AUP? Can I report them? If it's legal, we should be doing it as well.
White Hat / Black Hat SEO | | CsmBill0 -
Linking my pages
Hello everybody, i have a small dilemma and i am not shore what to do. I am (my company) the owner of 10 e-commerce web sites. On every site i have a link too the other 9 sites and i am using an exact keyvoerd (not the shop name).Since the web stores are big and have over a 1000 pages, this means thet all my sites have a lot off inbound links (compared with my competiton). I am woried that linking them all together could be bad from Googles point of wiev. Can this couse a problem for me, should i shange it? Regardes, Marko
White Hat / Black Hat SEO | | Spletnafuzija0 -
How to transform an excel file on a txt file to send the Google Dissavow
I have a disallow file made on excel with lots of columns off information. I want to transform to txt file saving it from excel, but the result file seems understandable Can someone helpme on how to transform an excel file on the Google Dissavow file format for the final import
White Hat / Black Hat SEO | | maestrosonrisas0 -
Content box (on page content) and titles Google over-optimization penalty?
We have a content box at the bottom of our website with a scroll bar and have posted a fair bit of content into this area (too much for on page) granted it is a combination of SEO content (with links to our pages) and informative but with the over optimization penalty coming around I am a little scared if this will result in a problem for us. I am thinking of adopting the process of this website HERE with the content behind a more information button that drops down, would this be better as it could be much more organised and we will be swopping out to more helpful information than the current 50/50 (SEO – helpful content) or will it be viewed the same and we might as well leave it as is and lower the amount of repetition and links in the content. Also we sell printed goods so our titles may be a bit over the top but they are bring us a lot of converting traffic but again I am worried about the new Google release this is an example of a typical title (only an example not our product page) Banner Printing | PVC Banners | Outdoor Banners | Backdrops | Vinyl Banners | Banner Signs Thank you for any help with these matters.
White Hat / Black Hat SEO | | BobAnderson0 -
How does someone rank page one on google for one domain for over 150 keywords?
A local seo is exclaiming his fantastic track record for a pool company(amonst others) in our local market. Over 150 keywords on page one of google. I checked out a few things using some moz tools and didn't find anything that would suggest that this has come from white hat strategies, tactics or links etc. Interested in how he is doing this and if it is white hat? Thanks, C
White Hat / Black Hat SEO | | charlesgrimm0 -
For traffic sent by the search engines, how much personalization/customization is allowed on a page if any?
If I want to better target my audience so I would like to be able to address the exact query string coming from the search engine. I'd also like to add relevant sections to the site based in the geo area they live in. Can I customize a small portion of the page to fit my visitors search query and geo area per the IP address? How much can I change a web page to better fit a user and still be within the search engine's guidelines?
White Hat / Black Hat SEO | | Thos0030