Robots.txt file in Shopify - Collection and Product Page Crawling Issue
-
Hi, I am working on one big eCommerce store which have more then 1000 Product. we just moved platform WP to Shopify getting noindex issue. when i check robots.txt i found below code which is very confusing for me. **I am not getting meaning of below tags.**
- Disallow: /collections/+
- Disallow: /collections/%2B
- Disallow: /collections/%2b
- Disallow: /blogs/+
- Disallow: /blogs/%2B
- Disallow: /blogs/%2b
I can understand that my robots.txt disallows SEs to crawling and indexing my all product pages. ( collection/*+* ) Is this the query which is affecting the indexing product pages?
Please explain me how this robots.txt work in shopify and once my page crawl and index by google.com then what is use of Disallow:
Thanks.
-
Make sure products are in your sitemap and it has been re-submitted. You can also submit your products to request indexing for them in Google Search Console.
-
Thank you for replying,
But, our main issue is that we have already crawled all collection pages but the product pages haven't crawled yet. Now we don't figure out that whether it's robots.txt issue or other crawling issue?
For example: "www.abc.com/collection/" page is crawled but "www.abc.com/collection/product1/" page hasn't crawled.
Please reply me some tips here.
-
While you may not want context indexed, it's still valuable to be crawled and access your most important content like products.
If you are blocking your /collections pages, Google will not be able to see that page's meta robots set to noindex, causing an issue for you. You may consider allowing robots to crawl your /collections pages but noindex them if they are low value or duplicative.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Changing Links to Spans with Robots.txt Blocked Redirects using Linkify/jQuery
Hi, I was recently penalized most likely because Google started following javascript links to bad neighborhoods that were not no-followed. The first thing I did was remove the Linkify plugin from my site so that all those links would disappear, but now I think I have a solution that works with Linkify without creating crawlable links. I did the following: I blocked access to the Linkify scripts using robots.txt so that Google won't execute the scripts that create the links. This has worked for me in the past with banner ads linking to other sites of mine. At least it appears to work because those sites did not get links from pages running those banners in search console. I created a /redirect/ directory that redirects all offsite URLs. I put a robots.txt block on this directory. I configured the Linkify plugin to parse URLs into span elements instead of a elements and add no follow attributes. They still have an href attribute, but the URLs in the href now point to the redirect directory and the span onclick event redirects the user. I have implemented this solution on another site of mine and I am hoping this will make it impossible for Google to categorize my pages as liking to any neighborhoods good or bad. Most of the content is UGC, so this should discourage link spam while giving users clickable URLs and still letting people post complaints about people that have profiles on adult websites. Here is a page where the solution has been implemented https://cyberbullyingreport.com/bully/predators-watch-owner-scott-breitenstein-of-dayton-ohio-5463.aspx, the Linkify plugin can be found at https://soapbox.github.io/linkifyjs/, and the custom jQuery is as follows: jQuery(document).ready(function ($) { 2 $('p').linkify({ tagName: 'span', attributes: { rel: 'nofollow' }, formatHref: function (href) { href = 'https://cyberbullyingreport.com/redirect/?url=' + href; return href; }, events:{ click: function (e) { var href = $(this).attr('href'); window.location.href = href; } } }); 3 });
White Hat / Black Hat SEO | | STDCarriers0 -
Difference between anchor text pointing to an article in our section pages and the title of our article
My concern is described more in details in the following hypothetic scenario(basically this is the same method that CNN site applies to its site): In one page i have a specific anchor text e.g. "A firefighter rescued a young boy" and this one is linked to an article which if you enter you will see that it has a different title than the anchor text/short title that i mentioned above. So the internal titlte of the article is "A firefighte rescued a young boy in Philippines while it was rainy". I want to know whether this is a good SEO practice or not. Regards, Christos
White Hat / Black Hat SEO | | DPG_Media0 -
Internal Links to Ecommerce Category Pages
Hello, I read a while back, and I can't find it now, that you want to add internal links to your main category pages. Does that still apply? If so, for a small site (100 products) what is recommended? Thanks
White Hat / Black Hat SEO | | BobGW0 -
Should I redirect old pages
I have taken over SEO on a site. The old people built thousands of pages with duplicate content. I am converting site to wordpress and was wondering if I should take the time to 301 redirect all 10,000 or so pages with duplicate content. The 10,000 pages all have links back to different to different pages as well as to the homepage. Should I just let them go to a 404 page not found.
White Hat / Black Hat SEO | | Roots70 -
HOW TO: City Targeted Landing Pages For Lead Generation
Hi guys, So one of my clients runs a web development agency in San Diego and for lead generation purposes we are thinking of creating him city targeted landing pages which will all be on different domains ie. lawebdesginstudio / sfwebdesigngurus I plan to register these 20-30 domains for my client and load them all up on a my single linux server I have from godaddy. I noticed however today using google's keyword tool that roughly only 5-10 cities have real traffic worth trying to capture to turn into leads. Therefore I am not sure if its even worth building those extra 20 landing pages since they will receive very little traffic. My only thought is, if I do decide to build all 30 landing pages, then I assume I will have a very strong private network of authority websites that I can use to point to the clients website. I mean I figure I can rank almost all of them page 1 top 5 within 2-3 months. My question is: 1. Do city targeted micro sites for the purpose of lead generation still work? If so are there any threads that have more info on this topic? 2. Do you suggest I interlink all 30 sites together and perhaps point them all to the money site? If so i'm wondering if I should diversify the ip's that I used to register the domains as well as the whois info. Thanks guys, all help is appreciated!
White Hat / Black Hat SEO | | AM2130 -
404checker.com / crawl errors
I noticed a few strange crawl errors in a Google Webmaster Tools account - further investigation showed they're pages that don't exist linked from here: http://404checker.com/404-checker-log Basically that means anyone can enter a URL into the website and it'll get linked from that page, temporarily at least. As there are hundreds of links of varying quality - at the moment they range from a well known car manufacturer to a university, porn and various organ enlargement websites - could that have a detrimental effect on any websites linked? They are all nofollow. Why would they choose to list these URLs on their website? It has some useful tools and information but I don't see the point in the log page. I have used it myself to check HTTP statuses but may look elsewhere from now on.
White Hat / Black Hat SEO | | Alex-Harford0 -
The Link in Profile Page is it good BackLink or not?
Well, i see that we need 200mozpoints to be able to put our Website Link (DOFOLLOW) into our profile in SEOMOZ.. the way i know it, it would be a good BackLink for my site. Here is the questions, please do answer from top to bottom, because if you have answered "NOT GOOD" for the first question, then the rest of the question will definitely be "NOT GOOD" too Every single back link source i used below (for question #2 and #3), comes from a good domain (it is an extremely wellknown website in Indonesia) 1. Is the DOFOLLOW link from my SEOMOZ Profile Page , a good back link? 2. is the DOFOLLOW from http://www.indonesiaindonesia.com/m4g1c14n a good back link 3. is the DOFOLLOW from http://www.kaskus.us/member.php?u=10407 (click the Contact Info), a good back link? okay, only if you answered the first 3 questions with "It is a good backlink, and it will definitely help your SEO Standing for your site", then i ask you my real question.. i was planning to use the service from http://www.monsterbacklinks.com , and i asked them to show me what kind of "High Quality Backlink" they will be giving me, here is their reply, 10 examples of profile they use to backlink to one of their client Domain PR 4--http://www.sanramon.org/user/12548
White Hat / Black Hat SEO | | IKT
Domain PR 5--http://extratasty.com/profile/42069/paulc4312
Domain PR 5--http://www.bug.co.uk/forums/members/paulc4312.html
Domain PR 5--http://www.offspring.com/forums/member.php?u=84973
Domain PR 5--http://www.massify.com/profiles/paulcpaul
Domain PR 6--http://www.gamezone.com/member/159751/
Domain PR
5--http://www.indyarocks.com/profile/profile_vview_main.php?uid=6155724
Domain PR 6--http://classic.mapmywalk.com/user_profile?u=866130762956343886
Domain PR 5--http://www.netbookreviews.com/forum/members/paulc4312.html
Domain PR 5--http://www.thepoint.com/users/paul-c-2/profile
Domain PR 5--http://forums.cagepotato.com/members/paulc4312.html In my eyes, all of those links are as good as the one link coming from SEOMOZ Profile, hell in fact i have already purchased from them the 750 High Quality BackLink package (cost 197$), but my PayPal is being lock down just now, because i login to my account from both my cellphone and pc (they think my account is hacked)... so will i increase my SEO Standing if i used their service? if they are, i will finalized my purchase tomorrow (after i settled the problem with paypal) Their FAQ Page is also very convincing .. such these 2 questions Will I get penalized for paying you to do my backlinks? There is no way you will get penalized for paying us to do your backlinks. It is possible to get penalized for paying people to put links on their sites but that's not what you're buying from us. When purchasing from us you are paying us to place thousands of free backlinks. There is absolutely no way Google can penalize you for this. Will Google ban/sandbox me for getting so many backlinks? We have never had any problems with getting sanboxed or banned by google. None of our customers have had any problems either. If our methods of placing backlinks were to get a site penalized or banned then we would be sending thousands of links towards our competitors sites. But since our methods work great for increasing search engine rankings, we would never use our backlinking on our competitors because that will damage our rankings and boost theirs. Please enlighten me 🙂0 -
How to improve on page Optimization ?
How to Optimize our website in terms of onpage optimisation, how relavent the onpage should be to that of the search engine. will onpage optimisatrion will help us to rank better in organic search results.
White Hat / Black Hat SEO | | AkshayaPatra0