Robots.txt file in Shopify - Collection and Product Page Crawling Issue
-
Hi, I am working on one big eCommerce store which have more then 1000 Product. we just moved platform WP to Shopify getting noindex issue. when i check robots.txt i found below code which is very confusing for me. **I am not getting meaning of below tags.**
- Disallow: /collections/+
- Disallow: /collections/%2B
- Disallow: /collections/%2b
- Disallow: /blogs/+
- Disallow: /blogs/%2B
- Disallow: /blogs/%2b
I can understand that my robots.txt disallows SEs to crawling and indexing my all product pages. ( collection/*+* ) Is this the query which is affecting the indexing product pages?
Please explain me how this robots.txt work in shopify and once my page crawl and index by google.com then what is use of Disallow:
Thanks.
-
Make sure products are in your sitemap and it has been re-submitted. You can also submit your products to request indexing for them in Google Search Console.
-
Thank you for replying,
But, our main issue is that we have already crawled all collection pages but the product pages haven't crawled yet. Now we don't figure out that whether it's robots.txt issue or other crawling issue?
For example: "www.abc.com/collection/" page is crawled but "www.abc.com/collection/product1/" page hasn't crawled.
Please reply me some tips here.
-
While you may not want context indexed, it's still valuable to be crawled and access your most important content like products.
If you are blocking your /collections pages, Google will not be able to see that page's meta robots set to noindex, causing an issue for you. You may consider allowing robots to crawl your /collections pages but noindex them if they are low value or duplicative.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Clean-up Question after a wordpress site Hack added pages with external links from a massive link wheel?
Hey All, Thought I would throw this out to ensure I am dotting my "i's" and crossing my "t's"..... Client WordPress site was hacked injected 3-4 pages that cross linked to hundreds (affiliate junk spam link wheel). Pages were removed, 3rd party cleared all malware/viruses. Heavy duty firewall and security monitoring are in place. Hacked pages are now showing as 404. No penalties, ranking issues....If anything there was a temporary BOOST in rankings due to the large link-wheel type net that the pages were receiving....That has since leveled out rankings. I guess my question is, in your opinion is it best to let those pages 404, I am noticing a large amount of links going to them from all over the world from this large link net that was built. I find the temptation to 301 re-direct deleted pages to the homepage difficult...lol..{the temptation is REAL}. Is there anything I am missing? Any other steps that YOU would take? I am assuming letting those pages 404 would be the best bet, as in time they will roll off index.... Thank you in advance, I appreciate any feedback or opinions....
White Hat / Black Hat SEO | | Anthony_Howard0 -
Want to know Best Method to fix keyword cannibalization issue?
I have a website that has been experiencing keyword cannibalization issue since last 2-3 months. We have one main key search term to bring our website TOP ranking, but we have been seeing our website’s 2 different pages ranking strangely sometime for 1st page& sometime for 2nd page that one main key search term. As e.g.:
White Hat / Black Hat SEO | | Aman_123
our main key search term 1st page rank sometime instead 2nd page
our main key search term 2nd page rank sometime instead page I am looking for best solution here to get this fixed..0 -
More pages is good for SEO? Is this true?
Hi Guys I have a question, I was told the more pages I have the better for SEO, Is this true?
White Hat / Black Hat SEO | | edward-may0 -
Controlling crawl speed/delay through dynamic server-code and 503's
Lately i'm experiencing performance trouble caused by bot traffic. Although Googlebot is not the worst (it's mainly bingbot and ahrefsbot), they cause heavy server load from time to time. We run a lot of sites on one server, so heavy traffic on one site impacts other site's performance. Problem is that 1) I want a centrally managed solution for all sites (per site administration takes too much time), which 2) takes into account total server-load in stead of only 1 site's traffic and 3) controls overall bot-traffic in stead of controlling traffic for one bot. IMO user-traffic should always be prioritized higher than bot-traffic. I tried "Crawl-delay:" in robots.txt, but Googlebot doesn't support that. Although my custom CMS system has a solution to centrally manage Robots.txt for all sites at once, it is read by bots per site and per bot, so it doesn't solve 2) and 3). I also tried controlling crawl-speed through Google Webmaster Tools, which works, but again it only controls Googlebot (and not other bots) and is administered per site. No solution to all three of my problems. Now i came up with a custom-coded solution to dynamically serve 503 http status codes to a certain portion of the bot traffic. What traffic-portion for which bots can be dynamically (runtime) calculated from total server load at that certain moment. So if a bot makes too much requests within a certain period (or whatever other coded rule i'll invent), some requests will be answered with a 503 while others will get content and a 200. Remaining question is: Will dynamically serving 503's have a negative impact on SEO? OK, it will delay indexing speed/latency, but slow server-response-times do in fact have a negative impact on the ranking, which is even worse than indexing-latency. I'm curious about your expert's opinions...
White Hat / Black Hat SEO | | internetwerkNU1 -
Where is the rule here that forces any home page URLs that include several versions of explicit page name references, such as default.htm or index.html, to redirect to the canonical homepage?
RewriteRule ^search/([a-zA-Z0-9,-]+)/sort-([a-z0-9-]+)/(1)?$ /search/$1/sort-$2 [R=301]
White Hat / Black Hat SEO | | esiow2013
RewriteRule ^search/([a-zA-Z0-9,-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=search&q=$1&page=$5&sort=$2 [L]
RewriteRule ^search/([a-zA-Z0-9,-]+)/(1)?$ /search/$1 [R=301]
RewriteRule ^search/([a-zA-Z0-9,-]+)((/)([0-9]+))?$ index.php?cmd=search&q=$1&page=$4 [L]
RewriteRule ^([r|R])-([0-9]+)((-)([0-9a-zA-Z-]+))?$ index.php?cmd=propertyPage&id=$2&group=1 [L]
RewriteRule ^([s|S])-([0-9]+)((-)([0-9a-zA-Z-]+))?$ index.php?cmd=propertyPage&id=$2&group=2 [L]
RewriteRule ^reset-password$ index.php?cmd=retrievePassword [L]
RewriteRule ^reset-password/([0-9]+)/([a-zA-Z0-9]+)$ index.php?cmd=retrievePassword&act=retrievePasswordConfirm&code=$1&key=$2 [L]
RewriteRule ^map-page.php$ index.php?cmd=mapPage [L]
RewriteRule ^advertise.php$ index.php?cmd=advertiseWithUs [L]
RewriteRule ^my-account.php$ index.php?cmd=protectedMyAccount [L]
RewriteRule ^bookmarked-properties(/((R|S)-([0-9]+)))?$ index.php?cmd=protectedBookmarkedProperties&id=$2 [L]
RewriteRule ^bookmarked-properties/([0-9]+)$ index.php?cmd=protectedBookmarkedProperties&page=$1 [L]
RewriteRule ^bookmarked-properties/delete/([0-9,]+)$ index.php?cmd=protectedBookmarkedProperties&act=protectedBookmarkedProperties_delete&id=$1 [L]
RewriteRule ^register.php$ index.php?cmd=register [L]
RewriteRule ^about-us.php$ index.php?cmd=aboutUs [L]
RewriteRule ^news(/([0-9]+)(.*))?$ index.php?cmd=news&id=$2&title=$3 [L]
RewriteRule ^registration/confirm/key=([a-zA-Z0-9]{150})/code=([a-zA-Z0-9]{5})$ index.php?cmd=register&act=registerConfirm&key=$1&code=$2 [L]
RewriteRule ^terms-and-conditions.php$ index.php?cmd=termsAndConditions [L]
RewriteRule ^privacy-policy.php$ index.php?cmd=privacyPolicy [L]
RewriteRule ^contact-us.php$ index.php?cmd=contactUs [L]
RewriteRule ^sitemap.php$ index.php?cmd=siteMap [L]
RewriteRule ^members/login.php$ login.php [L]
RewriteRule ^rss$ index.php?cmd=rss [L]
RewriteRule ^resources.php$ index.php?cmd=links [L]
RewriteRule ^links.php$ /resources.php [R=301]
RewriteRule ^resources2.php$ index.php?cmd=links2 [L]
RewriteRule ^results.php$ index.php?cmd=results [L]
RewriteRule ^residential-properties((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=1&page=$3 [L]
RewriteRule ^commercial-properties((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=2&page=$3 [L]
RewriteRule ^short-term-rentals((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=1&str=1&page=$3 [L]
RewriteRule ^buy((/)([0-9]+))?$ index.php?cmd=divisionsRoot&group=2&page=$3 [L]
RewriteRule ^rent((/)([0-9]+))?$ index.php?cmd=divisionsRoot&group=1&page=$3 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&str=1&sort=$2&page=$5 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&str=1&sort=$4&page=$7 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&str=1&sort=$5&page=$8 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&str=1&page=$7 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&str=1&page=$6 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&str=1&page=$4 [L]
RewriteRule ^rent/residential-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=1&page=$8 [L]
RewriteRule ^buy/residential-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=2&page=$8 [L]
RewriteRule ^lease/commercial-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=1&page=$8 [L]
RewriteRule ^buy/commercial-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=2&page=$8 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&sort=$2&group=1&page=$5 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&sort=$2&group=2&page=$5 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&sort=$2&group=1&page=$5 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&sort=$2&group=2&page=$5 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&group=1&page=$7 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&group=2&page=$7 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&group=1&page=$7 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&group=2&page=$7 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&group=2&page=$4 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&group=2&page=$4 [L]
RewriteRule ^rent/residential-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2®ion=$3&city=$4&area=$5&category=$1&group=1&page=$8 [L]
RewriteRule ^buy/residential-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2®ion=$3&city=$4&area=$5&category=$1&group=2&page=$8 [L]
RewriteRule ^lease/commercial-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2®ion=$3&city=$4&area=$5&category=$1&group=1&page=$8 [L]
RewriteRule ^buy/commercial-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2®ion=$3&city=$4&area=$5&category=$1&group=2&page=$8 [L]
RewriteRule ^rent/residential-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2&category=$1&group=1&page=$5 [L]
RewriteRule ^buy/residential-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2&category=$1&group=2&page=$5 [L]
RewriteRule ^lease/commercial-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2&category=$1&group=1&page=$5 [L]
RewriteRule ^buy/commercial-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2&category=$1&group=2&page=$5 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()0-9_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&area=$5&category=$1&sort=$6&page=$9 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()0-9_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&area=$5&category=$1&sort=$6&page=$9 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&category=$1&sort=$5&page=$8 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&category=$1&sort=$5&page=$8 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&area=$5&category=$1&page=$8 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&area=$5&category=$1&page=$8 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&category=$1&page=$7 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&category=$1&page=$7 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2&category=$1&sort=$3&page=$6 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2&category=$1&sort=$3&page=$6 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2&category=$1&page=$5 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2&category=$1&page=$5 [L]
RewriteRule ^rent/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain_categories&mainDivision=1&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain_categories&mainDivision=1&country=$1&group=2&page=$4 [L]0 -
Dealing with internal pages with bad backlinks - is this approach OK?
Hi all, I've just been going through every page of my company website, and found a couple of internal pages with nasty backlinks/profiles. There are a significant number of article marketing and rubbish directory pages pointing to these internal pages. These internal pages have low PR, yet are performing well in terms of SERPs. I was planning to: (1) change URLs - removing current (soon to be former) URLs from Google via Webmaster Tools. Then (2) remove website's 404 for a while so nasty links aren't coming anywhere near the website (hopefully nasty links will fail to find website and broken links will result in link removal - that's my thinking anyway). PS. I am not planning to implement any kind of redirect from the old URLs. Does this sound like a sensible approach, or may there be problems with it? Thanks in advance, Luke
White Hat / Black Hat SEO | | McTaggart0 -
Being penalized for unnatural links, determining the issue, and fixing the problem. What to do?
Hi all, A client has been penalised, having received the message in Google Webmasters last week, along with two more yesterday. It seems the penalty is for something specific: “As a result, for this specific incident we are taking very targeted action on the unnatural links instead of your site as a whole“. This is the first time I've had to deal with this so I'll be a bit layman about it The penalty, firstly, seems to be for the old domain, from which there is a re-direct to the current one. This redirect has been in place since Feb 2012 (no link building has been done for the old domain since then). In Webmasters, I have the old and new domains set up separately and the messages are only coming for the old (but affecting the new, obviously). I need to determine if it’s the old or new URL I’m being hit for, or would that even matter? Some questionable links I can see in WM: There is an affiliate for whom WM is showing 154,000 links (all followed) from their individual products listings to the client’s site (as a related product) but they’re linking to the new domain if that matters. Could this affiliate be an issue? There is also Updowner, which has added 2000+ links unbeknownst to me but apparently they are discounted by Google. I see a ton of recent directory submissions - right up until last week - that I am not responsible for. Could that be intentional spam targeting? I did also use a 3<sup>rd</sup> party link building company for Feb, March and April who ‘manually’ submitted the new domain to directories and social bookmarking sites. Could this be issue? For what kind of time-scale are penalties usually imposed - how far back (or how recently) are they penalising for? Ranking were going really well until this happened last Thursday. Will directories with non-followed links effect us negatively - one such one has over 2000 links. What is the most conclusive way to determine which are the poor, penalty-incurring links pointing to us? I know I now have to contact all the dodgy directories the site is now listed on to get links removed, but any and all advice on how to rectify this, along with determining what had gone wrong, will be most appreciated. Cheers, David
White Hat / Black Hat SEO | | Martin_S0 -
Shadow Page for Flash Experience
Hello. I am curious to better understand what I've been told are "shadow pages" for Flash experiences. So for example, go here:
White Hat / Black Hat SEO | | mozcrush
http://instoresnow.walmart.com/Kraft.aspx#/home View the page as Googlebot and you'll see an HTML page. It is completely different than the Flash page. 1. Is this ok?
2. If I make my shadow page mirror the Flash page, can I put links in it that lead the user to the same places that the Flash experience does?
3. Can I put "Pinterest" Pin-able images in my shadow page?
3. Can a create a shadow page for a video that has the transcript in it? Is this the same as closed captioning? Thanks so much in advance, -GoogleCrush0