Robots.txt file issues on Shopify server
-
We have repeated issues with one of our ecommerce sites not being crawled. We receive the following message:
Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
Are you aware of an issue with robots.txt on the Shopify servers? It is happening at least twice a month so it is quite an issue.
-
I leave you 2 url, but it seems that the robots do not let you modify it.
They propose you to put pages as non-index....
https://community.shopify.com/c/Shopify-Discussion/How-can-I-edit-robots-txt/td-p/432485/page/4
https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines
greetings
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz is showing old results for site craw issues
Moz is showing old results for site craw issues, the ondemand craw works but I can't fix these issues without knowing what causing them.
Moz Pro | | paulwildweb0 -
Solving URL Too Long Issues
Moz.com is reporting that many URL's are to long, these particularly affect Product URL's where the URL is typically https://www.domainname.com/collections/category-name/products/product-name, (You guessed it we're using Shopify). However, we use Canonicals that ignore all most of the URL and are just structured https://www.domainname.com/product-name, so Google should be reading the Canonical and not the long-winded version. However, Moz cannot seem to spot this... does anyone else have this problem and how to solve so that we can satisfy the Moz.com crawl engine?
Moz Pro | | Tildenet0 -
CSV file messed up
I cannot convert my exported CSV file to a proper excel sheet. The data is mixed, so converting doesn't work. Some rows have all data in the first cell (column), some rows have data in first AND second cell.. Anyone a solution? yaIKsZz
Moz Pro | | nans2 -
Does SeoMoz realize about duplicated url blocked in robot.txt?
Hi there: Just a newby question... I found some duplicated url in the "SEOmoz Crawl diagnostic reports" that should not be there. They are intended to be blocked by the web robot.txt file. Here is an example url (joomla + virtuemart structure): http://www.domain.com/component/users/?view=registration and the here is the blocking content in the robots.txt file User-agent: * _ Disallow: /components/_ Question is: Will this kind of duplicated url errors be removed from the error list automatically in the future? Should I remember what errors should not really be in the error list? What is the best way to handle this kind of errors? Thanks and best regards Franky
Moz Pro | | Viada0 -
Google Analytic Caching Issue?
Two weeks ago, I started testing a campaign and linked it to my Google analytics account. At the time there were 4 domains linked to the GA account. And I could see all 4 domains at the drop down menu that asked me to choose one to connect to. One week ago, I added a 5 more domains to GA. When I log directly into GA, I see the total of all 9 domains, and all are verified and collecting data. Today at pro seomoz I started a new campaign, for one of the new domains recently added to GA. When I try to connect the campaign to GA, only the ORIGINAL 4 domains are visible on the drop down menu. None of the 5 new domains are visible. So I am unable to connect to the new domain. At campaign settings, I disconnected from GA, and connected again, but could not see the new domains on the drop down menu. I tried to use both FF and IE. I can only assume that this is a caching issue at pro.seomoz. If so, how to flush cache and get current again. Please advise
Moz Pro | | microonae0 -
File permissions save. any suggestions
Hi - something not related directly to SEO - but a vital factor Need to know for any tool - by which i can take backup or save existing file permissions of all my web site files.. God forbid - in event of any unfortunate circumstance if some mis hap happens or file permissions changed accidentally - how can i get them back to the original. Any tool by which i can save existing file permissions of filezilla thanks
Moz Pro | | Modi0 -
Open Site Explorer Issue - Pullng up No-follow links when settings ask for Follow + 301 Redirect..
Anybody else having this issue? Here lately when I am doing competitive research on open site explorer I set it to only pull up followed + 301 redirects and it will still pull up no-follow competitors links. Can anybody help me out here?
Moz Pro | | axzm0 -
Rel Canonical issues for two urls sharing same IP address
Our client built a wordpress site on url A, then opted for a better url B. Rather than moving all the wordpress files/website over to the new url B, they just contacted GoDaddy, who hosted BOTH urls under the same IP address. When I do a term target on url B, I'm flagged for rel canonical use. I can only get a B grade for each keyword. (I've also tried using url A, but I get the same flag and B grade results). I'm not sure if this set-up will thwart our seo efforts for the site, because only the homepage comes up when you type in url B anyway. Every subsequent page displays the original url A. Somewhere, wordpress is also adding a rel canonical link on the homepage source to url A, too, which we can't seem to edit. So, question is: is it ok to leave this set up as is with both urls hosted on the same IP address, or should we move the whole site over to the desired url B? Thanks much!
Moz Pro | | GravitateOnline0