Robots.txt file issues on Shopify server
-
We have repeated issues with one of our ecommerce sites not being crawled. We receive the following message:
Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
Are you aware of an issue with robots.txt on the Shopify servers? It is happening at least twice a month so it is quite an issue.
-
I leave you 2 url, but it seems that the robots do not let you modify it.
They propose you to put pages as non-index....
https://community.shopify.com/c/Shopify-Discussion/How-can-I-edit-robots-txt/td-p/432485/page/4
https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines
greetings
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz is showing old results for site craw issues
Moz is showing old results for site craw issues, the ondemand craw works but I can't fix these issues without knowing what causing them.
Moz Pro | | paulwildweb0 -
I am having a duplicate title -pagination issue Need Help
In WordPress I added %%page%% saved changes and ran a test in Moz and screaming frog, it doesn't show any duplicates. But when I open the site up in a new browser the duplicate titles are still there. No access to the PHP file due to the theme the client chose also. Any suggestions anyone?
Moz Pro | | Strateguyz0 -
Site explorer Issue
Hello, I'm looking to see in the Site Explorer the links coming from directories such as BOW, yahoo etc. I'm listed there from almost 1 year and these links are not listed, the same with my competitors. I'm missing something? Thank you Claudio
Moz Pro | | SharewarePros0 -
Broken CSV files?
Hi, I have been downloading fiels from seoMOZ for quite a while and I keep getting broken CSV files. ON my Mac, I wrote a script that cleaned them out but I cannot (don't know how) do this on the Windows side. The problem is that some of the record lines get broken up mainly in the "Anchor Text" field (maybe the anchor text in question contained a return). This of course messes up my file when I open it in Excel. Is there a way get/open these files without having them messed up like this? P..S. I am also seeing problem with text encoding, I would like to see the anchor text that is being diaplayed on the referring site rather than its HTML/Unicode/UTF-8 code value. TIA, Michel
Moz Pro | | analysestc0 -
Google Analytic Caching Issue?
Two weeks ago, I started testing a campaign and linked it to my Google analytics account. At the time there were 4 domains linked to the GA account. And I could see all 4 domains at the drop down menu that asked me to choose one to connect to. One week ago, I added a 5 more domains to GA. When I log directly into GA, I see the total of all 9 domains, and all are verified and collecting data. Today at pro seomoz I started a new campaign, for one of the new domains recently added to GA. When I try to connect the campaign to GA, only the ORIGINAL 4 domains are visible on the drop down menu. None of the 5 new domains are visible. So I am unable to connect to the new domain. At campaign settings, I disconnected from GA, and connected again, but could not see the new domains on the drop down menu. I tried to use both FF and IE. I can only assume that this is a caching issue at pro.seomoz. If so, how to flush cache and get current again. Please advise
Moz Pro | | microonae0 -
SEOmoz tool Issue?
Hi Mozzers, I am doing a web maintenance task for a client and it's been weeks that Moz is detecting 49 duplicate pages ( contact page). I thought resolving the issue when creating the xml sitemap and excluding those duplicates. The moz tool would still detect them, so I went in making a search with some of these duplicate to check if they were indexed but non of them were indexed. So my question is has anyone recently experienced similar issues? Is the moz tool not 100% accurate? Thanks for sharing your thoughts and answers
Moz Pro | | Ideas-Money-Art0 -
How can I submit my sitemap file to SEOmoz? (I mean the same file I send to google and bing, not the sitemap page)
I'm new to SEOmoz. As far as I understand, SEOmoz goes out to my site and crawls it to find all the pages. Is there anyway I can submit my sitemap page to SEOmoz just like I submit it to Google or Bing? Thanks for your help! Fernando
Moz Pro | | fsantospt0 -
Does Rogerbot respect the robots.txt file for wildcards?
Hi All, Our robots.txt file has wildcards in it, which Googlebot recognizes. Can anyone tell me whether or not Rogerbot recognizes wildcards in the robots.txt file? We've done a Rogerbot site crawl since updating the robots.txt file and the pages that are set to disallow using the wildcards are still showing. BTW, Googlebot is not crawling these pages according to Webmaster Tools. Thanks in advance, Robert
Moz Pro | | AC_Pro0