Have a Robots.txt Issue
-
I have a robots.txt file error that is causing me loads of headaches and is making my website fall off the SE grid. on MOZ and other sites its saying that I blocked all websites from finding it. Could it be as simple as I created a new website and forgot to re-create a robots.txt file for the new site or it was trying to find the old one? I just created a new one.
Google's website still shows in the search console that there are severe health issues found in the property and that it is the robots.txt is blocking important pages. Does this take time to refresh? Is there something I'm missing that someone here in the MOZ community could help me with?
-
Hi primemediaconsultants!
Did this get cleared up?
-
You not always have to do this, if you would go to domain.com/robots.txt then it should be removed maybe already. If that's the case you should be starting to see an increase in the number of pages crawled in Google Search Console.
-
This seems very helpful as I did remove it, and fetch as google, but i'm a complete novice. How do you clear server cache?
-
What does your robots.txt file contain? (or share the link)
Try removing it, clearing server cache and fetching as google again.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Shopify Website Page Indexing issue
Hi, I am working on an eCommerce website on Shopify.
Intermediate & Advanced SEO | | Bhisshaun
When I tried Indexing my newly created service pages. The pages are not getting indexed on Google.
I also tried manual indexing of each page and submitted a sitemap but still, the issue doesn't seem to be resolved. Thanks0 -
Site moved. Unable to index page : Noindex detected in robots meta tag?!
Hope someone can shed some light on this: We moved our smaller site (into the main site ( different domains) . The smaller site that was moved ( https://www.bluegreenrentals.com)
Intermediate & Advanced SEO | | bgvsiteadmin
Directory where the site was moved (https://www.bluegreenvacations.com/rentals) Each page from the old site was 301 redirected to the appropriate page under .com/rentals. But we are seeing a significant drop in rankings and traffic., as I am unable to request a change of address in Google search console (a separate issue that I can elaborate on). Lots of (301 redirect) new destination pages are not indexed. When Inspected, I got a message : Indexing allowed? No: 'index' detected in 'robots' meta tagAll pages are set as Index/follow and there are no restrictions in robots.txtHere is an example URL :https://www.bluegreenvacations.com/rentals/resorts/colorado/innsbruck-aspen/Can someone take a look and share an opinion on this issue?Thank you!0 -
Issues with the apperance of cross country sitelinks
Hi Moz community, My questions is related to the international SEO, esepecially sitelinks. The problem is that the users from US see in the search results the sitelinks which comes from different countries, e.g. users from US see the sitelinks from Australia or the sitelinks from our international website, which has obviously no specification. I must say, that we've done everything to be in accordance with Google interantional SEO recommendations, hraflang & lang attributes, properly set location in GSC. All of these were tripplechecked. I also need to say, that it happens only to the websites that include content written in English nad French. All other branches show proper sitelinks. It think Google can't properly locate the content, if the language is the same regardless country. Previously it could be solved with disavow tool, but today's I don't know about any manual action that could deal with the issue. I also noticed that some other pages are affected with the same issue. To better understand the issue, please see the image link. The image shows the results from US. Despite the location, it shows sitelinks form UK or International website. Do you have similar experience? I will be thankful for any help. 1NNtJ
Intermediate & Advanced SEO | | eset0 -
Redirect issue launching duplicate product categories on another TLD
Dear Mozzerz We run this e-commerce website (superstar.dk) where we are selling all different kinds of wristwatches from different brand names (Casio, Garmin, Suunto etc). We just bought another website selling watches (xxx.com) and therefore we would like to move some of the content from superstar.dk to the new website xxx.com, making superstar.dk into a more niche website. So we are basically taking a brand with all the products in it and shutting it down on superstar.dk and instead launching it on xxx.com. Superstar.dk will still be running, just with a more niche product- and brand selection. So my question is, should we redirect all the old product categories that we are shutting down to the new website on another TLD where we are opening them again and the same for the products (e.g. superstar.dk/garmin -> xxx.com/garmin)? Or would it be better to keep the redirects within the same website/TLD (e.g. superstar.dk/garmin -> superstar.dk)? A few examples:
Intermediate & Advanced SEO | | superstardenmark
superstar.dk/garmin -> xxx.com/garmin
superstar.dk/suunto -> xxx.com/suunto
etc..
superstar.dk/product1 -> xxx.com/product1
superstar.dk/product2 -> xxx.com/product2
etc.0 -
Massive URL blockage by robots.txt
Hello people, In May there has been a dramatic increase in blocked URLs by robots.txt, even though we don't have so many URLs or crawl errors. You can view the attachment to see how it went up. The thing is the company hasn't touched the text file since 2012. What might be causing the problem? Can this result any penalties? Can indexation be lowered because of this? ?di=1113766463681
Intermediate & Advanced SEO | | moneywise_test0 -
Blocking out specific URLs with robots.txt
I've been trying to block out a few URLs using robots.txt, but I can't seem to get the specific one I'm trying to block. Here is an example. I'm trying to block something.com/cats but not block something.com/cats-and-dogs It seems if it setup my robots.txt as so.. Disallow: /cats It's blocking both urls. When I crawl the site with screaming flog, that Disallow is causing both urls to be blocked. How can I set up my robots.txt to specifically block /cats? I thought it was by doing it the way I was, but that doesn't seem to solve it. Any help is much appreciated, thanks in advance.
Intermediate & Advanced SEO | | Whebb0 -
Issues in Migrating to CMS
My number one client is migrating a formerly HTML/Dreamweaver site to an open source CMS (CMS Made Simple.) We have execellent rankings right now, and I am concerned about what we will lose, and how to ensure the rankings stay. Any guidance? I have already asked the developer to maintain the page names and structure, and tag the CMS output pages.htm . I've run a few weak spiders over the staged CMS site, and all the pages are being picked up. What else can I do, we are getting ready to launch. THANKS!
Intermediate & Advanced SEO | | jayt0 -
How to resolve Duplicate Page Content issue for root domain & index.html?
SEOMoz returns a Duplicate Page Content error for a website's index page, with both domain.com and domain.com/index.html isted seperately. We had a rewrite in the htacess file, but for some reason this has not had an impact and we have since removed it. What's the best way (in an HTML website) to ensure all index.html links are automatically redirected to the root domain and these aren't seen as two separate pages?
Intermediate & Advanced SEO | | ContentWriterMicky0