Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
SEO Best Practices regarding Robots.txt disallow
-
I cannot find hard and fast direction about the following issue:
It looks like the Robots.txt file on my server has been set up to disallow "account" and "search" pages within my site, so I am receiving warnings from the Google Search console that URLs are being blocked by Robots.txt. (Disallow: /Account/ and Disallow: /?search=). Do you recommend unblocking these URLs?
I'm getting a warning that over 18,000 Urls are blocked by robots.txt. ("Sitemap contains urls which are blocked by robots.txt"). Seems that I wouldn't want that many urls blocked. ?
Thank you!!
-
mmm it depends.
it's really hard for me to answer without knowing your site but I would say that you're in the good direction. You want to provide google more ways to reach your quality content.
Now do you have any other page that is bringing bots there via a normal user navigation or is it all search driven?
While google can crawl pages that discovered via internal/external links it can't reproduce searches by typing in your nav bar, so I doubt those pages should be extremely valuable unless you link to them somehow. In that case you may want to keep google crawling them.
A different thing would be if you want to "index" them, as being searches they are probably aggregating different information already present on the site. For indexation purposes you may want to keep them out of the index while still allowing the bot to run through them.
Again beware of the crawl budget, you don't want google to be wandering around millions of search results instead of your money pages, unless you're able to let them crawl only a sub portion of that.
I hope this made sense
-
Thank you for your response! I'm going to do a bit more research but I think I will disallow "account", but unblock "search". The search feature on my site pulls up quality content, so seems like I would want that to be crawled. Does this sound logical to you?
-
That could be completely normal. Google sends a warning because you're giving conflicting directions as you are preventing them to crawl pages (via robots) you asked them to index (via sitemap).
They do not know how important those pages may be for you so you are the one that needs to assess what to do net.
Are those pages important for you? Do you want them to be in the index? if that's the case change your robots.txt rule, if not then remove them from the sitemap.
About the previous answer robots text is not used to block hackers but quite the opposite. Hackers can easily find via the robots txt which are the pages you'd like to block and visit them as they may be key pages (ex. wp-admin), but let's not focus on that as hackers have so many ways to find core pages that it's not the topic. Robots txt is normally used to avoid duplication issues and to prevent google from crawling low value pages and waste crawl budget.
-
Typically, you only want robots.txt to block access points that would allow hackers into your site like an admin page (e.g. www.examplesite.com/admin/). You definitely don't want it blocking your whole site. A developer or webmaster would be better at speaking to the specifics, but that's the quick, high-level answer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it best practice to have a canonical tags on all pages
The website I'm working on has no canonical tags. There is duplicate content so rel=canonicals need adding to certain pages but is it best practice to have a tag on every page ?
Intermediate & Advanced SEO | | ColesNathan0 -
What does Disallow: /french-wines/?* actually do - robots.txt
Hello Mozzers - Just wondering what this robots.txt instruction means: Disallow: /french-wines/?* Does it stop Googlebot crawling and indexing URLs in that "French Wines" folder - specifically the URLs that include a question mark? Would it stop the crawling of deeper folders - e.g. /french-wines/rhone-region/ that include a question mark in their URL? I think this has been done to block URLs containing query strings. Thanks, Luke
Intermediate & Advanced SEO | | McTaggart0 -
Mega Menu Navigation Best Practice
First off, I'm a landscape/nature/travel photographer. I mainly sell prints of my work. I'm in the process of redesigning my website, and I'm trying to decide whether to keep the navigation extremely simple or leave the drop-down menu for galleries. Currently, my navigation is something like this: Galleries
Intermediate & Advanced SEO | | shannmg1
> Gallery for State or Country (example: California)
> Sub-region in State or Country (example: San Francisco)
Blog
Prints
About
Contact Selling prints is the top priority of the website, as that's what runs the business. I have lots of blog content, and I'm starting to build some good travel advice, etc. but in reality, the galleries, which then filter down to individual pages for each photo with a cart system, are the most important. What I'm struggling to decide is whether to leave the sort of "mega menu" for the galleries, or to do away with them, and have the user go to the overall galleries page to navigate further into the site. Leaving the mega menu intact, the galleries page becomes a lot less important, and takes out a step to get to the shopping cart. However, I'm wondering if the amount of galleries in the drop down menu is giving TOO many choices up front as well. I also wonder how changing this will affect search. Any thoughts on which is better or is it really just a matter of preference?0 -
"noindex, follow" or "robots.txt" for thin content pages
Does anyone have any testing evidence what is better to use for pages with thin content, yet important pages to keep on a website? I am referring to content shared across multiple websites (such as e-commerce, real estate etc). Imagine a website with 300 high quality pages indexed and 5,000 thin product type pages, which are pages that would not generate relevant search traffic. Question goes: Does the interlinking value achieved by "noindex, follow" outweigh the negative of Google having to crawl all those "noindex" pages? With robots.txt one has Google's crawling focus on just the important pages that are indexed and that may give ranking a boost. Any experiments with insight to this would be great. I do get the story about "make the pages unique", "get customer reviews and comments" etc....but the above question is the important question here.
Intermediate & Advanced SEO | | khi50 -
Domain Alias SEO
We have 5 domain alias of our existing sites
Intermediate & Advanced SEO | | unibiz
All 5 domain alias are domain alias of our main site. It means, all domain alias will have exactly same site and contents
Like Main domain: www.mywebsite.com
DomainAlias: www.myproduct.com, www.myproduct2.com, www.myproduc3.com
And if anybody will open our site www.myproduct.com, it will open same website which I have in primary site what can i do to rank all website without any penalty....i s there any way? This is domain alias of in hosting industry Thanks0 -
If you have an unlimited SEO budget, what would you do?
Here's a bit of background information: I've achieved the targets and is now being offered what is essentially an unlimited budget. I have a nice list of ideas but thought I would the brilliant people here at the SEOMOZ community what they would do. So as to promote as much response as possible, I'm going to keep my list to myself for now. And by "SEO", I mean I can do things like content strategy, blogging, infographics, etc. Shoot away!
Intermediate & Advanced SEO | | andrep0 -
Migrating online store to subdomain using shopify and effects on seo and energy down the road for seo
I'm looking for some clarity... Looking at using Shopify for an existing online store that we have to migrate. Setting up the store with shopify means we will be using a subdomain such as shop.mywebsite.com instead of mywebsite.com/shop. The following are points to consider when responding The client currently has an online store, however it's a proprietary shopping store and CMS that has since gone defunct and they need to migrate to an alternative in order to survive online against new CMS systems that allow the site and its content to be better optimized. There is a lot of existing SEO done on the current site that we don't want to loose PR on. There is roughly 2000 products Client has a fixed budget, dealing with checkout issues, custom work and various other "bugs" seems to be easier controlled with Shopify...thus budget can be used more on content/strategy and migration We want to run the main site in Wordpress and are wanting to use Shopify since it supports a gateway, has great features and seems like it would allow us to get more bang for the buck and can focus more on the main site and content strategy and drive traffic to the subdomain store if needed Or main concern is the effort of migrating 2000+ products to shopify and the traffic and PR it gives the current site will have a negative effect on the main domain itself. Should we really be considering this path? The domain is diveidc.com One main benefit to the subdomain is the ability to clearly segment products from the service portion of the site in the analytics and focus 2 clear strategies and track it in a very defined manner. We're really on the fence with this...any thoughts are welcome.
Intermediate & Advanced SEO | | MAGNUMCreative0 -
Does capitalization matter for SEO?
Two places capitalization comes into play: (1) on-page use (title, h1, body text, img alt text, etc) (2) external anchor text I didn't think it mattered from Google's point of view for on-page usage (is this correct?) but I notice that OpenSiteExplorer' s 'anchor text distribution' tab shows different counts for the same keyword if it's capitalized in different ways (eg seomoz.org is listed separate from SEOmoz.org). Is that just OSE or does Google treat the keyword/phrase different based on its capitalization, too? And if so, then should I be creating external links to my site with the 'regular' and 'Capitalized' versions of my key phrases?
Intermediate & Advanced SEO | | scanlin1