Does using robots.txt to block pages decrease search traffic?
-
I know you can use robots.txt to tell search engines not to spend their resources crawling certain pages.
So, if you have a section of your website that is good content, but is never updated, and you want the search engines to index new content faster, would it work to block the good, un-changed content with robots.txt? Would this content loose any search traffic if it were blocked by robots.txt? Does anyone have any available case studies?
-
If you block the pages from being crawled, you are also telling the search engines to not index the pages (they don't want to include something they haven't looked at). So yes, the traffic numbers from organic search will change if you block the pages in robots.txt.
-
Agreed, that is a better solution, but, I am still wondering if you block something with robots.txt, will that lead to a decrease in traffic? What if we have some duplicate content that is highly trafficked, if we block it with robots.txt, will the traffic numbers change?
-
You certainly don't want to block this content!
One thing I'd consider is the if-modified-since header, or other headers. Here are two articles that explain more about the concept of using headers to tell the search engines " this hasn't changed, don't bother crawling it". I haven't personally used this, but have read about it in many places.
http://www.feedthebot.com/ifmodified.html
http://searchengineland.com/how-to-improve-crawl-efficiency-with-cache-control-headers-88824
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Archive pages structure using a unique hierarchical taxonomy, could be good for SEO?
Hi, Preamble:
Intermediate & Advanced SEO | | danielecelsa
We are creating a website where people look for professionals for some home working. We want to create a homepage with a search bar where people write the profession/category (actually it is a custom taxonomy) that they need, like ‘plumbers’, and a dropdown/checkbox filter where they can choose the city where they need the plumber.
The result page is a list of plumber agencies in the city chosen. Each agency is a Custom Post Type for us. Furthermore, we are hardly working to make our SEO ranking as high as possible.
So, for example, we know that it is important to have a well-done Archive Page for each Taxonomy term, besides a well-done Results Page.
Also, we know it is bad for SEO to have duplicated pages or (maybe) similar pages, ranking for the same (or maybe also similar) keywords. Proposed Structure:
So, what we are thinking is to have this structure:
A unique hierarchical taxonomy that INCLUDES the City AND the profession! That means that our taxonomy ‘taxonomy_unique’ has terms like: ‘Rome’, ‘Paris’, ‘Dublin’ as father and also terms like ‘Plumbers’, ‘Gardeners’, ‘Electricians’ which are sons of some City father! So we will have the term 'Plumbers' son of 'Rome' and we will have also the term 'Plumbers' son of 'Paris'. Each of these two taxonomy terms (Rome/Plumbers and Paris/Plumbers) will have an archive page that we want to make ranking for the keywords ‘Plumbers in Rome’ and ‘Plumbers in Paris’ respectively. It is easier to think of it imagining the breadcrumbs. They will be:
Home > Rome > Plumbers
and
Home > Paris > Plumbers Both will have: a static content (important for SEO), where we describe the plumber profession with a focus on the city, like ‘Find the best Plumbers in Rome’ vs ‘Find the best Plumbers in Paris' a 'dynamic' content - below - that is a list of Custom Post Types which have that taxonomy term associated. Furthermore, also 'Rome' and 'Paris' are taxonomy terms that have their own archive page. In those pages, we are thinking to show the Custom Post Types (agencies) associated with that taxonomy term as a father OR maybe just a list of the 'sons' of that father, so links to those archive pages 'sons').
In both cases, there should be also a static content talking maybe about the city and the professionals it offers in general. Questions:
So what we would like to understand is: Is it bad from an SEO perspective to have 2 URLs that look like this:
www.mysite.com/Rome/Plumbers
and
www.mysite.com/Naples/Plumbers
where the static content is really similar and it is something like that:
“Are you looking for the best plumbers in the city of Rome”
and
“Are you looking for the best plumbers in the city of Naples”? Also, these kinds of pages will be much more than 2, one for each City.
We are doing that because we want the two different pages to rank high in two different cities, but we are not sure if Google likes that. On the other hand, each City will have one page for each kind of job, so:
www.mysite.com/Rome/Plumbers
www.mysite.com/Rome/Gardeners
www.mysite.com/Rome/Electricians
So the same question, does Google like this or not? About 'Rome' and 'Paris' archive pages, does Google prefer a list of Custom Post Types that have that father term associated as taxonomy, or a list of the archive pages 'sons', with links to those pages? What do you think about this approach? Do you think this structure could be good from an SEO perspective, or maybe there could be something better alternatively? Hoping everything is clear, we really appreciate anyone dedicating its time and leaving feedback.
Daniele0 -
Google Manual Penalty Lifted - Why is my website still decreasing on traffic?
Hi there, I was hoping that somebody has a potential answer to this or if anyone else has experienced this issue. Our website has recently hit by a manual penalty (structured data wasn't matching the content on the page) After working hard on this to fix the issue across the site, we submitted a reconsideration request which was approved by Google a few days later. I understand that not all websites recover and it doesn't guarantee rankings will go back to normal, but it seems as if the traffic is continuing to drop at an even quicker rate. There's a number of small technical optimisations that have been briefed into the dev team such as: Redirecting duplicate versions, fixing redirects on internal links, There's also work on-page running in the background fixing up keyword cannibalization, consolidating content keyword mapping and ensuring the internal link structure is sound. Has this happened to anyone else before? If so, how did you recover? Any suggestions/advice would be really appreciated. Thank you
Intermediate & Advanced SEO | | dbutler9120 -
301 Redirect to Home Page or Sub-Page?
What do you think about 301 redirect of good expired domain to a sub-page instead of the home page? I'm doing this so I don't hurt my brand name. Let me know your thoughts please. Thank you
Intermediate & Advanced SEO | | JuanWork0 -
Search engine blocked by robots-crawl error by moz & GWT
Hello Everyone,. For My Site I am Getting Error Code 605: Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag, Also google Webmaster Also not able to fetch my site, tajsigma.com is my site Any expert Can Help please, Thanx
Intermediate & Advanced SEO | | falguniinnovative0 -
How to make Google index your site? (Blocked with robots.txt for a long time)
The problem is the for the long time we had a website m.imones.lt but it was blocked with robots.txt.
Intermediate & Advanced SEO | | FCRMediaLietuva
But after a long time we want Google to index it. We unblocked it 1 week or 8 days ago. But Google still does not recognize it. I type site:m.imones.lt and it says it is still blocked with robots.txt What should be the process to make Google crawl this mobile version faster? Thanks!0 -
Mobile Search Results Include Pages Meant Only for Desktops/Laptops
When I put in site:www.qjamba.com on a mobile device it comes back with some of my mobile-friendly pages for that site(same url for mobile and desktop-just different formatting), and that's great. HOWEVER, it also shows a whole bunch of the pages (not identified by Google as mobile-friendly) that are fine for desktop users but are not supposed to exist for the mobile users, because they are too slow. Until a few days ago those pages were being redirected for mobile users to the home page. I since have changed that to 404 not founds. Do we know that Google keeps a mobile index separate from the desktop index? If so, I would think that 404 should work.. How can I test whether the 404 not founds will remove a url so they DON'T appear on a mobile device when I put in site:www.qjamba.com (or a user searches) but DO appear on a desktop for the same command.
Intermediate & Advanced SEO | | friendoffood0 -
Will Using Attributes For Landing Pages In Magento Dilute Page Rank?
Hello Mozzers! We have an ecommerce site built on Magento. We would like to use attribute filters in our layered navigation for landing page purposes. Each page will have a unique URL, Meta Title and Meta Description. For example: URL: domain.com/art/abstract (category is Art, attribute is Abstract) Title: Abstract Art For Sale Meta: Blah Blah Blah Currently these attribute pages are not being indexed by google as they are set in google parameters. We would like to edit google parameters to start indexing some of the attribute filters that users search for, so they can be used as landing pages. Does anyone have experience with this? Is this a good idea? What are the consequences? Will this dilute Page Rank? Could this destroy the world? Cheers! MozAddict
Intermediate & Advanced SEO | | MozAddict0 -
Will blocking urls in robots.txt void out any backlink benefits? - I'll explain...
Ok... So I add tracking parameters to some of my social media campaigns but block those parameters via robots.txt. This helps avoid duplicate content issues (Yes, I do also have correct canonical tags added)... but my question is -- Does this cause me to miss out on any backlink magic coming my way from these articles, posts or links? Example url: www.mysite.com/subject/?tracking-info-goes-here-1234 Canonical tag is: www.mysite.com/subject/ I'm blocking anything with "?tracking-info-goes-here" via robots.txt The url with the tracking info of course IS NOT indexed in Google but IT IS indexed without the tracking parameters. What are your thoughts? Should I nix the robots.txt stuff since I already have the canonical tag in place? Do you think I'm getting the backlink "juice" from all the links with the tracking parameter? What would you do? Why? Are you sure? 🙂
Intermediate & Advanced SEO | | AubieJon0