Multiple robots.txt files on server
-
Hi!
I have previously hired a developer to put up my site and noticed afterwards that he did not know much about SEO. This lead me to starting to learn myself and applying some changes step by step.
One of the things I am currently doing is inserting sitemap reference in robots.txt file (which was not there before). But just now when I wanted to upload the file via FTP to my server I found multiple ones - in different sizes - and I dont know what to do with them? Can I remove them? I have downloaded and opened them and they seem to be 2 textfiles and 2 dupplicates. Names:
robots.txt (original dupplicate)
robots.txt-Original (original)
robots.txt-NEW (other content)
robots.txt-Working (other content dupplicate)Would really appreciate help and expertise suggestions. Thanks!
-
So what's the best policy if a site uses an e-commerce platform like Magento, which has a robots file, but also has a Wordpress blog installed to another folder. eg: /blog and uses a plugin like YOAST which generated a robots file of the Wordpress installation.
Then you have 2 robots files, is this detrimental or no big deal?
-
Thanks very much for the help!
-
Thanks very much for the help!
-
Keep a backup and remove them.
Search engines are only going to look at the file which is exactly called robots.txt variations of file name will be ignored.
Do make sure the entries are correct in the main one though, you don't want Google crawling admin pages or other confidential areas of the site.
-
Hi, thanks for the answer and help!
Well, I only have one domain that has a webpage and no subdomains active (no blog-subdomain or similar) - so how can I configure that to the situation? Can I just remove all and upload the one I want, maybe?
-
That's a good question, EMS. The robots.txt protocol can get kind of
confusing when you think about it too long, and it sounds like you've
thought about this a bit. However, in this case, it might help to
look at robots.txt from the perspective of the spider.When a spider finds a URL, it takes the whole domain name (everything
between 'http://' and the next '/'), then sticks a '/robots.txt' on
the end of it and looks for that file. If that file exists, then the
spider should read it to see where it is allowed to crawl.In your case, Googlebot, or any other spider, should try to access
three URLs: domainA.com/robots.txt, domainB.domainA.com/robots.txt,
and domainB.com/robots.txt. The rules in each are treated as
separate, so disallowing robots from domainA.com/ should result in
domainA.com/ being removed from search results while
domainB.domainA.com/ remains unaffected, which does not sound like not
something you want.The problem you might have with the setup you have described is this--
in order to keep domainB.domainA.com out of the results, you would
need to have domainB.domainA.com/robots.txt exclude robots, while
domainB.com/robots.txt welcomes them. This means that you would need
to have a way to make domainB.domainA.com/ and domainB.com/ serve
different information, and judging from what you've described, you
have not set up your server to do so yet.Of course, it is always possible that I have assumed to much about
your situation, so it is a good idea to use Google's robots.txt
analysis tool (see http://www.google.com/support/webmasters/bin/topic.py?topic=8475
) to see if your robots.txt files already produce the results you
want.If using robots.txt files doesn't solve the problem, and assuming that
you want to continue hosting all of your content on domainA.com, one
strategy you really should look into would be setting up a 301
redirect from the pages on domainB.domainA.com/ to domainB.com/ . If
you need more advice on how to do this with your server software, your
hosting company's tech support would definitely be the best place to
start, but this group is here to help if more isues arise.Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Htaccess mod rewrite from server address
Hi there, I'm not massively experienced with creating mod rewrite rules and I'm worried I've got this wrong as its slightly different to what I'm used to. The web dev and content creators were working on http://5.10.105.45/~isea/ to create content. I want to redirect all URL's to www.iseasurfwear.co.uk. This is what I've written //Rewrite to www
Technical SEO | | BlueTree_Sean
Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} ^5.10.105.45/~isea/[nc]
RewriteRule ^(.*)$ http://www.iseasurfwear.co.uk/$1 [r=301,nc] Can anyone tell me if this is correct?0 -
Duplicate homepage content across multiple websites
Hi, I work for a company that runs 30+ membership based websites on separate domains and across multiple markets. The homepage for each site contains a section of content that highlights the site benefits and features. While each website serves a different market/niche, this section of content is essentially the same as each website offers the same benefits and features. What is the best way to avoid duplicate content issues while still being able to show the same section of content across 30+ sites? This particular section of content isn't valuable from an SEO perspective, but the rest of the content on that page is. Any ideas or suggestions would be much appreciated. Thanks
Technical SEO | | CupidTeam0 -
Could multiple languagues on one site be bad for SEO???
Our site is has content in English and in Spanish. The spanish side was translated by me, Spanish is my first language, so i know that the translations are good and its original content. We were Pandalized/Penguinnized pretty bad earlier this year. We have completely cleaned our site of anything that could be considered thin content or grey hat techniques. An associate is telling me that we need to put the spanish version of the site on its own domain, does this make sense to anyone? The spanish side of the site gets only about 5% of the visitors, bu i still don't see the logic in taking all those pages and putting them on a different domain. Would this help recover from Panda/Penguin. Thanks
Technical SEO | | 858-SEO0 -
Proper structure for site with multiple catagories of same products
Hi, we have products (trophies and awards) that can be catagorized in many ways. Using Award Medals as an example: - Medals by type: 1 1/2", 2", etc. - Medals by sport Baseball, Basketball, Cheer - Medals by Style Color, Gold, Silver, Bronze Right now, we have an Award Medals section off of our home page. The section has a decent page rank, but should be much better (I think). My guess is that we are loosing page range since we have separate sections with the groups above as we want our customers to be able to find the medals easily. Unfortunately, when we setup our site 10 years ago, we organized by type and this is what is hanging off the home page. The other groupings we added more recently. I have attached a snap shot of what the sections look like. We would like customers to find an individual medal when they do a Google search. For example a search for Baseball Medals. In Goggle, they likely would not search for 1 1/2" medals. My question is this: Can we keep the same structure we have today (to enable customer flexibility) but improve page rank and also have the sections like basball medals rank well? I have thought about using canonical tags, but the pages are not the same - in one case it is all baseball medals, in another it is all 1 1/2" medals, etc. Thanks for your help!!
Technical SEO | | trophycentraltrophiesandawards0 -
Internal search : rel=canonical vs noindex vs robots.txt
Hi everyone, I have a website with a lot of internal search results pages indexed. I'm not asking if they should be indexed or not, I know they should not according to Google's guidelines. And they make a bunch of duplicated pages so I want to solve this problem. The thing is, if I noindex them, the site is gonna lose a non-negligible chunk of traffic : nearly 13% according to google analytics !!! I thought of blocking them in robots.txt. This solution would not keep them out of the index. But the pages appearing in GG SERPS would then look empty (no title, no description), thus their CTR would plummet and I would lose a bit of traffic too... The last idea I had was to use a rel=canonical tag pointing to the original search page (that is empty, without results), but it would probably have the same effect as noindexing them, wouldn't it ? (never tried so I'm not sure of this) Of course I did some research on the subject, but each of my finding recommanded one of the 3 methods only ! One even recommanded noindex+robots.txt block which is stupid because the noindex would then be useless... Is there somebody who can tell me which option is the best to keep this traffic ? Thanks a million
Technical SEO | | JohannCR0 -
Multiple URLs in CMS - duplicate content issue?
So about a month ago, we finally ported our site over to a content management system called Umbraco. Overall, it's okay, and certainly better than what we had before (i.e. nothing - just static pages). However, I did discover a problem with the URL management within the system. We had a number of pages that existed as follows: sparkenergy.com/state/name However, they exist now within certain folders, like so: sparkenergy.com/about-us/service-map/name So we had an aliasing system set up whereby you could call the URL basically whatever you want, so that allowed us to retain the old URL structure. However, we have found that the alias does not override, but just adds another option to finding a page. Which means the same pages can open under at least two different URLs, such as http://www.sparkenergy.com/state/texas and http://www.sparkenergy.com/about-us/service-map/texas. I've tried pointing to the aliased URL in other parts of the site with the rel canonical tag, without success. How much of a problem is this with respect to duplicate content? Should we bite the bullet, remove the aliased URLs and do 301s to the new folder structure?
Technical SEO | | ufmedia0 -
Are multiple links devalued on the same domain?
I'm in negotiations to get links placed on a popular blog with good stats. I'm allowed to pick older posts on the site, and I get to pick the anchor text. Is it best practice to diversify the links by having different keywords pointing to different pages or am I better off pointing as many links as I can at one page (varying anchor text)? Also, is it best to pick a more recent blog post, or is it ok to pick one from say, 2009?
Technical SEO | | MichaelWeisbaum0 -
What is the sense of robots.txt?
Using robots.txt to prevent search engine from indexing the page is not a good idea. so what is the sense of robots.txt? just for attracting robots to crawl sitemap?
Technical SEO | | jallenyang0