Two Robots.txt files
-
Hi there
Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one.
Thanks in advance!
Cheers,
Satla -
Thank Riera
-
Hi Satia,
You mentioned that one robots.txt file placed in inside the server and search engine unable to see the file. If search engine won't see robots.txt file then what is the use of that robots.txt file?
AFAIK it must be placed under root directory an there is no way to keep two files with the same name. So you must have only one robots.txt and that should be placed under root directory.
Hope this helps.
Thanks
-
Hi Satla,
You're going to need to get rid of that 2nd version ASAP. The official standard for a robots.txt file is all lower case in the file name, so that's most likely what bots are seeing. But to err on the side of caution, I'd remove any possibilities of a "disallow: /" and remove that Robots.txt version.
Some servers are case sensitive, so you could run into issues here as well.
-
Hi,
I don't find any good reason why there are two files. There should be just one, where you specify everything you'd like to done.
If the tech team doesn't want to correct and leave just one file, may be because they are lazy or there might be some other issue that if they delete one file, the hole site blows up.Here, I leave you 2 Moz's articles about the robots.txt file.
What is Robots.txt? - Moz Learn
Learn About Robots.txt with Interactive Examples - Moz BlogTake into account that the name of the file must be in lower case. I've never seen any different and the servers are usually case sensitive with the filenames.
Hope its helpful.
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Two links to different page with same link label
What will be the impact in Google if I have two links in the same page pointing to different pages, but with the same label.
On-Page Optimization | | kjerstibakke0 -
How we can rank two keywords
Here is my example. I developed a property inspection app, where the keyword I try to rank is property inspection app, and property inspection software. Property Inspection Software got slightly more traffic than app. I would like to rank both of this keyword high for my main website, but there is only one title, one H1 tag I can use to have keyword, where if I use "App" there is no way to have "Software" on the title, etc. How can I deal with such a case? Any help will be appreciate.
On-Page Optimization | | terrysun0 -
One Company, Two Brands with Two Blogs, but One WP Panel for Blog?
I work with a company that has 2 brands. Both brands have separate sites (currently on a WP multisite install). We want each brand to have its own blog, but for ease of content creation have ONE wp install to create the blog content and depending on what category is clicked (Brand 1 and/or Brand 2), it will publish to that sites blog. 2 questions: 1. Is one WP install for blog syndication for 2 separate sites advisable (as client is requesting)? Or should we just bite the bullet and have each site have it's separate posting through it's own WP install? 2. Sometimes one blog post will be published to BOTH blogs (i.e. category Brand 1 and Brand 2 clicked OR if we use two separate wp installs for each site, publish to both blogs). Is using a rel=canonical for the original post (we need to decide which brand takes precedence) sufficient to overcome duplicate content problem? Thanks in advance! Stephan
On-Page Optimization | | stephanwb0 -
My company's product is referred to by two different names (SVN and Subversion). When cleaning up our Title tags, is it OK to use either name to keep the title tags around 70 characters?
I am cleaning up title tags that are too long or not correct. In our title tag we reference our product (a version of OSS source code). This product is often referred to as both SVN or Subversion. When writing Title tags is it OK to use one or the other depending on the length of the Title Tag? For instance: Contact Us | Free SVN & Git Hosting | Bug & Issue tracking | CloudForge vs **About CloudForge | Free Subversion & Git Hosting | Bug Tracking ** | |
On-Page Optimization | | CollabNet0 -
Optimizing for two keywords
Hi there, my question is: I need to optimze two pages related to almost the same concept but different orientations. One keyword is "emocional intelligence development" and the other word is "emocional intelligence workshops" I can't mix the two pages in one unique page due to web strucutre reasons. Is it possible to optimize each web for each keyword independently although they have the concept emotional intelligence in common? Thanks
On-Page Optimization | | juanmiguelcr0 -
Do I need a robots meta tag on the homepage of my site?
Is it recommended to include on the homepage of your site site? I would like Google to index and follow my site. I am using WordPress and noticed my homepage is not including this meta tag, therefore wondering if I should include it?
On-Page Optimization | | asc760 -
Question regarding two versions of a page (redirect)
At one point in my SEO journey I had noticed some duplicate issues going on in Google webmaster tools (I could have been reading it wrong.) It looked like WMT was saying I had 2 versions of some of pages...it looked like pages that ended in a '/' ranking differently than pages that have no slash at the end. So I downloaded the redirect plugin and redirected many non / pages to the page that end with the "/". This was probably a mistake. but I'm trying to clean all of that up. (I'm also noticing that SEOMoz tools tells me my site has many, many redirected url's. I'm not sure why. I was looking at http://www.opensiteexplorer.org/pages?site=noahsdad.com%2F this morning at my top pages and it shows 2 versions (one being redirected) of the page about my son who was born with Down syndrome. (Noahsdad.com/story and Noahsdad.com/story/) I don't know if I've built rankings for those pages separately and it would now hurt me if I did away with the redirect. I'm not sure why I did the redirects (I must have read something online somewhere.) I'm wondering if someone wouldn't mind looking at the report and letting me know what the best way to 'clean' this stuff up...Should I go into the redirect software and delete all of the redirects where the non / page is being redirected to the / page? Thanks.
On-Page Optimization | | NoahsDad0 -
Robots.txt file
Does it serve any purpose if we omit robots.txt file ? I wonder if spider has to read all the pages, why do we insert robots.txt file ?
On-Page Optimization | | seoug_20050