Two Robots.txt files
-
Hi there
Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one.
Thanks in advance!
Cheers,
Satla -
Thank Riera
-
Hi Satia,
You mentioned that one robots.txt file placed in inside the server and search engine unable to see the file. If search engine won't see robots.txt file then what is the use of that robots.txt file?
AFAIK it must be placed under root directory an there is no way to keep two files with the same name. So you must have only one robots.txt and that should be placed under root directory.
Hope this helps.
Thanks
-
Hi Satla,
You're going to need to get rid of that 2nd version ASAP. The official standard for a robots.txt file is all lower case in the file name, so that's most likely what bots are seeing. But to err on the side of caution, I'd remove any possibilities of a "disallow: /" and remove that Robots.txt version.
Some servers are case sensitive, so you could run into issues here as well.
-
Hi,
I don't find any good reason why there are two files. There should be just one, where you specify everything you'd like to done.
If the tech team doesn't want to correct and leave just one file, may be because they are lazy or there might be some other issue that if they delete one file, the hole site blows up.Here, I leave you 2 Moz's articles about the robots.txt file.
What is Robots.txt? - Moz Learn
Learn About Robots.txt with Interactive Examples - Moz BlogTake into account that the name of the file must be in lower case. I've never seen any different and the servers are usually case sensitive with the filenames.
Hope its helpful.
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Meta Robots information
Hi, I have a question about the Meta Robots information Accoarding to the Moz bar, our page uses the meta robots noodp and noydir. Our competitor uses
On-Page Optimization | | AdoBike
INDEX,FOLLOW I read that noodp and noydir are dated and not used anymore. Is it wise to use INDEX FOLLOW instead for better SEO? Thanks in advance!1 -
Need to change permalink structure - need advice on these two options
I have a client with a robust Wordpress blog that ranks well for a number of high-volume terms. We are doing a site rebuild, and they are adamant they don't want to lose any SERP ranking. However, their permalink structure contains the date the post was published, i.e. https://www.company.com/blog/2018/10/28/blog-post-title The date inclusion is not ideal for obvious reasons, but also because it's creating 404 links when pre-scheduling with Hubspot. Option 1: Change the permalink structure for all posts, past and future, and implement 301 redirects The only reason I'm not outright doing this is because there will be some slippage in the rankings, given that redirects only pass on a percentage of juice. The client doesn't want any slippage. Option 2: Create a new instance of Wordpress in a new subdirectory and continue the blog from there This will mandate a new structure for the blog posts; now posts will be children of /industry-blog/ instead of /blog/, but the old posts will remain as they are under /blog/ (and with dates intact). But I don't know if this will cause some hierarchical confusion that will negatively impact both the old blog and the new blog. Any advice given is appreciated. Please feel free to correct me if I've gotten anything wrong, I've only been practicing SEO for a few years now.
On-Page Optimization | | Marce5210 -
How we can rank two keywords
Here is my example. I developed a property inspection app, where the keyword I try to rank is property inspection app, and property inspection software. Property Inspection Software got slightly more traffic than app. I would like to rank both of this keyword high for my main website, but there is only one title, one H1 tag I can use to have keyword, where if I use "App" there is no way to have "Software" on the title, etc. How can I deal with such a case? Any help will be appreciate.
On-Page Optimization | | terrysun0 -
Need suggestion: Should the user profile link be disallowed in robots.txt
I maintain a myBB based forum here. The user profile links look something like this http://www.learnqtp.com/forums/User-Ankur Now in my GWT, I can see many 404 errors for user profile links. This is primarily because we have tight control over spam and auto-profiles generated by bots. Either our moderators or our spam control software delete such spammy member profiles on a periodic basis but by then Google indexes those profiles. I am wondering, would it be a good idea to disallow User profiles links using robots.txt? Something like Disallow: /forums/User-*
On-Page Optimization | | AnkurJ0 -
Optimizing for two keywords
Hi there, my question is: I need to optimze two pages related to almost the same concept but different orientations. One keyword is "emocional intelligence development" and the other word is "emocional intelligence workshops" I can't mix the two pages in one unique page due to web strucutre reasons. Is it possible to optimize each web for each keyword independently although they have the concept emotional intelligence in common? Thanks
On-Page Optimization | | juanmiguelcr0 -
Is it best to optimize your site for just one or two keywords?
My company/website makes and sells a product that's not that competitive but still has about 20 key words/phrases that people search for. My site is not a huge site maybe 35 pages after you include the blog posts.We sell samples off the site but it's mostly used as a brochure but we also want it to be a successful tool at bringing in leads. Should I optimize for the most popular key word phrases focusing on only one or two per page and forget about the rest or should I try to optimize for as many keywords as possible on all pages or should I optimize for just the few (3-5) heavy hitting keywords but on all pages? Right now I've got it optimized for around 3 keyword phrases for the whole site and only 1 or 2 per page with the most popular phrases on the most important pages.
On-Page Optimization | | JAARON0 -
URL with two forward slashes //
We have a potential client with a URL structure in this fashion: http://www.site-url.com//cpage/page.html pretty strange, right? my question is: How bad are the 2 forward slashes // for SEO? How bad is it to have that extra layer of /cpage in the URL? this doesn't appear to serve any other purpose than making the URL longer than necessary.
On-Page Optimization | | Motava0
