What are your thoughts on security of placing CMS-related folders in a robots.txt file?
-
So I was just about to add a whole heap of CMS-related folders to my robots.txt file to exclude them from search, and thought "hey, I'm publicly telling people where my admin folders are"...surely that's not right?!
Should I leave them out of the robots.txt file, and hope for the best that they never get indexed? Should I use noindex meta data on every page?
What are people's thoughts?
Thanks,
James
PS. I know this is similar to lots of other discussions around meta noindex vs. robots.txt, but I'm after specific thoughts around the security aspect of listing your admin folders in a robots.txt file...
-
surly your admin folders are secured?, it would not matter if someone knows where they are.
-
As a rule, you want to avoid using robots.txt files whenever possible. It does not consistently protect you from crawlers and when it does block crawlers it kills any PR on those pages.
If you can block those pages with a noindex tag, it would be a preferable solution.
With respect to security for a CMS site, it really needs to be a comprehensive effort. Many site owners take a couple steps and then have a false-sense of security. Here are a few thoughts:
-
try the site address with /administrator after it to access Joomla and other sites
-
try the site address or blog with /wp-admin/ after it to access Joomla sites
-
make up a webpage and try accessing it to view the site's 404 page
-
right-click on a page and choose View Page Source. Often you will see the name of the CMS clearly listed. Other times you will see clear clues such as /wp/ in folder names. Other times you will find unique extensions such as Yoast SEO which will give you an idea of the CMS
Once a bad guy knows which CMS is in use, they know the default folder structure and more. The point is it requires a lot more effort then most people realize to hide the CMS in use. I applaud your effort, but be very thorough about it. There is a lot more involved then simply covering your robots.txt file.
-
-
I found three options for you: http://www.techiecorner.com/106/how-to-disable-directory-browsing-using-htaccess-apache-web-server/
I think if you do it with.htacces that is a folder specific file than nobody will be able to detect where admin contet is located.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Role of Robots.txt and Search Console parameters settings
Hi, wondering if anyone can point me to resources or explain the difference between these two. If a site has url parameters disallowed in Robots.txt is it redundant to edit settings in Search Console parameters to anything other than "Let Googlebot Decide"?
Technical SEO | | LivDetrick0 -
When to file a Reconsideration Request
Hi all, I don't have any manual penalties from Google but do have a unnatural links message from them back in 2012. We have removed some of the spammy links over the last 2 years but we're now making a further effort and will use the disavow tool once we've done this. Will this be enough once I submit the file or should I / can I submit a Reconsideration Request as well? Do I have to have a manual penalty item in my webmaster account to be able to submit a request? Thanks everyone!
Technical SEO | | KerryK0 -
CMS on autopilot is happily creating duplicate pages - advice?
Hi, our ecommerce CMS (Magento) is creating a bunch of pages with very little content and no user value like this: http://goo.gl/UU2vl This particular example is the by product of a product filtering page, which has the format www.mywebsite/explore/index/loaddata/id/10/. These pages have no content other than images - also the pages don't have page titles and are therefore being flagged in webmaster tools as requiring HTML improvements We also have CMS auto generated pages like this: www.mysite.comhttp/review/product/list/id/7 where the page is effectively a duplicate of the product page, and this is giving us pages being flagged by webmastertools as having duplicate title tags. Should we exclude these two type of page via robots.txt or take another approach, like not worry about them 🙂 many thanks, any help gratefully received.
Technical SEO | | w1ll1am0 -
Google Places Reviews
Has anyone had any delays on Google+ reviews to show up? We have multiple clients who have not received a new review in over two months. These are good accounts with good Zagat scores with 15+ good reviews from real customers. Our clients have asked their clients and have confirmed that there has been reviews left recently. However no new reviews have shown up in the past 60+ days.
Technical SEO | | CaseyKluver0 -
File name same as folder name, ok?
Is it ok to have a folder and file name to be both the same e.g domain.com/xyz-products/ domain.com/xyz-products.php File name would be a page that lists a number of products and then within the folder there would be x-product.php, y-product.php etc
Technical SEO | | NeilD0 -
How narrowly geo targeted should your Google Places page be?
Hi Mozers I'm still struggling with my London based client with two locations and one business. Basically she has a location in W1W 'Westminster' and a location in 'WD!' Borehamwood. Has anyone any good resources of input concerning geotargeting. I've done some searching but can't get quite the help I'm seeking. I'd like to make the Pages cover a 5mile radius and be highly specific to their locations. Is this the right way to proceed? Thanks
Technical SEO | | catherine-2793880 -
Need Help writing 301 redirects in .htaccess file
SEOmoz tool shows me 2 errors for duplicate content pages (www.abc.com and www.abc.com/index.html). I believe, the solution to this is writing 301 redirects I need two 301 redirects 1. abc.com to www.abc.com 2. /index.html to / (which is www.abc.com/index.html to www.abc.com) The code that I currently have is ................................................... RewriteEngine On
Technical SEO | | WebsiteEditor
RewriteCond %{HTTP_HOST} ^abc.com
RewriteRule (.*) http://www.abc.com/$1 [R=301,L] Redirect 301 http://www.abc.com/index.html http://www.abc.com ...................................................... but this does not redirect /index.html to abc.com. What is wrong here? Please help.0 -
Confused about robots.txt
There is a lot of conflicting and/or unclear information about robots.txt out there. Somehow, I can't make out what's the best way to use robots even after visiting the official robots website. For example I have the following format for my robots. User-agent: * Disallow: javascript.js Disallow: /images/ Disallow: /embedconfig Disallow: /playerconfig Disallow: /spotlightmedia Disallow: /EventVideos Disallow: /playEpisode Allow: / Sitemap: http://www.example.tv/sitemapindex.xml Sitemap: http://www.example.tv/sitemapindex-videos.xml Sitemap: http://www.example.tv/news-sitemap.xml Is this correct and/or recommended? If so, then how come I see a list of over 200 or so links blocked by robots when Im checking out Google Webmaster Tools! Help someone, anyone! Can't seem to understand this robotic business! Regards,
Technical SEO | | Netpace0