Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Cgi-bin folder

Technical SEO

1241

Alick300 last edited by

Is there cg-bin folder of any site related to SEO in anyways and why we always block Cgi-bin folder in robots.txt.
1 Reply Last reply
Reply Quote 0
matbennett last edited by

As eyepaq says - it's for holding executable "program" files. search engiens don't need access so it gets blocked. If it's empty you can even remove it.
1 Reply Last reply
Reply Quote 1
eyepaq last edited by

No. No correlation with SEO.

However the same way you block your temp or admin folder, there is no need in letting any bot crawling your cgi-bin folder.

Cgi-bin folder holds executable file - you don't want external sources to see it anyway - it's "very" private from security reasons. You need more then knowing what's inside the folder to do harm but keeping it hidden is the wise call. Better safe then sorry.
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Folders in url structure?

Hello, Revamping an out-of-date website and am wondering if I need to include the folders (categories) in the url structure? The proposed structure has 8 main folders. I've been reading that Google is ok if the folder is not included in the url, but is it really? The hesitation I have is that the urls are getting long and the main folder only has only a sub folder beneath it. So, /folder-name/facility-name/treatment-overview. This looks too long, doesn't it? Thanks!
Technical SEO | | lfrazer123

0
Should Sitemaps be placed in the sub folder they reference?

I have a sitemap-index.xml file in the root. I then have several sitemaps linked to from the index in example.com/sitemaps/sitemap1.xml, example.com/sitemaps/sitemap2.xml, etc. I have seen on other sites that for example a sitemap containing blogs where the blogs are located at example.com/blog/blog1/ would be located at example.com/blog/sitemap.xml. Is it necessary to have the sitemap located in the same folder like this? I would like to have all sitemaps in a single sitemap folder for convenience but not if it will confuse search engines. My index count for URLs in some sitemaps has dropped dramatically in Google Webmaster Tools over the past month or so and I'm not sure if this is having an effect. If it matters, I have all sitemap files, including the index, listed in the robots.txt file.
Technical SEO | | Giovatto

0
Sub-domain or sub-folder for a blog?

Traditional thinking suggests sub-domains are treated as separate sites and so don't pass on link juice, but I've heard mixed opinions. I'm very much a believer in sub-folders but I'm interested to hear some other opinions. Thoughts?
Technical SEO | | underscorelive

0
Pointing a sub-domain to a sub-folder in htaccess

I have a client who currently uses shopify for there blog. I want to set them up with a separate blog away from the shopify system and host it in Australia. I know the best option is using a subfolder but as the shopify system is an unmoveable CMS can I somehow point my subdomain to a subfolder and get the benefits of the domain name? I could do this by using the rewrite rule in the htaccess file. If I was to do this would it end up cloaking the URL's of the articles?
Technical SEO | | acs111

0
Use of Location Folders

I'd like to understand the pro's and con's of using a location subfolder as an SEO strategy (example: http://sqmedia.us/Dallas/content-marketing.html), where the /Dallas folder is holding all of my keyword rich page titles. The strategy is to get local-SEO benefits from the use of the folder titled /Dallas (a folder which is unnecessary in the over all structure of this site), but how much is this strategy taking away from the page-title keyword effectiveness?
Technical SEO | | sqmedia

0
OK to block /js/ folder using robots.txt?

I know Matt Cutts suggestions we allow bots to crawl css and javascript folders (http://www.youtube.com/watch?v=PNEipHjsEPU) But what if you have lots and lots of JS and you dont want to waste precious crawl resources? Also, as we update and improve the javascript on our site, we iterate the version number ?v=1.1... 1.2... 1.3... etc. And the legacy versions show up in Google Webmaster Tools as 404s. For example: http://www.discoverafrica.com/js/global_functions.js?v=1.1
http://www.discoverafrica.com/js/jquery.cookie.js?v=1.1
http://www.discoverafrica.com/js/global.js?v=1.2
http://www.discoverafrica.com/js/jquery.validate.min.js?v=1.1
http://www.discoverafrica.com/js/json2.js?v=1.1 Wouldn't it just be easier to prevent Googlebot from crawling the js folder altogether? Isn't that what robots.txt was made for? Just to be clear - we are NOT doing any sneaky redirects or other dodgy javascript hacks. We're just trying to power our content and UX elegantly with javascript. What do you guys say: Obey Matt? Or run the javascript gauntlet?
Technical SEO | | AndreVanKets

0
External Microsite VS Internal Folder

We would like to create either a new website or a new section of our existing website that will feature (in time) a lot of content including a forum, video training, tutorials and downloadable resources. Logistically, it would be much easier to create this in a new site (we'll call it newproduct.com) and refer people to the new site. We would, however, like to keep all of that content on our existing site for the sake of content building and SEO. Should we: Duplicate the content and use no index no follow and/or rel canonical? Host all of the content on our site and set up a vanity domain (www.newproduct.com) to point people to the deep linked area (www.mainsite.com/product/newproductinfo)? Host the content only on an external site with the occasional link back to our main site? I realize there are other options but they're mostly variants of the above. Our main objectives are to make it easy for people to get to while leveraging the new content for SEO purposes. What are the pros and cons of these different approaches? What seems to make the most sense? Thank you!
Technical SEO | | BeijerElectronics

0
Keywords in file names vs folder names

We understand the value of a keyword phrase included in the URL. Is there more value to having that phrase in the folder name of the URL or the file name or does it matter? Example: http://www.biztoolsone.com/website-design.php or http://www.biztoolsone.com/website-design/ Which is best? Thanks, Wick Smith
Technical SEO | | wcksmith

0