Cgi-bin folder
-
Is there cg-bin folder of any site related to SEO in anyways and why we always block Cgi-bin folder in robots.txt.
-
As eyepaq says - it's for holding executable "program" files. search engiens don't need access so it gets blocked. If it's empty you can even remove it.
-
No. No correlation with SEO.
However the same way you block your temp or admin folder, there is no need in letting any bot crawling your cgi-bin folder.
Cgi-bin folder holds executable file - you don't want external sources to see it anyway - it's "very" private from security reasons. You need more then knowing what's inside the folder to do harm but keeping it hidden is the wise call. Better safe then sorry.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
CGI Redirects
Trying to 301 old legacy files like oldsite.com/green/red.cgi/blue/ using this htaccess code: Redirect 301 /green/red.cgi/blue/ http://www.newsite.com/summary-page/
Technical SEO | | SoulSurfer8
Instead it's redirecting to: newsite.com/red.cgi/blue/ FYI oldsite.com's htaccess file does not have any global 301 rules that would conflict Does anyone know if cgi files require a different 301 syntax? Thanks!0 -
Should Sitemaps be placed in the sub folder they reference?
I have a sitemap-index.xml file in the root. I then have several sitemaps linked to from the index in example.com/sitemaps/sitemap1.xml, example.com/sitemaps/sitemap2.xml, etc. I have seen on other sites that for example a sitemap containing blogs where the blogs are located at example.com/blog/blog1/ would be located at example.com/blog/sitemap.xml. Is it necessary to have the sitemap located in the same folder like this? I would like to have all sitemaps in a single sitemap folder for convenience but not if it will confuse search engines. My index count for URLs in some sitemaps has dropped dramatically in Google Webmaster Tools over the past month or so and I'm not sure if this is having an effect. If it matters, I have all sitemap files, including the index, listed in the robots.txt file.
Technical SEO | | Giovatto0 -
Htaccess code to 301 redirect a folder change
Hi, I need some help to redirect all my site as there was a folder change. eg, the old structure was www.mysite.com/stuff-1/bags.html and I need it to go to the same structure without the "-1" eg: /stuff/bags.html
Technical SEO | | Paul_MC
The "bags.html" will be lots of different products, so this would be a wildcard? What would the htaccess code need to be? Thanks0 -
Pointing a sub-domain to a sub-folder in htaccess
I have a client who currently uses shopify for there blog. I want to set them up with a separate blog away from the shopify system and host it in Australia. I know the best option is using a subfolder but as the shopify system is an unmoveable CMS can I somehow point my subdomain to a subfolder and get the benefits of the domain name? I could do this by using the rewrite rule in the htaccess file. If I was to do this would it end up cloaking the URL's of the articles?
Technical SEO | | acs1110 -
New URL or Folder Off Existing Site
I am working on a project that is promoting dining in a particular region of the southwest for a destination marketing company. The parent Web site is an authority in the region and ranks well for almost all terms related to the leisure experience in the region. A completely separate Web site was built to promote this culinary program as it involves a committee of different stakeholders, but it’s solely focused on the region. My question is this. The site is on a different CMS, etc., but the overall experience on the site is similar to the parent DMO site in terms of creative. The client has a brand new domain that they purchased for this initiative, but we are also considering mapping the parent site URL to the new culinary site. Parent: www.regionalsite.com New Themed Site: www.regionalsite.com/theme/ Or www.themeurl.com My fear is that if I take the approach of the new URL that it will take forever for the site to build any link clout at all, as the client doesn’t really get the fact that working a link strategy is so critical. However, I know that having links from the regional site over to the theme URL will have an impact. Also, if I do take the approach of mapping the URL to a new folder off of the parent domain, do I risk that 2<sup>nd</sup> tier links on the micro-site will have a challenge indexing as they will essentially be on tier 3? Any advice would be appreciated.
Technical SEO | | VERBInteractive0 -
What are your thoughts on security of placing CMS-related folders in a robots.txt file?
So I was just about to add a whole heap of CMS-related folders to my robots.txt file to exclude them from search, and thought "hey, I'm publicly telling people where my admin folders are"...surely that's not right?! Should I leave them out of the robots.txt file, and hope for the best that they never get indexed? Should I use noindex meta data on every page? What are people's thoughts? Thanks, James PS. I know this is similar to lots of other discussions around meta noindex vs. robots.txt, but I'm after specific thoughts around the security aspect of listing your admin folders in a robots.txt file...
Technical SEO | | James-Distinction0 -
Crawling image folders / crawl allowance
We recently removed /img and /imgp from our robots.txt file thus allowing googlebot to crawl our image folders. Not sure why we had these blocked in the first place, but we opened them up in response to an email from Google Product Search about not being able to crawl images - which can/has hurt our traffic from Google Shopping. My question is: will allowing Google to crawl our image files eat up our 'crawl allowance'? We wouldn't want Google to not crawl/index certain pages, and ding our organic traffic, because more of our allotted crawl bandwidth is getting chewed up crawling image files. Outside of the non-detailed crawl stat graphs from Webmaster Tools, what's the best way to check how frequently/ deeply our site is getting crawled? Thanks all!
Technical SEO | | evoNick0 -
Geotargeting a folder in GWT & IP targeting
I am curently managing a .com that targets Canada and we will soon be launching a .com/us/ that will target the US. Once we launch the /us/ folder, we want to display the /us/ content to any US IP. My concern is that Google will then only index the /us/ content, as their IP is in the US. So, if I set up .com and .com/us/ as two different sites in GWT, and geotarget each to the Country it is targeting, will this take care of the problem and ensure that Google indexes the .com for Canada, and the /us/ for the US? Is there any alternative method (that does not include using the .ca domain)? I am concerned that Google would not be able to see the .com content if we are redirecting all US traffic to .com/us/. Any examples of this online anywhere?
Technical SEO | | bheard0