Irrelevant Subfolders?
-
I have my root domain and as I am a web designer, I use subfolders as sandboxes for client websites. I would not like these to be searched though because they are just demos, ideas and such. Would this be a robots.txt matter or can I have SEOMoz wok with a campaign where I can exclude subfolders? Thank you in advance! : )
-
Hi Wil,
If you want to exclude all robots from crawling these pages you can exclude them from Robots.txt.
If you want to exclude Rogertbot you can find the info on how to do it on: http://www.seomoz.org/dp/rogerbot
I hope that helped,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt in subfolders and hreflang issues
A client recently rolled out their UK business to the US. They decided to deploy with 2 WordPress installations: UK site - https://www.clientname.com/uk/ - robots.txt location: UK site - https://www.clientname.com/uk/robots.txt
Technical SEO | | lauralou82
US site - https://www.clientname.com/us/ - robots.txt location: UK site - https://www.clientname.com/us/robots.txt We've had various issues with /us/ pages being indexed in Google UK, and /uk/ pages being indexed in Google US. They have the following hreflang tags across all pages: We changed the x-default page to .com 2 weeks ago (we've tried both /uk/ and /us/ previously). Search Console says there are no hreflang tags at all. Additionally, we have a robots.txt file on each site which has a link to the corresponding sitemap files, but when viewing the robots.txt tester on Search Console, each property shows the robots.txt file for https://www.clientname.com only, even though when you actually navigate to this URL (https://www.clientname.com/robots.txt) you’ll get redirected to either https://www.clientname.com/uk/robots.txt or https://www.clientname.com/us/robots.txt depending on your location. Any suggestions how we can remove UK listings from Google US and vice versa?0 -
Best Topography for eCommerce Site Product Pages (flat nav/off the root OR in products subfolder) ?
Hi Im SEO'ing a Shopify site (new/not yet live) at the moment and all the products are in a 'Products' subfolder along the lines of: domain.com/products/blue-widgets/ etc I understand that many ecommerce SEO's these days go 'Flat Navigation' with all products 'off the root' rather than in a sub folder. Then they communicate product & categories/departmental relationships via breadcrumbs & other internal linking etc In the case of a platform like Shopfy is this a good idea or is it best to leave 'as is' and the 'Products' subfolder is a perfectly good place for the product pages ? All Best Dan
Technical SEO | | Dan-Lawrence0 -
International architecture: Country specific subfolders > domain mapping to tld
Hi Ive got a clients dev saying they are setting up with country/language specific subfolders (as i recommended) BUT now they are saying they want to set up on network.domain.com (for example) and then each language will have its own sub-folder BUT will be domain mapped to the TLD as and when they get them. I have asked them to clarify since sounds a bit strange since thought best to have domain.com then /uk and /us etc etc and sure ok to forward country specific TLD's to these subfolders. Its this new subdomain (network.) thats concerning me and mapping rather than forwarding (or is it the same thing) but anyone know off hand if above sounds ok or also thinks a bit strange or know issues with such a set up ? many thanks dan
Technical SEO | | Dan-Lawrence0 -
Why would Google rank a highly irrelevant page in the top 15 especially for a seemingly important keyword?
While searching for "Blog writing service reviews", I found that a web page that's not even optimized for the query is ranking within top 15 search results. Upon checking the source code, I found that the webpage has been optimized for product reviews services. Plus, the website is only 11 months old, got 7 digit Alexa rank and has PR 1. Why would Google rank such a page in top 15?
Technical SEO | | suskanchan0 -
Correct linking to the /index of a site and subfolders: what's the best practice? link to: domain.com/ or domain.com/index.html ?
Dear all, starting with my .htaccess file: RewriteEngine On
Technical SEO | | inlinear
RewriteCond %{HTTP_HOST} ^www.inlinear.com$ [NC]
RewriteRule ^(.*)$ http://inlinear.com/$1 [R=301,L] RewriteCond %{THE_REQUEST} ^./index.html
RewriteRule ^(.)index.html$ http://inlinear.com/ [R=301,L] 1. I redirect all URL-requests with www. to the non www-version...
2. all requests with "index.html" will be redirected to "domain.com/" My questions are: A) When linking from a page to my frontpage (home) the best practice is?: "http://domain.com/" the best and NOT: "http://domain.com/index.php" B) When linking to the index of a subfolder "http://domain.com/products/index.php" I should link also to: "http://domain.com/products/" and not put also the index.php..., right? C) When I define the canonical ULR, should I also define it just: "http://domain.com/products/" or in this case I should link to the definite file: "http://domain.com/products**/index.php**" Is A) B) the best practice? and C) ? Thanks for all replies! 🙂
Holger0 -
Can SEOMoz crawl a single page as oppose to an entire subfolder?
I would like the following page to be crawled: http://www.ob.org/_programs/water/water_index.asp Instead, SEOMoz changes the page to the following subfolder which is an invalid url: http://www.ob.org/_programs/water/
Technical SEO | | OBIAnalytics0 -
Set base-href to subfolders - problems?
A customer is using the <base>-tag in an odd way: <base href="http://domain.com/1.0.0/1/1/"> My own theory is that the subfolders are added as the root because of revision control. CSS, images and internal links are used like this:
Technical SEO | | Vivamedia
internal link I ran a test with Xenu Link Sleuth and found many broken links on the site, but I can't say if it is due to the base-tag. I have read that the base-tag may cause problems in some browsers, but is this usage of base-tag bad in some SEO-perspective? I have a lot of problems with this customer and I want to know if the base-tag is a part of it.0 -
Moving forum to subfolder
Hi, we're thinking of moving our very well established forum to a subfolder on our main domain. I have read that this would be much better for seo reasons/domain authority rather than using a sub domain. Is this correct? Our forum has 15,000 users and gets roughly over 502,087 impressions from 101,147 visits a month. Thanks, James
Technical SEO | | Wedideas0