Welcome to the Q&A Forum

We have a very large store with 278,146 individual product pages. Since these are all various sizes and packaging quantities of less than 200 product categories my feeling is that Google would be better off making sure our category pages are indexed. I would like to block all product pages via robots.txt until we are sure all category pages are indexed, then unblock them. Our product pages rarely change, no ratings or product reviews so there is little reason for a search engine to revisit a product page. The sales team is afraid blocking a previously indexed product page will result in in it being removed from the Google index and would prefer to submit the categories by hand, 10 per day via requested crawling. Which is the better practice?

(Note, the site links are from a sandbox site and has very low DA or PA) If you look at this page, you will see at the bottom a lengthy article detailing all of the properties of the product categories in the links above. http://www.aspensecurityfasteners.com/Screws-s/432.htm My question is, is there more SEO value in having the one long article in the general product category page, or in breaking up the content and moving the sub-topics as content to the more specific sub-category pages? e.g. http://www.aspensecurityfasteners.com/Screws-Button-Head-Socket-s/1579.htm
http://www.aspensecurityfasteners.com/Screws-Cap-Screws-s/331.htm
http://www.aspensecurityfasteners.com/Screws-Captive-Panel-Scre-s/1559.htm

(There was a similar older question on the forum, but it wasn't really answered so please forgive me if this looks like a repeated question) Looks like keyword stuffing, but it isn't We have a massive web store with 80k "commodity products" (and this amount will only increase) which aren't sold the same way normal products are sold (i.e. by brand and model). Commodity products are sold by specification, therefore their product names are actually descriptions of the product. In our case, industrial fasteners (nuts, bolts, washers, screws, etc) sold in bulk. If you click on the link below, you will see that our catalog involves a tremendous amount of repetition, where the products all appear the same, but are varying by dimensions and/or package quantities. The solutions the web store software offers to solve this problem cause issues for us (i.e. displaying the dimensions and quantities ONLY under a common header) but more importantly, we are concerned that search engines are seeing this as keyword stuffing and penalizing the pages. http://www.aspenfasteners.com/Step-Bolts-Inch-Standard-s/407.htm If we can't change the presentation of the page, should we be concerned and if so, how do we let a search engine know that the repetition is legitimate?

Our site has far too many pages for our 10K page PRO account which are not SEO worthy. In fact, only about 2000 pages qualify for SEO value. Limitations of the store software only permit me to use robots.txt to sculpt the rogerbot site crawl. However, I am having trouble getting this to work. Our biggest problem is the 35K individual product pages and the related shopping cart links (at least another 35K); these aren't needed as they duplicate the SEO-worthy content in the product category pages. The signature of a product page is that it is contained within a folder ending in -p. So I made the following addition to robots.txt: User-agent: rogerbot
Disallow: /-p/ However, the latest crawl results show the 10K limit is still being exceeded. I went to Crawl Diagnostics and clicked on Export Latest Crawl to CSV. To my dismay I saw the report was overflowing with product page links: e.g. www.aspenfasteners.com/3-Star-tm-Bulbing-Type-Blind-Rivets-Anodized-p/rv006-316x039354-coan.htm The value for the column "Search Engine blocked by robots.txt" = FALSE; does this mean blocked for all search engines? Then it's correct. If it means "blocked for rogerbot? Then it shouldn't even be in the report, as the report seems to only contain 10K pages. Any thoughts or hints on trying to attain my goal would REALLY be appreciated, I've been trying for weeks now. Honestly - virtual beers for everyone! Carlo

Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis. If this was possible would we be getting more immediate results?

Having problems with this question, old edits get saved as copies of the question, even this one had to be edited twice, originally an old version of an edited question. How do I DELETE a question I authored (like this one) I see no button for it.

I believe that the Google and Bing crawlbots understand wildcards for the "disallow" URL's in robots.txt - does Roger?