Welcome to the Q&A Forum

@terentyev - sorry, can't edit my questions once submitted and I wait for approval (why?) the statement should read my question SHOULD be very specific, whereas my original question was much more general - you answered that question very nicely. Sorry for any misunderstanding

@terentyev thanks for the reply. We have no reason to believe these URL's are backlinked. These aren't consumer products that individual are interested in, our site is a wholesale B2B selling very narrow categories in bulk quantities typically for manufacturing. Therefore, almost zero chance for backlinks anywhere for something as specific as a particular size/material/package quantity of a product.

We have already initiated a canonicalization project started but we are stuck between two concerns from sales, 1) we can't wait for canonicalization (which is complex) we need sales now and 2) don't touch robots.txt because MAYBE the individual products are indexed.

So that is why my question is very specific - do we KNOW that Google will immediately de-index URL's blocked by robots.txt?

We have a very large store with 278,146 individual product pages. Since these are all various sizes and packaging quantities of less than 200 product categories my feeling is that Google would be better off making sure our category pages are indexed.

I would like to block all product pages via robots.txt until we are sure all category pages are indexed, then unblock them. Our product pages rarely change, no ratings or product reviews so there is little reason for a search engine to revisit a product page.

The sales team is afraid blocking a previously indexed product page will result in in it being removed from the Google index and would prefer to submit the categories by hand, 10 per day via requested crawling.

Which is the better practice?

What if the long, detailed content is for a page for which you cannot possibly win rankings for because huge, long-standing enterprises with high PA and DA already dominate the first few SERPs? Would it not be better to move the details of the content to the specific pages where they apply?

(Note, the site links are from a sandbox site and has very low DA or PA)

If you look at this page, you will see at the bottom a lengthy article detailing all of the properties of the product categories in the links above.

http://www.aspensecurityfasteners.com/Screws-s/432.htm

My question is, is there more SEO value in having the one long article in the general product category page, or in breaking up the content and moving the sub-topics as content to the more specific sub-category pages? e.g.

http://www.aspensecurityfasteners.com/Screws-Button-Head-Socket-s/1579.htm
http://www.aspensecurityfasteners.com/Screws-Cap-Screws-s/331.htm
http://www.aspensecurityfasteners.com/Screws-Captive-Panel-Scre-s/1559.htm

Thanks Tim. Do you have any examples of what those problems might be? With such a large catalog managing those rel canonical tags will be difficult (I don't even know if the store allows them, it's a hosted store solution and little code customization is allowed).

Thanks, Hurf. Thanks for the excellent reply. Unfortunately, with Volusion as our web-store we don't have the option of displaying anything but the full product name on listings by category unless we engage a Volusion partner to modify our store or we migrate 80K products to a different and more complex product/option structuring model.

For the sake of the clients, 12-products-per-page would be maddening. Text search is very ineffective when it comes to parsing product dimensions, so we produce long listings, sorted by size so the user can quickly locate their product.

In the short term, I think it makes most sense to pay for custom programming to reduce the product description to just the dimension and quantity when displayed on a category page.

(There was a similar older question on the forum, but it wasn't really answered so please forgive me if this looks like a repeated question)

Looks like keyword stuffing, but it isn't

We have a massive web store with 80k "commodity products" (and this amount will only increase) which aren't sold the same way normal products are sold (i.e. by brand and model). Commodity products are sold by specification, therefore their product names are actually descriptions of the product. In our case, industrial fasteners (nuts, bolts, washers, screws, etc) sold in bulk.

If you click on the link below, you will see that our catalog involves a tremendous amount of repetition, where the products all appear the same, but are varying by dimensions and/or package quantities. The solutions the web store software offers to solve this problem cause issues for us (i.e. displaying the dimensions and quantities ONLY under a common header) but more importantly, we are concerned that search engines are seeing this as keyword stuffing and penalizing the pages.

http://www.aspenfasteners.com/Step-Bolts-Inch-Standard-s/407.htm

If we can't change the presentation of the page, should we be concerned and if so, how do we let a search engine know that the repetition is legitimate?

We have a similar problem, but much more complex to handle as we have a massive catalog of 80,000 products and growing.

The problem occurs legitimately because our catalog is so large that we offer different navigation paths to the same content.

http://www.aspenfasteners.com/Self-Tapping-Sheet-Metal-s/8314.htm

http://www.aspenfasteners.com/Self-Tapping-Sheet-Metal-s/8315.htm

(If you look at the "You are here" breadcrumb trail, you will see the subtle differences in the navigation paths, with 8314.htm, the user went through Home > Screws, with 8315.htm, via Home > Security Fasteners > Screws).

Our hosted web store does not offer us htaccess, so I am thinking of excluding the redundant navigation points via robots.txt.

My question: is there any reason NOT to do this?

Hi Andres!

Sorry, I thought I answered this earlier. If I understand correctly wildcards ARE allowed, according to this reply to my question on the topic: http://www.seomoz.org/q/does-rogerbot-read-url-wildcards-in-robots-txt

Hope THIS reply sticks this time!

Our site has far too many pages for our 10K page PRO account which are not SEO worthy. In fact, only about 2000 pages qualify for SEO value. Limitations of the store software only permit me to use robots.txt to sculpt the rogerbot site crawl. However, I am having trouble getting this to work. Our biggest problem is the 35K individual product pages and the related shopping cart links (at least another 35K); these aren't needed as they duplicate the SEO-worthy content in the product category pages.

The signature of a product page is that it is contained within a folder ending in -p. So I made the following addition to robots.txt:

User-agent: rogerbot
Disallow: /-p/

However, the latest crawl results show the 10K limit is still being exceeded. I went to Crawl Diagnostics and clicked on Export Latest Crawl to CSV. To my dismay I saw the report was overflowing with product page links:

e.g. www.aspenfasteners.com/3-Star-tm-Bulbing-Type-Blind-Rivets-Anodized-p/rv006-316x039354-coan.htm

The value for the column "Search Engine blocked by robots.txt" = FALSE; does this mean blocked for all search engines? Then it's correct. If it means "blocked for rogerbot? Then it shouldn't even be in the report, as the report seems to only contain 10K pages.

Any thoughts or hints on trying to attain my goal would REALLY be appreciated, I've been trying for weeks now. Honestly - virtual beers for everyone!

Carlo

Still waiting for my custom crawl launched on April 28th (it's May 2nd) to complete... seriously? If SEOmoz is overwhelmed, first of all, congratualations on being so popular, but I am not getting any timely data at all.

I have wondered about this - my impression is that since Google is quite insistent on tracking fully-qualified sub-domains that the impact of creating CNAME records for a related sub-domain would be limited. Our blog is hosted via dreamhost, whereas our store is on Volusion - a prepackeaged shopping cart solution. To implement a blog in the same domain, we'd have to create a CNAME record like blog.mysite.com and of course, have a link to it from www.myseite.com.

So far, I've always believed that this would only allow for - as you said - "a portion of the value" of the links... is there any reason to believe otherwise?

Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis.

If this was possible would we be getting more immediate results?

Having problems with this question, old edits get saved as copies of the question, even this one had to be edited twice, originally an old version of an edited question.

How do I DELETE a question I authored (like this one) I see no button for it.

I believe that the Google and Bing crawlbots understand wildcards for the "disallow" URL's in robots.txt - does Roger?