Large site with content silo's - best practice for deep indexing silo content
-
Thanks in advance for any advice/links/discussion. This honestly might be a scenario where we need to do some A/B testing.
We have a massive (5 Million) content silo that is the basis for our long tail search strategy. Organic search traffic hits our individual "product" pages and we've divided our silo with a parent category & then secondarily with a field (so we can cross link to other content silo's using the same parent/field categorizations).
We don't anticipate, nor expect to have top level category pages receive organic traffic - most people are searching for the individual/specific product (long tail). We're not trying to rank or get traffic for searches of all products in "category X" and others are competing and spending a lot in that area (head).
The intent/purpose of the site structure/taxonomy is to more easily enable bots/crawlers to get deeper into our content silos. We've built the page for humans, but included link structure/taxonomy to assist crawlers.
So here's my question on best practices. How to handle categories with 1,000+ pages/pagination. With our most popular product categories, there might be 100,000's products in one category. My top level hub page for a category looks like www.mysite/categoryA and the page build is showing 50 products and then pagination from 1-1000+.
Currently we're using rel=next for pagination and for pages like www.mysite/categoryA?page=6 we make it reference itself as canonical (not the first/top page www.mysite/categoryA).
Our goal is deep crawl/indexation of our silo. I use ScreamingFrog and SEOMoz campaign crawl to sample (site takes a week+ to fully crawl) and with each of these tools it "looks" like crawlers have gotten a bit "bogged down" with large categories with tons of pagination. For example rather than crawl multiple categories or fields to get to multiple product pages, some bots will hit all 1,000 (rel=next) pages of a single category. I don't want to waste crawl budget going through 1,000 pages of a single category, versus discovering/crawling more categories.
I can't seem to find a consensus as to how to approach the issue. I can't have a page that lists "all" - there's just too much, so we're going to need pagination. I'm not worried about category pagination pages cannibalizing traffic as I don't expect any (should I make pages 2-1,000) noindex and canonically reference the main/first page in the category?). Should I worry about crawlers going deep in pagination among 1 category versus getting to more top level categories?
Thanks!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's the best tool to use to compare competirors
A client of ours has asked us to compare their search rankings to competitors. What's the best tool to use in SEOMoz to do this?
Moz Pro | | BillyBobGriffin0 -
Issue Using MozScape wtih SEOGadet's Link Excel Extension
Howdy Everyone! Since i've seen the MozCon presentation by Richard Baxter at SEOGadget, i've been completely amazed by the links extension for Excel. I tried this morning to give it a try. Unfortunately, I cannot get it working, and was hoping that someone here could help me out. I know it's out of the usual "realm" of questions, but I figure it's worth a try 🙂 I successfully installed the addon and entered my Access id (as "member-xxxxxxxxxx" member is in my id) and the secret key. I then downloaded the "OSE" excel spreadsheet" just to make sure I get all the calls right (as I know the doc works) Once I do this, and enter anything in, I get the "an error occured: the remote server returend an error: 401 unauthorized. I then went into the config file (as the setup doc suggests) and disabled run time caching (or at least set "SEOMOZ_API_use_cache_YN":"N") and the SEOMOZ_API_timeout to 100000 from 60000. I have also tried uninstalling and reinstalling the addin, along with regenerating the MozScape API Key Anyway, i'm not an excel wiz, and would appreciate any help that I could get on this. I'm also about to experiment with SEO Tools For Excel if anywone wants to check that out. Thanks in advance Zach Russell
Moz Pro | | Zachary_Russell0 -
Can you change RankTracker's setting?
Is it possible to change RankTrackers settings to see if my keywords are in the top 500 spots instead of only the top 50?
Moz Pro | | theLotter0 -
Sorting Dupe Content Pages
Hi, I'm no excel pro, and I'm having a bit of a challenge interpreting the Crawl Diagnostics export .csv file. I'd like to see at a glance which of my pages (and I have many) are the worst offenders for dupe content – ie. which have the most "Other URLs" associated with them. Thanks, would appreciate any advice on how other people are using this data, and/or how 'Moz recommends to do it. 🙂
Moz Pro | | ntcma0 -
Moving from a dynamic site to nopcommerce
Hi, First question since becoming a member so be gentle with me ;o) We are moving from a site using a dynamically generated ecommercetemplate to a nopcommerce site. I have two questions about this I know that 301 redirects are the best way to pass "link juice" but with the site being dynamically generated a lot of these links will simply disappear when we move to nop, meaning that they wont actually be there in order to add a 301. Any advice on this would be appreciated. How can i get a list of the pages on my current site which have the best rank in order to create the redirects. Unfortunately due to some technical issues we had with google analytics we were unable to install it so dont really have any analytics to give us some extra info. I was hoping that there would be somewhere within seomoz where i could be directed. Many thanks Chris
Moz Pro | | cjhamill0 -
What's been the best SERP rank tracker for you?
I do a lot of my SERP ranking tracking via Aaron Wall's Rank Checker, but I'm realizing it's taking up too much time. I was looking at Raven's tools and a few others. What's been your favorite thus far? If you do reply, can you talk about: price capabilities time savings b/c of it graphical UI other thoughts I like SEOMoz's UI, but I can only manage 5 domains and I don't have a huge budget yet. Also, if there's a rank checker out there that incorporates google places results, that would be great too! Maybe too good, eh?
Moz Pro | | SeattleOrganicSEO0 -
Site rank checking tool
Is there a tool where I can enter a URL and it will tell me all the terms a site ranks for? Basically I recently put up a new website and I want to know what terms I rank in the top 50 for in Google
Moz Pro | | KevinPatrick0 -
What is the best/your favorite management software?
Currently my company uses basecamp with a splash of google docs a little bit of Paymo, and Raventools for project management. (We use other tools like SEOmoz, but not so much for management as SEO tasks) The ideal features we are looking for are: Collaborative document editing and sharing(Google Docs) Task lists and project organization (basecamphq) Time Tracking on a per task basis (Paymo) (SEO tracking software is great too, but it doesn't have to be integrated with the project management directly. Currently using Raventools and SEOMoz along with some smaller tools.) The project management side can be completely separate from the SEO tool side, but it would be great if there was one simple interface that all of this could be done from. Any suggestions? Are there features I'm missing in my current software that could bring them up to this level?
Moz Pro | | MarloSchneider0