How deep should I let my forum be spidered
-
I run quote a niche website that's been running since late 1999 and over that time I've built up something like 4000 resources which consist of either text articles or image galleries and reviews along side another few thousand news stories relating to the niche interest. On top of the main site I also have a forum which isn't especially optimised for SEO and I was wondering, whilst was cleaning it up, whether anyone has any tips / suggestion / best practices for forum SEO.
Because it is all UGC the quality of the posts can be quite weak so I was wondering whether I should block robots completely from the forum, which seems a little harsh, whether I should let the whole forum be spidered (which seems a little excessive and potentially a bad thing) or whether I should restrict things to that only the main index and perhaps one page of topics and their posts be accessible to robots and then nofollow the rest?
Any thoughts?
-
Hey mate.
UGC can actually be very high quality content, and subsequently I would let Google run wild through all the content.
Forums can be tricky and you would want to make sure Google doesnt index any duplicate content.
I would use this tool, http://home.snafu.de/tilman/xenulink.html to crawl the forum. Export to Excel and then assess all the URL structures and see if it's picking up any strange parameters or duplicate content.
I would block the weird paramters in Google webmaster tools, and if technically possible 301 all the duplicate content issues. The problem with restricting things in robots.txt is that you might actually block link juice accidentally.
Hope that helps - my background is more in ecommerce but im sure we'll have similar site structure issues.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
There is a copy of our website that is ranking. How can I let Google know our website is the authentic site?
I just found another copy of my old website and have no way to take it down. Unfortunately, it's ranking so he didn't place it as a nofollow. (My boss hired someone to redevelop our website before I came on board and never finished the project). So, could this be hurting us? I tried to look to see if we were being penalized and couldn't find that we were. Also, ever since we migrated to a new domain name, our ranking is tumbling. I've redirected properly and tested to make sure they're resolving correctly and they are. I have no idea what is going on. We've virtually lost all ranking. Any help would be much appreciated.
On-Page Optimization | | npuffer790 -
Ranking a forum topic
We can rank a e-commerce product page,
On-Page Optimization | | kevinbp
http://moz.com/blog/building-deep-links-into-ecommerce-pages Q: But how do we rank a forum topic?
For example i want to rank this page, http://moz.com/community/q/does-a-link-in-facebook-count-as-a-backlinkHow do i go about doing it? Is it similar to ranking a website?1 -
Content in forum signatures being spidered, does it matter?
Hello, first post here, just started with SEOmoz so hope it's relevant. Searched a fair bit on this without getting a good answer either way so interested to get some opinions. The core of the site I run is a forum dedicated to collecting, for the sake of argument let's say cars. A good percentage of the users have signatures which list their collection, for example 1968 Car A - 1987 Car B - 1998 Car D and so on.... These signatures lists can be 20 items or more, some hotlink the signautres back to the relevant post on the forum, some not. The signatures show on every post on which the user makes. What I'm noting is a) SEOMoz is reporting a LOT of links on every forum page, due mainly to these signatures I guess. and of more interest b) The content of the signatures is being spidered. So for example of you search for '1968 Car A' you might get a couple of good results directly relevant to '1968 Car A' from my site, but you also get a lot of other non-relevant threads as results because the user just happens to have posted on them. Obviously this is much more apparent on the site google search. So what is the best approach? Leave as is? Hide the signatures from the BOTs? Another approach?
On-Page Optimization | | rutteger0 -
Does Commented HTML Code Get Spidered?
I have a change, perhaps temporary, to my web store that I will be making. I am wondering if the code that I will comment out () gets spidered?
On-Page Optimization | | lbohen0 -
Too many forum posts - may I delete the oldest ones?
Hello, We've got product pages with comments. We have collected so many comments for the last two years on these pages so they made problems to our website. We plan to cut every posts older than one year to solve this problem. Could it be bad for SEO? (Yes, these are really unique and relevant posts...) How could I predict the consequences? Thanks!
On-Page Optimization | | DDL0 -
Are Forum Links To Website Causing Rankings To Drop ?
For many years our top keywords hve ranked in top 5. Usually top 3 for a few in particular. The last few months those particular keywords have dropped to 25 +. Our other rankings are doing pretty decent and are remaiing the same. We are members of about 6 forums since they relate to our website and sometime we have contests on them. I do know that the forums are based on those particular keywords and are linking to our website, as well as other forum members linking to us. Each forum has roughly 300 links to our website. It does not seem as if our competitors are affected by this, as they are members of the forums as well. What could be causing this for these certain keywords? If those forums were affected by Google's new algorythm would this have an affect on our website?
On-Page Optimization | | hfranz0 -
How deep should I go with a directory site?
I am creating a new site which has a directory component. Based on what I have ready I am inclined to keep the site architecture as flat as possible. However, the natural layout that I have come up with in my head has the directory listings 5 or 6 pages deep in the site structure. I saw in another post that someone in a similar situation was suggesting that going deep like this is fine so long as there are many internal links to the deeper pages to indicate that they are important. Should I make a conscious effort to make the site architecture as flat as possible? Are there any specific guides/resources that address this particular issue that I should be aware of? Thanks!
On-Page Optimization | | fastestmanalive0 -
Should I show and allow following of deep navigation links?
Hello mozzers, I am currently designing an e-commerce system for various companies, my pilot being www.containerpadlocks.co.uk. I you look on the left in the navigation, you will see only the top level categories are visible (and in the html). Sub categories are only shown if viewing a parent category, or a product in that category/subcategory. My question would be, should I be listing all my subcategories in that navigation (hidden from the user) to allow Google to index my pages easily, and will this promote more PageRank and internal links for my site, or will this just dilute my page juice? Many Thanks,
On-Page Optimization | | jcarter
James0