How deep should I let my forum be spidered
-
I run quote a niche website that's been running since late 1999 and over that time I've built up something like 4000 resources which consist of either text articles or image galleries and reviews along side another few thousand news stories relating to the niche interest. On top of the main site I also have a forum which isn't especially optimised for SEO and I was wondering, whilst was cleaning it up, whether anyone has any tips / suggestion / best practices for forum SEO.
Because it is all UGC the quality of the posts can be quite weak so I was wondering whether I should block robots completely from the forum, which seems a little harsh, whether I should let the whole forum be spidered (which seems a little excessive and potentially a bad thing) or whether I should restrict things to that only the main index and perhaps one page of topics and their posts be accessible to robots and then nofollow the rest?
Any thoughts?
-
Hey mate.
UGC can actually be very high quality content, and subsequently I would let Google run wild through all the content.
Forums can be tricky and you would want to make sure Google doesnt index any duplicate content.
I would use this tool, http://home.snafu.de/tilman/xenulink.html to crawl the forum. Export to Excel and then assess all the URL structures and see if it's picking up any strange parameters or duplicate content.
I would block the weird paramters in Google webmaster tools, and if technically possible 301 all the duplicate content issues. The problem with restricting things in robots.txt is that you might actually block link juice accidentally.
Hope that helps - my background is more in ecommerce but im sure we'll have similar site structure issues.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
On-Page Optimization | | Jacksons_Fencing0 -
Do Search Engine Spiders Read Commented Out Content?
Do Search Engine Spiders Read Commented Out Content? Is commented out content detrimental?
On-Page Optimization | | lbohen0 -
Title tags for deep pages
Just pondering what is current best practice for Title tags of pages buried deep within my website? Say I have a page about 'Cheese's of the world' and from that page there is a page about 'Cheshire Cheese' how would you suggest to structure title tags Would for example this be ok - Cheshire Cheese | Cheese's of the World | Brand name Or is this better - Cheshire Cheese | Brand name Just wondering as I'm redesigning my site currently and looking at everything! Ted PS - I like cheese 🙂
On-Page Optimization | | Jon-C0 -
Too many forum posts - may I delete the oldest ones?
Hello, We've got product pages with comments. We have collected so many comments for the last two years on these pages so they made problems to our website. We plan to cut every posts older than one year to solve this problem. Could it be bad for SEO? (Yes, these are really unique and relevant posts...) How could I predict the consequences? Thanks!
On-Page Optimization | | DDL0 -
Html and css errors - what do SE spiders do if they come across coding errors? Do they stop crawling the rest of the code below the error
I have a client who uses a template to build their websites (no problem with that) when I ran the site through w3c validator it threw up a number of errors, most of which where minor eg missing close tags and I suggested they fix them before I start their off site SEO campaigns. When I spoke to their web designer about the issues I was told that some of the errors where "just how its done" So if that's the case, but the validator still registers the error, do the SE spiders ignore them and move on, or does it penalize the site in some way?
On-Page Optimization | | pab10 -
Relating forum discussions to improve internal linking
Hi all, I have a matter i would like a discussion about, since I am looking for a good solution. First the case: I have a site with a large community based discussion board. We daily have 40 to 80 active forum threads with average of 200 posts. As it is right now each thread page shows, at the bottom of the thread, the 20 threads with latest activity. From a SEO point of view this is not the best solution, since all thread pages passes link juice to the latest 20 threads. However, the threads change daily so the juice is sprayed around all over the place. What I want to do is related forum threads, such that each thread at the bottom shows up to 10 threads which could be of interest to the reader. In this way one thread will have more or less the same threads at the bottom at all time, unless the relevancy is better for other threads, causing som minor changes to happen over time. The question is, how can one do this? In the backend, the forum has two tables. One holding threads and one holding posts relating to the threads. All in all the system has 66.500 threads, and in total 469.000 posts. Every thread has a title, and the posts are of varying length. The threads have categories, but they are not so distinct that a thread in one category can not be related to a thread in another category. So I would like to make the relevancy from the title of the thread and the content of the forum posts. As of yet I have not come up with a good solution, and i will look forward to reading any feedback for this. I will answer any questions as fast as possible, to get a good discussion goint here. Best regards, Rasmus
On-Page Optimization | | rasmusbang0 -
Site architecture for spatial location: Countries, states, regions: How deep should I go?
Hi, Based on the answers to my question about how to put the spatial location in the URL I'm now thinking about whether and how to flatten my information architecture. My main content is trails and courses. For both categories I have most content for Vancouver, BC (over 100 trails). I have some trails from California and more trails from other areas in BC (5-20 trails for 3 separate counties). My current site architecture is: trails -> country -> state/province -> county/regional district -> list of trails. So a trail page is 5 clicks away from the root. My course structure is: courses -> course list (I have far fewer courses but need to start structuring them) I did a search for site:example.com and found that my course pages rank most highly (probably because I have more inbound links for them) then I get workout pages then I get trail pages last of all. I want to be set up to scale for the rest of the world but I think I have to start winning in my local area first. What ideas might be good for a better site architecture? I'm thinking of doing this: trails -> location page -> list of trails for county. The location page would be a single page with a tree hierarchy from country to county - nicely styled to help the user. Something like: Canada -> British Columbia -> -> Greater Vancouver -> -> Okanagan-Similikameen -> -> Squamish-Lilloet United States -> California -> -> Marin I would make the urls be /trail/ca-bc-greater-vancouver/baden-powell-trail. I'm considering whether /trails/ca-bc/ (i.e. to get the state) should return a list of the counties. I'm worried about duplicate content for doing this. Curiously, my competitors don't have this structure at all. Access to their trails is by searching. Thoughts? Many thanks in advance
On-Page Optimization | | esarge0 -
Are Forum Links To Website Causing Rankings To Drop ?
For many years our top keywords hve ranked in top 5. Usually top 3 for a few in particular. The last few months those particular keywords have dropped to 25 +. Our other rankings are doing pretty decent and are remaiing the same. We are members of about 6 forums since they relate to our website and sometime we have contests on them. I do know that the forums are based on those particular keywords and are linking to our website, as well as other forum members linking to us. Each forum has roughly 300 links to our website. It does not seem as if our competitors are affected by this, as they are members of the forums as well. What could be causing this for these certain keywords? If those forums were affected by Google's new algorythm would this have an affect on our website?
On-Page Optimization | | hfranz0