How deep should I let my forum be spidered
-
I run quote a niche website that's been running since late 1999 and over that time I've built up something like 4000 resources which consist of either text articles or image galleries and reviews along side another few thousand news stories relating to the niche interest. On top of the main site I also have a forum which isn't especially optimised for SEO and I was wondering, whilst was cleaning it up, whether anyone has any tips / suggestion / best practices for forum SEO.
Because it is all UGC the quality of the posts can be quite weak so I was wondering whether I should block robots completely from the forum, which seems a little harsh, whether I should let the whole forum be spidered (which seems a little excessive and potentially a bad thing) or whether I should restrict things to that only the main index and perhaps one page of topics and their posts be accessible to robots and then nofollow the rest?
Any thoughts?
-
Hey mate.
UGC can actually be very high quality content, and subsequently I would let Google run wild through all the content.
Forums can be tricky and you would want to make sure Google doesnt index any duplicate content.
I would use this tool, http://home.snafu.de/tilman/xenulink.html to crawl the forum. Export to Excel and then assess all the URL structures and see if it's picking up any strange parameters or duplicate content.
I would block the weird paramters in Google webmaster tools, and if technically possible 301 all the duplicate content issues. The problem with restricting things in robots.txt is that you might actually block link juice accidentally.
Hope that helps - my background is more in ecommerce but im sure we'll have similar site structure issues.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate content because of member only restrictions on a forum.
Our website's Community Forum links to the membership profile pages, which by default are blocked for non-members. https://www.foodbloggerpro.com/community/ https://www.foodbloggerpro.com/community/member/1301/ We're getting warnings in Moz for duplicate content (and errors) on these member profile pages. Any ideas for how we can creatively solve this problem? Should we redirect those pages or just beef them up with more content? Just ignore it and assume that search spiders will be smart enough to figure it out? See attached video for further explanation. Community_Area.mp4
On-Page Optimization | | Bjork0 -
Do Search Engine Spiders Read Commented Out Content?
Do Search Engine Spiders Read Commented Out Content? Is commented out content detrimental?
On-Page Optimization | | lbohen0 -
Content in forum signatures being spidered, does it matter?
Hello, first post here, just started with SEOmoz so hope it's relevant. Searched a fair bit on this without getting a good answer either way so interested to get some opinions. The core of the site I run is a forum dedicated to collecting, for the sake of argument let's say cars. A good percentage of the users have signatures which list their collection, for example 1968 Car A - 1987 Car B - 1998 Car D and so on.... These signatures lists can be 20 items or more, some hotlink the signautres back to the relevant post on the forum, some not. The signatures show on every post on which the user makes. What I'm noting is a) SEOMoz is reporting a LOT of links on every forum page, due mainly to these signatures I guess. and of more interest b) The content of the signatures is being spidered. So for example of you search for '1968 Car A' you might get a couple of good results directly relevant to '1968 Car A' from my site, but you also get a lot of other non-relevant threads as results because the user just happens to have posted on them. Obviously this is much more apparent on the site google search. So what is the best approach? Leave as is? Hide the signatures from the BOTs? Another approach?
On-Page Optimization | | rutteger0 -
Advice with keywords - category - Forum
Hiya guys Everyone has been really good to me on here, just wanted a bit of advice with the keywords on my forum. my website is a nightlife forum for the UK, each city has its own section. Each section has a eg: _What's on in Birmingham? Club Nights, Upcoming Events, Promotions _ as the Title category, Should I drop the Club Nights, Upcoming Events, Promotions and put that in the description of the forum. So it'll just be What's on in Birmingham? with a description Find Club night information, Upcoming events and pr............. eg Just wondering if it was to stop searches been made, like, Club nights in Birmingham etc. from being targeted. Your thoughts please guys Thanks for reading Lukescotty
On-Page Optimization | | Lukescotty0 -
Getting 403 error in forum
Hi all, I am getting 403 error for my site where it is throwing error for the following url http://www.topuniversityforum.in/members/member id/ignore and it is showing 7 similar url for 7 user ids. I want to know how can i resolve it and if it is going to have any negative effect on its ranking.
On-Page Optimization | | akhilendra0 -
Site architecture for spatial location: Countries, states, regions: How deep should I go?
Hi, Based on the answers to my question about how to put the spatial location in the URL I'm now thinking about whether and how to flatten my information architecture. My main content is trails and courses. For both categories I have most content for Vancouver, BC (over 100 trails). I have some trails from California and more trails from other areas in BC (5-20 trails for 3 separate counties). My current site architecture is: trails -> country -> state/province -> county/regional district -> list of trails. So a trail page is 5 clicks away from the root. My course structure is: courses -> course list (I have far fewer courses but need to start structuring them) I did a search for site:example.com and found that my course pages rank most highly (probably because I have more inbound links for them) then I get workout pages then I get trail pages last of all. I want to be set up to scale for the rest of the world but I think I have to start winning in my local area first. What ideas might be good for a better site architecture? I'm thinking of doing this: trails -> location page -> list of trails for county. The location page would be a single page with a tree hierarchy from country to county - nicely styled to help the user. Something like: Canada -> British Columbia -> -> Greater Vancouver -> -> Okanagan-Similikameen -> -> Squamish-Lilloet United States -> California -> -> Marin I would make the urls be /trail/ca-bc-greater-vancouver/baden-powell-trail. I'm considering whether /trails/ca-bc/ (i.e. to get the state) should return a list of the counties. I'm worried about duplicate content for doing this. Curiously, my competitors don't have this structure at all. Access to their trails is by searching. Thoughts? Many thanks in advance
On-Page Optimization | | esarge0 -
Should I let Google index tags?
Should I let Google index tags? Positive? Negative Right now Google index every page, including tags... looks like I am risking to get duplicate content errors? If thats true should I just block /tag in robots.txt Also is it better to have as many pages indexed by google or it's should be as lees as possible and specific to the content as much as possible. Cheers
On-Page Optimization | | DiamondJewelryEmpire0 -
How deep should I go with a directory site?
I am creating a new site which has a directory component. Based on what I have ready I am inclined to keep the site architecture as flat as possible. However, the natural layout that I have come up with in my head has the directory listings 5 or 6 pages deep in the site structure. I saw in another post that someone in a similar situation was suggesting that going deep like this is fine so long as there are many internal links to the deeper pages to indicate that they are important. Should I make a conscious effort to make the site architecture as flat as possible? Are there any specific guides/resources that address this particular issue that I should be aware of? Thanks!
On-Page Optimization | | fastestmanalive0