How deep should I let my forum be spidered
-
I run quote a niche website that's been running since late 1999 and over that time I've built up something like 4000 resources which consist of either text articles or image galleries and reviews along side another few thousand news stories relating to the niche interest. On top of the main site I also have a forum which isn't especially optimised for SEO and I was wondering, whilst was cleaning it up, whether anyone has any tips / suggestion / best practices for forum SEO.
Because it is all UGC the quality of the posts can be quite weak so I was wondering whether I should block robots completely from the forum, which seems a little harsh, whether I should let the whole forum be spidered (which seems a little excessive and potentially a bad thing) or whether I should restrict things to that only the main index and perhaps one page of topics and their posts be accessible to robots and then nofollow the rest?
Any thoughts?
-
Hey mate.
UGC can actually be very high quality content, and subsequently I would let Google run wild through all the content.
Forums can be tricky and you would want to make sure Google doesnt index any duplicate content.
I would use this tool, http://home.snafu.de/tilman/xenulink.html to crawl the forum. Export to Excel and then assess all the URL structures and see if it's picking up any strange parameters or duplicate content.
I would block the weird paramters in Google webmaster tools, and if technically possible 301 all the duplicate content issues. The problem with restricting things in robots.txt is that you might actually block link juice accidentally.
Hope that helps - my background is more in ecommerce but im sure we'll have similar site structure issues.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
On-Page Optimization | | Jacksons_Fencing0 -
Do deep pages issues affect homepage chances of ranking?
Hello community, I have a general question: let's say you have some issues in deep pages, like duplicate pages without a canonical tag, or missing description or missing titles, etc.. Will these issues affect the chances of the homepage ranking if the homepage is optimized (no duplicates, canonical, good keywords volume, alt text, etc.) and has none of the issues present in deep pages within the site? Gracias.
On-Page Optimization | | EduardoRuiz0 -
Duplicate links from forum what to do?
After a crawl it found over 5k errors and over 5k warnings. Those are: Duplicate page content; Duplicate page title; Overly-Dynamic URLs; Missing Meta descr; Title Element too long. All those come from domain.com/forum/ I don't need SEO on forum so what should I do? What could be an easy solution to this? No index? No follow? Please help
On-Page Optimization | | OVJ0 -
Using Transcription Service For Videos - Have Question Around Search and Spiders
Hi All, So I have put together a weekly video series on security topics. I have read an SEOmoz post around how you can boost SEO by adding the transcription to the page, which makes perfectly good sense. My question is, can I include the first couple of paragraphs and then have a "read the full transcription" so when the user clicks, the rest of the content appears? Do the search engine spiders only crawl the first two paragraphs in this instance or do they crawl the whole thing even though the entire content is not on the page? I dont mind making the page longer and including the entire transcription if it is easier for SEO but if there is no difference, than I think the first option would be the best user experience. Thanks for the help Pat
On-Page Optimization | | PatBausemer0 -
SeoMoz Pro - Keyword stuffing but its a forum
hiya guys Just going through the Seomoz Pro on page thing One of the flagged items is keyword stuffing, I have 22 keywords (stuffed) but they want it down to 15, the thing is, my website is a forum. and the keyword stuffing is from the title name of each categories eg its a nightlife forum so Birmingham nightlife forum Brighton nightlife forum Leeds nightlife forum Manchester nightlife forum u get the picture To avoid this keyword stuffing, should I change some of the forum names to eg leeds nightlife forum > leeds nightlife forums? just by adding the 's that will half the keywords, but its not solving the problem in the long run, if i was to add more cities and areas with the XXXX nightlife foru. What you reckon guys? Cheers guys Luke
On-Page Optimization | | Lukescotty0 -
Html and css errors - what do SE spiders do if they come across coding errors? Do they stop crawling the rest of the code below the error
I have a client who uses a template to build their websites (no problem with that) when I ran the site through w3c validator it threw up a number of errors, most of which where minor eg missing close tags and I suggested they fix them before I start their off site SEO campaigns. When I spoke to their web designer about the issues I was told that some of the errors where "just how its done" So if that's the case, but the validator still registers the error, do the SE spiders ignore them and move on, or does it penalize the site in some way?
On-Page Optimization | | pab10 -
Site architecture for spatial location: Countries, states, regions: How deep should I go?
Hi, Based on the answers to my question about how to put the spatial location in the URL I'm now thinking about whether and how to flatten my information architecture. My main content is trails and courses. For both categories I have most content for Vancouver, BC (over 100 trails). I have some trails from California and more trails from other areas in BC (5-20 trails for 3 separate counties). My current site architecture is: trails -> country -> state/province -> county/regional district -> list of trails. So a trail page is 5 clicks away from the root. My course structure is: courses -> course list (I have far fewer courses but need to start structuring them) I did a search for site:example.com and found that my course pages rank most highly (probably because I have more inbound links for them) then I get workout pages then I get trail pages last of all. I want to be set up to scale for the rest of the world but I think I have to start winning in my local area first. What ideas might be good for a better site architecture? I'm thinking of doing this: trails -> location page -> list of trails for county. The location page would be a single page with a tree hierarchy from country to county - nicely styled to help the user. Something like: Canada -> British Columbia -> -> Greater Vancouver -> -> Okanagan-Similikameen -> -> Squamish-Lilloet United States -> California -> -> Marin I would make the urls be /trail/ca-bc-greater-vancouver/baden-powell-trail. I'm considering whether /trails/ca-bc/ (i.e. to get the state) should return a list of the counties. I'm worried about duplicate content for doing this. Curiously, my competitors don't have this structure at all. Access to their trails is by searching. Thoughts? Many thanks in advance
On-Page Optimization | | esarge0 -
How deep should I go with a directory site?
I am creating a new site which has a directory component. Based on what I have ready I am inclined to keep the site architecture as flat as possible. However, the natural layout that I have come up with in my head has the directory listings 5 or 6 pages deep in the site structure. I saw in another post that someone in a similar situation was suggesting that going deep like this is fine so long as there are many internal links to the deeper pages to indicate that they are important. Should I make a conscious effort to make the site architecture as flat as possible? Are there any specific guides/resources that address this particular issue that I should be aware of? Thanks!
On-Page Optimization | | fastestmanalive0