As Devanur says, this will achieve your goal. It's worth reiterating that there is nothing inherently wrong with /index.php URLs as long as you cannot access the same content without /index.php. For instance, if www.site.com/page1/index.php exists as well as www.site.com/page1/, then this is duplicate content and should be fixed. I imagine this is your current situation because this is most common when /index.php is being added to URLs.

However, if only one version of every page loads and that version has the /index.php extension, this is not automatically bad. It's preferable for the extension not to be there for the sake of URL tidiness and because this does move the content one folder-level away from the root (not a huge issue, but probably best avoided) however.

If you go through 301 redirects to shift the old URLs to the new ones without /index.php, your rankings should not suffer. There might be a little bit of ranking fluctuation as Google indexes the new URLs and acknowledges the redirects, but nothing permanent. It's worth noting that this is not an absolute rule, however, and that there is always a risk of lowered rankings or rankings not returning to what they were before after a 301 redirect though.

Cheers,

Jane

Hi there,

Sorry, can you clarify - are you saying that your client supposedly has links on pages like http://www.sammorganhomes.com/wp-fav/backup/supplement/semitruckleasing.html, but that those pages redirect to the spam site?

Hosting these pages as subfolders instead of subdomains should be fine, and is the "traditional" way of presenting new content with the highest likelihood that the content will inherit authority from its parent domain from an SEO point of view. They should rank just as well as if they were on subdomains or separate domains.

Separate installations and themes should not make a difference; Google will be absolutely clear in its understanding that these sub-brands are related to the main brand and to each other, so assuming you are fine with this, different design and CMS installations will not make a difference.

Hi there,

The first question, as the others have stated, is whether the content has been duplicated. If so, that's a problem no matter where the websites are located.

However, if you have two totally different websites from a content point of view on the same topic, hosting them together should not be a big problem. The only issue would be Google's understanding that the two resources were owned by the same entity, and therefore ranking them together might not constitute the most diverse user experience for people searching on the topic. Google reads registration information as well, and also understands that some servers / hosting services host hundreds of thousands of websites so some topic cross-over is natural without the sites being owned by the same person or company. If you had two sites selling pet food on the same dedicated server for the sole purpose of having two sites rank for the same pet food queries, I'm not 100% sure Google would ignore their hosting though.

Hi there,

Realistically, the tag should be used for duplicates, yes. How "duplicated" a page is, is subjective: a page with 50% of the same content as another page is probably going to count as duplicated as far as Google goes... where that line of duplication acceptability goes isn't something any of us really know.

For pages where the content is totally different besides the header and footer, you technically shouldn't use canonicalisation. However, experiments have shown that Google honours the tag, even if the pages aren't duplicates. Dr. Pete did an experiment when the tag came out (admittedly a few years ago) where he showed that you could radically reduce the number of pages Google had indexed for a site by canonicalising everything to the home page. I personally had a client do this by accident a couple of years ago, and sure enough, their number of indexed pages dropped very quickly, along with all the rankings those pages had. As an ecommerce site that was ranking for clothing terms, this was very very bad. It took about six weeks to get those rankings back again after we fixed the tags, and the tags were fixed within about five days (should have been quicker but our urgent request went into a dev queue).

So the answer would be that Google seems to honour the tag no matter the content of the pages, but I am pretty sure that if you asked a Googler, they'd tell you that it should only be used for dupes or near-dupes.

Hi Alan,

I replied to you about this on another thread, but I wanted to reiterate:

My SEO firm claims that Google looks at link profile only when there is a Penguin update and that these only occur once or twice a year.

This is not correct. Google only looks at removing or handing out Penguin penalties during a Penguin update (1, 2 or 3 times per year is the norm), but regular sites with regular link development can improve at all times. Link building might take a month or so to really show progress, but Google crawls and updates its indices almost constantly. It's simply not true that you will only see ranking improvements once or twice a year.

Similarly, you can lose rankings if you lose links that were counting, outside of a Penguin update.

Regarding DA and PA - yes, you can rank well with any combination of these. They are not indicators of how well you will rank, but a gauge of what your site's authority is relative to every other site Moz analyses, based upon backlinks. DA and PA are configured in a way that is meant to best represent how Moz believes Google values backlink profiles, but clearly Moz doesn't have access to Google's actual algorithm or measures of which links Google counts / discounts / values / doesn't value.

Also, "can I rank well" depends on what query you want to rank for. You will easily outrank a far higher authority website if your site is much more relevant to the query (although it's annoying to see high authority websites pop into SERPs for queries that they're not particularly relevant for, and it does happen from time to time).

If you have relatively low authority and you're trying to rank against similar websites with DA / PA scores in the 70s or so, your job is likely to be a lot harder. That said, it can still be done with focus on the specific niches you're interested in and work on improving the number and quality of links pointing to the site.

Hope this helps.

Cheers,

Jane

If you are happy for the second page to still exist but not rank, you should use the canonical tag to point the second page to the first one. This will lend the first page the majority of the strength of the second page and perhaps improve its authority and ranking as a result. However, the second page will no longer be indexed because the canonical tag tells Google: "ignore this page over here; it should be considered the same as the canonical version, here."

Again, this can benefit the first page, but it does mean that the second page will no longer rank at all. Only do this if you are okay with that scenario.

Cheers,

Jane

Hi Teddy,

Does this seem to have been resolved, or are you still receiving the notice about an invalid URL?

Hi Joanne,

Sorry for not totally expanding on what I meant as the outcome using javascript! Yes, you're "hiding" portions of the mega menu. There is speculation that search engines might execute JS commands to find out if there is content behind them, but at the moment this still seems a valid way to "remove" links from Google's total link count. Keep and eye on whether this changes though (and acknowledge that it might not be totally reliable, even now).

Perhaps check out this tutorial for JS menus: http://www.siteground.com/tutorials/menu/javascript.htm

Cheers,

Jane

Hi Damien,

So if I'm reading this correctly, the consultant is saying that due to the size of the site (tens of thousands of pages), and the need to categorise its content, that subdomains are the best choice.

I would say that there are far bigger websites using categories within subfolders, notably big retailers, e.g.

http://www.marksandspencer.com/c/beauty, http://www.marksandspencer.com/c/food-and-wine, http://www.marksandspencer.com/c/mands-bank

http://www.waitrose.com/home/inspiration.html, http://www.waitrose.com/home/wine.html, http://www.waitrose.com/content/waitrose/en/home/tv/highlights.html (<-- the last one being a crappy version, but a subdomain nonetheless)

and so do websites that deal with providing content for very different audiences:

http://www.ncaa.com/schools/tampa, http://www.ncaa.com/championships/lacrosse-men/d1/tickets, http://www.ncaa.com/news/swimming-men/article/2014-03-29/golden-bears-and-coach-david-durden-earn-third-national-title, http://www.ncaa.com/stats/football/fbs

Has the consultant provided examples of other websites doing this that would take on the same structure?

There are hundreds of examples of websites whose structure / categories are properly understood despite existing in subdirectories, so I'm still sceptical that this is a necessity.

This is not to say that a subdomain approach wouldn't work and is definitively bad or anything, I'm just not really convinced that the reasoning is strong enough to move content away from the root domain.

I disagree about user experience - from a user's perspective, the only difference between subfolders and subdomains is the URL they can see in the address bar. The rest is aesthetic. You can do or not do everything you'd do with the design of a website using subdirectories that you'd do with a website(s) employing subdomains. For example, just because content sits on www.icy-veins.com/wow/, its navigation wouldn't have to link to www.icy-veins.com/hearthstone/ or mention the other brand in any way if you don't want to. You can still have separate conversion funnels, newsletter sign-ups, advertising pages, etc.

Hi Rohit,

Just to follow up and add to what others have said: if you built links from third-party websites, pointing to your site, Google should crawl the third-party websites and see the links pointing to you within 15 days. If Google is not crawling the site more regularly than once every 15 days, the websites are probably lower in quality and not great sources of backlinks to begin with.

If you are using Moz / Ahrefs, etc. are slower in crawling and indexing than Google. Moz often updates once every 30 days, so could pick up these news links in its next crawl. Ahrefs tends to be slightly faster, but is still not instantaneous in picking up new links.

As others have said above, combating scrapers is a big reason, but you're relying on lazy scrapers not removing the tags.

Another reason is to prevent rogue content management systems from attaching unnecessary query strings to URLs, creating pages that can end up loading infinite times under different URLs. A canonical tag in the source file would mean that any number of duplicate pages point back to the original. The same reason goes for sites that have issues with redirecting www / non-www URLs to the correct version, or who deal with the same problem regarding secure / insecure URLs. In all these cases, the canonical tag is a bandaid, not a cure - it would be better to fix the underlying problem of the rogue CMS, incorrect redirection, etc. but the canonical tag (self-referencing) is there if you need it.

Google doesn't seem to have an issue with it, but if you're at all concerned about the other engines, use with care.

Hi Prashant,

Yes - any URLs that are different are different in Google's eyes, unless the modifier is a # symbol.

So if you have www.example.com/key#value12345 and www.example.com/key#valuexyzabc, then Google sees these as the same, i.e. www.example.com/key. They will ignore everything after the # character.

All other query strings, etc., mean that the URL has changed and if the pages on those URLs are the same, it's duplicate content.

I hope this helps.

Cheers,

Jane

Hi there,

Have you looked into the SEO guides available here on Moz? There are a lot of important on and off page factors to get right in order to gain good rankings. Have a look at the Beginner's Guide to SEO - it is not specifically about forums, but all the points included are important and count for what you're looking to do.

As Chris says, you need to look a off-page optimisation as well. It is very difficult to have a site rank well with little or no off-page work. Check out the link development category on the blog as well - there are years of great content about off-page work in there.

Cheers,

Jane

William and Gary sum it up really well here - directory listings can be useful, but they should be sought for other reasons than SEO. Noah and Eduardo are also right - the safest thing to do is to acquire these links with the nofollow tag.

If you want to take a risk or two, the highest quality directories probably won't hurt you if they're followed links. Google's stance on these things is pretty hard line though, so if you're working on behalf of a larger company or client, perhaps it's best to play it safe.

Hi Prashant,

This sounds like removal due to duplication rather than DMCA - the omission is usually noted as being because of DMCA notices if they are the reason, e.g. http://img.labnol.org/images/2008/07/googlesearchdmcacomplaint.png

Google likely sees these as duplicates, or near-dupes, as David has said,

Hi Conor,

Chris's answer here is perfect.

You definitely don't just want to link from the old site to the new; you want the old site to be "gone" and to serve a 301 redirect to tell search engines "this resource has permanently moved, and is now over here ----> [new site]".

301 redirects are vital; 302s do not pass authority from one URL to another (there is speculation that a 302 that has been in place for a very long time might be treated as a 301 by Bing, but let's go with the fact that they're next to useless in most cases for passing SEO value).

The 301 redirect should be implemented on a page by page basis, meaning that the home page at www.oldiste.com redirects to www.newsite.com. www.oldsite.com/page1.html redirects to that page's new location, e.g. www.newsite.com/page1.html, and so forth. As Chris says, you do not want www.oldsite.com/page23.html to redirect to the home page of www.newsite.com. The only case in which you'd want to do this is if /page23.html no longer exists on the new site.

How this is done depends on what type of server your site sits on. If you are using an Apache server, the web developers will do this in the site's .htaccess file. If you are using a Microsoft IIS server, this will be done in a control panel. The web development company will clearly know which server you're using and where this will be done.

Some servers and hosts still implement 302 redirects as standard and not 301s, which is very frustrating from an SEO point of view. There should be nothing more difficult about creating a 301 than a 302.

The short version is: 301 the old site to the new if you want to transfer the authority from the old site to the new one. This will ensure that you bring most of that authority with you.

Hi again,

Yep - your non-www and www pages are both resolving... e.g. http://cheaptubes.com/ and http://www.cheaptubes.com/ bring up the same content. Also, http://cheaptubes.com/default.htm and http://www.cheaptubes.com/default.htm is also a duplicate of the home page.

Internally, I am seeing the same thing, e.g. http://www.cheaptubes.com/carbon-nanotubes-prices.htm and http://cheaptubes.com/carbon-nanotubes-prices.htm - same page, one on the www subdomain ("www." is a subdomain like any other, just with an extremely common name) and one just sitting on the root.

The solution here is either to 301 redirect the non-www version of the site to the www version for every page, or to use the canonical tag to point from the non-preferred versions to the "canonical" versions. More information on this is available here.

You also have a situation where upper-case URLs will resolve as well as lower case ones, e.g. http://www.cheaptubes.com/SWNTs.htm and http://www.cheaptubes.com/swnts.htm (as well as http://cheaptubes.com/swnts.htm!).

URLs should only be allowed to resolve with one case, preferably lower. The upper / mixed case should 301 redirect to the proper version.

Essentially, the "two versions of the site" issue is the biggest problem, with all pages being available on at least two URLs - one with www and one without. There are other tidiness issues like /default.htm bringing up the home page as well.

Does this make sense? Let me know if this is not clear.

Best,

Jane

EGOL and Samuel raise some very good points here. I feel like you're also somewhat at risk of looking like you're trying to deceive the audience if you're deliberately divorcing the community from the brand site and image, but using it to drive the community to commercial action. I am not sure how subtle they are planning on making that connection, but subtlety in that manner rarely works: it's a lot more "honest" and appreciated to cultivate a community and openly market a product, as Moz has done.

Using the Moz example, this site did not start out as a tool / analytics provider, so the community actually came first. When I was a full time employee, we were largely an SEO consultancy. The blog was a labour of love as far as Rand and the staff were concerned. In 2007 when we first introduced Premium (now Pro) accounts, we clearly used the community (which was pretty substantial, even back then) to market the service. To all but a very few negative folks, this went down fine.

If Rand had started a separate company and website hosting and selling SEO tools and subscriptions, and had dropped hints and posts every now and again here to promote it, it would not have been met with nearly as much favour.

The SEO points about two sites being twice as hard to maintain, market and rank than one also apply, but I would question whether a smaller brand can build a totally separate community that is a) big enough, and b) independent enough not to annoy users if they figure out the commercial intent behind it. If there's one thing that's slightly worse than no community, it's an empty one.

Hi Damien,

So if I'm reading this correctly, the consultant is saying that due to the size of the site (tens of thousands of pages), and the need to categorise its content, that subdomains are the best choice.

I would say that there are far bigger websites using categories within subfolders, notably big retailers, e.g.

http://www.marksandspencer.com/c/beauty, http://www.marksandspencer.com/c/food-and-wine, http://www.marksandspencer.com/c/mands-bank

http://www.waitrose.com/home/inspiration.html, http://www.waitrose.com/home/wine.html, http://www.waitrose.com/content/waitrose/en/home/tv/highlights.html (<-- the last one being a crappy version, but a subdomain nonetheless)

and so do websites that deal with providing content for very different audiences:

http://www.ncaa.com/schools/tampa, http://www.ncaa.com/championships/lacrosse-men/d1/tickets, http://www.ncaa.com/news/swimming-men/article/2014-03-29/golden-bears-and-coach-david-durden-earn-third-national-title, http://www.ncaa.com/stats/football/fbs

Has the consultant provided examples of other websites doing this that would take on the same structure?

There are hundreds of examples of websites whose structure / categories are properly understood despite existing in subdirectories, so I'm still sceptical that this is a necessity.

This is not to say that a subdomain approach wouldn't work and is definitively bad or anything, I'm just not really convinced that the reasoning is strong enough to move content away from the root domain.

I disagree about user experience - from a user's perspective, the only difference between subfolders and subdomains is the URL they can see in the address bar. The rest is aesthetic. You can do or not do everything you'd do with the design of a website using subdirectories that you'd do with a website(s) employing subdomains. For example, just because content sits on www.icy-veins.com/wow/, its navigation wouldn't have to link to www.icy-veins.com/hearthstone/ or mention the other brand in any way if you don't want to. You can still have separate conversion funnels, newsletter sign-ups, advertising pages, etc.