Remove false urls

[webnology]

Hi,

we are using expressionengine as our CMS. Every url that gets tracked, which does not lead to a real page, gets directed to the home page. This gives us a lot of duplicate content. For every wrong link, we get +1 in duplicate content.

Is there a way to list the false urls, so Moz does not use them anymore in the statistics, or that Google does not crawl them anymore?

Where would be the best place to do this?

All help appreciated,
Michel

[webnology]

Hi,

thank you for taking the time to respond. The issues with broken paths are fixed, but those pages seem to be still alive in Google/Moz. I guess adjusting the robots file will be the best option here.

Thanks,
Michel

[Cyrus-Shepard]

It's been a few years since I was good at EE, but this problem plagued me as well.

There's no easy workaround to my knowledge, but erhaps the most direct way is to exclude those URLs via your robots.txt file, and then use Google's URL removal tool to wipe them from the index.

Adding them to your robots.txt file should take care of your duplicate content problems in Moz, and the removing them from Google's index should help with the rest.

Another option is to redirect these through your .htaccess file via 301. I won't get into the technicalities here, but it's another route to consider.

[Kara.Wallace]

Hi Michel,

I guess my first suggestion would be to fix the issue at the source - so you no longer have the broken linking in the first place.

If that is not an option and you can access the broken pages in EE, you could use the rel="canonical" to specify the preferred version of the duplicate page or use "noindex" on those pages to tell Moz and Google to not crawl those pages anymore.

If you cannot access the broken pages, you could set up 301 redirects so that the homepage content isn't being duplicated on other URLs... but that the broken links just permanently redirect to the homepage URL.

Does that make sense?

Mike