How can I block unwanted urls being indexed on google?

[VipinLouka78]

Hi,

I have to block unwanted urls (not that page) from being indexed on google. I have to block urls like example.com/entertainment not the exact page example.com/entertainment.aspx . Is there any other ways other than robot.txt? If i add this to robot.txt will that block my other url too? Or should I make a 301 redirection from example.com/entertainment to example.com/entertainment.aspx. Because some of the unwanted urls are linked from other sites.

thanks in advance.

[VipinLouka78]

Hi Nick,

Thanks so much!

Is there any advantage for shorter URLs over long ones? My site is on IIS6, How can I remove the .aspx extension in IIS6?

[Nick_Ker]

Since some of the unwanted URLs are linked from other sites, use 301 redirects.

Usually, people want to remove the file extensions like .aspx for cleaner looking, shorter URLs.

You can remove the .aspx extension from all pages with a rewrite like this in .htaccess:

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}.aspx -f
RewriteRule ^(.*)$ $1.aspx

If you want to keep the URLs with .aspx, rather than without, I think you will have to use individual 301 redirects.

[seohive-222720]

Vipin

My understanding is that if you block anything from Robots.txt that you prevent link juice from flowing for the pages.  If the pages in question have links to them then I agree with Highland to 301 redirect them to the correct pages.

shivun

[Highland]

301 is definitely the way to go, especially if you have inbound links to the unwanted pages. That will preserve most of your rank and transfer it to the actual page.

As far as blocking goes, you can use a robots.txt or a robots meta tag on your page with a NOFOLLOW,NOINDEX