Question about Robot.txt

[paumer80]

I just started my own e-commerce website and I hosted it to one of the popular e-commerce platform Pinnacle Cart. It has a lot of functions like, page sorting, mobile website, etc. After adjusting the URL parameters in Google webmaster last 3 weeks ago, I still get the same duplicate errors on meta titles and descriptions based from Google Crawl and SEOMOZ crawl. I am not sure if I made a mistake of choosing pinnacle cart because it is not that flexible in terms of editing the core website pages. There is now way to adjust the canonical, to insert robot.txt on every pages etc. however it has a function to submit just one page of robot.txt. and edit the .htcaccess. The website pages is in PHP format.

For example this URL:

www.mycompany.com has a duplicate title and description with www.mycompany.com/site-map.html (there is no way of editing the title and description of my sitemap)

Another error is

www.mycompany.com has a duplicate title and description with http://www.mycompany.com/brands?url=brands

Is it possible to exclude those website with "url=" and my "sitemap.html" in the robot.txt? or the URL parameters from Google is enough and it just takes a lot of time.

Can somebody help me on the format of Robot.txt. Please? thanks

[paumer80]

Thank you for your reply. This surely helps. I will probably edit the htaccess.

[Andropenis_Australia]

That's the problem with most sitebuilder type prgrams, they are very limited.

Perhaps look at your site title, and page titles. Usually the site title will be the included on all of your webpages followed by the page title so you could simply name your site www.yourcompany.com then add an individual page title to each page.

A robots.txt file is not supposed to be added to every page and only tells the bots what to crawl, and what not to.

If you can edit the htaccess, you should be able to get to the individual pages and insert/change the code for titles, just be aware that doing it manually can work, but sometimes when you go back to make an edit in the builder it may undo all of your manual changes, if that's the case, get your site perfect, then do the individual code changes as the last change.

Hope this helps.

[paumer80]

I have no way of adding those too. Ooops thanks for the warning. I guess I would have to wait for Google to filter out the parameters.

Thanks for your answer.

[irvingw]

You certainly don't want to block your sitemap file in robots.txt. It takes some time for Google to filter out the parameters and that is the right approach. If there is no way to change the title, I wouldn't be so concerned over a few pages with duplicate titles. Do you have the ability to add a noindex,follow meta tag on these pages?