Welcome to the Q&A Forum

Thank you for the reply. We want to allow all crawling, except for rogerbot in the community folder.

I have updated the robots.txt to the following, does this look right?:

User-agent: *
Disallow:

User-agent: rogerbot
Disallow: /community/

User-agent: Mediapartners-Google
Disallow:

Sitemap: http://www.faithology.com/sitemap.xml

view the robots here: http://www.faithology.com/robots.txt

Could you please review our robots.txt file and let me know if this is correct.

www.faithology.com/robots.txt

Thank you!

For our company website faithology.com we are attempting to block out any urls that contain a ? mark to keep google from seeing some pages as duplicates.

Our robots.txt is as follows:

User-Agent: *
Disallow: /*?

User-agent: rogerbot
Disallow: /community/

Is the above correct?  We are wanting them to not crawl any url with a "?" inside, however we don't want to harm ourselves in seo.  

Thanks for your help!

Hello everyone.

I have been reading through this site for a while and tried to put everything together that I have learned so far.  Would any of you mind looking at our site and providing any pointers or areas we can still improve on or areas I completely missed.  I appreciate any feedback you can give!

Our site is faithology.com

Thanks again!

Brandon

SEScout.com or micrositemasters.com