Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Roger Bot

Moz Pro

2805

Bush_JSM Subscriber last edited by

Hi Mozzers,

I have a dev site that I want to run your crawl text on (Roger Bot) but I want to ensure the other engines don't crawl it.

What is the Robots.txt line that I need to make sure only Roger bot can get in and not Google etc?

Please advise

Thanks

Gareth
1 Reply Last reply
Reply Quote 0
kyleNeedham last edited by

HI Gareth, Your robots.txt should look like this; User-agent: * Disallow: / User-agent: rogerbot Allow: /
1 Reply Last reply
Reply Quote 1
intSchools last edited by

User-agent: *

Disallow: /

User-agent: rogerbot

Allow: /
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Unsolved The Moz.com bot is overloading my server

rogerbot moz crawler

The Moz.com bot is overloading my server. How to solve it?
Moz Pro | | paulavervo

0
What Are Roger's Super Powers?

I ended up with a box of SEOMOZ swag. (Thanks! As to how this came to pass...I shall draw a veil, as they used to say in Victorian novels.) My upstairs neighbour Max, age 5, enjoys a rich fantasy life and is very much into superheroes and costumes. Naturally, he ended up with a lot of the Roger stickers. Alas, I was unable to answer all of Max's questions. When he asked: "What does Roger do?" I replied: "Roger makes your computer work." Pretty good, I thought. But then Max asked: "What does the antenae do?" I was kind of stumped. Then it got worse. Max asked what Roger's superpowers are and if he could beat Spiderman. I tried to change the subject. Max wasn't impressed. What are the answers? Enquiring five year old minds want to know!
Moz Pro | | DanielFreedman

6
Our Duplicate Content Crawled by SEOMoz Roger, but Not in Google Webmaster Tools

Hi Guys, We're new here and I couldn't find the answer to my question. Here it goes: We had SEOMoz's Roger Crawl all of our pages and he came up with quite a few erros (Duplicate Content, Duplicate Page Titles, Long URL's). Per our CTO and using our Google Webmaster Tools, we informed Google not to index those Duplicate Content Pages. For our Long URL Errors, they are redirected to SEF URL's. What we would like to know is if Roger is able to know that we have instructed Google to not index these pages. My concern is Should we still be concerned if Roger is still crawling those pages and the errors are not showing up in our Webmaster Tools Is there a way we can let Roger know so they don't come up as errors in our SEOMoz Tools? Thanks so much, e
Moz Pro | | RichSteel

0
Will SEOMoz offer URL data relating to Bot visits

Does SEOMoz in the future plan to report on Bot visits for each URL, when they are spidered and when they appear in for example Google's index ?
Moz Pro | | NeilTompkins

0
Hug From Roger

I have long since had enough Moz Points to get my hug and STILL have nothing. WHERE'S MY HUG!!!??
Moz Pro | | TheGrid

1
Why is Roger crawling pages that are disallowed in my robots.txt file?

I have specified the following in my robots.txt file: Disallow: /catalog/product_compare/ Yet Roger is crawling these pages = 1,357 errors. Is this a bug or am I missing something in my robots.txt file? Here's one of the URLs that Roger pulled: <colgroup><col width="312"></colgroup>
| example.com/catalog/product_compare/add/product/19241/uenc/aHR0cDovL2ZyZXNocHJvZHVjZWNsb3RoZXMuY29tL3RvcHMvYWxsLXRvcHM_cD02/ Please let me know if my problem is in robots.txt or if Roger spaced this one. Thanks! |
Moz Pro | | MeltButterySpread

0
Crawl test. Bot crawled only 200 or so links when it should have crawled thousands

Hi everyone, I just recieved my crawl test report and its only given me 200 or so URL's when my site has thousands, any thoughts?
Moz Pro | | Ev84

0
SEOmoz Bot indexing JSON as content

Hello, We have a bunch of pages that contain local JSON we use to display a slideshow. This JSON has a bunch of<a links="" in="" it. <="" p=""></a> <a links="" in="" it. <="" p="">For some reason, these</a><a links="" that="" are="" in="" json="" being="" indexed="" and="" recognized="" by="" the="" seomoz="" bot="" showing="" up="" as="" legit="" for="" page. <="" p=""></a> <a links="" that="" are="" in="" json="" being="" indexed="" and="" recognized="" by="" the="" seomoz="" bot="" showing="" up="" as="" legit="" for="" page. <="" p="">One example page this is happening on is: http://www.trendhunter.com/trends/a2591-simplifies-product-logos . Searching for the string '<a' yields="" 1100+="" results="" (all="" of="" which="" are="" recognized="" as="" links="" for="" that="" page="" in="" seomoz),="" however,="" ~980="" these="" json="" code="" and="" not="" actual="" on="" the="" page.="" this="" leads="" to="" a="" lot="" invalid="" our="" site,="" super="" inflated="" count="" on-page="" page. <="" span=""></a'></a> <a links="" that="" are="" in="" json="" being="" indexed="" and="" recognized="" by="" the="" seomoz="" bot="" showing="" up="" as="" legit="" for="" page. <="" p="">Is this a bug in the SEOMoz bot? and if not, does google work the same way?</a>
Moz Pro | | trendhunter-159837

0