Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Roger Bot

Moz Pro

2791

Bush_JSM Subscriber last edited by

Hi Mozzers,

I have a dev site that I want to run your crawl text on (Roger Bot) but I want to ensure the other engines don't crawl it.

What is the Robots.txt line that I need to make sure only Roger bot can get in and not Google etc?

Please advise

Thanks

Gareth
1 Reply Last reply
Reply Quote 0
kyleNeedham last edited by

HI Gareth, Your robots.txt should look like this; User-agent: * Disallow: / User-agent: rogerbot Allow: /
1 Reply Last reply
Reply Quote 1
intSchools last edited by

User-agent: *

Disallow: /

User-agent: rogerbot

Allow: /
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Unsolved The Moz.com bot is overloading my server

rogerbot moz crawler

The Moz.com bot is overloading my server. How to solve it?
Moz Pro | | paulavervo

0
How do I redeem a tweet from Roger?

Last week, I reached over 100 mozPoints and I've been waiting by my virtual mailbox for days for a tweet from Roger. Level mozPoints Benefits **Aspirant - ** ** - 100 - 199** ** - A tweet from Roger (the week you reach 100 MozPoints)** The past few days have been sad. I've been moping around the office feeling terribly alone, wondering why I've received no Twitter based recognition for my efforts. The radio has been playing 'Careless Whisper' on repeat and it hasn't stopped raining outside. Can anyone help? (Attached an image of my sadness for reference) fYHP3
Moz Pro | | seanginnaw

0
Why SEOmoz bot consider these as duplicate pages?

Hello here, SEOmoz bot has recently marked the following two pages as duplicate: http://www.virtualsheetmusic.com/score/PatrickCollectionFlPf.html?tab=mp3 http://www.virtualsheetmusic.com/score/PatrickCollectionFlPf.html?tab=pdf I don't personally see how these pages can be considered duplicate since their content is quite different. Thoughts??!!
Moz Pro | | fablau

0
Our Duplicate Content Crawled by SEOMoz Roger, but Not in Google Webmaster Tools

Hi Guys, We're new here and I couldn't find the answer to my question. Here it goes: We had SEOMoz's Roger Crawl all of our pages and he came up with quite a few erros (Duplicate Content, Duplicate Page Titles, Long URL's). Per our CTO and using our Google Webmaster Tools, we informed Google not to index those Duplicate Content Pages. For our Long URL Errors, they are redirected to SEF URL's. What we would like to know is if Roger is able to know that we have instructed Google to not index these pages. My concern is Should we still be concerned if Roger is still crawling those pages and the errors are not showing up in our Webmaster Tools Is there a way we can let Roger know so they don't come up as errors in our SEOMoz Tools? Thanks so much, e
Moz Pro | | RichSteel

0
Will SEOMoz offer URL data relating to Bot visits

Does SEOMoz in the future plan to report on Bot visits for each URL, when they are spidered and when they appear in for example Google's index ?
Moz Pro | | NeilTompkins

0
Hug From Roger

I have long since had enough Moz Points to get my hug and STILL have nothing. WHERE'S MY HUG!!!??
Moz Pro | | TheGrid

1
SEOmoz bot and "noindex"

As a recent newbie to SEOmoz, I've been implementing some suggestions and doing a general tidy up. I removed URL's from our robots txt, and rolled out instead the noindex meta tag to pages we don't want indexed. But surprised to see issues that are now flagged from the last crawl by the moz bot from pages that have this meta tag? Does the SEOmoz bot not ignore this tag? Just want to make sure I've implemented it correctly, so the google bot does ignore it. Meta tag syntax is and is placed below the title tag. cheers Steve
Moz Pro | | sjr4x4

0
Seomoz Spider/Bot Details

Hi All Our website identifies a list of search engine spiders so that it does not show them the session ID's when they come to crawl, preventing the search engines thinking there is duplicate content all over the place. The Seomoz has bought a over 20k crawl errors on the dashboard due to session ID's. Could someone please give the details for the Seomoz bot so that we can add it to the list on the website so when it does come to crawl it won't show it session ID's and give all these crawl errors. Thanks
Moz Pro | | blagger

1