Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Roger Bot

Moz Pro

2791

Bush_JSM Subscriber last edited by

Hi Mozzers,

I have a dev site that I want to run your crawl text on (Roger Bot) but I want to ensure the other engines don't crawl it.

What is the Robots.txt line that I need to make sure only Roger bot can get in and not Google etc?

Please advise

Thanks

Gareth
1 Reply Last reply
Reply Quote 0
kyleNeedham last edited by

HI Gareth, Your robots.txt should look like this; User-agent: * Disallow: / User-agent: rogerbot Allow: /
1 Reply Last reply
Reply Quote 1
intSchools last edited by

User-agent: *

Disallow: /

User-agent: rogerbot

Allow: /
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Unsolved The Moz.com bot is overloading my server

rogerbot moz crawler

The Moz.com bot is overloading my server. How to solve it?
Moz Pro | | paulavervo

0
Have a Campaign, but only states 1 page has been crawled by SEOmoz bots. What needs to be done to have all the pages crawled?

We have a campaign running for a client in SEOmoz and only 1 page has been crawled per SEOmoz' data. There are many pages in the site and a new blog with more and more articles posted each month, yet Moz is not crawling anything, aside from maybe the Home page. The odd thing is, Moz is reporting more data on all the other inner pages though for errors, duplicate content, etc... What should we do so all the pages get crawled by Moz? I don't want to delete and start over as we followed all the steps properly when setting up. Thank you for any tips here.
Moz Pro | | WhiteboardCreations

0
Where is my Hug from Roger?

I just remembered that when I got 50 mozPoints I had to receive a hug from Roger. So, I need my hug!
Moz Pro | | ditoroin

2
What Are Roger's Super Powers?

I ended up with a box of SEOMOZ swag. (Thanks! As to how this came to pass...I shall draw a veil, as they used to say in Victorian novels.) My upstairs neighbour Max, age 5, enjoys a rich fantasy life and is very much into superheroes and costumes. Naturally, he ended up with a lot of the Roger stickers. Alas, I was unable to answer all of Max's questions. When he asked: "What does Roger do?" I replied: "Roger makes your computer work." Pretty good, I thought. But then Max asked: "What does the antenae do?" I was kind of stumped. Then it got worse. Max asked what Roger's superpowers are and if he could beat Spiderman. I tried to change the subject. Max wasn't impressed. What are the answers? Enquiring five year old minds want to know!
Moz Pro | | DanielFreedman

6
Why SEOmoz bot consider these as duplicate pages?

Hello here, SEOmoz bot has recently marked the following two pages as duplicate: http://www.virtualsheetmusic.com/score/PatrickCollectionFlPf.html?tab=mp3 http://www.virtualsheetmusic.com/score/PatrickCollectionFlPf.html?tab=pdf I don't personally see how these pages can be considered duplicate since their content is quite different. Thoughts??!!
Moz Pro | | fablau

0
SEOmoz bot and "noindex"

As a recent newbie to SEOmoz, I've been implementing some suggestions and doing a general tidy up. I removed URL's from our robots txt, and rolled out instead the noindex meta tag to pages we don't want indexed. But surprised to see issues that are now flagged from the last crawl by the moz bot from pages that have this meta tag? Does the SEOmoz bot not ignore this tag? Just want to make sure I've implemented it correctly, so the google bot does ignore it. Meta tag syntax is and is placed below the title tag. cheers Steve
Moz Pro | | sjr4x4

0
Crawl test. Bot crawled only 200 or so links when it should have crawled thousands

Hi everyone, I just recieved my crawl test report and its only given me 200 or so URL's when my site has thousands, any thoughts?
Moz Pro | | Ev84

0
SEOmoz Bot indexing JSON as content

Hello, We have a bunch of pages that contain local JSON we use to display a slideshow. This JSON has a bunch of<a links="" in="" it. <="" p=""></a> <a links="" in="" it. <="" p="">For some reason, these</a><a links="" that="" are="" in="" json="" being="" indexed="" and="" recognized="" by="" the="" seomoz="" bot="" showing="" up="" as="" legit="" for="" page. <="" p=""></a> <a links="" that="" are="" in="" json="" being="" indexed="" and="" recognized="" by="" the="" seomoz="" bot="" showing="" up="" as="" legit="" for="" page. <="" p="">One example page this is happening on is: http://www.trendhunter.com/trends/a2591-simplifies-product-logos . Searching for the string '<a' yields="" 1100+="" results="" (all="" of="" which="" are="" recognized="" as="" links="" for="" that="" page="" in="" seomoz),="" however,="" ~980="" these="" json="" code="" and="" not="" actual="" on="" the="" page.="" this="" leads="" to="" a="" lot="" invalid="" our="" site,="" super="" inflated="" count="" on-page="" page. <="" span=""></a'></a> <a links="" that="" are="" in="" json="" being="" indexed="" and="" recognized="" by="" the="" seomoz="" bot="" showing="" up="" as="" legit="" for="" page. <="" p="">Is this a bug in the SEOMoz bot? and if not, does google work the same way?</a>
Moz Pro | | trendhunter-159837

0