Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Moz crawler

Moz Bar

1049

Emanuele_Ricci last edited by

I have a site which is in a non production status.

Crawlers are blocked vis robot txt.

User-agent: *
Disallow: /

I WANT TO MAKE A CRAWLING TEST WITH MOZ CRAWLER (RogerBot) ,
how can I allow your crawler to get in and prevent other crawlers from indexing the site?

Thanks

memok
1 Reply Last reply
Reply Quote 0
maxweb last edited by

Try this:

User-agent: *
Disallow: /
User-agent: rogerbot
Allow: /
1 Reply Last reply
Reply Quote 3
GPainter last edited by

did you take a look through - http://moz.com/learn/seo/robotstxt

I'll give you a clue -

Allow a specific web crawler to visit a specific web page

Disallow: /no-bots/block-all-bots-except-rogerbot-page.html User-agent: rogerbot Allow: /no-bots/block-all-bots-except-rogerbot-page.htmlmore info -http://moz.com/help/pro/rogerbot-crawler&http://moz.com/help/pro/what-is-rogerbot-
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Moz is not showing traffic stats..;(

I have connected my analytics account in moz. But it is not showing my traffic stats.
Moz Bar | | daniyal_ahmed

0
What does it mean when Moz KW explorer returns a negative keyword difficulty score (eg -2)?

I recently did a keyowrd difficulty lookup in Moz keyword explorer as usual and for the first time a saw several (out of 100s) of negative keyowrd scores (generally -2). Is this a bug, what does it mean?
Moz Bar | | FlagshipCons

1
Moz Pro Question: Does the amount of keywords you are allowed to search reset each month?

I am a Moz Pro subscriber and I really love the new Keyword Explorer tool. One question I have that I couldn't find a clear answer was regarding the number of monthly keyword queries. Do they reset each month? I hope they do.
Moz Bar | | joemaclean

0
How non-US Moz customers will use Keyword Explorer after the Keyword Difficulty tool is retired?

The new Moz Keyword Explorer looks good but its search volume is US based and completely useless for non-US websites. This is from Rand's post: "while the tool can search any Google domain in any country, the volume numbers will always be for US-volume. In the future, we hope to add volume data for other geos as well." In the Keyword Difficulty tool, Moz shows Google search volume data, which is similar to what I see in the Google Keyword Planner and Google Search Console. For example, keyword X in the Australian search market has 6-7k searches in the Google Keyword Planner and 8k searches in Moz. The very same keyword has 118k-300k search volume in the new Keyword Explorer! Obviously this new search volume is not useful in the Australian market. I often used the Keyword Difficulty tool to identify new keyword opportunities but what can I do to complete the same tasks after they retire the tool?
Moz Bar | | Gyorgy.B

2
Moz Crawl Test Trying to Crawl Contact Form Submit Button Location?

Moz Crawl Test for some reason is trying to Crawl a contact form Widget Submit Location. My obvious guess is that obviously the crawl cannot submit to the required fields…..I believe this because they're only kicking back these errors on the pages I have a contact form widget on. http://crawfordspest.com/pest-control/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
http://crawfordspest.com/tree-services/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
http://crawfordspest.com/lawn-care/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
http://crawfordspest.com/specialty-services/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404 Can you shed any insight to this? I'm a bit worried that I'll have to complete gut the contact form which was one of the major requests my client requested. Or in a worse scenario make all fields not required. It would let so much spam in. I have never seem anything like this at all. But I've learned a lot from Moz, and with major errors like 404 damage Domain Authority greatly. I've fixed 404 issues with newly acquired clients existing sites and tracked through Moz and the domain authority flies up once these errors are fixed. Along with fixing what Webmaster Tools through Google reports back. ..... Let me know if you have any expertise on this matter.
Moz Bar | | Funk-Creative-Media

0
Using Moz to find blogger with good PR.

Hi, My manager has recently informed me that you can use Moz in order to find blogger by category and their page rank. Is this possible? The reason for this is because we are looking for blogger with a good page rank and who are relevant to our sector, which is days out, outdoor activities, zipwires, adventure etc. We then want to contact them to see if we can invite them to write about our Tree Top Adventure and zip trekking activities. Any help would be much appreciated.
Moz Bar | | GoApe_2014

0
Rel Canonical and Moz Crawl

we have Rel Canonical tags set up on a few pages. When viewing the page source, the tags are correct. However, Moz Crawl results show the opposite. for example the page source, correctly shows, URL X with a Rel canonical Tag of URL Y
but.. Moz crawl is showing URL Y with a Rel Canonical Tag of URL X ..any thoughts why this would happen? which should i trust more?
Moz Bar | | S.S.N

0
Moz Keyword Ranking Report

Hi, Moz Keyword Ranking report shows different results then what i see on my desktop for some keywords. There wasn't much difference on most of them until now so ignored some of the differences but now i see huge differences. For instance, on my Google UK I see the keyword ranking 1 or 2 but Moz reports show 22 or higher than that . Search Engine is Google UK. Would you please advise if I am doing anything wrong here? Thank you,
Moz Bar | | Rubix

0