Moz crawler
-
I have a site which is in a non production status.
Crawlers are blocked vis robot txt.
User-agent: *
Disallow: /I WANT TO MAKE A CRAWLING TEST WITH MOZ CRAWLER (RogerBot) ,
how can I allow your crawler to get in and prevent other crawlers from indexing the site?Thanks
memok
-
Try this:
User-agent: *
Disallow: /
User-agent: rogerbot
Allow: / -
did you take a look through - http://moz.com/learn/seo/robotstxt
I'll give you a clue -
Allow a specific web crawler to visit a specific web page
Disallow: /no-bots/block-all-bots-except-rogerbot-page.html User-agent: rogerbot Allow: /no-bots/block-all-bots-except-rogerbot-page.htmlmore info -http://moz.com/help/pro/rogerbot-crawler&http://moz.com/help/pro/what-is-rogerbot-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Accuracy of Moz Keyword Tool
I understand that no search tool is entirely accurate, but I'm seriously beginning to doubt the accuracy of the Moz keyword tool search volumes for keywords. For example: I entered "largest police departments" and it says 0-10 per month in the US, despite the fact that that term is directly responsible for 200 hits per day. Many alterations of that keyword come up with 0-10 or not determined. This is not a small error -- and it repeats over and over for other search terms that I have already ranked for in the US. It's gigantically inaccurate in a massive scale.
Moz Bar | | Luke_Proctor0 -
Utilizing the Moz Location in keywords vs including the location in the actual keyword phrase
I've searched the Moz blog as well as Q&A forums - but I specifically have a question about utilizing the Moz Location in keywords vs including the location in the actual keyword phrase. I've done both and get different rankings. For instance, I have "it company" with the Location set to Corpus Christi, Texas and our rank is #3. Then I also have "it company corpus christi" with the Location set to Corpus Christi, Texas and our rank is #6. Additionally, I threw in "it company corpus christi" with the Location set to National and our rank is #7. What is the best practice and what are the determining factors for the differences? Thank you in advance!
Moz Bar | | LBoxerger0 -
Moz on-page grader can't see keyword in H1
Hi, Trying to do grade for whatiswhere.com, keyword 'poi search'. The tool can't see keyword in h1, but It is there. Could you please check what is the problem? Thank you, Andrei.
Moz Bar | | Anazar_20010 -
Am I the only one seeing the Moz's points ranking with mixed rankings?
Hello everyone. It's been several weeks since the problem persists, Here: https://moz.com/community/users
Moz Bar | | Gaston Riera
There are rankings that are incorrect and some users that have the same ranking than other users. I've attached some pics. GR 25603b49292acd5d68f7dd6a43977f94 6d8fea557f6ed2de23466f6c7df850eb0 -
Https address has different result that http in Page Optimization Score toll in Moz PRO
The following url
Moz Bar | | TrueluxGroup
https://www.whichledlight.com/t/gu10-led-bulbs has (100 score for keyword 'GU10 LED')
has different on page opmisation score results to
http://www.whichledlight.com/t/gu10-led-bulbs (73 score for keyword 'GU10 LED') Anyone know if we've set something up wrong?
Also, is this even something to worry about, does google treat them differently? We're using the Page Optimisation Tool in Moz Pro ** UPDATE ** It's worth mentioning we are using emberjs, so the website is a single page application.
We use prerender to render the pages for google.0 -
MozBar is displaying different numbers than Moz Pro
We have over 15 accounts with Moz Pro, and I just noticed that almost every single one has a different DA when comparing the MozBar to their campaign. Any idea why this is? The DA of most of our sites seems to have dropped across the board, in some cases somewhat severely.
Moz Bar | | anchour0 -
Pro.moz.com referrals and transactions
Our Google Analytics account reports a significant number of visits and transactions as referrals from pro.moz.com. That is nice and we appreciate this but where is the traffic generated from ? (my first MOZ question)
Moz Bar | | silverpewter0 -
Why still the moz index showing "Next Update on August 26, 2013"?
I have been waiting to see the updated authority metrics from Opensiteexplorer.org. But still it is showing old data. and showing "next index update: August 26, 2013" as the date is 27th August today. Is there any delay in update?
Moz Bar | | Bala_K2
