Moz crawler
-
I have a site which is in a non production status.
Crawlers are blocked vis robot txt.
User-agent: *
Disallow: /I WANT TO MAKE A CRAWLING TEST WITH MOZ CRAWLER (RogerBot) ,
how can I allow your crawler to get in and prevent other crawlers from indexing the site?Thanks
memok
-
Try this:
User-agent: *
Disallow: /
User-agent: rogerbot
Allow: / -
did you take a look through - http://moz.com/learn/seo/robotstxt
I'll give you a clue -
Allow a specific web crawler to visit a specific web page
Disallow: /no-bots/block-all-bots-except-rogerbot-page.html User-agent: rogerbot Allow: /no-bots/block-all-bots-except-rogerbot-page.htmlmore info -http://moz.com/help/pro/rogerbot-crawler&http://moz.com/help/pro/what-is-rogerbot-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Inaccurate results in MOZ Rank Tracker
Hello, We recently ran a test on Rank Tracker and got a result indicating that the page in question was in position #17. We did a manual check on a Google USA SERP for the query, and found that we weren't in results where the tool had indicated. Does anyone have any ideas on what might be causing this discrepancy? Thanks!
Moz Bar | | Critical_Mass0 -
That URL is inaccessible Moz grader?
Hi all, I'm having some issues getting my site graded www.balihaiphoto.com + Kauai wedding photographer with On Page grader, where as when I enter another photographer example www.jmoellerphoto.com and getting results. Is there any reason this is going on that I can correct? Many thanks for any help all! -Jon
Moz Bar | | Jon_Gibb0 -
How can I find duplicate pages from a Moz Crawl?
We have many duplicate pages that show up on the Moz Crawl, and we're trying to fix these but it's very difficult because I can't see a way to isolate the code where the duplicate is found. For instance, http://experiencemission.org/immersion/ is one of our main pages, and the crawl shows one duplicate of http://experiencemission.org/immersion. It appears that one of our staff manually edited the source code in one of our pages but forgot the trailing slash. This would be an easy fix but the problem is that this page is linked to internally on our website 2423 times, so it's next to impossible to find the code that is incorrect. We have many other pages with this same basic problem. We know we have duplicates, but it's next to impossible to isolate them. So my question is this: When viewing the Moz Crawl data is there any way to see where a specific duplicate page link is located on our website? Thanks for any and all help!
Moz Bar | | expmission0 -
Moz's on-page grader
Hello, I am having an issue with Moz's on-page grader. Grading the page, http://www.babyklar.dk/tigerspring/26-ugers-springet.html for the keyword 'Tigerspring 26 uger' I am receiving the error message: 'Exact keyword is used in page title' - tagging it as a highly important but easy fix. But I only have the words "Tigerspring 26 uger" in the title, so how can this not grade as Exact keyword? Is there somthing wrong with the system? Or is it me? Thanks very much. Best regards
Moz Bar | | Babyklar
Martin0 -
Suggestion: Moz Domain Authority should take disavow into account
Since Moz is trying to predict how Google ranks your site, and Google claims to take the disavow file into account, I'd like to suggest that Moz allow webmasters to upload their disavow file. I imagine this data would be useful to Moz in determining Domain Authority (they may even think of other ways to use it and might even help come to a conclusion on the great debate) and it gives a chance for sites to improve their Moz DA when they are bombarded by spammy links. I'd love to hear the community's thoughts on this idea, as well as the what Wizards of Moz have to say.
Moz Bar | | YairSpolter1 -
Signed up for moz reports - have received Moz error report - need someone who is capable to take report and perform cleanup edits within Joomla site?
Looking for someone in the US - please contact me at Mary@workingwebsolutions.com If available and interested in task. Thanks Mary
Moz Bar | | PortlandWebDesign0 -
Moz Conversion Tracking
Is there a place here on Moz where goals and conversions can be set up or viewed? If the latter, does Moz just take the data from Analytics? Any information on this would be appreciated. Thanks,
Moz Bar | | xvpn9020 -
Given that I am currently using a bot sniffer...How can I identify the MOZ bot in order to whitelist it?
MOZ is currently blocked from crawling my sites because I use a bot sniffer. Does anyone know how I can properly identify the MOZ bot in order to whitelist it? MOZ is using Amazon web services and thus employs thousands of dynamic IPs to crawl.
Moz Bar | | Felix_LLC0