Moz crawler
-
I have a site which is in a non production status.
Crawlers are blocked vis robot txt.
User-agent: *
Disallow: /I WANT TO MAKE A CRAWLING TEST WITH MOZ CRAWLER (RogerBot) ,
how can I allow your crawler to get in and prevent other crawlers from indexing the site?Thanks
memok
-
Try this:
User-agent: *
Disallow: /
User-agent: rogerbot
Allow: / -
did you take a look through - http://moz.com/learn/seo/robotstxt
I'll give you a clue -
Allow a specific web crawler to visit a specific web page
Disallow: /no-bots/block-all-bots-except-rogerbot-page.html User-agent: rogerbot Allow: /no-bots/block-all-bots-except-rogerbot-page.htmlmore info -http://moz.com/help/pro/rogerbot-crawler&http://moz.com/help/pro/what-is-rogerbot-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is Moz more accurate than Ahrefs ?
When I check my this website https://joinpakarmy.com.pk/ on Moz it gives me DA 3
Moz Bar | | joinpakarmy234
But when I check it out on Ahrefs, it tells me there is no authority for this site.
So who is more accurate in the result?
thanks0 -
Moz not updating the spam score metrics
Hi Experts, I have done everything to come over the following metrics flagged as spam on my website by Moz, few months ago. But, Moz has not updated the spam score yet. ✓Low MozTrust or MozRank Score - improved from 2 to 4.5
Moz Bar | | jamesh.rich01
✓Large Site with Few Links - My website have more than 6K backlinks
✓Small Proportion of Branded Links - My website have a good amount of branded backlinks
✓Thin Content - Every webpage on website has more than 500 words content
✓External Links in Navigation - There is no external link in navigation other than social media links
✓No Contact Info - The proper address has alreay been placed on website footer
✓Low Number of Pages Found - I am wondering if there are any standard score or number of links to reach to remove these flag?
Also, please suggest some ways that will help me improve moz spam score at faster rate. Thanks for your help in advance!"0 -
How do can the crawler not access my robots.txt file but have 0 crawler issues?
So I'm getting this errorOur crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.https://www.evernote.com/l/ADOmJ5AG3A1OPZZ2wr_ETiU2dDrejywnZ8kHowever, Moz is saying I have 0 Crawler issues. Have I hit an edge case? What can I do to rectify this situation? I'm looking at my robots.txt file here: http://www.dateideas.net/robots.txt however, I don't see anything that woudl specifically get in the way.I'm trying to build a helpful resource from this domain, and getting zero organic traffic, and I have a sinking suspicion this might be the main culprit.I appreciate your help!Thanks! 🙂
Moz Bar | | will_l0 -
Why does the moz crawl test lists page twice?
Hi, I'm running into an issue where some crawlers list my pages twice, once with a trailing slash, once without. I first saw it on a few pages with screaming frog, then saw it happen on all my pages with the moz crawler. The site is www.kidsandart.org and its on Squarespace. I grepped the sitemap.xml I submitted to google webmaster and got 167 distinct pages, all of them without a trailing slash. Any insights on why this is happening, and how to regard moz crawler results would be appreciated. thanks Tom
Moz Bar | | tpushpathadam0 -
MoZ vs Alexa & Moz vs Google
My colleague is continuously arguing with me why i went for Moz and why not for Alexa, He also says when Google is there then why Moz. I tried on my part to convince him but he has his own learning. Can anybody help me make him understand otherwise my job will become hard if he remains doubtful about Moz? Looking for down to earth and an honest feedback. Thanks Tanveer
Moz Bar | | Sequelmed0 -
Moz / more changes on the way?
I love Moz and the community and all the tools here. I admit I haven't rolled around in all the new things rolled out a few months ago. I thought there were more changes on the way but I wasn't sure if those already happened and I missed them or if I need to be patient? Affiliate program, client reporting? Thanks for any response. Have a great weekend! Matthew
Moz Bar | | Mrupp441 -
Connect to Google Analytics problem (Moz Analytics)
Hello - I recently received the Beta invite to Moz Analytics (very excited). I set up the campaign for our primary site and entered our Google Analytics profile - however, Moz does not seem to accept the profile when I click the "Save" button. All of our profiles are listed w/in the drop-down menu, but after selecting the profile we wish to link/connect to - nothing happens. So we're not receiving any stats w/in Moz from our GA acct. Please help. I'm really anxious to begin delving into Moz Analytics! Thanks
Moz Bar | | ryankish0 -
Moz Rank Tracker doesn't work with "PHRASE" Keywords!?
Hello, If at http://ranktracker.moz.com trying to track phrase KW - the system wont accept it. But adding [EXACT] match with [] - works well. Have I missed something? Cheers.
Moz Bar | | SEOisSEO0