Moz crawler
-
I have a site which is in a non production status.
Crawlers are blocked vis robot txt.
User-agent: *
Disallow: /I WANT TO MAKE A CRAWLING TEST WITH MOZ CRAWLER (RogerBot) ,
how can I allow your crawler to get in and prevent other crawlers from indexing the site?Thanks
memok
-
Try this:
User-agent: *
Disallow: /
User-agent: rogerbot
Allow: / -
did you take a look through - http://moz.com/learn/seo/robotstxt
I'll give you a clue -
Allow a specific web crawler to visit a specific web page
Disallow: /no-bots/block-all-bots-except-rogerbot-page.html User-agent: rogerbot Allow: /no-bots/block-all-bots-except-rogerbot-page.htmlmore info -http://moz.com/help/pro/rogerbot-crawler&http://moz.com/help/pro/what-is-rogerbot-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
MOZ Tool not reading my xml sitemap.
I have submitted an xml sitemap to Search Console back in December, and it's been approved, but the moz tool says there's no xml sitemap. Please help!!!
Moz Bar | | SEO_Tap220 -
My page cant be crawled by MOZ
Hi everyone, I have a page named zita.vn. Im using Moz to do SEO for the page, but when I use On-page grader in MOZ, I got an error that the page cant be accessed. Additionally, I got an issue with page crawling. Google search still crawl my page normally. Please help. Thanks a lot.
Moz Bar | | zita.vn0 -
Moz crawler finding my homepage multiple times
Hi and thank you in advance for your help! I have a Moz Pro campaign running (I am a complete Moz novice by the way) for one of my websites (balloonsutah.com). After crawling my site, the Moz crawler informed me that I have 3 pages with duplicate content. While I am not sure why exactly this is happening, the crawler indexed my homepage 3 times under different url's. -balloonsutah.com
Moz Bar | | Keenan-Price
-balloonsutah.com/
-balloonsutah.com/index.html I checked my FTP server and I cannot figure out for the life of me why the crawler is finding anything other than the index.html file. I suppose I need to do something regarding a rel="Canonical" but I am not terribly familiar with that either. Any suggestions would be greatly appreciated!
Keenan0 -
Has MOZ stopped including Text to Code Ratio?
My Moz extension no longer displays text to code ratio. Anyone else encounter this or know why it's gone? I see it in videos and descriptions and I used to see it in my extension, but it is no longer there.
Moz Bar | | dan.bertone0 -
MOZ crawl test is not reporting on all the pages on my site.
I've run the crawl test one of the sites I've taken over SEO for, however its only picking all the pages. For instance it indexes all the pages under xxxxx/us but none under xxxxx/au or xxxxx/uk The pages are being indexed as they're ranking in Google. Thanks.
Moz Bar | | ahyde0 -
Understanding Onsite Elements and Moz's Onpage Grader
I'm currently helping a friend with his website. One of the pages I am targeting for his business is: http://stalbansdentist.com.au/dentist-cairnlea/ The main keyword this page is targeting will be "dentist cairnlea" (Cairnlea is a suburb near Melbourne, Australia) To perfect onsite optimization, I've used the Moz Onpage Grader. Generally, its straightforward, but I have encountered some issues... Appropriate Use of Rel Canonical The page does have the following tag: <link rel="<a class="attribute-value">canonical</a>" href="http://stalbansdentist.com.au/dentist-cairnlea/" />. I thought this would suffice. What exactly do I need to do to fix this critical problem? Appropriate Characters in the URL I don't understand this one. The URL has completely appropriate characters. Why does Moz Onpage Grader insist that URLs here need to be appropriated? No More Than One H1 Tag There appears to be only one H1 tag on this page. Is Moz just wrong on this one or perhaps a little delayed? There were 2 H1 tags before one was deleted and a recheck done with this problem remaining on Moz's onsite check. Any assistance here on these 3 points and just understanding the Moz Onpage grader would be appreciated!
Moz Bar | | Gavo0 -
Optimise your Pages in Moz Crawl - where do the keywords come from?
I am just going through my first Crawl stats from the MOZ analytics Under the pages to optimise section I have pages that I have optimised for my best keyword with an A grade that are showing as an F grade and suggesting a different keyword? Where is this keyword coming from? I am assuming that my page has been analysed and a better keyword has been recommended? Can anyone advise? Thanks Roger
Moz Bar | | rnperki0 -
Moz is reporting a broken link error but GWT is not
My latest Moz report is showing a 404 for: http://www.fateyes.com/how-will-googles-hummingbird-affect-your-search-ranking/”ht
Moz Bar | | gfiedel
(and showing the link this way with the characters after the last / which are not part of the page URL) Google Webmaster Tools says we have no errors. I'm wondering why there is this descrepancy and I'm wondering how I can track down where this link is originating from on our site. I've tried downloading screamingfrog and deeptrawl to no avail (java issues). I've also tried a couple of services online and installed Broken Link Checker plugin with no luck finding it. Any suggestions? Thanks in advance!0