Keyword Grader & OmniExplorer_Bot Question
-
I found this in several of my clients Robots.txt
"User-Agent: OmniExplorer_Bot
Disallow: /"
Right now I cannot grade any of them that are on a particular platform that has that written in.
Would this block the keyword grader?
Thanks!
-
I'm going to reach out to our vendor to try and resolve the issue. In your experience, what would be the best way to explain this problem to their tech team and try to resolve it?
-
User-Agent: bingbot
Crawl-delay: 5
User-Agent: msnbot
Crawl-delay: 5
User-Agent: Googlebot-Mobile
Allow: /mobi
User-Agent: OmniExplorer_Bot
Disallow: /
-
Hey,
Mike is correct that our crawler is named RogerBot.
I took a look at one of your campaigns that was only crawling one and it appears that it's returning an access denied page.
This is likely something based on the server itself for RogerBot since it looks like it's returning full source to other User-Agents.
I hope that helps track down the issue.
Cheers,
Joel. -
Moz's crawler is rogerbot so that shouldn't block it as far as I know. What does the rest of your robots.txt look like? Maybe something in the format of it is accidentally disallowing more bots than you meant to block.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How does Moz determine the number of keywords evident on a given page
How does Moz identify which keywords are evident on page and how many that can be on that page. Can any of these identified keywords lie on any other pages within site and to what degree-thanks
Moz Bar | | bickybeanie0 -
Is there an automated way to import Keyword Rank Tracker data in Google Data Studio?
I am creating an SEO Dashboard in Google Data Studio.
Moz Bar | | Lvet
While it's easy to import data from Google Search Console, I have troubles in importing data from Mozzarellas tools. I thought about uploading a CSV file with the ranking to a Google Sheet Document, but I cannot find a way to do this at regular intervals with manual intervention.0 -
Moz On-Page Grader doesn't pick up my Title, URL, Meta, H1, Body, IMG ALT's....does this mean Google won't?
Good morning, As my title says, 'Moz On-Page Grader doesn't pick up my Title, URL, Meta, H1, Body, IMG ALT's....does this mean Google won't?' My URL is www.refusedcarfinance.com and I'm currently targeting the Keyword 'bad credit car finance'. I am using Yoast SEO and have the keyword in my title, meta, content, h1's etc. Any advice would my much appreciated. Kindest regards, Joshua
Moz Bar | | RocketStats0 -
Keyword Difficulty tool showing Google search volume or Bing?
Hi, I plugged in a bunch of keywords to know their search volume, chose Google from the drop down and then chose English from the next dropdown and clicked on Check dificulty. It shoes me "Bing Search Volume (Exact Match)" on top but shows the name of the keyword and "for Google CA" on the left. Is the search volume for Google? or Bing? and the difficulty as well? Google or Bing? "
Moz Bar | | SpaMedica0 -
Tool for seeing targeted keywords?
Hello Friends, Is there a tool that provides a list including targeted keywords and there ranking for those keywords? For example if I type in a website url it shows me which keywords they targeted, on which page and their ranking position. Thanks for helping out a foreign noobie;)
Moz Bar | | grobro0 -
Keyword difficulty - benchmarks
hi, anyone know some good benchmarks? how many hours a certain keyword needs with what percentage? or how to really interpret the %? thanks, Walter
Moz Bar | | Walterman1 -
On Page Grader Changed?
Hi, Could anyone tell me if the on page grader has changed? I remember seeing 8 boxes at the top but now i'm only seeing 6... Before I remember seeing a H2-H4 and a Strong/B Box? Thanks
Moz Bar | | dmxsta0 -
Optimise your Pages in Moz Crawl - where do the keywords come from?
I am just going through my first Crawl stats from the MOZ analytics Under the pages to optimise section I have pages that I have optimised for my best keyword with an A grade that are showing as an F grade and suggesting a different keyword? Where is this keyword coming from? I am assuming that my page has been analysed and a better keyword has been recommended? Can anyone advise? Thanks Roger
Moz Bar | | rnperki0