Keyword Grader & OmniExplorer_Bot Question
-
I found this in several of my clients Robots.txt
"User-Agent: OmniExplorer_Bot
Disallow: /"Right now I cannot grade any of them that are on a particular platform that has that written in.
Would this block the keyword grader?
Thanks!
-
I'm going to reach out to our vendor to try and resolve the issue. In your experience, what would be the best way to explain this problem to their tech team and try to resolve it?
-
User-Agent: bingbotCrawl-delay: 5
User-Agent: msnbotCrawl-delay: 5
User-Agent: Googlebot-MobileAllow: /mobi
User-Agent: OmniExplorer_BotDisallow: /
-
Hey,
Mike is correct that our crawler is named RogerBot.
I took a look at one of your campaigns that was only crawling one and it appears that it's returning an access denied page.
This is likely something based on the server itself for RogerBot since it looks like it's returning full source to other User-Agents.
I hope that helps track down the issue.
Cheers,
Joel. -
Moz's crawler is rogerbot so that shouldn't block it as far as I know. What does the rest of your robots.txt look like? Maybe something in the format of it is accidentally disallowing more bots than you meant to block.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keyword Explorer - root domain (NEW): Domains outside US & Canada - no or no meaningful results
Basically I really like and would love to use the new root domain feature in the Keyword Explorer in my daily work. I usually deal with small to medium sized sites in Switzerland. Looking these up unfortunately it either gives me back nonsense (i.e. keywords that merely have nothing to do with the site; sites are very well keyword optimized) or no data ("There are no results for your search") at all. Is this going to change in the future (...honestly) or its simply due to the fact that MOZ focuses primarily on US & Canada and maybe huge sites in Europe? Thanks for your input. Cheers, Cesare
Moz Bar | | Cesare.Marchetti0 -
Keyword explorer not working
Hi there, The keyword explorer tool has been malfunctioning for the last week. Moz barely seem to have any data (Search Volume) for any keywords I search for, particularly for suggested keywords. It's rather frustrating as I can't explore any keyword opportunities using the Moz keyword tool... I'm based in the UK BTW. Thanks I've attached a screenshot as an example. ZX3Kw3H.jpg
Moz Bar | | SEONOW1231 -
On-Page Grader Url is inaccessible
Hi everybody. I'm trying to use on -page grader for https://www.upscaledinnerclub.com and get "Sorry, but that URL is inaccessible." Robots.txt are empty, another thread on MOZ was talking about DNS check - it's all good. So, I can't figure out why this is happening. Also I am trying the same for another website https://www.regexseo.com - the same story. Common thing is that they both are on Google App Engine. And at first i thought that was the problem. Bu then i checked this one : https://www.logitinc.com/ and it's working, even though this website is on GAE as well. None of these website have robots.txt or any differences in setup or settings. Any thoughts?
Moz Bar | | DmitriiK0 -
Keyword research in Moz - Am I missing something?…
I'm fairly new at SEO and am still trying out MOZ. I'm finding myself wishing that Moz had a broader set of tools for keyword research, but am wondering if maybe I'm just missing something. Am I correct in saying that there is no place within Moz that you can get keyword ideas, select/organize keywords, find new/related keywords? I've appreciated the keyword difficulty tool and have played around with Moz Analytics keyword tracking, but these don't help with the broader keyword research aspect. The various tutorials talk about using Google AdWords Keyword Planner and other tools to do that job. But with all that Moz can do, it just seems backwards to do lots of copying-pasting from other tools into Moz, or to have to rely on old excel spreadsheets to organize keywords. It seems that life would be much easier if Moz would be able to pull that kind of external research information in, like Raven does. That way you do most of your research within Moz – allowing you to organize/evaluate potential keywords BEFORE choosing the ones you would like to test for difficulty or track. That way you could simply select the desired keywords and add them to your tracked list. The way it is now, even the one research tool that Moz does have (keyword difficulty) does not allow for an easy way to track or manage a chosen set of keywords. For example, after doing some keyword difficulty analysis, I've chose 5 keywords. To add these 5 to the tracked keywords in a specific campaign I can't just select them and say "track keywords > in xxx campaign". From what I've understood, I must copy-and-paste them into the manage keyword section of the desired campaign. To me this all feels a bit awkward. But since I'm new at this, maybe I'm just missing something? Or maybe it's unrealistic to expect a single tool to address all my SEO needs?
Moz Bar | | Mike_E0 -
Keyword Ranking Report Blank
I have a custom ranking report set up to run every month. The module that shows the overall changes (how many improved, how many dropped, number in rank #1-3, etc.) populates with data, but the keyword ranking for tracked keywords listing is blank for all keywords, all search engines. I haven't changed anything in my campaigns - why is this reporting blank? Thanks!
Moz Bar | | GranthamUniversity0 -
Keyword Rankings vs Rank Tracker
Hello, I have a question, what is the difference between these tools? I thought both are the same, just that you can see you competitors ranks on the keyword rankings... The keyword rankings tool is telling me that my competitor is ranking in some keywords.. If I go to the rank tracker tool and type the keyword and competitor url it gives me that that keyword is not in the top 50, when keyword ranking is giving me position #3 I am using in both the same Search Engine with the same country (Google Mexico) Thank you
Moz Bar | | pixelarmx0 -
On-Page Grader not working?
I've been using the On-Page Grader tool for a while now, and have always had great success using it to better optimize my pages. But just recently I've noticed that it's not detecting any changes I make to any of my pages.
Moz Bar | | CMinc
For example: One of my pages was coming up with Exact Keyword is Used in Page Title. So I changed it and uploaded it to the site and then clicked the re-grade page button. The only problem is that it's not seeing any of the changes that I made.1 -
Moz Rank Tracker - showing deleted subdomains results for entered keywords
We have recently deleted subdomains because they were causing duplicate page errors but now the Moz rank tracker is showing the subdomain key word results even instead of the exact url we entered. So for example wine cellar is showing up for a deleted subdomain (winecellar.vigilantinc.com) instead of the entered exact url of (vigilantinc.com/winecellars/) Please advise. Thanks!
Moz Bar | | KristyFord0