Keyword Grader & OmniExplorer_Bot Question
-
I found this in several of my clients Robots.txt
"User-Agent: OmniExplorer_Bot
Disallow: /"
Right now I cannot grade any of them that are on a particular platform that has that written in.
Would this block the keyword grader?
Thanks!
-
I'm going to reach out to our vendor to try and resolve the issue. In your experience, what would be the best way to explain this problem to their tech team and try to resolve it?
-
User-Agent: bingbot
Crawl-delay: 5
User-Agent: msnbot
Crawl-delay: 5
User-Agent: Googlebot-Mobile
Allow: /mobi
User-Agent: OmniExplorer_Bot
Disallow: /
-
Hey,
Mike is correct that our crawler is named RogerBot.
I took a look at one of your campaigns that was only crawling one and it appears that it's returning an access denied page.
This is likely something based on the server itself for RogerBot since it looks like it's returning full source to other User-Agents.
I hope that helps track down the issue.
Cheers,
Joel. -
Moz's crawler is rogerbot so that shouldn't block it as far as I know. What does the rest of your robots.txt look like? Maybe something in the format of it is accidentally disallowing more bots than you meant to block.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keyword Lists - Omitting Keywords
I've had trouble adding keywords to Keyword Lists - specifically when I add a large list of keywords (400+), and only 50 or 75 keywords are analyzed. Is there a reason this happens?
Moz Bar | | brianglassman0 -
On-page Grader API?
Hi guys, I have a massive list of keywords i want to optimise against a specific URL. I'm checking them manually with the MOZ on-page grader to determine if the keyword is optimised for the URL (Grade A score). I was wondering if there is a way to pull the scores automatically for each URL with the MOZ api? Or does anyone know of any tools which can automate this process? Cheers, Chris
Moz Bar | | jayoliverwright1 -
Keyword Difficulty Tool - Bulk Data of Specifics?
For the keyword difficulty tool, if I want to see the details, I have to click on the keyword in order to see the list of the competitors and their DA and RLRDs. Is there a way to download this list for all of the keywords and their DA and RLRDs? The only bulk I can download is the list of keywords in my difficulty tool list along with their Bing search volume and keyword difficulty %. Thanks!!!1
Moz Bar | | Titan5520 -
Pending problem with Keyword Difficulty
Hi, Is there any problems with the Keyword Difficulty tool? It was working fine all morning but now it appears to be stuck on Pending. I have tried Firefox and Chrome and both have the same problems. Thanks Rachael
Moz Bar | | GlitzyBingo1 -
Keyword volume
I started using the keyword difficulty tool, I can see the keyword difficulty but how can I see the search volume of each keyword?
Moz Bar | | amielsosa0 -
Bing data in the keyword difficulty and SERP analysis tool
Hi Guys, I'm wondering about the exact match bing data in the Moz KWD & SERP analysis tool. We're looking at the ranking difficulty for given terms in Google but getting exact match bing numbers. Is the tool estimating Google searches from the bing data (perhaps a simple formula assuming maybe 10% search totals for Bing, 60% for Google or whatever is closest these days) or is it just raw bing numbers that we then have to upsize in order to estimate search numbers in Google? Hope that makes some kind of sense! In other words - how are you guys using that Bing data when considering terms to target using this tool? Look forward to hearing your thoughts.
Moz Bar | | lovealbatross
Cheers
James0 -
How is this possible, Keyword Difficulty tool has some problems
Hello guys, I have some concerns or misunderstanding. If I am about to look for [software development] using Google first, it shows that the Exact match for the keyword is 16,574, then I choose Bing and I had the same number 16,574. Yahoo is the showing the same figure. Do I miss something in here or this tool doesn't work? odd_zps2388b5a9.png
Moz Bar | | Askalexis0 -
How to print segmented keyword lists to PDF?
Hi Moz forum, We're trying to print segmented keyword lists to PDF. Using the 'segmented lists' feature, we have created segmented lists for several different keyword groupings. We are looking to print each list individually as a PDF, but the feature seems to print ALL keywords from the entire campaign. How can we print segmented keyword lists to PDF? Thanks
Moz Bar | | BoomDialogue690