Moz crawler showing pages blocked by robots.txt
-
I've blocked a large number of pages which Moz were showing as duplicate or giving 404's in our robots.txt using /?key and /?p etc.
However Moz crawler is still showing as being an issue. I assumed Roger picked up the robots.txt file, or is that not the case?
-
Not really an expert in robots.txt - but you could always try to simulate Rogerbot using a tool like Screaming Frog (the tool is free if you crawl less than 500 pages).
You can enter a custom user agent. Rogerbot seems to be rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) and try crawling the pages on your site. In default mode, the tool will only crawl pages that are allowed by the robots.txt. If the tool crawls your pages, then something is wrong with the robots.txt - if not, than you should contact Moz staff to check with them.
-
Yep. Checked in the tester at the time and did some more spot checks this morning.
I used Disallow: /*?key which I thought was generally recognised by bots?
-
Did you check these pages with the robots.txt tester in Webmastertools to be sure that these pages are really blocked for bots? Did you exclude all bots or only the Googlebot?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Links in link lists not being discovered by Moz
I uploaded a couple of link lists to Moz about a week ago and have been tracking them. Most of the links on the list are from significant 90+ DA domains such as Wikipedia, Medium, etc. So far, Moz only seems to have discovered about 4 the 200 or so links I loaded. Do we know how long it takes for these to be discovered by Moz? Quite new to this, so any help understanding timelines for this would be appreciated.
Link Explorer | | RobertPhillips180 -
Why Moz Link Explorer is showing wrong data?
Hi everyone. I am using Moz Link Explorer to find relevant links from competitors and when I narrow it down to Link Type: Follow Status: Active I get the list. All good here... BUT ... Inspecting the actual link on the page there is a no=follow attached to it. Is this feature still working properly on the new Link Explorer or is a bug? Thanks!
Link Explorer | | Sorin_T0 -
Moz is not indexing all my backlinks
Why is there a difference in the backlinks of my website when checked in moz.com and SEMrush.com.
Link Explorer | | mzakaria0 -
What is the difference between - This page v this subdomain v this root domain in OSE
What is the difference between - This page v this subdomain v this root domain in OSE, while checking website backlinks See inbound links to the page, subdomain, or root domain you've entered and analyze the linking pages.
Link Explorer | | Green.landon0 -
Dofollow link from Moz user profile still not showing in OpenSiteExplorer
Hi everyone! Refering to my last question http://moz.com/community/q/dofollow-link-from-moz-user-profile-not-showing-in-opensiteexplorer, the last update was on March 10 and I'm still searching my dofollow link from my Moz user profile http://moz.com/community/users/625691 after have 200 MozPoints, Opensiteexplorer only found the nofollow link from the icon, but nothing from the anchor text of my URL. Is possible that only take the first link to my blog and ignore the rest if it have the same href? 😞 Thanks! fTdwaO6.png
Link Explorer | | rubenalonsoes1 -
Why are none of the links we've received from press and other sources from the last two months showing on our Inbound Links in OSE???
It's really hard to judge if the DA of our site according to MOZ is 1) up to date and 2) accurate when for example there are SO many inbound links still not recorded, that were published months ago.
Link Explorer | | TVape0 -
Open Site Explorer doesn't show local directories or citation links?
Why is it that when I run a link analysis for my local business sites I don't see any links from Yelp, Google+, Facebook, any local citations? Regardless of them being follow or no follow isn't the hyperlink to a website from a Yelp profile considered a link? Same with any other local business directory? I know the company is listed on a few dozen because I can see it manually but it's not showing up in Moz? Why is it that all local directories and citations with website are omitted? And since it is not there, where is a good place to go to discover these things? I have tried whitespark. It was okay but honestly I wasn't terribly impressed. The data was good, the UI was just okay, but with smaller clients it was an initial check and then very occasional checks that needed to happen. We would go months without using it.
Link Explorer | | bricegump0
