Moz crawler showing pages blocked by robots.txt
-
I've blocked a large number of pages which Moz were showing as duplicate or giving 404's in our robots.txt using /?key and /?p etc.
However Moz crawler is still showing as being an issue. I assumed Roger picked up the robots.txt file, or is that not the case?
-
Not really an expert in robots.txt - but you could always try to simulate Rogerbot using a tool like Screaming Frog (the tool is free if you crawl less than 500 pages).
You can enter a custom user agent. Rogerbot seems to be rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) and try crawling the pages on your site. In default mode, the tool will only crawl pages that are allowed by the robots.txt. If the tool crawls your pages, then something is wrong with the robots.txt - if not, than you should contact Moz staff to check with them.
-
Yep. Checked in the tester at the time and did some more spot checks this morning.
I used Disallow: /*?key which I thought was generally recognised by bots?
-
Did you check these pages with the robots.txt tester in Webmastertools to be sure that these pages are really blocked for bots? Did you exclude all bots or only the Googlebot?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Links in link lists not being discovered by Moz
I uploaded a couple of link lists to Moz about a week ago and have been tracking them. Most of the links on the list are from significant 90+ DA domains such as Wikipedia, Medium, etc. So far, Moz only seems to have discovered about 4 the 200 or so links I loaded. Do we know how long it takes for these to be discovered by Moz? Quite new to this, so any help understanding timelines for this would be appreciated.
Link Explorer | | RobertPhillips180 -
If I have a MOZ PRO account, do I still need Screaming Frog?
Hi, I have a MOZ PRO account which is brilliant for me. I am about to move my site from HTTP to HTTPS and was looking for a smooth way to list every page and its details so I can make sure I do the correct redirects etc.. before the switch to SSL. Do I need to get Screaming Frog, or can the tools with MOZ Pro ease my path just as well? Thanks
Link Explorer | | SeoSheikh0 -
Open Site Explore page titles show as "No Title"
This has been asked many times but I cannot find an answer. On Open Site Explorer, most URLs I enter show as "No title" which means you have to hover over the URL to see which page is being referred to. I know you'd want an example, so here's your own site Moz.com 🙂 CWrkxZQ
Link Explorer | | clifra0 -
Moz Spam Score 9/17 when there are no links
Hi I was just looking at a competitors new site in OSE and it had a DA and PA of 1 which I would expect and also no links, which again I would expect, basically the guy used to design stuff for me and thinks he can do a better job so he is giving it ago. But he has a spam score of 9/17 but no links - how is this even possible. Thanks Andy
Link Explorer | | Andy-Halliday0 -
Pagination & Duplicate Page Title
I was under the impression that pagination resolved duplicate page title issues with Google, but Moz is flagging them up as duplicates. Is this just a case of Moz not being able to understand pagination or am I missing something? i.e. correctly formatted pagination: http://thespacecollective.com/space-memorabilia/page/4
Link Explorer | | moon-boots1 -
Does Moz allow you to upload your disavow to take into account when analysing links?
I was wondering if moz allows you to upload your disavow file and if it has a consideration in the data supplied? Otherwise how can we accurately use moz to understand the true analytics of our site?
Link Explorer | | MoneySite0 -
How Is a Page Crawled by Moz When Moz Says 'No Links'?
As above, really. I've crawled a new client's site to find the Moz crawler has identified a handful of 404 errors. The Moz crawler says these pages have '0 linking domains', and OSE has no data for these pages. So how are these pages being crawled by Moz and what should I advise my client?
Link Explorer | | xerox4320
