Moz crawler showing pages blocked by robots.txt
-
I've blocked a large number of pages which Moz were showing as duplicate or giving 404's in our robots.txt using /?key and /?p etc.
However Moz crawler is still showing as being an issue. I assumed Roger picked up the robots.txt file, or is that not the case?
-
Not really an expert in robots.txt - but you could always try to simulate Rogerbot using a tool like Screaming Frog (the tool is free if you crawl less than 500 pages).
You can enter a custom user agent. Rogerbot seems to be rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) and try crawling the pages on your site. In default mode, the tool will only crawl pages that are allowed by the robots.txt. If the tool crawls your pages, then something is wrong with the robots.txt - if not, than you should contact Moz staff to check with them.
-
Yep. Checked in the tester at the time and did some more spot checks this morning.
I used Disallow: /*?key which I thought was generally recognised by bots?
-
Did you check these pages with the robots.txt tester in Webmastertools to be sure that these pages are really blocked for bots? Did you exclude all bots or only the Googlebot?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
MOZ DA IS NOT INCRAESING
Hey my website on business cards its DA is not increasing. is there any way to increase its DA.
Link Explorer | | cartwrightkar0 -
How to tell moz about the domain redirects?
If I redirected my previous domain to my new domain, how can I tell moz that the previous domain is redirected?
Link Explorer | | LannaRosse0 -
I couldn't check Page authority or Domain Authority for any page, what can be reason?
I am trying to check DA and PA for page but not able to do this. Tried it before two days and it was working fine, problem occurring today. I want help regarding this issue, is anyone other facing this problem. Kindly help me out (provide some other way to check DA/PA).
Link Explorer | | JessicaThompson1 -
Domain Authority (DA) in Moz Pro changed only within the last 1-2 months?
Has anyone noticed that the Domain Authority (DA) as reported in Moz Pro has changed only within the last 1-2 months? We have screen shots showing plots of DA vs competitors w/ line graph 2 months ago starting in NOV 2017 which today starting JAN 2018 and comparing shows DA up to 50% different!
Link Explorer | | Amplitude_Digital
The change is seen both in the Links Overview and under the Spam Score sections still marked "NEW". Can Moz confirm that it's only recently within the last 2 months that in Moz Pro the NEW DA numbers have retroactively been updated even though the new Link Explorer has been publicly out since APR 30 from https://moz.com/community/q/moz-s-new-link-explorer-including-our-revamped-index-and-da-pa-scores-is-now-open-to-everyone? Look at the top green line starting ~12 months ago on both graphs, w/ old below 40 and new above 50. We've seen even greater differences for other tracked domains. Thanks! view0 -
Internal Equity-Passing Links not getting crawled in Moz Open Site Explorer?
Internal Equity-Passing Links not getting crawled in Moz Open Site Explorer. What is the cause of this? We've checked the robots.txt and htaccess file, but so far we can't find anything that would be blocking Moz from crawling the internal links. We manage loads of other clients on this platform and this is the first time we've run into this issue. What else can I check?
Link Explorer | | OozleMedia0 -
Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?
We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries. It's madness. I can't figure out what is causing these URLs to generate to be crawled in the first place! Does this sound familiar to anyone out there, any constructive suggestions? Robots text or maybe meta robots tags that would resolve this crawl issue?
Link Explorer | | RegistrarCorp0 -
OpenSiteExplorer is showing all link root domains or not?
I want to know open site explorer is show all link root domain or not? As my point of view missing some linking root domain of my website on opensiteexplorer,so, Why Its not showing all link root domain?
Link Explorer | | renukishor0 -
SEOmoz not showing latest backlinks
Hi, How come open site explore(come with SEOmoz tool) not showing new backlinks created by me a month ago? I faced same issue while using other tools for the same. All of them show up old backlinks but not the new ones created recently. However, the Google Webmaster Tool updates me with this recent development. Any reason?
Link Explorer | | acelerar0