Moz crawler showing pages blocked by robots.txt
-
I've blocked a large number of pages which Moz were showing as duplicate or giving 404's in our robots.txt using /?key and /?p etc.
However Moz crawler is still showing as being an issue. I assumed Roger picked up the robots.txt file, or is that not the case?
-
Not really an expert in robots.txt - but you could always try to simulate Rogerbot using a tool like Screaming Frog (the tool is free if you crawl less than 500 pages).
You can enter a custom user agent. Rogerbot seems to be rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) and try crawling the pages on your site. In default mode, the tool will only crawl pages that are allowed by the robots.txt. If the tool crawls your pages, then something is wrong with the robots.txt - if not, than you should contact Moz staff to check with them.
-
Yep. Checked in the tester at the time and did some more spot checks this morning.
I used Disallow: /*?key which I thought was generally recognised by bots?
-
Did you check these pages with the robots.txt tester in Webmastertools to be sure that these pages are really blocked for bots? Did you exclude all bots or only the Googlebot?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How Does Moz Spam Score Increase and Decrease?
Hello Moz Community, I'm facing spam score issue on my site General Queen according to MOZ my website spam score is 30% but anybody can explain how can i decrease it? Firstly I disavowed all low-quality and high spam score links (6 months ago) but my spam score was not decreased after that I created some high-quality and 0 spam score backlinks but still, my spam score is same 😞 According to Moz link explorer, my domain backlinks spam score ratio was below; Stats of 14 July 2019 .. Screen shot is attached 1-30% .. 45.3%
Link Explorer | | BloggerSEObd
31-60% .. 32.1%
61-100% .. 22.6% Stats of 13 June 2020 .. Screen shot is attached 1-30% .. 65.1%
31-60% .. 30.7%
61-100% .. 4.2% You can see a huge difference in the backlinks spam score in it. But why my spam score is still 30%? Looking forward to your reply. Thanks & Regards, abu1NiB XGLkr5z1 -
Is there a way to download a report showing all meta descriptions for our web pages?
I see how to look at data for web pages with meta descriptions that have been flagged as being less than optimal. Is there a way to do a complete download of the meta descriptions on all our pages? (good meta as well as not so good) Thanks! Lydia
Link Explorer | | Lifespan-Moz0 -
After how long Moz show the matrices of a new domain
I want to know the Matrices of this site " Clash of lights ". Its a new domain how can i find its matrices like DA PA etc in Moz. Please guide me through it.
Link Explorer | | Muhammadahamd0 -
Moz Pro: Filter inbound links by partial anchor text?
My site has been targeted by a spam farm with hundreds of different domains, all linking to images on our CDN with similar variations of anchor text, eg: get free high quality hd wallpapers wedding cake makers
Link Explorer | | James_NZ
get free high quality hd wallpapers hairstyle makeover
get free high quality hd wallpapers living room cafe
etc Is it possible within Moz Pro to filter all incoming links with anchor text including "free high quality hd wallpapers" so that I can disavow all of the domains en masse? So far I've only been able to display/download the list of links exactly matching the full anchor text which is very time-consuming with 100+ variations. Regards,
James0 -
Sudden Spike in 404 Pages Not Found in Moz Crawl But No Errors in WMT
Recently I received a spike in errors from the Moz crawler. When I looked into the matter I noticed that all the URI's looked right but then I looked a little closer and there was a /page/2 and /page/3 in front of the URI's. I'm running a WordPress website. Immediately I thought to myself this must be some kind of caching or permalinks error. So I disabled all my plugins including W3 Total Cache and ran the Integrity Link Crawler for the Mac and found that the errors were still popping up. 404-errors-ncworkercomp.png?dl=0
Link Explorer | | NCCompLawyer0 -
403 errors in Moz but not in Google Search Console
Hello, Moz is showing that one of the sites I manage has about ten 403 errors on main pages, including the home page. But when I go to Google Search Console, I'm not getting any 403 errors. I don't know too much about this site (I handle the SEO for a few sites as a contractor for a digital marketing agency), but I can see that it's a WordPress site (I'm not sure if that's relevant). Can I assume this a Moz issue only? Thanks, Susannah Noel
Link Explorer | | SusannahK.Noel0 -
How do I fix 885 Duplicate Page Content Errors appearing in my Moz Report due to categories?
Hi There, I want to set up my Moz report to send directly to a client however there are currently 885 duplicate page content errors displaying on the report. These are mostly caused by an item listed in multiple 'categories' and each category is a new pages/URL. I guess my questions are: 1. Does Google see these as duplicate page content? Or does it understand the categories are there for navigation purposes. 2. How do I clear these off my Moz report so that the client doesn't panic that there are some major issues on the site Thanks for your advice.
Link Explorer | | skehoe0 -
Duplicate Home Page
'Content that is identical (or nearly identical) to content on other pages of your site forces your pages to unnecessarily compete with each other for rankings.' Can anyone tell me why this is the case? I know for a fact that their are no other duplicates of my home page, however moz disagrees. Cheers for any help. **Taylor **
Link Explorer | | Taylor1230