Moz crawler showing pages blocked by robots.txt
-
I've blocked a large number of pages which Moz were showing as duplicate or giving 404's in our robots.txt using /?key and /?p etc.
However Moz crawler is still showing as being an issue. I assumed Roger picked up the robots.txt file, or is that not the case?
-
Not really an expert in robots.txt - but you could always try to simulate Rogerbot using a tool like Screaming Frog (the tool is free if you crawl less than 500 pages).
You can enter a custom user agent. Rogerbot seems to be rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) and try crawling the pages on your site. In default mode, the tool will only crawl pages that are allowed by the robots.txt. If the tool crawls your pages, then something is wrong with the robots.txt - if not, than you should contact Moz staff to check with them.
-
Yep. Checked in the tester at the time and did some more spot checks this morning.
I used Disallow: /*?key which I thought was generally recognised by bots?
-
Did you check these pages with the robots.txt tester in Webmastertools to be sure that these pages are really blocked for bots? Did you exclude all bots or only the Googlebot?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to tell moz about the domain redirects?
If I redirected my previous domain to my new domain, how can I tell moz that the previous domain is redirected?
Link Explorer | | LannaRosse0 -
Website showing High Spam Score
I am having a website https://www.hemporganic.net/ Suddenly I am noticing the website showing around 42% spam score. I simply cannot understand the reason behind it. The website contains only one backlink and that too it is guest posting backlink. Please explain to me why this is occurring with my site.
Link Explorer | | stay.hungary10 -
How many pages Google crawl in free version
I am using moz pro 30 days trial version.Can Anybody tell me how many pages moz crawl in a day or in a week.Because its two days and only 2 pages they crawled. Thanks
Link Explorer | | VarinderS0 -
403 errors in Moz but not in Google Search Console
Hello, Moz is showing that one of the sites I manage has about ten 403 errors on main pages, including the home page. But when I go to Google Search Console, I'm not getting any 403 errors. I don't know too much about this site (I handle the SEO for a few sites as a contractor for a digital marketing agency), but I can see that it's a WordPress site (I'm not sure if that's relevant). Can I assume this a Moz issue only? Thanks, Susannah Noel
Link Explorer | | SusannahK.Noel0 -
Do other sites linking to my page with a low domain authority affect my rankings?
Hello, my page authority 30, I see other page with an authority of 21 for example linking to us. Is this good or bad, does this affect our ranking? A lot of these don't have anchor text, I'm guessing we should reach out to these webmaster to include anchor text with the keywords we want to drive traffic to this page or make as descriptive as possible? Thanks! Laura
Link Explorer | | lauramrobinson320 -
Rogerbot and Dotbot blocked by WP Engine hosting
We recently discovered a problem with our MOZ account which led to checking our host's settings. Turns out that WP Engine (who we love to death) automatically BLOCK Rogerbot and Dotbot from crawling sites they host. What? Yeah exactly expect I was cursing a little too. So we went through the steps of getting that sorted one bot at a time but a week on and we're still getting 403 returns via OSE. This is more of a discussion than a question expect what I would like to see is a force crawl button so that we can test that we're no longer serving a 403 error from our host MOZ. I thought WP Engine were huge and I'm surprised this has not come up before but there's no info on it anywhere. Is there a way to force a mozbots crawl??
Link Explorer | | wearehappymedia0 -
My Facebook / Twitter Shares Not Showing Up
In Open Site Explorer, my Google+ shares are showing up but any shares on Twitter and Facebook are not. I am wondering if anyone has any ideas as to why this might be happening? My site is www.riddelllaw.com. Thanks
Link Explorer | | bpurdue0 -
Why Moz doesn't offer tool similar like SEOprofiler Link Disinfection?
Hi Moz, As I have been your member & using pro tool for more than an year, it's been amazing experience working and using your each tool's function tool, wehther to check on-page grader, rank tracking, on-page crawling issue, keyword difficulty and the one my best for link analysis is Open Site Explorer etc. I came here to ask about something which I didn't find here than other providers is most popular Bad Link Detection Tool. As **SEOprofiler Link Disinfection **offers their customers to use this to get all of their bad or spammy links harming them to rank high or getting away from ranking goal. Let me know, are you planning anything similar to this to bring soon in your tools and provide your members to take benefits of such type function as well OR should i go to SEOprofiler for this? Best,
Link Explorer | | Futura
Teginder Ravi0