Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?
-
We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries. It's madness. I can't figure out what is causing these URLs to generate to be crawled in the first place! Does this sound familiar to anyone out there, any constructive suggestions? Robots text or maybe meta robots tags that would resolve this crawl issue?
-
Hey,
I'm not sure if this is resolved for you, but as you suggest you can do something with robots.txt. Specifically, you could use wildcards to capture these URLs and tell Rogerbot (Moz's crawler) to ignore them. Here's a great Stackoverflow query to get you started and details on how block Rogerbot, you can take a look here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I couldn't check Page authority or Domain Authority for any page, what can be reason?
I am trying to check DA and PA for page but not able to do this. Tried it before two days and it was working fine, problem occurring today. I want help regarding this issue, is anyone other facing this problem. Kindly help me out (provide some other way to check DA/PA).
Link Explorer | | JessicaThompson1 -
Error Message on Moz Crawler
Hi all, Just ran into this issue, when analysing this site. Just got this message when using MOZ "Page Optimisation Error". Anyone know why? It seems to be working fine on other SEO analyser tools. Website is: www.sbpcreativemedia.com.au Thanks in advance! luXS8V5
Link Explorer | | Dushala0 -
Duplicate content, despite having correct canonical tags
Hi there We've run a report today which shows we have 157 pages with duplicate content. Out of these 157, 134 have canonical tags so I'm not sure why they're showing up in the reports as being duplicate. Here's an example, this page: https://www.havaianasaustralia.com.au/Accessories?category=beach-towels .....has an canonical of: http://www.havaianasaustralia.com.au/beach-towels-havaianas-accessories But it's still showing as duplicated in your reports. Any support would be appreciated. Thanks.
Link Explorer | | Havs10 -
301/302 Target?
On a Moz crawl I see a column that says "301/302 Target" I understand 301s and 302s, but what does this 301/302 mean (shows up in column T on the csv esport). Thanks!
Link Explorer | | CaseyMcG0 -
403 errors in Moz but not in Google Search Console
Hello, Moz is showing that one of the sites I manage has about ten 403 errors on main pages, including the home page. But when I go to Google Search Console, I'm not getting any 403 errors. I don't know too much about this site (I handle the SEO for a few sites as a contractor for a digital marketing agency), but I can see that it's a WordPress site (I'm not sure if that's relevant). Can I assume this a Moz issue only? Thanks, Susannah Noel
Link Explorer | | SusannahK.Noel0 -
Learn how to use Moz's Spam Score metric to identify high risk links. Get your Daily SEO Fix.
Almost every site has a few bad links pointing to it but risky links can have a negative impact on your search engine rankings. Watch The Moz Daily SEO Fix: How to Use Spam Score to Identify High Risk Links to learn how to spot those spammy links and what to do with them. And, if you have more questions about Spam Score, check out Rand’s blog post: "Spam Score: Moz’s New Metric to Measure Penalization Risk." This video is part of The Moz Daily SEO Fix tutorial series--Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, and to subscribe to future ones, make sure to visit the Daily SEO Fix channel on YouTube.
Link Explorer | | kellyjcoop3 -
Why is Moz not crawling my backlinks
Hi my website www.dealwithautism.com is 3 months old and has been on DA 1 and PA1 ever since, even though the site is actively developed with quality content (a couple of posts already have 1k+ fb likes acquired editorially, while that doesnt necessarily improve SERP, it sure tells you that the post is engaging). In contrast another site of mine, www.deckmymac.com which is hardly ever managed, not have more than 15 posts and just 1 backlink, has DA 14. Running an on page analysis on www.dealwithautism.com I observed that Moz has not identified any backlinks nor social signals (except G+). However, according to Webmasters, I have 57 links, 51 of them to the root. Even Majestic is able to report 32+ backlinks. So what am I missing? Certainly, at this stage my website doesn't deserve DA 1, or does it?
Link Explorer | | DealWithAutism0 -
Confused about the link metrics showing up in MOZ & OSE
For my campaign "itelligence Applications" which is for site itelligencegroup.com/uk/ - MOZ seems to be giving me different stats on links, depending on which tab I look it in the Link Analysis section. Then when I look at OSE, the numbers are totally different again, not just a little bit different. Example: In MOZ Link Analysis, on the "inbound links" tab, it says there are 129 Root Domains and 6832 Total External Inbound Links. If I then look at the Competitive Metrics screen, it says there are 46,284 external followed links and 240 total linking root domains. If I then go to OSE for the same site, it says there are 14 Root Domains and 686 total links. I have no idea why these numbers vary so much or how to even report them to my client. Am I missing something here? Any help would be gratefully received! Maureen
Link Explorer | | mfrgolfgti0