Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?
-
We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries. It's madness. I can't figure out what is causing these URLs to generate to be crawled in the first place! Does this sound familiar to anyone out there, any constructive suggestions? Robots text or maybe meta robots tags that would resolve this crawl issue?
-
Hey,
I'm not sure if this is resolved for you, but as you suggest you can do something with robots.txt. Specifically, you could use wildcards to capture these URLs and tell Rogerbot (Moz's crawler) to ignore them. Here's a great Stackoverflow query to get you started and details on how block Rogerbot, you can take a look here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz link explorer is how much reliable and accurate as compared to others link explorer tools like Ahref
Hi.. I need to check backlinks for one of my client website duplicatebills through Moz Pro link explorer tool then its show that linking domain to this website are 90 but when i cross check these links from other tools like Ahref free backlinking tool then its show a different figure (ahref show that linking domain to this website are 103).
Link Explorer | | dbills
I am confused that which one correct. And MOZ link explorer tools in reliable to which extent?0 -
Problem with third party cookies while using MOZ?
Hello, I am using Moz Pro and I am happy to have this tool for my SEO work. But I am having a little problem while working on my blog site https://thepoolparadise.com/ some times the Moz does not work and asks to allow third-party cookies, wherein the settings third-party cookies are always allowed. Please guide me in this matter. Thank you.
Link Explorer | | Niujbuj760 -
Hello There I have a site the problem with this site is I have taken countless backlinks but it is not reflecting on Moz. Is there any way to help Moz find my links?
Hello There I have a site health related..the problem with this site is I have taken countless backlinks but it is not reflecting on Moz. Is there any way to help Moz find my links? herpescurecare.com
Link Explorer | | thelancethealth0 -
Why is my backlink not detected here, but it appears on google search console? I have added the moz robots
Why is my backlink not detected here, but it appears on google search console? I have added the moz robots
Link Explorer | | gunawanjord0 -
Expired Domain Redirect
hi guys have recently bought a few expired domains with good juice and redirected them to help boost pages on my site how long will these take to show in the moz link explorer?
Link Explorer | | Staunton_Rook0 -
6 Months and still no PA/DA or links by Moz Explorer?
Hi, It's been 6 months and my site is still showing no PA/DA or backlinks according to moz opensite explorer. I know that it has backlinks as I created them myself, and I know its crawlable and the links is there as GWT tells me so. Also you can check for yourself on ahrefs.com. Is Moz indexed pages that much smaller then the other crawlers? The site in question is amanandvan247.com. Thanks.
Link Explorer | | Marvellous1 -
Does Moz allow you to upload your disavow to take into account when analysing links?
I was wondering if moz allows you to upload your disavow file and if it has a consideration in the data supplied? Otherwise how can we accurately use moz to understand the true analytics of our site?
Link Explorer | | MoneySite0 -
How long does it take for moz to index new domain?
Hi, Two months ago we changed our site www.jicht.nl to jicht.nl. Everthing is redirected properly but the page authority remains 1. How long does is take rogerbot to index the site without www? Google rankings are also dropping so there might be a bigger issue. Thanx
Link Explorer | | AlkaVitae0