Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?
-
We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries. It's madness. I can't figure out what is causing these URLs to generate to be crawled in the first place! Does this sound familiar to anyone out there, any constructive suggestions? Robots text or maybe meta robots tags that would resolve this crawl issue?
-
Hey,
I'm not sure if this is resolved for you, but as you suggest you can do something with robots.txt. Specifically, you could use wildcards to capture these URLs and tell Rogerbot (Moz's crawler) to ignore them. Here's a great Stackoverflow query to get you started and details on how block Rogerbot, you can take a look here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my website Backlinks not getting Crawl by Moz?
Hi, I've the query related to backlinks of my website. Some of the high authority sites give backlinks to my website but these links are still not showing in Moz. My website is 6 months old and also continuously getting backlinks from high authority sites but still these are not showing on Moz and also not improving DA and PA of my website. I've also attached the screenshot of Moz link explorer results please check and guide me what to do with website so Moz will consider it and give some authority. And also guide me How much Moz takes time to crawl website backlinks and shown in their link explorer. Website URL: https://www.welderexpert.com Moz Expert Suggestions needed. Thanks, overview?site=welderexpert.com⌖=domain
Link Explorer | | KOidue0 -
Why is Link Explorer showing the same domain strength for domain and sub-domain? Shouldn't it be different?
I believe in the past the sub-domain score was different than the domain score in Link Explorer. Now I am seeing that it is reporting the same score. I have to use exact page to get a sense of how strong the sub-domain is.
Link Explorer | | Liamis0 -
What does "Number of Linking Root Domains" in Open Site Explorer actually mean?
I've read through the forum, but I'm confused as to what "Number of Linking Root Domains" means in the Linking Domains report. For example, if the root domain is Moz.com and the number is 135,000 does that mean there are 135,000 links from Moz pointing back to my site or does it mean there are 135,000 sites linking back to Moz? Any help is appreciated. Thank you!
Link Explorer | | karrabarron1 -
Top Pages Report References URLs from a Different Domain
I am running a Top Pages Report using Open Site Explorer and it's returning that the top pages for examplesiteA.org are examplesiteB.com. Important to note that examplesiteA.org was a dormant domain for a few years. We recently purchased it and are in the process of reviving it. Has anyone experienced this before? Would this suggest that canonical link elements were put in place prior to it going dormant and that the top pages for this site are now assumed to be on examplesiteB.com?
Link Explorer | | seoaustin0 -
Why are no-follow links in my blog comments across the web showing up as "equity-passing"?
When I make comments on blog posts (and there's a link in my comment or my name is the link to my site), the links are always no-follow (as they should be). But, when I check Open Site Explorer, the new links show up as equity-passing. Are they actually passing equity or is this a mistake?
Link Explorer | | infotrust20 -
Clicking on "Filter" on "Just discovered"tab errors out
IF you click on the "fiter" button on the "just discovered" tab in OSE, page returns message "no links found." This is weird because that message displays even when you didn't actually filter for anything, but just clicked on the button.
Link Explorer | | ALLee0 -
What is the difference between a "Just Discovered" link and an "Established" link in OSE?
What is the difference between a "Just Discovered" link and an "Established" link in OSE? How long does it generally take for a just discovered link to migrate in to the established link section? Are there other parameters that need to be met before a link can be referred to as established? Sorry for all the questions in one post, but I've had the same "just discovered" links in OSE for quite some time and have yet to see them migrate over to the "established" link section. I was hoping that when these links become established that it would help improve my site DA. Can anyone shed any light on this issue or have any other comments? Thanks MOZ!
Link Explorer | | GalcoIndustrial1 -
Getting this on OSE - wp-content/cache/page_enhanced/
Hi there, In using OSE, it switches to www.mysite.com/wp-content/cache/page_enhanced/www.mysite.com/_index Is there something I need to do in my .htaccess or some other file to fix this? thanks chris
Link Explorer | | jcbradley110