Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?
-
We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries. It's madness. I can't figure out what is causing these URLs to generate to be crawled in the first place! Does this sound familiar to anyone out there, any constructive suggestions? Robots text or maybe meta robots tags that would resolve this crawl issue?
-
Hey,
I'm not sure if this is resolved for you, but as you suggest you can do something with robots.txt. Specifically, you could use wildcards to capture these URLs and tell Rogerbot (Moz's crawler) to ignore them. Here's a great Stackoverflow query to get you started and details on how block Rogerbot, you can take a look here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz isn't crawling all my backlinks.
Moz isn't crawling all my backlinks. It's showing only 29 referring domain when I have more than 200 referring domains linking to my website. My website URL is 360gisthub.com.ng 360gisthub.com.ng
Link Explorer | | Mustybay0 -
Backlinks aren't shown
Hi, my site has 15 backlinks from sites with 50-90 da. and only 4 of them are shown on the moz's backlink checker, While on google console and on different platforms, I do see most of them. All of the links are dofollow. I even tried the link tracking list and all the domains are shown as not linking to my domain, while they are. Any idea what is happening? Thank you.
Link Explorer | | Yair1231230 -
Page Authority higher on my old (redirected) domain than new domain?
Hi everyone. I moved the domain my blog was on about 18 months ago, and it's ranking in Google never recovered. I've noticed in the 'Inbound Links' tool that my old domain (jamescrowley.co.uk) shows a page authority of 26, while my new domain (jamescrowley.net) shows a page authority of 15. Any idea why that would be? 301 redirects have been in place for the whole 18 months, so I'd have thought the authority would have been 'passed on' by now? Many thanks James
Link Explorer | | james.crowley0 -
Sudden Spike in 404 Pages Not Found in Moz Crawl But No Errors in WMT
Recently I received a spike in errors from the Moz crawler. When I looked into the matter I noticed that all the URI's looked right but then I looked a little closer and there was a /page/2 and /page/3 in front of the URI's. I'm running a WordPress website. Immediately I thought to myself this must be some kind of caching or permalinks error. So I disabled all my plugins including W3 Total Cache and ran the Integrity Link Crawler for the Mac and found that the errors were still popping up. 404-errors-ncworkercomp.png?dl=0
Link Explorer | | NCCompLawyer0 -
403 errors in Moz but not in Google Search Console
Hello, Moz is showing that one of the sites I manage has about ten 403 errors on main pages, including the home page. But when I go to Google Search Console, I'm not getting any 403 errors. I don't know too much about this site (I handle the SEO for a few sites as a contractor for a digital marketing agency), but I can see that it's a WordPress site (I'm not sure if that's relevant). Can I assume this a Moz issue only? Thanks, Susannah Noel
Link Explorer | | SusannahK.Noel0 -
MOZ competitor analysis tool
Hi Guys Has anyone also noticed that the MOZ competitor analysis tool does not always pick up on all back-links? If I put two competitors into MOZ and Ahrefs I get so many more back-link info from Ahrefs. Has anyone else experienced the same issue? Thanks Jon
Link Explorer | | NaughtyChilli0 -
None of the pages crawled contain an email address or links to a social profile...
I'm trying to reduce the amount of spam flags our website has using the Open Site Explorer tool. Currently, we have 2/17 flags based on: Large Site with Few Links - We found very few sites linking to this site, considering its size No Contact Info - None of the pages crawled contain an email address or links to a social profile The "few links" can be ignored, we're working on this. We don't have a visible email address on the website and we don't particularly want one. We prefer customers to fill out our online form or to call us. Moz say "none of the pages crawled contain an email address OR links to a social profile" - we do have social buttons on every page of the website, but these are official Facebook and Twitter buttons that are rendered with Javascript, so don't actually appear in the page source on load. If we replace these with actual links to our pages using Facebook and Twitter icons, will this flag be removed since Moz are saying "or links to a social profile" - making it sound optional. Thanks!
Link Explorer | | LiamMcArthur0