Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?
-
We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries. It's madness. I can't figure out what is causing these URLs to generate to be crawled in the first place! Does this sound familiar to anyone out there, any constructive suggestions? Robots text or maybe meta robots tags that would resolve this crawl issue?
-
Hey,
I'm not sure if this is resolved for you, but as you suggest you can do something with robots.txt. Specifically, you could use wildcards to capture these URLs and tell Rogerbot (Moz's crawler) to ignore them. Here's a great Stackoverflow query to get you started and details on how block Rogerbot, you can take a look here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why some of my backlinks are not showing on moz
why some of my backlinks are not showing on moz. These backlinks are profile and web 2.0 backlinks, i have tried many solutions but none of them worked for me. my site is solution4people is there any way to show all the backlinks of my site on moz . Thanks alot.
Link Explorer | | tzoubaier0 -
Links in link lists not being discovered by Moz
I uploaded a couple of link lists to Moz about a week ago and have been tracking them. Most of the links on the list are from significant 90+ DA domains such as Wikipedia, Medium, etc. So far, Moz only seems to have discovered about 4 the 200 or so links I loaded. Do we know how long it takes for these to be discovered by Moz? Quite new to this, so any help understanding timelines for this would be appreciated.
Link Explorer | | RobertPhillips180 -
Need help understanding this Moz Chart comparing link metrics against competitors...
Why is my website at 60% and what does it mean? And more importantly, what needs to be done to fix (or not fix) this situation??? Please help... Image of chart attached kcd5r40
Link Explorer | | SamCitron0 -
Inbound links not found in Moz Pro(though they exist)
These sites have many inbound links but Moz Pro finds no such links. I also used SimilarWebPro Trial to confirm the existing of such links. (these sites have IP block so it requires an VPN to access with Japan IP) osakado.cc
Link Explorer | | Yuji_m
oskd.biz Could anyone figure out why Moz can't find those links to those sites? Any tricks in place to hide those links from SeoMoz? If I pay to upgrade SimilarWebPro but Id rather stay with SeoMoz Pro. thanks in advance.0 -
Does MOZ have a Flash test tool?
I want to test my websites and see if they use Flash, is there a flash check tool like on SEO tool kit here on MOZ? Thanks, Lance
Link Explorer | | BlueprintMM0 -
Internal Pages marked as Spam by Moz
Hello,
Link Explorer | | Umesh-Chandra
We use Moz for our website,Open site explorer marks many of the internal pages of the website as spam. Even pages such as careers, privacy policy etc. Please let us know why is this the case ? Also does these pages have impact on overall rankings of the website ?, If yes, What should we do about it ? Please clarify. Thanks.0 -
Only two pages crawled
Hi community, I'm having trouble viewing data for all of our pages. Moz is only crawling two of our pages, and we have a lot more than that. Any suggestions? Thanks, Emily
Link Explorer | | SkuidEmily0 -
Why don't links from Reddit show up in Site Explorer?
Some are follow and some are no follow. They show up in webmaster tools and not in site explorer. In addition why do some other links show up in webmaster tools but not site explorer?
Link Explorer | | RafeTLouis0