Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?
-
We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries. It's madness. I can't figure out what is causing these URLs to generate to be crawled in the first place! Does this sound familiar to anyone out there, any constructive suggestions? Robots text or maybe meta robots tags that would resolve this crawl issue?
-
Hey,
I'm not sure if this is resolved for you, but as you suggest you can do something with robots.txt. Specifically, you could use wildcards to capture these URLs and tell Rogerbot (Moz's crawler) to ignore them. Here's a great Stackoverflow query to get you started and details on how block Rogerbot, you can take a look here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Issue with former site redirect
Hello, After 2 months of starting a new service with a new domain name and a rather flat curve, I decided to invest in auditing tools. To my surprise, when analyzing links with 'Link Explorer', I got the following warning : "You entered the URL crapahute.com which redirects to www.krapahute.com. Click here to analyze www.krapahute.com instead."... So I understand I bought a domain name that was previously in use, 8 years ago. And logically, when they moved from crapahute.com to krapahute.com, the former site owners set a (probably 401) redirect between their former name and their new one. Probably that in the last 8 years they dropped crapahute.com since I was able to buy the domain name last year. I can't understand the warning I'm receiving. since I thought this redirect would become inactive when they dropped the domain name. On the contrary, if I analyze the backlinks to krapahute.com I can indeed see : crapahute.com [no anchor text] 3 2 0 --
Link Explorer | | x_all
redirect More Info
Date first seen - Date lost - Link target
2016-09-05 - N/A - www.krapahute.com So my question is : where is this warning coming from ? I performed some nslookup and dig on my domain and their's just to find out that every thing seems to look normal. Meanwhile, if I type "crapahute" in Google (and most other search engines), my site arrives in 12th page (!), and if I type "crapahute.com", the site "krapahute.com" appears first. Thank you for those reading up to that point ! Any hint on what could cause this ? Should I change of domain name ? Thank you !0 -
After how long Moz show the matrices of a new domain
I want to know the Matrices of this site " Clash of lights ". Its a new domain how can i find its matrices like DA PA etc in Moz. Please guide me through it.
Link Explorer | | Muhammadahamd0 -
Inbound links for https://www.example.com versus just example.com
Hi all, Wondering the difference between the Inbound Links stats for https://www.example.com versus only using example.com. When I type in the https:// version, I see 3 Spam Flags - for the example.com version, there are none. There are also spammy links pointing to the https version as opposed to the non-https version when I use this tool: https://moz.com/researchtools/ose/ Anyone know the difference and which should be regarded when fixing spam issues?
Link Explorer | | netamorphosis0 -
Does MOZ have a Flash test tool?
I want to test my websites and see if they use Flash, is there a flash check tool like on SEO tool kit here on MOZ? Thanks, Lance
Link Explorer | | BlueprintMM0 -
301/302 Target?
On a Moz crawl I see a column that says "301/302 Target" I understand 301s and 302s, but what does this 301/302 mean (shows up in column T on the csv esport). Thanks!
Link Explorer | | CaseyMcG0 -
DA hasn't Moved over
We moved our site over a couple of weeks ago from bi-mi.com to bimi.co - its a magento site. We followed instructions to wait 2 weeks and upload a coming soon page etc to help google realise it was not just a parked domain. We moved everything over and then set a hta record up to redirect the "bi-mi" base domain to "bimi" , we then logged teh site move within web master tools also and I set the bimi.co preferred url etc. However its been about a week and half now and the DA is still showing as 1 and bi-mi (the old site that is redirecting) is showing as 18 Is it just patience needed here or have any of you also come across this issue?
Link Explorer | | Kelly33301 -
May I know multiple campaigns set up, with same site url is allowable in Moz campaign or not?
Hi Guys, I have a question regarding moz campaign. I am a Moz Pro member, have taken Standard Plan. (Time being, am not in a position to upgrade to a higher plan) I have utilized only one campaign slot among total 5 slots. I have added xyz.com website in one campaign, there I have added 350 keywords which is the maximum count we can add in a a single campaign. Still, I have plenty of keywords to track for same website xyz.com. I am not planning to use the remaining 4 campaign slots for time being for any other websites. May I know, if I can add same xyz.com website in remaining campaign slots for tracking the remaining keywords for my website?
Link Explorer | | zco_seo0 -
Getting Different PA/DA for 'www' and 'non-www'?
Can anyone explain why we're seeing different DA/PA for our website when viewing in OSE for 'www' and 'non-www'? This is for our site whiteboardcreations.com Our 'www' is resolving at DA 37 | PA 47
Link Explorer | | WhiteboardCreations
Our 'non-www' is resolving at DA 37 | PA 44 Thanks! - Patrick0