Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?
-
We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries. It's madness. I can't figure out what is causing these URLs to generate to be crawled in the first place! Does this sound familiar to anyone out there, any constructive suggestions? Robots text or maybe meta robots tags that would resolve this crawl issue?
-
Hey,
I'm not sure if this is resolved for you, but as you suggest you can do something with robots.txt. Specifically, you could use wildcards to capture these URLs and tell Rogerbot (Moz's crawler) to ignore them. Here's a great Stackoverflow query to get you started and details on how block Rogerbot, you can take a look here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz show me that my backlinks lost every day in my some important keywords
hi, moz show me that my backlinks lost every day in my some important keywords like: ثبت تغییرات شرکت ، ثبت شرکت ، .... but , i think that's not correct. can you explain to me that is for what? my website is: https://vanak.org thank you....
Link Explorer | | alirezatrade01230 -
Moz can't crawl our site
Moz can't crawl our site because of an error in the robots.txt, we've tried everything in the troubleshooting guide but nothing works - I believe its a server error but have no idea how to fix it pls help
Link Explorer | | SigneerHFS0 -
Moz crawling http rather than https site
Our site is secure but when I ask moz to crawl it by giving the root domain including https moz insists on crawling the non secure version. How do i force it to crawl the secure version?
Link Explorer | | media12340 -
DA/PA Fluctuations: How to Interpret, Apply, & Understand These ML-Based Scores
Howdy folks, Every time we do an index update here at Moz, we get a tremendous number of questions about Domain Authority (DA) and Page Authority (PA) scores fluctuating. Typically, each index (which release approximately monthly), many billions of sites will see their scores go up, while others will go down. If your score has gone up or down, there are many potential influencing factors: You've earned relatively more or less links over the course of the last 30-90 days.
Link Explorer | | randfish
Remember that, because Mozscape indices take 3-4 weeks to process, the data collected in an index is between ~21-90 days old. Even on the day of release, the newest link data you'll see was crawled ~21 days ago, and can go as far back as 90 days (the oldest crawlsets we include in processing). If you've done very recent link growth (or shrinkage) that won't be seen by our index until we've crawled and processed the next index. You've earned more links, but the highest authority sites have grown their link profile even more
Since Domain and Page Authority are on a 100-page scale, the very top of that represents the most link-rich sites and pages, and nearly every index, it's harder and harder to get these high scores and sites, on average, that aren't growing their link profiles substantively will see PA/DA drops. This is because of the scaling process - if Facebook.com (currently with a DA of 100) grows its link profile massively, that becomes the new DA 100, and it will be harder for other sites that aren't growing quality links as fast to get from 99 to 100 or even from 89 to 90. This is true across the scale of DA/PA, and makes it critical to measure a site's DA and a page's PA against the competition, not just trended against itself. You could earn loads of great links, and still see a DA drop due to these scaling types of features. Always compare against similar sites and pages to get the best sense of relative performance, since DA/PA are relative, not absolute scores. The links you've earned are from places that we haven't seen correlate well with higher Google rankings
PA/DA are created using a machine-learning algorithm whose training set is search results in Google. Over time, as Google gets pickier about which types of links it counts, and as Mozscape picks up on those changes, PA/DA scores will change to reflect it. Thus, lots of low quality links or links from domains that don't seem to influence Google's rankings are likely to not have a positive effect on PA/DA. On the flip side, you could do no link growth whatsoever and see rising PA/DA scores if the links from the sites/pages you already have appear to be growing in importance in influencing Google's rankings. We've done a better or worse job crawling sites/pages that have links to you (or don't)
Moz is constantly working to improve the shape of our index - choosing which pages to crawl and which to ignore. Our goal is to build the most "Google-shaped" index we can, representative of what Google keeps in their main index and counts as valuable/important links that influence rankings. We make tweaks aimed at this goal each index cycle, but not always perfectly (you can see that in 2015, we crawled a ton more domains, but found that many of those were, in fact, low quality and not valuable, thus we stopped). Moz's crawlers can crawl the web extremely fast and efficiently, but our processing time prevents us from building as large an index as we'd like and as large as our competitors (you will see more links represented in both Ahrefs and Majestic, two competitors to Mozscape that I recommend). Moz calculates valuable metrics that these others do not (like PA/DA, MozRank, MozTrust, Spam Score, etc), but these metrics require hundreds of hours of processing and that time scales linearly with the size of the index, which means we have to stay smaller in order to calculate them. Long term, we are building a new indexing system that can process in real time and scale much larger, but this is a massive undertaking and is still a long time away. In the meantime, as our crawl shape changes to imitate Google, we may miss links that point to a site or page, and/or overindex a section of the web that points to sites/pages, causing fluctuations in link metrics. If you'd like to insure that a URL will be crawled, you can visit that page with the Mozbar or search for it in OSE, and during the next index cycle (or, possibly 2 index cycles depending on where we are in the process), we'll crawl that page and include it. We've found this does not bias our index since these requests represent tiny fractions of a percent of the overall index (<0.1% in total). My strongest suggestion if you ever have the concern/question "Why did my PA/DA drop?!" is to always compare against a set of competing sites/pages. If most of your competitors fell as well, it's more likely related to relative scaling or crawl biasing issues, not to anything you've done. Remember that DA/PA are relative metrics, not absolute! That means you can be improving links and rankings and STILL see a falling DA score, but, due to how DA is scaled, the score in aggregate may be better predictive of Google's rankings. You can also pay attention to our coverage of Google metrics, which we report with each index, and to our correlations with rankings metrics. If these fall, it means Mozscape has gotten less Google-shaped and less representative of what influences rankings. If they rise, it means Mozscape has gotten better. Obviously, our goal is to consistently improve, but we can't be sure that every variation we attempt will have universally positive impacts until we measure them. Thanks for reading through, and if you have any questions, please leave them for us below. I'll do my best to follow up quickly.13 -
How to force moz to crawl my backlinks?
I have some good number number of backlinks in my webmaster tools. But, open site explorer is showing very few backlinks. How to force moz to crawl all the backlinks? Or is there any way to submit backlinks to moz?
Link Explorer | | sankar7890 -
Why are no-follow links in my blog comments across the web showing up as "equity-passing"?
When I make comments on blog posts (and there's a link in my comment or my name is the link to my site), the links are always no-follow (as they should be). But, when I check Open Site Explorer, the new links show up as equity-passing. Are they actually passing equity or is this a mistake?
Link Explorer | | infotrust20 -
Getting this on OSE - wp-content/cache/page_enhanced/
Hi there, In using OSE, it switches to www.mysite.com/wp-content/cache/page_enhanced/www.mysite.com/_index Is there something I need to do in my .htaccess or some other file to fix this? thanks chris
Link Explorer | | jcbradley110