Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?
-
We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries. It's madness. I can't figure out what is causing these URLs to generate to be crawled in the first place! Does this sound familiar to anyone out there, any constructive suggestions? Robots text or maybe meta robots tags that would resolve this crawl issue?
-
Hey,
I'm not sure if this is resolved for you, but as you suggest you can do something with robots.txt. Specifically, you could use wildcards to capture these URLs and tell Rogerbot (Moz's crawler) to ignore them. Here's a great Stackoverflow query to get you started and details on how block Rogerbot, you can take a look here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I want to know how PA works in MOZ?
I want to know how PA works in MOZ? Like I have a website https://myinfo.pk its DA is same on all posts but when I see PA of IESCO Bill Page and LESCO Bill page. Both are different. Why PA is different of many posts when DA is same. Is there any specific reason?
Link Explorer | | ehtishamkhan0 -
What is difference between Moz data than ahrefs?
hi, On ahrefs it's showing me that my web Fit Food has 1.2k backlinks and 140 referring domain. while on Moz it's totally different stats which making me confuse to understand. So is Domain rating is different than Domain Authority?
Link Explorer | | SophiaRPereira1 -
Page Authority higher on my old (redirected) domain than new domain?
Hi everyone. I moved the domain my blog was on about 18 months ago, and it's ranking in Google never recovered. I've noticed in the 'Inbound Links' tool that my old domain (jamescrowley.co.uk) shows a page authority of 26, while my new domain (jamescrowley.net) shows a page authority of 15. Any idea why that would be? 301 redirects have been in place for the whole 18 months, so I'd have thought the authority would have been 'passed on' by now? Many thanks James
Link Explorer | | james.crowley0 -
Moz tool to estimate a website's organic traffic?
Moz has tons of great tools, so I was quite suprised there is nothing similar to Alexa's siteinfo tool, or SimilarWeb's site comparison tool. Am I missing something? Or does such a tool exists at Moz? Or is there a different way of estimating my competitor's traffic? It would be great if Moz would have something like that.
Link Explorer | | GrandRapids0 -
Fastest way to get new backlinks recognised by Moz?
Hi all, I have seen a few discussions / advice pieces around this, but wondered what the current view is on the fastest way to get new backlinks recognised by Moz? I have seen people say that the content needs to be shared (e.g. on Twitter), and advice around creating Bitly links. I am not convinced that these techniques are working. On a related note, my website is very large. I have around 2.5m pages. Most of these pages are 3-4 clicks from the home page, but I have a small amount of important content much higher up in the site (1-2 clicks from the home page). Does Moz crawl pages nearer the home page more frequently, or does it make a pass of all pages on the site before it starts at the beginning again? If it makes a complete pass of all pages, then presumably making my site smaller will help to get Moz to see new content more quickly? Is there any other value to be gained in making a large site smaller? Many thanks in advance, Neil.
Link Explorer | | NHARRIS10 -
Spam Score and crawling of my site
Hello, I'm trying to analyze the spam score of my site which is 9/17 Actually I have few backlinks and all of them have a low spam score (max 4/17, just one). I think there's some kind of issue with the crawler since I get strange spam factors: Large Site with Few Links (likely true, I recently deleted a lot of tags used once) Low Number of Pages Found (wasn't it a "Large Site"??) Low Number of Internal Links (I got a considerable number) No Contact Info (I have a link to my facebook in the menu and a "contacts" page) Thin Content (It's just a blog with min 300 words per post, why thin?) Site Link Diversity is Low (likely true) Ratio of Followed to Nofollowed Subdomains (likely true) Low MozTrust or MozRank Score (true) Ratio of Followed to Nofollowed Domains (likely true) Can you please help me to understand it, is it a crawling problem or similar? If needed I will post the url of the website. Thank you so much Marco
Link Explorer | | MarcoBP0 -
When requesting a CSV for an internal 302 redirect report, the data isn't aligning with the UI. Does anyone have any suggestions?
When requesting a CSV for an internal 302 redirect report, the data isn't aligning with the UI. Does anyone have any suggestions?
Link Explorer | | Merkle-Impaqt0 -
Why are none of the links we've received from press and other sources from the last two months showing on our Inbound Links in OSE???
It's really hard to judge if the DA of our site according to MOZ is 1) up to date and 2) accurate when for example there are SO many inbound links still not recorded, that were published months ago.
Link Explorer | | TVape0