Is there a way to control a crawl?
-
I need to decrease our campaigns to 1,000 crawls we are currently at 10000 per
-
Hey Josh!
With the latest increase of crawl limits for all campaign, unfortunately the lowest is set for 10,000. If you have less than 10,000 pages then we will only crawl the total amount of pages you have. Takeshi has great tips on excluding certain pages from being crawled in robots.txt
Our user-agent is rogerbot so an example would be
User-agent: rogerbot
Disallow: /categories/Hope this helps!
-
Yes I am talking about pages crawled.
-
Kind of what I was wondering. I think this is a product deal, since they referenced 10000.
-
Hi Josh!
Thanks for posting in Q&A! Just to clarify, are you referring to your _pages _crawled in Moz Analytics?
-
Are you asking about crawls or indexation?
I can't think of any reason why you would want to decrease crawls, but you could probably do so by getting backlinks removed, blocking sections of your site from being crawled through robots.txt or nofollow links, and decreasing the change frequency in your xml sitemaps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best way to manage multiple Google My Business accounts as a marketing agency?
As marketing agency we have for each client a Google My Business account. This means that we have more than 100 Google My Business accounts. We know that GA has a limit of 100 analytics accounts (free version), but is there a limit in Google My Business too? For now, we have created multiple accounts that own different Google My Business accounts of our clients. Can I just change all the different e-mail accounts of our marketing agency and change it to one? So we can manage all the Google My Business accounts of our clients in one e-mail account?
Image & Video Optimization | | conversal0 -
How to see SERPs the way that moz sees SERPs
I'm getting to grips with the New Keyword Explorer. I've found that one of my top keyword phrases has a 'Featured Snippet' result. However I'm struggling to see it in my own browser. Is there a clean way of viewing SERPs un enhanced by history, location etc? I'd also like to see the other SERPs features - is that possible? I'd like to be able to see the results as Moz clearly can view them! Thank you Mozers C09Fb
Image & Video Optimization | | Catherine_Selectaglaze0 -
What is the best way to optimise the website of a service area business websites for local search?
Service area businesses are a little caught in the middle when it comes to local optimisation and can really struggle to So what approaches would your recommend for optimising a service area business: Assuming the following, perhaps obvious, goals: Make sure your site visitors can clearly understand if their location is covered by your service areas Compete for search traffic for service + location searches (including those with an implided location). One common tactic that I see being used is to generate humongous lists of locations, maybe in a footer. Obviously the more areas you want to include the less relevant (and more spammy looking) your targetting looks. And of course, with loads of locations, you're not going to be able to get them all into your titles (even if you wanted to!) Maybe if you've got locations that are more inportant to you you might create specific landing pages for these location? If you've got the kind of business where the locations you target are determined by the locations you can reasonably get to during a working day it's really tough to find a realistic way to target specific location on your site. I'd be really interested to hear how you approach these kinds of sites.
Image & Video Optimization | | DougRoberts0 -
Best Way to Replace Lost Links
We found a particular client that was using a 3rd party backlinker to build links (they were outsourced). The backlinks appeared to be low quality, just basic text and no substantial content surrounding it, usually like a forum profile. The backlinker was swapped out. However, we've been suffering from a big loss of links. We anticipated that quite a few of the links the previous crew were managing would be lost, however, we've lost over 1500. Some of that is due to the former SEO company using sitesays.com as part of their strategy. That site has since removed all listings/links because they were hit by recent algorithm changes. Every week GWT has fewer and fewer links coming from that site. We expect this to continue until GWT has removed all links from that site. We are also losing ground (quickly) on local rankings too. We believe this is due to the high number of links lost. What is best practice when building links to counter this issue? Is there anything that can be done to stem the link purge and, if possible, gain ground on rankings again?
Image & Video Optimization | | AaronHenry0 -
Asking Robots to Crawl? Does this really work?
Curious, we have a competitor on one of our sites and they have the following code. Can't figure out how they are beating us, thought this might be it. Or perhaps its because they have an older copyright on their site?
Image & Video Optimization | | greenhornet770 -
Best way to publish same content on 2 blogs?
I currently have a couple of blogs and I want to put a post on one of them and then copy it onto my other blog, but give the credit and the link back to the original (lose you yet?) I want to write a blog and publish it on my site (http://goo.gl/WRwR8) After it's on the blog on my main e-commerce site, i want to copy it onto my wordpress blog (http://goo.gl/0SSti) What is the best way to accomplish this without creating duplicate content? Is there a way to state on the second wordpress blog that this post originated from the ecommerce site? I could really use some guidance... Thanks
Image & Video Optimization | | Prime850 -
Australian Service that Allows you to Control all your Local Listings
Hey, After a few stumbles upon a variety of services that allows you to manage all your local listings from the one profile (the latest one was Yext.com). It got me thinking - is there an Australian equivalent? My research revealed nothing. Thanks.
Image & Video Optimization | | LukeyJamo1 -
What are some effective ways to take up SERP real estate?
Hey Guys, I was wondering if you've seen or practiced some effective ways to take up real estate in the SERPS. Ideally I'd like to see more results from my own domain although I realize that G seems to only display 3 for any single domain. I'm looking for creative ways to take up more real estate other than subdomains. I recently saw a municipal website with 4 results (1 being a PDF) which suprised me. I have a couple in my head but what's worked best for you?
Image & Video Optimization | | reegs0