Is there a way to control a crawl?
-
I need to decrease our campaigns to 1,000 crawls we are currently at 10000 per
-
Hey Josh!
With the latest increase of crawl limits for all campaign, unfortunately the lowest is set for 10,000. If you have less than 10,000 pages then we will only crawl the total amount of pages you have. Takeshi has great tips on excluding certain pages from being crawled in robots.txt
Our user-agent is rogerbot so an example would be
User-agent: rogerbot
Disallow: /categories/Hope this helps!
-
Yes I am talking about pages crawled.
-
Kind of what I was wondering. I think this is a product deal, since they referenced 10000.
-
Hi Josh!
Thanks for posting in Q&A! Just to clarify, are you referring to your _pages _crawled in Moz Analytics?
-
Are you asking about crawls or indexation?
I can't think of any reason why you would want to decrease crawls, but you could probably do so by getting backlinks removed, blocking sections of your site from being crawled through robots.txt or nofollow links, and decreasing the change frequency in your xml sitemaps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Taking control over Google Places and Google+ Pages?
Hi we have a client who has about 40 different stores around the country. Most of the stores have their own Google+ Pages or Google Places. We want to take control of all 40 stores from a centralized location. So we can control their Google+ Pages/Google Places to improve them, and keep them uniformed with each other.How/Can can we do this? Does Google provide any way to do this? Thanks, Mark
Image & Video Optimization | | MBASydney0 -
Asking Robots to Crawl? Does this really work?
Curious, we have a competitor on one of our sites and they have the following code. Can't figure out how they are beating us, thought this might be it. Or perhaps its because they have an older copyright on their site?
Image & Video Optimization | | greenhornet770 -
Best way to use popular YouTube channel for SEO purposes
Hi all, We have a YouTube channel containing about 25 videos we created professionally a few years ago. They are all product features and are hosted on our site for applicable products (a very small percentage of our overall number of products). This channel was largely forgotten about for a few years. It has now reached over 160k views. What's the best way to utilize these videos for SEO benefits? Any techniques anyone has to maximize our ever increasing number of views? Thanks
Image & Video Optimization | | bradkrussell0 -
What is the best way to host a video on my site from a SEO perspective?
Hi, we are in the process of creating a video about a topic that is highly relevant for our customers. Once it is completed we want to offer the video to our customers so that they can embed it into their own websites for free. What is the best way to do this and still receive maximum SEO juice? If we use a a video service like youtube or vimeo, then the link will go to vimeo or youtube and not our site. One thing we are thinking about is asking our customers to link to our site in return for embedding the video, but if there is an automatic solution then this would be a lot better. Any tips will be highly appreciated! If you have anyother tips regarding video content then the same of course! Many thanks! Daan
Image & Video Optimization | | daan.loening0 -
Australian Service that Allows you to Control all your Local Listings
Hey, After a few stumbles upon a variety of services that allows you to manage all your local listings from the one profile (the latest one was Yext.com). It got me thinking - is there an Australian equivalent? My research revealed nothing. Thanks.
Image & Video Optimization | | LukeyJamo1 -
Best way to host video?
Hi, We are planning to host videos on our site for the first time. It looks like YouTube is a bit too over focused on ads these days for commercial use, can anyone recommend any better video hosting services? I also realize that hosting somewhere like YouTube has little SEO benefit, am I better off hosting them on our site if possible? Please share your experiences with the various services and methods of hosting. Thanks
Image & Video Optimization | | Maximise1 -
Is there a very quick way to geenrate the google top 100 for a any given keyword
I ma looking to produce a list of the top 100 websites in a list format from any given keyword. Bascially a process or ftool that will generate a list so i dont have to copy and paste all of them.....any ideas guys and gals
Image & Video Optimization | | raytax0 -
Is there an easy way to see how competitive a local search term is?
Is there a quick and easy way to see how competitive local search terms are? I am looking at helping my clients show up on the local search results. Some times all I have to do is claim a listing and they move right to the top. Sometimes I claim a listing and nothing happens so I get links and I get reviews and it takes awhile. I want to be able to put an accurate price point on the service I am offering. I have looked at the search volume and it hasn't been the best indicator because some industries are more competitve than others. Any help or suggestions would be appreciated.
Image & Video Optimization | | jimmyseo0