Value of having a good crawl budget?
-
Hi,
I've seen several questions where people give advice on how to increase the crawl budget. What I haven't seen anyone comment is what the value of this really is if you have many pages that doesn't get updated very often.
Take for example the typical agency page - 50 pages, most of them rarely gets updates. In a monthly basis normally 10% of the website gets updated.
Is there really any value then of having 100% of the website crawled on a daily basis?
-
Check out this awesome post by AJ Kohn
To answer your question...
"What I’ve observed over the last few years is that pages that haven’t been crawled recently are given less authority in the index. To be more blunt, if a page hasn’t been crawled recently, it won’t rank well."
"The pages that aren’t crawled as often are pages with little to no PageRank. CrawlRank is the difference in this very large pool of pages. You win if you get your low PageRank pages crawled more frequently than the competition."
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do YouTube videos in iFrames get crawled?
There seems to be quite a few articles out there that say iframes cause problems with organic search and that the various bots can't/won't crawl them. Most of the articles are a few years old (including Moz's video sitemap article). I'm wondering if this is still the case with YouTube/Vimeo/etc videos, all of which only offer iFrames as an embed option. I have a hard time believing that a Google property (YT) would offer an embed option that it's own bot couldn't crawl. However, let me know if that is in fact the case. Thanks! Jim
Technical SEO | | DigitalAnarchy0 -
Why is my crawl taking so long?
Hi There, My crawl for albertcuyp.nl is taking very long, it started on the 10th of april. I don't know whats going on but i think 2 weeks for a crawl is extremely long. Can you help me?
Technical SEO | | KnowHowww0 -
Is Buying Domains Good For SEO? Can I 301 redirect domains to an Original website?
I have a friend that purchased multiple domains related to their website. Each of these domains have the back ground of the original website and irrelevant content on them. Is is possible to redirect the various domains to certain pages on the original website. For example if the website is www.shoes.com and they purchased domains such as www.leathermensshoes.com and a few others related to the website. Is it SEO friendly to link the domains purchased to the original website?
Technical SEO | | TSpike10 -
Site maintenance and crawling
Hey all, Rarely, but sometimes we require to take down our site for server maintenance, upgrades or various other system/network reasons. More often than not these downtimes are avoidable and we can redirect or eliminate the client side downtime. We have a 'down for maintenance - be back soon' page that is client facing. ANd outages are often no more than an hour tops. My question is, if the site is crawled by Bing/Google at the time of site being down, what is the best way of ensuring the indexed links are not refreshed with this maintenance content? (ie: this is what the pages look like now, so this is what the SE will index). I was thinking that add a no crawl to the robots.txt for the period of downtime and remove it once back up, but will this potentially affect results as well?
Technical SEO | | Daylan1 -
Suggestions on good framework/code for building an optimized website?
There seem to be quite a few template, framework, and theme options for building a site optimized for search. I'm currently looking at Socrates and Genesis premium themes for Wordpress. Does anyone have experience or suggestions on these resources?
Technical SEO | | ksracer0 -
Good organic rankings but no shopping rankings
My site www.hair1direct.co.uk comes up well in the organic rankings but when I select shopping in Google it does appear. Why would this be? Thanks James
Technical SEO | | avecsys0 -
How to increase the crawl rate?
hello, Our site was hosted in North America and Google was crawling it reasonably fast. Since our traffic is mostly from India we moved it to India, now the crawling is terribly slow from Google. Is there anyway to fix the crawl rate(we have increased the crawl rate in GWT)
Technical SEO | | greyniumseo0 -
SeoMoz robot is not able to crawl my website.
Hi, SeoMoz robot crawls only two web pages of my website. I contacts seomoz team and they told me that the problem is because of Javascript use. What is the solution to this? Should I contact my webdesign company and ask them to remove Javascript code?
Technical SEO | | ashish2110