Value of having a good crawl budget?
-
Hi,
I've seen several questions where people give advice on how to increase the crawl budget. What I haven't seen anyone comment is what the value of this really is if you have many pages that doesn't get updated very often.
Take for example the typical agency page - 50 pages, most of them rarely gets updates. In a monthly basis normally 10% of the website gets updated.
Is there really any value then of having 100% of the website crawled on a daily basis?
-
Check out this awesome post by AJ Kohn
To answer your question...
"What I’ve observed over the last few years is that pages that haven’t been crawled recently are given less authority in the index. To be more blunt, if a page hasn’t been crawled recently, it won’t rank well."
"The pages that aren’t crawled as often are pages with little to no PageRank. CrawlRank is the difference in this very large pool of pages. You win if you get your low PageRank pages crawled more frequently than the competition."
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to allow bots to crawl all but WP-content
Hello, I would like my website to remain crawlable to bots, but to block my wp content and media. Does the following robots.txt work? I worry that the * user agent may conflict with the others. User-agent: *
Technical SEO | | Tom3_15
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/ User-agent: GoogleBot
Allow: / User-agent: GoogleBot-Mobile
Allow: / User-agent: GoogleBot-Image
Allow: / User-agent: Bingbot
Allow: / User-agent: Slurp
Allow: /0 -
When Should I Ignore the Error Crawl Report
I have a handful of pages listed in the Error Crawl Report, but the report isn't actually showing anything wrong with these pages. I am double checking the code on the site and also can't find anything. Should I just move on and ignore the Error Crawl Report for these few pages?
Technical SEO | | ChristinaRadisic0 -
Paid directory links--good or bad thing to do for Prof. Services sites?
Yes, I am pretty darn new to SEO and have heard that Google doesn't like Paid directory links. I have looked at some of my competition and they must have paid to get in certain directories as I don't see any backlinks posted on their home pages (usually a badge or icon must be placed on the homepage to avoid a fee.)
Technical SEO | | Stratocaster0 -
Remove more than 1000 crawl errors from GWT in one day?
In google webmasters tools you have the feature "Crawl Errors". This one displays the top 1000 crawl errors google have on your site. I have around 16k crawl errors at the moment, which all are fixed. But i can only mark 1000 of them as fixed each day/each time google crawls the site. (This as it only displays top 1000 errors. When i have marked those as fixed it won't show other errors for a while.) Does anyone know if it's possible to mark ALL errors as fixed in one operation?
Technical SEO | | Host10 -
Domain Crawl Question
We have our domain hosted by two providers - web.com for the root and godaddy for the subdomain. Why SEOMOZ is not picking up the total pages of the entire domain?
Technical SEO | | AppleCapitalGroup0 -
Mobile sitemaps - how much value?
Hi, We have a main www website with a standard sitemap. We also have a m. site for mobile content (the mobile site only contains our top pages and doesn't include the entire site). If a mobile client accesses one of our www pages we redirect to the m. page. If we don't have a m. version we keep them on the www site. Since we already have a www sitemap, is there much value in creating a mobile site map? The mobile site (although missing all pages) is pretty robust and contains most content people are looking for. Will the mobile sitemap help for Mobile searches (more so than our standard sitemap)? I'm also planning on rel canonical the m. pages to the www. pages (per other suggestios on SEOMoz) Thanks
Technical SEO | | NicB10 -
Too many 301 redirects - good or bad?
Hi, Currently, page A is redirecting to page B. I am in the process of developing new site for the same domain and this time page B will be redirected to page C. This is gonna happen on many pages. Is it correct or should i adopt some other strategy? Will it have adverse effect on the speed of my site? Page A -----> Page B ------> Page C Regards, Shailendra
Technical SEO | | IM_Learner0