Crawl Budget on Noindex Follow
-
We have a list of crawled product search pages where pagination on Page 1 is indexed and crawled and page 2 and onward is noindex, noarchive follow as we want the links followed to the Product Pages themselves. (All product Pages have canonicals and unique URLs) Orr search results will be increasing the sets, and thus Google will have more links to follow on our wesbite although they all will be noindex pages. will this impact our carwl budget and additionally have impact to our rankings?
Page 1 - Crawled Indexed and Followed
Page 2 onward - Crawled No-index No-Archive Followed
Thoughts?
Thanks,
Phil G
-
Check out Google's latest "handling" of pagination using rel=canonical, rel=next + rel=prev
http://www.youtube.com/watch?v=njn8uXTWiGgYou can now:
Page 1:
- canonical: page 1
- next: page 2
Page 2:
- canonical: page 2
- next: page 3
- prev: page 1
Page 3:
- canonical: page 3
- next: page 4
- prev: page 2
Page 4 (say last page):
- canonical: page 4
- prev : page 3
Another option is to have a "view all" page which lists all products & you can point a canonical to that page from all pages within the set
Hope that helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Suggested Screaming Frog configuration to mirror default Googlebot crawl?
Hi All, Does anyone have a suggested Screaming Frog (SF) configuration to mirror default Googlebot crawl? I want to test my site and see if it will return 429 "Too Many Requests" to Google. I have set the User Agent as Googlebot (Smartphone). Is the default SF Menu > Configuration > Speed > Max Threads 5 and Max URLs 2.0 comparable to Googlebot? Context:
Intermediate & Advanced SEO | | gravymatt-se
I had tried NetPeak SEO Spider which did a nice job and had a cool feature that would pause a crawl if it got to many 429. Long Story short, B2B site threw 429 Errors when there should have been no load on a holiday weekend at 1:00 AM.0 -
Crawl rate drop
Hi Guys, I have a crawl rate drop in webmastertools and can't figure out way. In the last month I removed a lot o duplicate pages that we don't need anymore, there were at least 1.5 million pages. Can this be a motive? D7O5x1l
Intermediate & Advanced SEO | | Silviu0 -
Best strategy to follow for a single service site
Can anyone share what they feel is the best strategy to follow for a single service site? Would you optimise and target the homepage for the primary service they offer or target a page one level lower and leave the homepage to target the Brand name? Links to any references or case studies would also be greatly appreciated, thank you!
Intermediate & Advanced SEO | | Marketing_Today0 -
Fix broken external links on noindex, follow pages no one visits?
Would you take the time to fix external links on your site on pages that are noindex, follow on pages that no one ever visits? The only reason to do it would be to present a tidier site to Google, but would it really care if those pages are noindex/folllow? The thing that makes it a non-trivial amount of work is that there are hundreds of these on a large site. Do you think Google cares, if they're noindex/follow? I know the safe answer is always fix everything, but really it has to get weighed against the likely benefit and other projects with a limited amount of time to work with. Best... Mike
Intermediate & Advanced SEO | | 945010 -
Crawl efficiency - Page indexed after one minute!
Hey Guys,A site that has 5+ million pages indexed and 300 new pages a day.I hear a lot that sites at this level its all about efficient crawlabitliy.The pages of this site gets indexed one minute after the page is online.1) Does this mean that the site is already crawling efficient and there is not much else to do about it?2) By increasing crawlability efficiency, should I expect gogole to crawl my site less (less bandwith google takes from my site for the same amount of crawl)or to crawl my site more often?Thanks
Intermediate & Advanced SEO | | Mr.bfz0 -
If I had an issue with a friendly URL module and I lost all my rankings. Will they return now that issue is resolved next time I'm crawled by google?
I have 'magic seo urls' installed on my zencart site. Except for some reason no one can explain why or how the files were disabled. So my static links went back to dynamic (index.php?**********) etc. The issue was resolved with the module except in that time google must have crawled my site and I lost all my rankings. I'm nowher to be found in the top 50. Did this really cause such an extravagant SEO issue as my web developers told me? Can I expect my rankings to return next time my site is crawled by google?
Intermediate & Advanced SEO | | Pete790 -
Is it safe to not have a sitemap if Google is already crawling my site every 5-10 min?
I work on a large news site that is constantly being crawled by Google. Googlebot is hitting the homepage every 5-10 minutes. We are in the process of moving to a new CMS which has left our sitemap nonfunctional. Since we are getting crawled so often, I've met resistance from an overwhelmed development team that does not see creating sitemaps as a priority. My question is, are they right? What are some reasons that I can give to support my claim that creating an xml sitemap will improve crawl efficiency and indexing if we are already having new stories appear in Google SERPs within 10-15 minutes of publication? Is there a way to quantify what the difference would be if we added a sitemap?
Intermediate & Advanced SEO | | BostonWright0 -
Page Crawling Check after Modification Done without staying 7 days
Page Crawling Check after Modification Done without staying 7 days. I have dome modification to my site and uploaded .so i wanna check remaining errors but Moz Crawl web site once per 7 days ,is there any way to check before that . Thank you
Intermediate & Advanced SEO | | innofidelity0