What is a good crawl budget?
-
Hi Community!
I am in the process of updating sitemaps and am trying to obtain a standard for what is considered "strong" crawl budget? Every documentation I've found includes how to make it better or what to watch out for. However, I'm looking for an amount to obtain for (ex: 60% of the sitemap has been crawled, 100%, etc.)
-
@blueprintmarketing I have a large website with Wordpress image folders going back to 2009.
I am currently redesigning my website, and I am trying to determine if there is any benefit to trying to shrink down / delete those images and image folders which I am no longer using.
I really do not have time to go through all of those image folders, and see which ones I am still using, and which ones I am not using anymore. I am hoping this does not matter?
Does anyone here know if this matters when it comes to Google's Crawl Budget?
All of the images are completely optimized and crunched, however, my question is whether it would be worth the time investment to go through every single folder and thousands of images and try to delete the ones which are not being referenced on any of my pages?
Does anyone have a definitive answer regarding Crawl Budget?
-
Can you give some inputs about the site [https://indiapincodes.net/](link url) I tried all recommendations, only 30% of the url is been indexed. would appreciate your time.
-
@yaelslater
Unless you have a huge site, I'm talking about half a million to one million pages. I Would not worry about True Google crawl budge anymore.However, if only 60% of URLs in your XML site map are being indexed, make sure they are indexable URLs if they're not index value, or else you should be able to click in the coverage section of Search Console. It will give you a reason why your URL was submitted by an XML site map or not noindex.
A recent study showed about 20% of URLs on all websites across the study were not indexed for one reason or another.But make sure there are only 200 URLs, no redirects 301, 302, or 404's or noindex nofollow URLs in the XML sitemap because obviously, Google does not put them into the index if the Search Console does not tell you the issue & you would like to share your domain with me, I'm sure I could figure it out.
I don't know if you're using a CDN and if you could share a little more with me especially the domain I can be a lot more helpful.
You could also use a tool like screaming frog and generate a new site map and make sure that is not the issue. If you're using Yoast, you can turn it on and off if you wanted to create a new site map.
You can create up to 500 pages for free using Screaming Frog SEO Spider it is paid after that https://www.screamingfrog.co.uk/xml-sitemap-generator/
Or if you want it or you can generate over 1000 URLs for free online I would recommend https://www.sureoak.com/seo-tools/google-xml-sitemap-generator
However, please keep in mind the sureoak tool has things like a "keyword density checker" that makes me feel like this site is giving out that information because that's not a real thing that Google considers unless you use the same word for every word in the document. Keyword density is one of those things that are not real
But the XML site map generator works just fineI hope this was of help,
Tom
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Sites website https://www.opcfitness.com/ title NOT GOOD FOR SEO
We set up a website https://www.opcfitness.com/home on google sites. but google sites page title not good for SEO. How to fix it?
Technical SEO | | ahislop5740 -
Submitted URL has crawl issue - Submitted URL seems to be a Soft 404 - but all looks fine
Google Search Console is showing some pages up as "Submitted URL has crawl issue" but they look fine to me. I have set them as fixed but after a month they were finally re-crawled and google states the issue persists. Examples are: https://www.rscpp.co.uk/counselling/175809/psychology-alcester-lanes-end.html
Technical SEO | | TommyNewmanCEO
https://www.rscpp.co.uk/browse/location-index/889/index-of-therapy-in-hanger-lane.html
https://www.rscpp.co.uk/counselling/274646/psychology-waltham-forest-sexual-problems.html There's also some "Submitted URL seems to be a Soft 404": https://www.rscpp.co.uk/counselling/112585/counselling-moseley-depression.html I also have more which are "pending", but again I couldn't see a problem with them in the first place. I'm at a bit of a loss as to what to do next. Any advice? Thanks in advance.0 -
Akamai's Edge Redirector good for SEO?
Hey guys, Just wondering if anyone has used/tested Akamai's new 'Edge Redirector' cloudlet?http://www.akamai.com/html/technology/edge-redirector.html It seems like it would be a better/faster option than redirects at the server level via htaccess.. thoughts? Thanks!,
Technical SEO | | wojkwasi
Woj1 -
Google crawl rate dropped after we activated CloudFront
Hello! Previously we've been using Amazon CloudFront for our static content (js, css etc). But to be able to reduce load on our origin servers and to be able to give our international users a good user experience we decided to deliver a couple of our sites through CloudFront. We noticed very nice drops in page load time, but when checking Google webmaster tools we noticed that all CloudFront-activated sites got a huge drop in pages crawled per day (from avg ~3500 to ~150). Also one of the sites have issues with the Google sitemaps (just marked as "Pending" in GWT) and no new pages or updated pages seems to be updated in the Google SERP. The rest of the sites gets some updates on the Google SERP, but very few compared to before CloudFront activation. Is there anybody here who have experience in full site delivery through CloudFront (or other CDNs) and effects on SEO/Google? Would be very glad for any insights or suggestions. The risk is that we need to remove CloudFront if this just continues.
Technical SEO | | Ludde0 -
Crawl Diagnostics Report 500 erorr
How can I know what is causing my website to have 500 errors and how I locate it and fix it?
Technical SEO | | Joseph-Green-SEO0 -
Have a client that migrated their site; went live with noindex/nofollow and for last two SEOMoz crawls only getting one page crawled. In contrast, G.A. is crawling all pages. Just wait?
Client site is 15 + pages. New site had noindex/nofollow removed prior to last two crawls.
Technical SEO | | alankoen1230 -
Interesting Dilemma: Making A (Good) Content Scrapper Rank
Greetings Moz'ers! I'm in the process of optimizing a non-profit's website that has an interesting dilemma: They lack the resources to create their own content so they search the web for videos, quotes, blogs, pictures, etc. to post on their website. As the "good guys" their cause is just but because they are essentially scrapping content from other sites they are considered "bad guys" in Google's eyes and are getting hit with duplicate content penalties. What they are doing isn't much different than what the major news sites do - they just want to be a good resource center. The content that they are using is relevant to the cause and they give credit on everything they post by linking to where the content came from. Any recommendations on how I can help them? Thanks, Jason
Technical SEO | | DragonSearch0 -
seo moz crawl diagnosis
Hi seomozzers, We are creating a brand new website for a client and I would like to run an seo moz crawl to fix what has been done wrong. So my question is it ok to run an SEO moz crawl with a dev URL? Are final URLs and dev URLs will give me the same results or not? Basically, Should I wait for getting the final URL or is it ok to run a crawl under a dev URL such as www.dev2.example.com or http://183.2564.2864? Thank you 🙂
Technical SEO | | Ideas-Money-Art0