Screaming frog Advice
-
Hi
I am trying to crawl my site and it keeps crashing.
My sys admins keeps upgrading the virtual box it sits on and it now currently has 8GB of memory, but still crashes.
It gets to around 200k pages crawl and dies.
Any tips on how I can crawl my whole site, can u use screaming frog to crawl part of a site.
Thanks in advance for any tips.
Andy
-
Thanks, I tried all the tips on the screaming frog site, but I have just tried to 2 pages a second and lets hope that work.
-
Hi Andy. There are quite a few settings you can adjust to make the server load less while the crawl is running. These can be found with descriptions here: http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
For example, by not checking Images, CSS, SWF, and Javascript you'll be able to lessen load substantially, or if you'd like to crawl just a portion of the site you can set it to not check links outside of the start folder.
To have even more control over the crawl, you can use regular expressions to exclude certain pages, or sections that match a given pattern. The page above is fairly robust, so it should help you dial back the crawler to be friendlier to your server. Cheers!
-
Hey there mate,
Sorry to hear that you are having issues. You can actually ask Screaming Frog to use more RAM. If you haven't done that yet please give it a go.
You can find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
If you want to crawl part of your site it can surely do that. You can exclude pages or whole sections.
Find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need advice on redirects
Hi, I have new web addresses for my subpages. None if them have external links. Should I do redirects to the new pages or just leave the old pages in 404 and let google crawl and rank the new page. I am asking because my current pages don’t have a good ranking and I am thinking starting with a clean url is better. Thank you,
Intermediate & Advanced SEO | | seoanalytics1 -
Can you give me some advices to rank this domain?
Hi Moz community, I've a coleague that's working to rank this site: www.devsar.com. The selected keywords are:
Intermediate & Advanced SEO | | Gaston Riera
Mobile development
Web development
Django Development
Python Development I've checked the site: It's fast and clean. Has a good PA and DA. It's responsive and good lookking. Meta description , title, hreflang.. everything in order. Link profile a little rare (checked with ahref.com), it's because someone made a mistake redirecting some expired domain Can you help me to help my mate out?
Thanks
GR.0 -
Need advice and smart solution for H1
Hi! I've been quite a long time trying to find a solution to the html structure of a webpage that I'am auditing right now. But I need your valuable help! The problem is the following: Actually, they have different sections inside. The structure is something like this Masters - Degrees - Programs - About us - News And if you go to the Masters Section you will find something like this Masters Master number 1 in Tourism (brand.com/master/master-number-1-tourism) Presentation
Intermediate & Advanced SEO | | teconsite
(brand.com/master/master-number-1-tourism) Objectives
(brand.com/master/master-number-1-tourism/objectives) Professional opportunites
(brand.com/master/master-number-1-tourism/professional-opportunities) Faculty
(brand.com/master/master-number-1-tourism/faculty) Qualification
(brand.com/master/master-number-1-tourism/qualification) Financial
(brand.com/master/master-number-1-tourism/financial) Master number 2 with a long name Presentation Objetives Profesisional opportunities Faculty ... Master number 3 in Sports and so on The Degrees section, has inside exactly the same structure with the same names. My doubt is related with the use of h1 tag What would be the best h1 strategy for each content page? Each master has 6 pages (presentation, objectives, faculty,...) For page Objetives,
brand.com/master/master-number-1-tourism/objectives If I choose to use as H1 just the word Objetives, what will happen is that I will have a lot of pages (one per master, degree or program), with the same H1, because each master will have its own page Objectives. If we have 10 masters + 10 degrees + 2 programs y will have 22 pages with the same H1 Objecives If I choose to use as H1 the following: "Objectives of the Master Number 1 in Tourisim and so on with long name" it will be difficult for the users to visually see the difference between the different pages. for instance Objectives of the Master Number 1 in Tourisim and so on with long name and Faculty of the Master Number 1 in Tourisim and so on with long name because they only differ in one word. What do you think of this solution? Objectives Master Number 1 in Tourisim and so on with long name is it correct to do this inside the h1? or would you use combinations of h1 and h2 like these
h1: Objectives
h2: Master Number 1 in Tourisim and so on with long name would be this appropiate?0 -
Advice on Content Marketing in a Tough Niche
Hello, In our niche, nobody links to the content/information with rare exceptions. Do you guys have any good articles/ideas for cases like this? The content that is linked to is once removed in subject matter from the content of our site, like if we sold shoes and had to write on different types of clothing stores. Looking for advice on what to do and how to figure out what to write about. We've probably got a descent budget this time but we're not sure how to go about this. Any advice is appreciated.
Intermediate & Advanced SEO | | BobGW0 -
Advice on Link Building?
I know webmasters shouldn't focus on link building but unfortunately there are some types of content that doesn't get shared as much as other. And for content to go viral, it ain't that easy and it's almost impossible in some smaller niches where you don't have the volume to go "viral". That said I know about the common link building techniques. I know I can submit guest posts but when you're competing with websites that have over 10,000 backlinks, there is no way I'm going to get close to this with guest posting and commenting on other blogs. One way I found for getting backlinks is to publish interviews. Most of the time, people/businesses you interview like to link to this type of content. Publishing value-added content about other businesses' products or services may get some backlinks in return but not that often. So other than that, can some of you share some "out-of-the-box" link building strategies? Thank you in advance!
Intermediate & Advanced SEO | | sbrault740 -
Advice needed on how to handle alleged duplicate content and titles
Hi I wonder if anyone can advise on something that's got me scratching my head. The following are examples of urls which are deemed to have duplicate content and title tags. This causes around 8000 errors, which (for the most part) are valid urls because they provide different views on market data. e.g. #1 is the summary, while #2 is 'Holdings and Sector weightings'. #3 is odd because it's crawling the anchored link. I didn't think hashes were crawled? I'd like some advice on how best to handle these, because, really they're just queries against a master url and I'd like to remove the noise around duplicate errors so that I can focus on some other true duplicate url issues we have. Here's some example urls on the same page which are deemed as duplicates. 1) http://markets.ft.com/Research/Markets/Tearsheets/Summary?s=IVPM:LSE http://markets.ft.com/Research/Markets/Tearsheets/Holdings-and-sectors-weighting?s=IVPM:LSE http://markets.ft.com/Research/Markets/Tearsheets/Summary?s=IVPM:LSE&widgets=1 What's the best way to handle this?
Intermediate & Advanced SEO | | SearchPM0 -
Conversion Rate Optimisation - advice before seeking out a specialist
Hello! I have a site I've been working on for a lovely client, a small business start up since Christmas. The site has a very simple layout, is ranking well and maintaining its positions, has solid social media, is receiving enough traffic and ranking for a number of terms. The problem is - conversions! The site just isn't converting. I have spoken with a few peers who have said advanced CRO will be too much for me to learn in terms of Psychology of Buying, learning about colors, fonts etc. I understand meta descriptions for example are something that I can do, I was wondering if anyone could give me advice on any other basic CRO techniques I could apply to the site before going to a specialist. Any advice would be MUCH appreciated - the moz community is always so helpful! Charlotte 🙂
Intermediate & Advanced SEO | | CharlotteWaller1 -
Renaming primary domain advice
Hi, We have a site www.anteater.com.au. It's a pest control company here in Sydney Australia. Now I've just renamed it to www.anteaterpestcontrolsydney.com.au yesterday for SEO purposes. I'm doing 301 redirects. My question is, is it worth it to do this? Will we have any advantage in the long run with just this URL change? Or would it be better to just leave it as is as www.anteater.com.au Any comments would be appreciated. Cheers.
Intermediate & Advanced SEO | | Peter.Huxley590
