Screaming frog Advice
-
Hi
I am trying to crawl my site and it keeps crashing.
My sys admins keeps upgrading the virtual box it sits on and it now currently has 8GB of memory, but still crashes.
It gets to around 200k pages crawl and dies.
Any tips on how I can crawl my whole site, can u use screaming frog to crawl part of a site.
Thanks in advance for any tips.
Andy
-
Thanks, I tried all the tips on the screaming frog site, but I have just tried to 2 pages a second and lets hope that work.
-
Hi Andy. There are quite a few settings you can adjust to make the server load less while the crawl is running. These can be found with descriptions here: http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
For example, by not checking Images, CSS, SWF, and Javascript you'll be able to lessen load substantially, or if you'd like to crawl just a portion of the site you can set it to not check links outside of the start folder.
To have even more control over the crawl, you can use regular expressions to exclude certain pages, or sections that match a given pattern. The page above is fairly robust, so it should help you dial back the crawler to be friendlier to your server. Cheers!
-
Hey there mate,
Sorry to hear that you are having issues. You can actually ask Screaming Frog to use more RAM. If you haven't done that yet please give it a go.
You can find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
If you want to crawl part of your site it can surely do that. You can exclude pages or whole sections.
Find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keyword rank drop, any advice?
Screen Shot 2021-08-26 at 19.02.18.png My search visibility dropped from around 13% a few weeks ago to 8.29%. I know that Google launched a bunch of updates in this past few weeks to ignore spam links, and I'm pretty sure that was the reason for the drop - some of the links to my site date back over 10 years and those links were garbage. Confusingly, at the same time, my Domain Authority went up by 1 to 32, then back down a week later. How can I restore my previous rank in the short term? We're designing a new site at the moment with vastly improved page speed, but I'm not sure what effect that will have yet (thespacecollective.com).
Intermediate & Advanced SEO | | moon-boots0 -
Advice on redirects for a category I want to reuse.
Hi, We have a current category set up that is starting to rank OK but we are going through a site re-build and this category URL will now better describe a new category of products. My dilemma is if I 301 redirect the current url to my new category I won't be able to use the URL for the new one. But if I don't redirect it will the pages that have already been ranked under this url then confuse customers and search engines. For example - Products and sub-categories under the URL /personalised-toys will now become /personalised-toys-for-boys but I want to use the /personalised-toys URL for a different set of sub categories and products. Any assistance or ideas or just definitely don't do it in a particular way would be greatly appreciated
Intermediate & Advanced SEO | | neil_stickergizmo0 -
I have an authority site with 90K visits per month. Now I have to change from non www to www. Will incur in any SEO issues while doing that? Could you please advice me on the best steps to follow to do this? Thank you very much!
Because I want to increase site speed, Siteground (my hosting) suggested I use Cloudflare Plus which needs my site to have www in order to work. I'm also using a cloud hosting. Im a bit scared of doing this, and thus decided to come to the community. I used MOZ for over 6 months now and love the tool. Please help me make the best possible decisions and what steps to follow. It would be much appreciated. Thank you!
Intermediate & Advanced SEO | | Andrew_IT0 -
Panda penalty removal advice
Hi everyone! I'm after a second (or third, or fourth!) opinion here! I'm working on the website www.workingvoices.com that has a Panda penalty dating from the late March 2012 update. I have made a number of changes to remove potential Panda issues but haven't seen any rankings movement in the last 7 weeks and was wondering if I've missed something... The main issues I identified and fixed were: Keyword stuffed near duplicate title tags - fixed with relevant unique title tags Copies of the website on other domains creating duplicate content issues - fixed by taking these offline Thin content - fixed by adding content to some pages, and noindexing other thin/tag/category pages. Any thoughts on other areas of the site that might still be setting off the mighty Panda are appreciated! Cheers Damon.
Intermediate & Advanced SEO | | Digitator0 -
Strange 404s in Screaming Frog
I just ran a website (Drupal) through screaming frog and the only 404s I found related to web pages which were the same as URLs already used on the website plus the company phone number so... www.company.com/[their phone number] - www.company.com/services[their phone number] - any ideas what might be causing this problem?
Intermediate & Advanced SEO | | McTaggart0 -
[Need advice!] A particular question about a subdomain to subfolder switch
Hello Moz Community! I really was hoping to get your help on a issue that is bothering me for a while now. I know there is a lot of about this topic but I couldn’t find a good answer for my particular question. We are running several web applications that are similar but are also different from each other. Right now, each one has its own subdomain (which was mainly due to technical reasons). Like this: webapp1.rootdomain.com, webapp2.rootdomain.com etc. Our root domain currently points with 301 to webapp1.rootdomain.com. Now, we are thinking about making two changes: changing to a subfolder level like this: rootdomain.com/webapp1 , rootdomain.com/webapp2 etc. Changing our rootdomain to a landing page (lisitng all the apps) and take out the 301 to webapp1 We want to do these changes mainly for SEO reasons. I know that the advantages are not so clear between subdomain/subfolder but we think it could be the right way to go to push the root domain and profit more from juice passing to the different apps. The problem is that we had a bad experience when we first switched from our first wep app (rootdomain.com) to an subdomain (webapp1.rootdomain.com) to set them equal with the other apps. Our traffic dropped a lot and it took us 6 weeks to get back on the same level as before. Maybe it was the 301 not passing all juice or maybe it was the switch to the subdomain. We are not sure. So, I guess my question is do you think it is the right thing to do for web apps to go with subfolders to pass more juice from root to subfolders? Will it bring again huge drops in traffic once we make that change? Is it worth taking that risk or initial drop because it will pay off in the future? Thanks a lot in advance! Your answers would help me a lot.
Intermediate & Advanced SEO | | ummaterial0 -
Advice on further SEO
I am frustrated by a lack of progress for a major keyword I want to rank for. I have made several pages, optimized with Onpage and even a whole site but I can't seem to get my ratings up. I am hoping somone can take a look at my pages and efforts and offer me some advice... Keyword is "National Currency" One site is devoted to this keyword: NationalCurrencyValues This site is ranked 30th and is down 9... and this page on another site is devoted to the same keyword ranked 26th is: http://www.antiquebanknotes.com/National-Currency.aspx
Intermediate & Advanced SEO | | Banknotes0 -
Any advice for setting up a Job Board?
Hi- I've got a big client who is setting up an in-house job board, ie. as part of their own site. They are a diverse company that offers a wide variety of roles across the whole country. The software they have chosen to use is not in any way SEO-focused. Therefore I'll need to recommend some modifications to the sitemap created by the web design team, within the time and budget available to me. At this stage I am thinking along the lines of determining the major geographical areas and job sectors and creating summary (landing) pages such as /jobs/california/electrical-engineering which contain any currently available roles for that geo+sector. I've tried to find guidance on job board SEO optimization or even case studies but haven't found much at all. This seems useful though: http://www.jobboardmount.com/cm/features/seo_dashboard Does anyone have any tips or links to useful information on job board SEO? Thanks in advance! Jules
Intermediate & Advanced SEO | | Juller1