Screaming frog Advice
-
Hi
I am trying to crawl my site and it keeps crashing.
My sys admins keeps upgrading the virtual box it sits on and it now currently has 8GB of memory, but still crashes.
It gets to around 200k pages crawl and dies.
Any tips on how I can crawl my whole site, can u use screaming frog to crawl part of a site.
Thanks in advance for any tips.
Andy
-
Thanks, I tried all the tips on the screaming frog site, but I have just tried to 2 pages a second and lets hope that work.
-
Hi Andy. There are quite a few settings you can adjust to make the server load less while the crawl is running. These can be found with descriptions here: http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
For example, by not checking Images, CSS, SWF, and Javascript you'll be able to lessen load substantially, or if you'd like to crawl just a portion of the site you can set it to not check links outside of the start folder.
To have even more control over the crawl, you can use regular expressions to exclude certain pages, or sections that match a given pattern. The page above is fairly robust, so it should help you dial back the crawler to be friendlier to your server. Cheers!
-
Hey there mate,
Sorry to hear that you are having issues. You can actually ask Screaming Frog to use more RAM. If you haven't done that yet please give it a go.
You can find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
If you want to crawl part of your site it can surely do that. You can exclude pages or whole sections.
Find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO advice with having a blog on sub domain.
Righto, so: I've been working on our company website www.nursesfornurses.com.au which is built on .asp which is a real pain because the site is built so messy and on a very dated CMS which means I have to go back to the dev every time I want to make a change. We've made the decision to move the site over to Wordpress in stages. So, (and I hope logically), i've started by making them a proper blog with better architecture to start targeting industry related keywords. I had to put it on a sub domain as the current hosting does not support Wordpress http://news.nursesfornurses.com.au/Nursing-news/
Intermediate & Advanced SEO | | 9868john
The previous blog is here: http://www.nursesfornurses.com.au/blog Its not live yet, so I'm just looking for SEO advice or issues I might encounter by having the blog on a sub domain. In terms of user experience, I realise that there needs a clearer link back to the main website, I'm just trying to work out the best way to do it... Any advice / criticism is greatly welcomed. Thanks0 -
Site Wide Footer Links Exception, Any Advice ?
I was reading the following Q&A on site wide footer links, http://moz.com/community/q/site-wide-links-from-another-domain-could-these-cause-a-problem I feel my situation is slightly different however,we have lots of international sites linking to each other through these links like our sites for different counties and languages so our German, French and Spanish sites, http://www.cirrusresearch.co.uk/ Our main UK site has always ranked very well and has never really had a problem despite always having had these followed sitewide footer links, Because of this we regularly get high amount of visitors performing English language searches from different counties and i don't think it is a bad thing having more country/language specific sites of ours available in the footer for visitors that may prefer a more localized site, Our main website has to be at least 10+ years old at least, has a lot of strong links compared to our competitors, but the smaller German and Spanish sites are relatively smaller in size and most only 1-2 years old, my big fear is that these smaller sites would not be able to stand on there own without these footer links from our main site, After reading the community question caused me to question this ?, should i take a leap of faith and no-follow all of these site wide footer links connecting all of our sites ? we never really had a problem ranking so i don't really see the need but would this be the best thing to do ? Thank you, James
Intermediate & Advanced SEO | | Antony_Towle0 -
Advice on Link Removal Services
Hello everyone, Due to the Penguin update my site unfortunately took a bit of a hit. A little while ago I submitted all of our questionable/bad links to the disavow tool, however I still wante to go back and delete any and all problematic links that are still out there. Ive looked into many services, however I haven't been too impressed. Removeem - The email addresses they provided weren't always valid, and their email tool didn't always deploy correctly - a lot of cross referencing and was not saving me any time. Link Detox - Free trial was a bust. They show you 10 links on the free trial, however for me, 9 of the 10 were all the same. Couldn't get a good feel of their system. Rmoov - Their tool is one where you upload your own links, and they help manage everything, however they DONT allow you to email through their system, so Im not sure how this helps my process if I have to do everthing manaully anyway. A lot of sites I see are also a full service approach that charge you based on how many links they remove, and this can get quite costly. I have also contacted:
Intermediate & Advanced SEO | | Lukin
Link Delete - No reponse from multiple email requests
Linkquidator - No response
Infatex - No response My questions to all of you are: Is there any company out there that you recommend that provide a self service tool [online or desktop driven]? Is this even an avenue I should explore, or should I compile my own list [as 3rd party algorithms are not always accurate] and reach out to sites manually? Is disavowing good enough and Im just spinning my wheeles trying to now get them all removed? Thanks!0 -
Please Review and Advice!
My site is WordPress Solution Please expert give me some guideline how can i improve my Website's SEO
Intermediate & Advanced SEO | | shakiel0 -
Redirect advice
My website has two versions of the homepage: http://www.nile-cruises-4u.co.uk/http://www.nile-cruises-4u.co.uk/index.cfmI wondered if I could set up a 301 redirect in the .htaccess file so that only the http://www.nile-cruises-4u.co.uk page was returned as the homepage?Colin
Intermediate & Advanced SEO | | NileCruises0 -
.htaccess question/opinion/advice needed
Hello, I am trying to achieve 3 different things on my .htaccess I just want to make sure I am doing it the right or best way because I don't have much experience working on this kind of files. I am trying to: a) Redirect www.mysite.com/index.html to www.mysite.com so I don't get a duplicate content/tag error. b) Redirect mysite.com to www.mysite.com c) Get rid of the file extensions; www.mysite.com/stuff.html to www.mysite.com/stuff This is the code that I'm currently using and it seems to work fine, however I would like someone with experience to take a look so I can avoid internal server errors and other kinds of issues. I grabbed each piece of code from different posts and tutorials. Options +FollowSymlinks
Intermediate & Advanced SEO | | Eblan
RewriteEngine on Index Rewrite RewriteRule ^index.(htm|html|php) http://www.mysite.com/ [R=301,L] RewriteRule ^(.*)/index.(htm|html|php) http://www.mysite.com/$1/ [R=301,L] RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule ^(.*)$ $1.html Options +FollowSymlinks
RewriteEngine on
Rewritecond %{http_host} ^mysite.com [nc]
Rewriterule ^(.*)$ http://www.mysite.com/$1 [r=301,nc] Thanks a lot!0 -
Advice on Getting this site ranking?
Hi there I'm looking to optimise this site for SEO -> Gets about 3,000 visits per day but all from branded searches. Gets virtually no 'keyword searches' It's just a landing page at the moment. Would you recommend I integrate a blog with it, so we can start targeting more long tail keywords (free football game etc) Any thoughts/advice appreciated 🙂 Thanks Howard
Intermediate & Advanced SEO | | HowardK0 -
Need advice re: selecting an Ecommerce platform
Hi. I am responsible for choosing an ecommerce platform and overseeing the implementation of a large ecommerce site. The site will have tens of thousands of products and will be fairly complex. Eventually the site will integrate with the suppliers back end inventory/order management system, which is some sort of custom Windows/.NET system. (I'm not very technical if you haven't noticed....) Primarily I want a platform that is SEO friendly, and I have to be sure that the site is developed properly from an SEO and usability perspective. I thought I would go with an asp.net solution (aspdotnetstorefront, specifically) to facilitate the future integration, but I am questioning this choice after reading some of the comments I have found here at SEOMoz. So is asp.net really a bad choice SEO-wise? I almost considered Magento, but was having trouble finding a good solution provider to work with. I also worried about integration issues down the road. I would appreciate any advice or input anyone may have. Thank you!
Intermediate & Advanced SEO | | sbg7810