Screaming frog Advice
-
Hi
I am trying to crawl my site and it keeps crashing.
My sys admins keeps upgrading the virtual box it sits on and it now currently has 8GB of memory, but still crashes.
It gets to around 200k pages crawl and dies.
Any tips on how I can crawl my whole site, can u use screaming frog to crawl part of a site.
Thanks in advance for any tips.
Andy
-
Thanks, I tried all the tips on the screaming frog site, but I have just tried to 2 pages a second and lets hope that work.
-
Hi Andy. There are quite a few settings you can adjust to make the server load less while the crawl is running. These can be found with descriptions here: http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
For example, by not checking Images, CSS, SWF, and Javascript you'll be able to lessen load substantially, or if you'd like to crawl just a portion of the site you can set it to not check links outside of the start folder.
To have even more control over the crawl, you can use regular expressions to exclude certain pages, or sections that match a given pattern. The page above is fairly robust, so it should help you dial back the crawler to be friendlier to your server. Cheers!
-
Hey there mate,
Sorry to hear that you are having issues. You can actually ask Screaming Frog to use more RAM. If you haven't done that yet please give it a go.
You can find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
If you want to crawl part of your site it can surely do that. You can exclude pages or whole sections.
Find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sreaming Frog vs. Yoast - meta description clash
Hi all, I'm working on a site where when I crawl it with SF, SF doesn't pick up on the meta description (as in the source code it IS blank). However, the meta description has been set via the Yoast Wordpress plugin and it does exist in the source code and is shown in the SERPs. The code looks like this: <title>Dining Table and Chairs set</title> So my question is: will this be affecting SEO and how the website is ranking if all the actual are blank? Thank you
Intermediate & Advanced SEO | | Bee1591 -
Subdomain optimization - advices
Hi, I need some specific advices on which is the best way to optimize the subdomain of a main domain. Besides meta title, description, etc. Br.
Intermediate & Advanced SEO | | Tormar0 -
I have an authority site with 90K visits per month. Now I have to change from non www to www. Will incur in any SEO issues while doing that? Could you please advice me on the best steps to follow to do this? Thank you very much!
Because I want to increase site speed, Siteground (my hosting) suggested I use Cloudflare Plus which needs my site to have www in order to work. I'm also using a cloud hosting. Im a bit scared of doing this, and thus decided to come to the community. I used MOZ for over 6 months now and love the tool. Please help me make the best possible decisions and what steps to follow. It would be much appreciated. Thank you!
Intermediate & Advanced SEO | | Andrew_IT0 -
Strange 404s in Screaming Frog
I just ran a website (Drupal) through screaming frog and the only 404s I found related to web pages which were the same as URLs already used on the website plus the company phone number so... www.company.com/[their phone number] - www.company.com/services[their phone number] - any ideas what might be causing this problem?
Intermediate & Advanced SEO | | McTaggart0 -
Hit by Google updates; Some good advice needed
Hi, Here`s my domain http://www.kent-website-designer.co.uk/. Registered in 2007. We have took a big hit from the updates in the last 6 months and its really affecting revenue. I know when you look at the site you may well think WOW this is 2007 SEO and youre right it hasnt been updated in some time as of last year we ranked very highly and it gave us enough business to concentrate on. However up until last year many of my competitors were using same onpage and offpage strategies....and probably a few of you were too! So now the inquiries and income is drying up. However, I provided myself with an income from my efforts, rather than be unemployed, so I want to get it back on track. I visited the Google webmaster forums to query a couple of webmaster account queries and basically got beat up by the rude and arrogant google forum admins. Basically they said I was a spam site who shouldnt be in business. How very nice! 1. I have EMD - but domain age should mean something? 2. I lost a few links from https://www.getsafeonline.org/partners-and-supporters/ in the last year which hasnt helped when they reorganised their content. Same with other trusted sites we lost links. We are left with low quality links. 3. Some CMS sites have replicated our footer links on a large scale, which wasnt intentional but may look as link spam, plus they arent no followed as G prefers. 4. Google seems to have become intelligent? Apparently it can detect content which is negative in outdated seo advice. How, can it understand context and meaning so older seo advice isdetected as spam content? 5. No pages are de indexed just a rank drop to 30 - 60 positions. 6. Over optimised H1`s? 7. Is Pipe command in titles now negative? So its sink or swim time I guess. The siteand domain is honest but neglected and probably should re align the business with what we can offer. We got away with that SEO but clearly things have changed. However with no grey or black hat at least we arent overly worried by removing links. Also looking for an SEO company who we can outsource with a white label solution in order to offer SEO. I dont need beating up, short and to the point critiques please. Pros and Cons Many thanks.
Intermediate & Advanced SEO | | xtopher661 -
Need advice for indexing a multilingual website
We are in the process of creating a Spanish subdomain of our website. I want to know what needs to be done in regard to meta tags, sitemap.xml and robots.txt so that Google and Bing will index both website properly and not causing the web page on the English site to lost rank. Our English site is www.mydomain.com with the Spanish site being es.mydomain.com We are planning to put a button or link on both sites so that visitors can switch between both sites. The two sites are similar but not all pages are mirror images.
Intermediate & Advanced SEO | | Qualbe-Marketing-Group0 -
Advice on Getting this site ranking?
Hi there I'm looking to optimise this site for SEO -> Gets about 3,000 visits per day but all from branded searches. Gets virtually no 'keyword searches' It's just a landing page at the moment. Would you recommend I integrate a blog with it, so we can start targeting more long tail keywords (free football game etc) Any thoughts/advice appreciated 🙂 Thanks Howard
Intermediate & Advanced SEO | | HowardK0 -
I need some blogging advice please!
My name is Matthew and I am a new PRO member and founder of my own Internet marketing company in KS. So far I love the interaction and tools and functionality of seomoz. I am a true student of seo and love the subject. My dilemma is I know a blog is an important piece of any good seo campaign but I know very little about HOW to blog well......this is my new site and blog page. I only have a couple articles so far but many more planned. http://sawwebmarketing.com/seo-blog/ When I read an article that would be particuarly beneficial for my visitors can i post or share that on MY blog (giving the author the credit of course) without google thinking its duplicate content? is there anything specific I need to do with my blog for google to "see" the new, fresh content that is being added to the site? I have seen "tagged" items at the bottom of some blogs. Is this important? Some blogs will have a word or string of 2-3 words that are a link to a specific website. Does this help me or just them or just people reading the blog? **All I know is articles I write need to be relevant to my site and interesting and ORIGINAL and of benefit to my site visitors. ** Any advice that would help insure my blog articles get me all the juice they can would be GREATLY appreciated! Thank you in advance! Matthew ps - my site only went live a couple days ago so I am still working on a few onpage items but ANY feedback about the site itself would be spectacular! Have a GREAT weekend!
Intermediate & Advanced SEO | | Mrupp440