Screaming frog Advice
-
Hi
I am trying to crawl my site and it keeps crashing.
My sys admins keeps upgrading the virtual box it sits on and it now currently has 8GB of memory, but still crashes.
It gets to around 200k pages crawl and dies.
Any tips on how I can crawl my whole site, can u use screaming frog to crawl part of a site.
Thanks in advance for any tips.
Andy
-
Thanks, I tried all the tips on the screaming frog site, but I have just tried to 2 pages a second and lets hope that work.
-
Hi Andy. There are quite a few settings you can adjust to make the server load less while the crawl is running. These can be found with descriptions here: http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
For example, by not checking Images, CSS, SWF, and Javascript you'll be able to lessen load substantially, or if you'd like to crawl just a portion of the site you can set it to not check links outside of the start folder.
To have even more control over the crawl, you can use regular expressions to exclude certain pages, or sections that match a given pattern. The page above is fairly robust, so it should help you dial back the crawler to be friendlier to your server. Cheers!
-
Hey there mate,
Sorry to hear that you are having issues. You can actually ask Screaming Frog to use more RAM. If you haven't done that yet please give it a go.
You can find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
If you want to crawl part of your site it can surely do that. You can exclude pages or whole sections.
Find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Advice needed on canonical paginated pages
Hi there. I use Genesis and StudioPress themes. I recently noticed that the canonical link for blog pages points to the first page on all paginated pages, which I understand is an SEO no-no. I found some code here that adds a unique canonical link to each paginated page but for categories only. It works fine. I only have one category for my site. My question is: is there a downside (or even upside) to not having a blog page and placing a link to my category page in the navigation bar instead, using the category page as the blog page? It looks good and works. What do you think? I find it odd that this seems to be an issue across the Internet and the only solution that comes up relies on the Yoast plugin, which I don't want to use (don't want to use a plugin for SEO). Thanks in advance.
Intermediate & Advanced SEO | | Nobody16165422281340 -
Probably basic, but how to use image Title and Alt Text - and confusing advice from Moz!
I've been doing SEO on my business's site for years and have got good results. I've always used image Titles and Alt Text text. Our blog posts are image-intensive, often with 100-200 pictures (not surprising since we're photographers). For any given blog post, I've tended to have a uniform image Title for each image and then a more specialised Alt Text tag giving a description. A typical image on one of our blog posts would be like this: Image filename: wedding-photography-at-so-and-so-venue-001.jpg .... 002, 003 etc Image Title Attribute: Wedding Photography at So-And-So-Venue by Our-Company-Name - this would be the same for every image in the blog post. Alternative Text: Bride and groom exchanging vows during wedding ceremony at so-and-so-venue - this would be tailed for each image. So my question is - is this right? The Moz help page for image SEO is actually incorrect in one aspect: https://moz.com/ugc/10-tips-for-optimizing-your-images-for-search "Alt text (short for “alternative text”) is used to highlight the identity of an image when you hover over it with your mouse cursor. It also shows as text to all users when there are problems rendering the image." This is not the case. Hovering over the image in Firefox, Chrome, Edge and Opera ALL display the Image Title, NOT Alt Text. Thoughts?
Intermediate & Advanced SEO | | robandsarahgillespie0 -
Can you give me some advices to rank this domain?
Hi Moz community, I've a coleague that's working to rank this site: www.devsar.com. The selected keywords are:
Intermediate & Advanced SEO | | Gaston Riera
Mobile development
Web development
Django Development
Python Development I've checked the site: It's fast and clean. Has a good PA and DA. It's responsive and good lookking. Meta description , title, hreflang.. everything in order. Link profile a little rare (checked with ahref.com), it's because someone made a mistake redirecting some expired domain Can you help me to help my mate out?
Thanks
GR.0 -
New g(TLD) advice needed
Hey all, I'm a bit confused by conflicting advice, need some direct input. We're quite experienced in SEO but that doesn't mean we can't get better 🙂 I manage a very old, well established, very generic TLD portal that ranks very highly in MANY keywords. (If you know our domain, I'd appreciate not naming it here) (145 1-3 ranks, 342 1-20 ranks) but there are also many topics we want to improve upon. Lets say, for example, I own gold.com, but I've failed to rank for 'gold events' and I acquired gold.events. What is the thought as to using some of the g(TLD)s versus the original .com? In the example events.gold.com or gold.events or gold.com/events/? I really can't find a consensus on which would bemost effective for SEO purposes. In a more general aspect of the same question, we own MANY "gold.newg(TLD)" domains and are conflicted as to best use of all of them. All advice greatly appreciated. Nat
Intermediate & Advanced SEO | | WorldWideWebLabs0 -
Wordpresss Bug? Duplicate pages yet again, Adviced Needed
I have work up this morning with a large number of missing meta description, when I looked at th pages (knowing I have to missing metas) I have duplicates of my pages that look like this, These pages normally look like www.finalduties.co.uk/ NAME OF POST now i have urls with numbers in, seems as though all my blog posts have duplicated.. Now could this be down to a wordpress bug? I am so fed up, I had all my errors all cleared up only to wake and have pages like this, which is going to cause me a problem because my pages are now duplicating.. please help, advice needed from someone that knows wordpress? | Father and son battle over 13th century inheritance http://www.finalduties.co.uk/?p=1006 1 0 Court of Appeal rules against RSCPA http://www.finalduties.co.uk/?p=1007 1 0 Life insurance policy payout can skip probate process http://www.finalduties.co.uk/?p=1008 1 0 Government clamping down on inheritance tax avoidance http://www.finalduties.co.uk/?p=1010 |
Intermediate & Advanced SEO | | Chris__Chris0 -
Please help me with your advice
Hi all, Couple years ago I started to build my business based on EMD domain. The intention was to create the source with the rich unique content. After a year of hard work the site achieved top 10 in Google and started to generate good amount of leads. Then Google announced the EMD Update and site lost the 90% of traffic (after Pandas updates our SERP was steady ) “ a new filter that tries to ensure that low-quality sites don’t rise high in Google’s search results simply because they have search terms in their domain names. ” But I don’t consider my site low-quality site, every page, every post is 100% unique and has been created only to share the knowledge with others… The site has EXCELLENT content from industry point of view.... Since the “ EMD Update “ I read hundreds , hundreds of different articles and opinions related to EMD update and finally I am confused and lost. What should I do… • Kill the site and start new one
Intermediate & Advanced SEO | | Webdeal
• Get more links, but what type of links and how I should get them
• Keep hoping and pray....
• Or do something else Please help me with your advice0 -
Need some urgent Panda advice. Open discussion about recovering from the Panda algorithm.
I have a site that has been affected by Panda, and I think I have finally found the problem. When I created this site in the year 2006, I bought content without checking it. Recently, when I went through the site I found out that this content had many duplicates around the web. Not 100% exact, but close to. The first thing I did is ask my best writer to rewrite these topics, as they are a must on my site. This is a very experienced writer, and she will make the categories and subpages outstanding. Second thing I did was putting a NOINDEX, FOLLOW robots meta in place for the pages I determined being bad. They haven't been de-indexed yet. Another thing I recently did is separate other languages and move these over to other domains (with 301's redirecting the old locations to the new.) This means that the site now has a /en/ directory in the URL which is no longer used. With this in mind I was thinking to relocate the NEW content, and 301 the old (to preserve the juice for a while.) For example: http://www.mysite.com/en/this-is-a-pandalized-page/ 301 to http://www.mysite.com/this-is-the-rewritten-page/ The benefits of doing this are: decreasing the amounts of directories in the URL getting rid of pages that are possibly causing trouble getting fresh pages added to the site Now, the advice I am looking for is basically this: Do you agree with the above? Or don't you agree? If you don't, please be so kind to include a reason with your answer. If you do, and have any additional information, or would like to discuss, please go ahead 🙂 Thanks, Giorgio PS: Is it proven that Panda is now a running update? Or is it still periodically executed?
Intermediate & Advanced SEO | | VisualSense1 -
We've just bought a new domain - need advice on the exact procedure to follow...
Hi guys, We've just bought a 3 letter .co.uk domain to replace our current 20 character old domain. Our existing domain is PR5 with quite a few links (that we can modify no problem) We're currently .301 redirecting the new domain to the old domain. I was looking at the procedure in one of the guides but as it's slightly different - is this the correct procedure? 1. prep the duplicate site on new domain and prep the individual htaccess .301 redirects 2. Add new the domain to google webmaster tools bing Webmaster centre 2. On the switchover date - modify all possible incoming links from external sites 3. On the switchover date - apply the .301 redirects and make the new site live 4. On the switchover date - apply the new sitemaps to google & bing 5. on the switchover date - fill out the change of address form in webmaster tools 6. Do the happy dance? many thanks in advance, Tony.
Intermediate & Advanced SEO | | posh_tiger0