Screaming frog Advice
-
Hi
I am trying to crawl my site and it keeps crashing.
My sys admins keeps upgrading the virtual box it sits on and it now currently has 8GB of memory, but still crashes.
It gets to around 200k pages crawl and dies.
Any tips on how I can crawl my whole site, can u use screaming frog to crawl part of a site.
Thanks in advance for any tips.
Andy
-
Thanks, I tried all the tips on the screaming frog site, but I have just tried to 2 pages a second and lets hope that work.
-
Hi Andy. There are quite a few settings you can adjust to make the server load less while the crawl is running. These can be found with descriptions here: http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
For example, by not checking Images, CSS, SWF, and Javascript you'll be able to lessen load substantially, or if you'd like to crawl just a portion of the site you can set it to not check links outside of the start folder.
To have even more control over the crawl, you can use regular expressions to exclude certain pages, or sections that match a given pattern. The page above is fairly robust, so it should help you dial back the crawler to be friendlier to your server. Cheers!
-
Hey there mate,
Sorry to hear that you are having issues. You can actually ask Screaming Frog to use more RAM. If you haven't done that yet please give it a go.
You can find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
If you want to crawl part of your site it can surely do that. You can exclude pages or whole sections.
Find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New Domain, No 301 Possible - Any Advice
A client of mine lost their domain when an ex business partner sold it out from under them. They've filed with WIPO, but in the meantime we're trying to figure out how to help them out. They had two really excellent links - one from the NY Times and one from a .edu website. I'm going to reach out to the authors of those articles (the articles are pretty old, so I doubt they'll change the links), but does anyone have any advice on how to let search engines know the new domain replaces the old without having the ability to do redirects? The content on the site is exactly the same - we were able to get the files over, happily. I've re-submitted the site for indexing, changed the domain links in Moz Local, changed in Analytics, and on all their social sites. Is there anything I'm not thinking of that can be done to let Google know that this new domain replaces the old? Thank you!
Intermediate & Advanced SEO | | newwhy0 -
Please help need some advice?
Can any of you guys please help me I have alerts on links coming in and it looks like recently someone did this, it looks maliciously done as it is only our domain mentioned and most are brand new posts? http://testosteroneclinicindenve53950.shotblogs.com/testosterone-clinic-in-denver-fundamentals-explained-6102386 http://claytondmnnp.ampedpages.com/Details-Fiction-and-testosterone-clinic-in-denver-16897309 http://vinylvehiclecarwrap38041.alltdesign.com/a-review-of-vinyl-vehicle-car-wrap-9574042 http://devinxccct.educationalimpactblog.com/1784474/little-known-facts-about-vinyl-vehicle-car-wrap http://keeganbsftf.ka-blogs.com/7488539/how-vinyl-vehicle-car-wrap-can-save-you-time-stress-and-money http://andybxoes.thezenweb.com/vinyl-vehicle-car-wrap-Fundamentals-Explained-17581028 http://kylerhfdzu.blogkoo.com/not-known-details-about-vinyl-vehicle-car-wrap-9029141 http://troyytkyn.timeblog.net/7695911/the-greatest-guide-to-vinyl-vehicle-car-wrap http://waylontyzab.pointblog.net/testosterone-clinic-in-denver-Secrets-16335972 http://testosteroneclinicindenve30516.onesmablog.com/Top-testosterone-clinic-in-denver-Secrets-17252737 http://emiliogkmop.blogofoto.com/7667522/top-guidelines-of-testosterone-clinic-in-denver http://caidenaczxt.blogs-service.com/7514172/testosterone-clinic-in-denver-fundamentals-explained http://daltonpyfms.mybjjblog.com/5-simple-statements-about-testosterone-clinic-in-denver-explained-6517932 Should I try to disavow these and submit to google or will google know our site which has been up for 5 years is not doing this? Should I do any of these https://tehnoblog.org/google-webmaster-tools-my-website-got-bombed-with-backlinks-what-to-do/
Intermediate & Advanced SEO | | BobAnderson0 -
302 > 302 > 301 Redirect Chain Issue & Advice
Hi everyone, I recently relaunched our website and everything went well. However, while checking site health, I found a new redirect chain issue (302 > 302 > 301 > 200) when the user requests the HTTP and non-www version of our URL. Here's what's happening: • 302 #1 -- http://domain.com/example/ 302 redirects to http://domain.com/PnVKV/example/ (the 5 characters in the appended "subfolder" are dynamic and change each time)
Intermediate & Advanced SEO | | Andrew_In_Search_of_Answers
• 302 #2 -- http://domain.com/PnVKV/example/ 302 redirects BACK to http://domain.com/example/
• 301 #1 -- http://domain.com/example/ 301 redirects to https://www.domain.com/example/ (as it should have done originally)
• 200 -- https://www.domain.com/example/ resolves properly We're hosted on AWS, and one of my cloud architects investigated and reported GoDaddy was causing the two 302s. That's backed up online by posts like https://stackoverflow.com/questions/46307518/random-5-alpha-character-path-appended-to-requests and https://www.godaddy.com/community/Managing-Domains/My-domain-name-not-resolving-correctly-6-random-characters-are/td-p/60782. I reached out to GoDaddy today, expecting them to say it wasn't a problem on their end, but they actually confirmed this was a known bug (as of September 2017) but there is no timeline for a fix. I asked the first rep I spoke with on the phone to send a summary, and here's what he provided in his own words: From the information gathered on my end and I was able to get from our advanced tech support team, the redirect issue is in a bug report and many examples have been logged with the help of customers, but no log will be made in this case due to the destination URL being met. Most issues being logged are site not resolving properly or resolving errors. I realize the redirect can cause SEO issues with the additional redirects occurring. Also no ETA has been logged for the issue being reported. I do feel for you since I now understand more the SEO issues it can cause. I myself will keep an eye out for the bug report and see if any progress is being made any info outside of this I will email you directly. Thanks. Issue being Experienced: Domains that are set to Go Daddy forwarding IPs may sometimes resolve to a url that has extra characters appended to the end of them. Example: domain1.com forwards to http://www.domain2.com/TLYEZ. However it should just forward to http://www.domain2.com. I think this answers what some Moz users may have been experiencing sporadically, especially this previous thread: https://moz.com/community/q/forwarded-vanity-domains-suddenly-resolving-to-404-with-appended-url-s-ending-in-random-5-characters. My question: Given everything stated above and what we know about the impact of redirect chains on SEO, how severe should I rate this? I told my Director that I would recommend we move away from GoDaddy (something I don't want to do, but feel we _**have **_to do), but she viewed it as just another technical SEO issue and one that didn't necessarily need to be prioritized over others related to the relaunch. How would you respond in my shoes? On a scale of 1 to 10 (10 being the biggest), how big of a technical SEO is this? Would you make it a priority? At the very least, I thought the Moz community would benefit from the GoDaddy confirmation of this issue and knowing about the lack of an ETA on a fix. Thanks!0 -
Need advice on overcoming a Google penalty
Here is the situation. Our website for our primary product (www.thetablift.com) has received a penalty by Google. Not long ago we had excellent rankings; (1st page) for some of our primary keywords, like "tablet stand". Now we are not in the index at all. Here is what happened (or at least what seems to have happened in my non-SEO opinion). Around October 2016, we had the "bright" idea to try and emulate a campaign that Eat 24 did, utilizing inexpensive traffic from advertisements on porn websites. The idea was a play on a joke we often hear about our product being perfect for certain activities where one needs to free one's hands while watching a screen. Of course this is not how we market our product (it is a best selling mainstream product), but we wanted to see if we could emulate the success of another mainstream brand that utilized this kind of non-mainstream advertising. The immediate result was a whole lot of traffic, but obviously the wrong kind, as it did not convert. So we pulled the plug after about 3 days. Flash forward several months later and we not only lost our great SEO rankings, but we were removed from Google's index entirely. I assume the reason for this is that somehow the website got dinged for being somehow related to porn. But of course it has nothing to do with that. So the question is: how do we go about getting un-penalized by Google? We had build up some solid SEO over the previous couple of years, and I'd like to get back to where we were, if possible. Oh, and this may or may not be relevant, but we also switched from www.tablift.com to www.thetablift.com a few months before we did this campaign. However, we used permanent redirects and did a textbook changeover, so I don't think that had any bearing. But I can't be sure. What are the steps to reverse this damage, if any? Thanks!
Intermediate & Advanced SEO | | csblev0 -
Advice for structuring hotel website
Hey guys, I am currently setting up a hotel booking website and I'm not so sure how to structure it. I have landing pages for: 1. Cities
Intermediate & Advanced SEO | | baresound
2. Sights
3. States The main keywords are mainly "Hotels in Cityname" or "Hotels near Sightname". What would be the best SEO friendly way of structuring the url? https://hotels-example.com/hotels/cities/cityname
https://hotels-example.com/hotels/sights/sightname
https://hotels-example.com/hotels/states/statename or https://hotels-example.com/hotels/cityname
https://hotels-example.com/hotels/sightname
https://hotels-example.com/hotels/statename or https://hotels-example.com/hotels-in-cityname
https://hotels-example.com/hotels-in-sightname
https://hotels-example.com/hotels-in-statename Or are there better ways of structuring it or am I just overthinking it? I would greatly appreciate any advice and suggestions 🙂 Best, Max0 -
Advice on URL structure for competing against EMDs of a hot keyword
Here is the question, illustrated with an example: A law client focuses on personal injury. Their domain is nondescript. The question comes into the URL structure for an article section of the site (I think I know what most people here will say, but want to raise this anyway). This section will have several hundred 'personal injury' articles at launch, with 100+ added each month by writers. Most articles do not mention 'personal injury' in the titles or in the content, but focus on the many areas in which people can hurt themselves :-). Spreading a single keyword emphasis across many pages/posts is considered poor form by many, but the counter-argument is that hundreds of articles, all with 'personal injury' in the URL, could increase the overall authority of the site for that term (and may compete more strongly with EMD competitors). For instance, let's say Competitor A has this article: www.acmepersonalinjury.com/articles/tips-if-in-car-accident And we had the following options: Option A: www.baddomain.com/articles/tips-if-in-car-accident Option B: www.baddomain.com/personal-injury-articles/tips-if-in-car-accident Of course, for the term "car accident", Option A seems on equal footing with the ACME competitor. But, what about the overall performance of the "personal injury" keyword (a HOT keyword in this space)? Would ACME always have an advantage (however slight) due to its domain? Would Option B help in this regard? The downside of course is that this pushes "car accident" further down in the URL string, making all articles perhaps less competitive on their individual keywords.
Intermediate & Advanced SEO | | warpsmith0 -
Wordpresss Bug? Duplicate pages yet again, Adviced Needed
I have work up this morning with a large number of missing meta description, when I looked at th pages (knowing I have to missing metas) I have duplicates of my pages that look like this, These pages normally look like www.finalduties.co.uk/ NAME OF POST now i have urls with numbers in, seems as though all my blog posts have duplicated.. Now could this be down to a wordpress bug? I am so fed up, I had all my errors all cleared up only to wake and have pages like this, which is going to cause me a problem because my pages are now duplicating.. please help, advice needed from someone that knows wordpress? | Father and son battle over 13th century inheritance http://www.finalduties.co.uk/?p=1006 1 0 Court of Appeal rules against RSCPA http://www.finalduties.co.uk/?p=1007 1 0 Life insurance policy payout can skip probate process http://www.finalduties.co.uk/?p=1008 1 0 Government clamping down on inheritance tax avoidance http://www.finalduties.co.uk/?p=1010 |
Intermediate & Advanced SEO | | Chris__Chris0 -
I need some blogging advice please!
My name is Matthew and I am a new PRO member and founder of my own Internet marketing company in KS. So far I love the interaction and tools and functionality of seomoz. I am a true student of seo and love the subject. My dilemma is I know a blog is an important piece of any good seo campaign but I know very little about HOW to blog well......this is my new site and blog page. I only have a couple articles so far but many more planned. http://sawwebmarketing.com/seo-blog/ When I read an article that would be particuarly beneficial for my visitors can i post or share that on MY blog (giving the author the credit of course) without google thinking its duplicate content? is there anything specific I need to do with my blog for google to "see" the new, fresh content that is being added to the site? I have seen "tagged" items at the bottom of some blogs. Is this important? Some blogs will have a word or string of 2-3 words that are a link to a specific website. Does this help me or just them or just people reading the blog? **All I know is articles I write need to be relevant to my site and interesting and ORIGINAL and of benefit to my site visitors. ** Any advice that would help insure my blog articles get me all the juice they can would be GREATLY appreciated! Thank you in advance! Matthew ps - my site only went live a couple days ago so I am still working on a few onpage items but ANY feedback about the site itself would be spectacular! Have a GREAT weekend!
Intermediate & Advanced SEO | | Mrupp440