Screaming Frog - What are your "go to" tasks you use it for?
-
So, I have just purchased screaming frog because I have some specific tasks that need completing. However, looking at Screaming Frog generally, there is so much information I was wondering for those who use it, what are the top key tasks you use it for. I mean what are your "go to" things you like to check, that perhaps are not covered by the Moz Crawl reports.
Just looking for things I perhaps hadn't thought about, that this might be useful for.
-
Ha ha, I know! It's like giving the developers a little present all wrapped up with a bow...here's the problem, and here's where to fix it
-
Allie,
That's a great example use-case. After my audits, clients are like "you found thousands of internal redirects and 404s - where are they?"
I'm like - hold on I have a spreadsheet of that!
-
I love Screaming Frog! One use case I've used recently is using it to find internal 404 errors prior-to and immediately-after a major site redesign.
After running a crawl, go to Bulk Export > Response Code > Client error (4xx) Inlinks and download the report. It shows the offending URL and the URL referring to it, which makes it easier to update the bad link.
I also have this page bookmarked, and it's my go-to guide:
-
It's one of the best tools so I feel like I use it "for everything." But some includes:
-
Title / meta duplication & finding parameters on ecomm stores
-
Title length & meta desc length
-
Removing meta keywords fields
-
Finding errant pages (anything but 200, 301, 302, or 404 status code)
-
Large sitemap export (most tools do "up to 500 pages." Useless.)
-
Bulk export of external links (what ARE we linking to??)
-
Quickly opening a page in Wayback Machine or Google cache
-
Finding pages without Analytics, as was mentioned.
I use Screaming Frog for tons of other things. Finding the AJAX escaped frag URL, identifying pages with 2 titles, 2 canonicals, 2 H1 tags, etc. Even seeing www & non-www versions live, links to pages that shouldn't be linked and http vs https.
Very cool tool - useful for pretty much everything! haha
-
-
That's awesome. Thanks. Will take a look at all those things this week.
-
I use SF religiously for all the audit work I do. I run a sample crawl (using Googlebot as the crawler) to check for all the standard stuff and go further.
My standard evaluation with SF includes:
- Redirect / dead end internal linking
- Redirect / dead end "external" links that point to site assets housed on CDN servers.
- URL hierarchical structure
- Internal linking to both http and https that can reinforce duplicate content conflicts
- Page Title/H1 topical focus relevance and quality
- Confusion from improperly "nofollowing" important pages (meta robots)
- Conflicts between meta robots and canonical tags
- Slow page response times
- Bloated HTML or image file sizes
- Thin content issues (word count)
- Multiple instances of tags that should only have one instance (H1 headline tags, meta robots tags, canonical tags)
-
That crawl path report is pretty cool, and it led me to the redirect chain report, which I have a few issues to resolve with that with a few multiple redirects on some old links. Fantastic stuff.
-
I am a big fan of Screaming frog myself. Apart from the real basic stuff (checking H1, titles,...etc) it's also useful to check if all your pages contain your analytics tag and to check the size of the images on the site (these things Moz can't do).
It's also extremely useful when you're changing the url structure to check if all the redirects are properly implemented.
Sometimes you get loops in your site, especially if you use relative rather than absolute links on your site - Screaming Frog has an extremely helpful feature: just click on the url and select "crawl path report" - which generates an xls which shows the page where the problem originates
It's also very convenient that you can configure the spider to ignore robots.txt / nofollow / noindex when you are test a site in a pre-production environment. Idem for the possibility to use regex to filter some of the url's while crawling (especially useful for big sites if the they aren't using canonicals or noindex where they should use it)
rgds,
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should you aim for Google to use your meta tags?
When updating meta titles and descriptions, I'm taking note of whether Google is displaying the set tag or changing it to copy from the page. Does this affect the ranking position if Google is having to change the tag? How much should I worry if Google is choosing to change every other page? Thanks!
On-Page Optimization | | Omar_aw0 -
Thoughts on adding "near me" to title tag for local SEO?
I want to lean out my title tags and will most likely be doing an A/B test. They currently have the "Near Me" modifier in there, which I believe Google can distinguish local SEO without it. Thoughts?
On-Page Optimization | | imjonny1230 -
Should I leave away the http when using the canonical tag
My site redirects http://www.anydomainblabla.com to www.anydomainblabla.com So I guess I have to do it like this Is that correct? Big thanks for your time.
On-Page Optimization | | MichaelJanik0 -
How to "on page" seo a small local service business - particularly headers
First off, let me apologize if this question is posted elsewhere, worded differently. I've looked around quite a bit and have been unable to find the answer. Basically, we are a small web design firm just getting our feet with with SEO. Most of our clients, especially initially, will be quite small, local, service businesses. For example: and electrician, a pet sitter, a retail printing and map store, a surgeon etc. Almost all of their sites will follow a basic "business card on the web" format... Home Page - About Us - Testimonials - Rates - FAQ - Contact Us - Etc So, from what I've read about on-page optimization, making sure my keywords are in the title, header, body, and meta description is one of the easiest and quickest things we can do for our clients. This is a straightforward concept for me when applied to the homepage. For example, take the local pet sitting business. Her keywords are: Pet sitting, Dog walking, and the city we live in, Anytown USA. So, I've used those keywords in all the appropriate places on the home page: title: Dog Walking and Pet Sitting Service in Anytown USA header: Dog Walking and Pet Sitting Service in Anytown USA first sentence of body: We are a professional Dog Walking and Pet Sitting Service in Anytown USA meta description: We are a professional Dog Walking and Pet Sitting Service in Anytown USA. At Business Name your furry friends become a part of our family. So, my question is: Do I also optimize the "about us" page? I've changed the title of all the pages to follow this format: Dog Walking and Pet Sitting in Anytown USA - Home Dog Walking and Pet Sitting in Anytown USA - About Us Dog Walking and Pet Sitting in Anytown USA - Rates Dog Walking and Pet Sitting in Anytown USA - FAQ Dog Walking and Pet Sitting in Anytown USA - Etc Easy enough so far. Also pretty easy for the meta description, and the body. However, how would I add keywords to the header without making it look ridiculous? We use wordpress with the genesis framework, and child themes from studiopress. The header is always prominently visible at the top of the page. Most people would expect to see the header be the same as the link they clicked on the nav bar: for example, on the "about us" page, people expect the header to be: "about us" Not: "dog walking and pet sitting in Anytown USA - About Us" Do I just not worry about the headers on the other pages? For that matter, I'd really like people to "land" on the home page, not any of the other pages, so should I not optimize them at all? Does optimizing the rest of the pages help the home page to show up higher in the SERPS? If I do end up optimizing the rest of the pages, should I use slightly different spellings of the keywords: like Dog walker instead of dog walking? Or pet sitter instead of pet sitting? I've repeatedly seen people talk about not using the same keywords on more than one page... but for most of these businesses there are really fairly few keywords. There just isn't that many different ways that someone is going to search for an electrician, or a plumber, or a pet sitter. By the second or third page that I optimize on one site, I imagine I'll start running out of different variations of the keywords. I recognize that a lot of what we'll do that will be most helpful to local clients has nothing to do with on page optimization (setting up google places, google+, yahoo + bing local, etc). I'd just like to make sure that I'm doing the on page stuff as perfectly as possible. Thanks for your time and responses! -Matt p.s. while I'm at it, let me ask another question about domain names as well. Right now the pet sitting client mentioned above is using: www.petcare_Anytown_.com After operating her business for the last year she realized she is much more interested in dog walking than pet sitting. We are in the processes of redesigning the site, and when finished, are considering moving it to: www.dogwalking_Anytown_.com My assumption is that as long as we use permanent redirects from the old site to the new one, we shouldn't lose much SEO value. Is this thinking correct? On a related note though: another article I read mentioned that using a brand name in the domain may actually be more useful than the keyword rich domains above. However, www._businessname._com happens to already be taken by a pet sitting business at the other end of the country. We could however use: www.businessnameAnytown.com Which one do you think would work better? The keyword/location domain, or the businessname/location domain? Thanks!
On-Page Optimization | | Webformix0 -
Should I use rel=canonical in this case
Hi SEO pros, I am working on a website competing on the keyword "USA maps" and would appreciate your advice and comments on the issue below. The site has one major page for USA maps and like 5-6 smaller pages under different categories, e.g. US travel maps (under Travel Maps category), US travel guides (under Travel Guides), US atlases (under Atlases), etc. The smaller pages do not rank in search results and are not optimized well for any keyword. Here are my questions: #1. Do you think that if I add rel=canonical to the main USA maps page from all small pages that will help get higher ranking of the main page? #2 Or should I better try to optimize these small pages for the keywords they target (e.g. "US travel maps") and try to send link juice to the main page from text link within the content? Thank you,
On-Page Optimization | | ParisChildress0 -
Not using H1's with keywords to simulate natural non SEO'd content?
There has been a lot of talk lately about making a website seem like it is not SEO'd to avoid over optimization penalties with the recent Google Algorithmic updates. Has anyone come across the practice of not using Headings (H1's, H2's etc..) properly to simulate that the current webpage isn't over optimized? I've come across a site that used to use multiple keywords within their headings & now they are using none. In fact they are marking their company name & logo as an H1 and non keyworded H2's such as our work or Contact. Is anyone holding back on their old SEO tactics to not seem over optimized to Google? Thanks!
On-Page Optimization | | DCochrane0 -
Which is better to use: apostrophe version or non-apostrophe version?
When targeting a keyword phrase that contains an apostrophe (in the correct spelling), is it better to use the non-apostrophe version if it has higher search volume.
On-Page Optimization | | ProjectLabs0 -
The Better Title to Use?
Hello Mozers- I am targeting the keywords "liposuction scottsdale", "liposuction phoenix", "liposuction mesa", "liposuction arizona". Out of the following two Titles below, one would you consider the better one? And Why? I am leaning towards the second example. If you have any more ideas that would be great. 1). Tummy Tuck Scottsdale, Phoenix, Mesa, Arizona 2).Tummy Tuck Scottsdale | Tummy Tuck Phoenix, Mesa, Arizona. Thanks!!
On-Page Optimization | | Red_Spot_Interactive0