Screaming Frog - What are your "go to" tasks you use it for?
-
So, I have just purchased screaming frog because I have some specific tasks that need completing. However, looking at Screaming Frog generally, there is so much information I was wondering for those who use it, what are the top key tasks you use it for. I mean what are your "go to" things you like to check, that perhaps are not covered by the Moz Crawl reports.
Just looking for things I perhaps hadn't thought about, that this might be useful for.
-
Ha ha, I know! It's like giving the developers a little present all wrapped up with a bow...here's the problem, and here's where to fix it
-
Allie,
That's a great example use-case. After my audits, clients are like "you found thousands of internal redirects and 404s - where are they?"
I'm like - hold on I have a spreadsheet of that!
-
I love Screaming Frog! One use case I've used recently is using it to find internal 404 errors prior-to and immediately-after a major site redesign.
After running a crawl, go to Bulk Export > Response Code > Client error (4xx) Inlinks and download the report. It shows the offending URL and the URL referring to it, which makes it easier to update the bad link.
I also have this page bookmarked, and it's my go-to guide:
-
It's one of the best tools so I feel like I use it "for everything." But some includes:
-
Title / meta duplication & finding parameters on ecomm stores
-
Title length & meta desc length
-
Removing meta keywords fields
-
Finding errant pages (anything but 200, 301, 302, or 404 status code)
-
Large sitemap export (most tools do "up to 500 pages." Useless.)
-
Bulk export of external links (what ARE we linking to??)
-
Quickly opening a page in Wayback Machine or Google cache
-
Finding pages without Analytics, as was mentioned.
I use Screaming Frog for tons of other things. Finding the AJAX escaped frag URL, identifying pages with 2 titles, 2 canonicals, 2 H1 tags, etc. Even seeing www & non-www versions live, links to pages that shouldn't be linked and http vs https.
Very cool tool - useful for pretty much everything! haha
-
-
That's awesome. Thanks. Will take a look at all those things this week.
-
I use SF religiously for all the audit work I do. I run a sample crawl (using Googlebot as the crawler) to check for all the standard stuff and go further.
My standard evaluation with SF includes:
- Redirect / dead end internal linking
- Redirect / dead end "external" links that point to site assets housed on CDN servers.
- URL hierarchical structure
- Internal linking to both http and https that can reinforce duplicate content conflicts
- Page Title/H1 topical focus relevance and quality
- Confusion from improperly "nofollowing" important pages (meta robots)
- Conflicts between meta robots and canonical tags
- Slow page response times
- Bloated HTML or image file sizes
- Thin content issues (word count)
- Multiple instances of tags that should only have one instance (H1 headline tags, meta robots tags, canonical tags)
-
That crawl path report is pretty cool, and it led me to the redirect chain report, which I have a few issues to resolve with that with a few multiple redirects on some old links. Fantastic stuff.
-
I am a big fan of Screaming frog myself. Apart from the real basic stuff (checking H1, titles,...etc) it's also useful to check if all your pages contain your analytics tag and to check the size of the images on the site (these things Moz can't do).
It's also extremely useful when you're changing the url structure to check if all the redirects are properly implemented.
Sometimes you get loops in your site, especially if you use relative rather than absolute links on your site - Screaming Frog has an extremely helpful feature: just click on the url and select "crawl path report" - which generates an xls which shows the page where the problem originates
It's also very convenient that you can configure the spider to ignore robots.txt / nofollow / noindex when you are test a site in a pre-production environment. Idem for the possibility to use regex to filter some of the url's while crawling (especially useful for big sites if the they aren't using canonicals or noindex where they should use it)
rgds,
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What heading tag to use on sidebars and footers
Hello, I have some awareness of how to use H1, H2 and H3.
On-Page Optimization | | kowston
H1 only once per page as the main page heading.
H2's should be subheadings, H3's are sub-sub headings of the and so on.
This structure gives hierarchy and opportunities to use additional keywords in an order of priority. I can clearly understand how this would work in an article but what about other content on the page such as global/frequently repeated elements like sidebars and footers? I see sites - and in particular, I have examed SEO focused sites - that use H3, H4 and H5 in these instances seemingly giving themselves scope to use at least H2 tags as part of the page content and break out of the structure hierarchy when dealing with sidebars and footers. I suppose this could signal theses headings are sections of the page that are less relevant than the main article content but that is just an assumption. I don't know what is correct.0 -
"Google-selected canonical different to user-declared" - issues
Hi Moz! We are having issues on a number of our international sites where Google is choosing our page 2 of a category as the canonical over page 1. Example; https://www.yoursclothing.de/kleider-grosse-groessen (Image attached). We currently use infinite loading, however when javascript is disabled we have a text link to page 2 which is done via a query string of '?filter=true&view=X&categoryid=X&page=2' Page 2 is blocked via robots.txt and has a canonical pointing at page 1. Due to Google selecting page 2 as the canonical, the page is no longer ranking. For the main keyphrase a subcategory page is ranking poorly. LqDO0qr
On-Page Optimization | | RemarkableAgency1 -
Linking back to the homepage im trying to rank - Using exact match anchor? Linking from footer?
Hello, Our site is an basically advertisements / listings website. Structure is as following <last 200="" adds=""> (homepage trying to rank) < category 1 > < category 1 > < category 1 > < category 1 > < category 1 ></last> My question - each of the categories links back within the menu back to homepage. The link text currently is last 200 adds. Can i use exact match anchor? Or should i use just last 200 ads? The issue is that one of my categorys (category 1) has already the exact match anchor im trying to rank for. So i can not use the same to link back to homepage. Im worried that google does not see any exact keyword anchor texts back to homepage hence will rank my homepage with lower strenght for that keyword . Im also worried that the category 1 page might now compete with the main homepage for this word (even tho at the moment category itself does not rank for this keyword) Can i link from footer back to homepage with an alternative keyword then to give some "context" to google more? Would this be spamming?
On-Page Optimization | | advertisingcloud0 -
Is there any tool available to check which website using Lazy Load?
Hello Experts, Is there any tool available to check which website using Lazy Load? Thanks!
On-Page Optimization | | wright3350 -
I'm using Canonical URL but still receiving message - Appropriate Use of Rel Canonical
Hello, I checked my site and it looks like everything is setup correctly for canonical url but I keep getting the message that it's not. Am I doing something wrong? SORRY I FIGURED IT OUT! THANK YOU! HOW DO I DELETE THIS?
On-Page Optimization | | seohlp440 -
Which pages should use rel="canonical" links?
I have many pages showing up as multiple content. Most of the them belong to product pages for my store, login pages that show up everywhere on the site, etc. I know that I need to use the rel=canonical link in the header but after searching the forum I'm still unsure of what pages need it. Is it the pages that I don't want searched/crawled by Google or the other way around? Thanks! Crystal
On-Page Optimization | | COfashionista0 -
Should I use this Facebook comment content on my related blog post?
I have a blog post that ranks pretty high for the term "justin bieber tickets". We are running a ticket giveaway and have received tons of responses on Facebook and G+. The responses are often poorly written in they sense that they are from younger fans, but it is a bunch of related content that I thought could be a "good "addition of unique content to the post. Is this a good idea in general? Is it still a good idea if the comments are poorly written and contain lots of slang an exclamation points? Is it bad form to put people's Facebook comments live on the web, even though it is a public page. Here is the post Example of what this would look like in the post >http://cl.ly/1Q3N0t091V0w3m2r442G Source of comments >http://www.facebook.com/SeatGeek Another less aggressive option would be to curate some of my favorite comments... Thanks for any thoughts.
On-Page Optimization | | chadburgess0 -
Should a no-follow tag be used on a ssl or trust seal?
I'm just wondering if there would be some benefit if this was followed since the purpose of it is to add trust to your site.
On-Page Optimization | | BradBorst0