Automated checking for broken links within content pieces
-
Hi, I am wondering if anyone can send me in the right direction on a system suggestion.
We have currently grown out amount of content pieces on our website and our manual checking if the links in the content pieces are still 200 status is becoming extremely time consuming. Does anyone have a recommendation of a system that will crawl your pages and check both the internal and external links within the content for a status code (404,200,etc)? Preferably something server side so it can just run on a schedule but really anything would be fine.
I have tried things like Screaming frog, etc and it just doesn't seem to be the right tool.
-
Try ScreamingFrog again Jonathan, it works great for these kind of things and should also be able to solve your use case.
-
Jonathan, I'm not sure why you're saying that Screaming Frog isn't the right tool--we use it with great success to check the internal links on the site. There are other tools that you can use, such as Integrity (on a Mac), or Xenu, which is an older link checker but still works.
-
Have you tried http://www.link-assistant.com/website-auditor/ as it checks for broken links and can be scheduled to run automatically. You can sit it on your own server or something like AWS. We ran it on a free instance of AWS for quite a while before upgrading and never had issues. We upgraded as we run quite a bit of software on there - still isn't huge costs involved.
Hope this helps!
Matt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help With Duplicated Content
Hi Moz Community, I am having some issue's with duplicated content, i recently removed the .html from all of our links and moz has reported it as being duplicated. I have been reading up about Canonicalization and would to verify some details, when using the canonical tag would it be placed in the /mywebpage.html or /mywebpage file? I am having a hard time to sort this out so any help from you SEO experts would be great 🙂 I have also updated my htaccess file with the following Thanks in advance
On-Page Optimization | | finelinewebsolutions0 -
Cornerstone Page And Outbound Links
I have a cornerstone page and 10 related articles that all have links to the cornerstone page. My question is, should the cornerstone page link back to those 10 articles as well or will it lose juice by doing so? Thanks in advance 😉
On-Page Optimization | | Humanovation0 -
What Should I Do With Low Quality Content?
As my site has definitely got hit by Panda, I am in the process of cleaning my website of low quality content. Needless to say, shitty articles are completed being removed but I think lots of this content is now of low quality because it is obsolete and dated. So what should I do with this content? Should I rewrite those articles as completely new posts and link from the old posts to the new ones? Or should I delete the old posts and do a 301 redirect to the new post? Or should I rewrite the content of these articles in place so I can keep the old URL and backlinks? One thing is that I've got a lot more followers than I used to so publishing a new post gets a lot more views, like and shares and whatnot from social networks.
On-Page Optimization | | sbrault741 -
Too many outbound links on a page?
We have a "Clients" page on our site with approximately 125 of our clients listed. We have a link to each client's website, so that's 125 links. I am rethinking this approach. Is there any value to having these outbound links? The SEOmoz PRO analysis tells me I have too many links on this page. I have read that more than 100 links on a page is too many, but that seemed to be referring to internal links. Any thoughts? Thanks!
On-Page Optimization | | nyc-seo0 -
Drop in Internal Links to Root
This morning I noticed in Google Webmaster Tools that my internal links to my root domain (Home Page), dropped from 428,000 to 58,000. It appears that this could be my header or footer links back to the home page that Google is not showing any more. My programmers claim they have not made any changes over the last 30 days. My rankings and traffic are normal. Any cause for concern?
On-Page Optimization | | tdawson090 -
Do we have too many links in our footer?
Hi guys, we have 41 links on our holiday(vacation) rental website, this seems too many when looking at best practice. 24 of these are links to community pages while 8 link to activities pages. The community and activity pages are also accessible from links on the top menu so they are not strictly necessary but do get 10% of site clickthroughs according to Google in-page analytics. I therefore do not want to remove the links if there is no good evidence that google will penalize us for this. What do you think would be best for our site? Thanks, John Tulley. footer.jpg
On-Page Optimization | | JohnTulley0 -
Duplicate content? Not sure.
Good news! I have my first real SEO gig and now I have to be able to actually deliver. I'm up for it but I want to be sure I'm seeing what I think I am before suggesting any changes. I'm working my way throught Danny Dover's excellent book SEO Secrets and learning tons! To see if there is duplicate content on the site, I've taken a sentence from one of the pages on the site and searched for it: i.e., site:storybooksforhealing.com "Some of the most quiet moments are often the most difficult after a loss. Mornings, late nights, time alone." The SERPs show 7 pages that have this text on it. It seems like this is duplicate content, right? This is a Wordpress website so what's happening is the actual page is here: www.storybooksforhealing.com/publish-cup-of-joy/ but there are several archive pages that show excerpts of this text, too. If this is duplicate content (first question) then how would I go about remedying it? Should I set the canonical reference to /publish-cup-of-joy page? Thank you for being patient with my NOOB questions.
On-Page Optimization | | ChristiMc0 -
Nofollow on these internal links?
On an x-cart ecommerce website we have, seomoz has picked up a lot of duplicate content, based on URLs that are different, but are essentially the same page. These come from Fitlers, that allow a page to show only certain colours and styles, reordering page by price etc, and also the page 2, page 3 etc of a category: All the below are '4ft-bedding.html' http://www.textilesdirect.co.uk/store/4ft-Bedding.html?filter=1&value=Pink http://www.textilesdirect.co.uk/store/4ft-Bedding.html?page=2 http://www.textilesdirect.co.uk/store/4ft-Bedding.html?sort=price&view_all=Y I've now changed all these internal links to rel="nofollow" on the a tag. Is that the correct and best way to sort? I might be mistaken on when I did this update and when the last report was ran, but on the SEOmoz crawling report, it still has the above as problem pages. thanks!
On-Page Optimization | | rowleysit-2598920