Automated checking for broken links within content pieces
-
Hi, I am wondering if anyone can send me in the right direction on a system suggestion.
We have currently grown out amount of content pieces on our website and our manual checking if the links in the content pieces are still 200 status is becoming extremely time consuming. Does anyone have a recommendation of a system that will crawl your pages and check both the internal and external links within the content for a status code (404,200,etc)? Preferably something server side so it can just run on a schedule but really anything would be fine.
I have tried things like Screaming frog, etc and it just doesn't seem to be the right tool.
-
Try ScreamingFrog again Jonathan, it works great for these kind of things and should also be able to solve your use case.
-
Jonathan, I'm not sure why you're saying that Screaming Frog isn't the right tool--we use it with great success to check the internal links on the site. There are other tools that you can use, such as Integrity (on a Mac), or Xenu, which is an older link checker but still works.
-
Have you tried http://www.link-assistant.com/website-auditor/ as it checks for broken links and can be scheduled to run automatically. You can sit it on your own server or something like AWS. We ran it on a free instance of AWS for quite a while before upgrading and never had issues. We upgraded as we run quite a bit of software on there - still isn't huge costs involved.
Hope this helps!
Matt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Internal Links Catalog (too many links)
I have several car catalogs that I use to direct my customers to the parts for their vehicle. Example:http://www.mikesfuel.com/Acura_c_69.html My customers like the approach. They can easily find their vehicle, then scan the columns for the parts that they need. Each part number is a link and the part numbers are sometimes posted several times in one column. All of the links are internal to the web site. My fear is that this is going to be treated as a link farm. I have considered using a PDF catalog with links, but that is going to be hugely expensive. What do you think? Thanks for any help Mike
On-Page Optimization | | mikescarb0 -
Use External Links
Hey 🙂 I noticed when analysing my pages that Moz gives the following advice about adding external links to my articles; "On any page specifically targeting a keyword, link externally to at least one (and possibly more than one) relevant, trusted resources as a best practice." As a small business I work pretty damn hard to get visitors to my website, so why on earth would I want to go to all that trouble just to send them away again to a trusted resouce? Secondly, what exactly is a "trusted resource"? Can I simply use search and use the top competitor, for example Moz or Wikipedia and does the anchor need to be an exact match or will a partial suffice. I say this because I already have the top spot for my longtail, so an exact match would be pointless. And lastly, I notice that pretty much all quality sites use external links to open in the same window i.e. not target=_blank, I never thought of it before today, but now that I'm considering using external linking in my articles I guess it's important to know the answer - i.e. Is this a best practice and does this give any seo benefit? Cheers, Lee :)
On-Page Optimization | | LeeC0 -
Link to social network
I have very powerfull social network but i do not link to them from my website to improve page speed and avoid external links on main page. Althought i do link from my twitter acounts , facebook and google to my site. Should i link to my twitter or social on main page. I do not want to send people to there. I want to send people from there to my page.
On-Page Optimization | | maestrosonrisas0 -
Links to Paywall from Content Pages
Hi, My site is funded by subscriptions. We offer lengthy excerpts, and then direct people to a single paywall page, something like domain.com/subscribe/ This means that most pages on the site links to /subscribe, including all of the high value pages that bring people in from Google. This is a page with an understandably high bounce rate, as most users are not interested in paying for content on the web. My question is are we being penalized in Google for having so many internal links to a page with a very high bounce rate? If anyone has worked with paywall sites before and knows the best practices for this, I'd be really grateful to learn more.
On-Page Optimization | | enotes0 -
Help With Duplicated Content
Hi Moz Community, I am having some issue's with duplicated content, i recently removed the .html from all of our links and moz has reported it as being duplicated. I have been reading up about Canonicalization and would to verify some details, when using the canonical tag would it be placed in the /mywebpage.html or /mywebpage file? I am having a hard time to sort this out so any help from you SEO experts would be great 🙂 I have also updated my htaccess file with the following Thanks in advance
On-Page Optimization | | finelinewebsolutions0 -
How to check duplicate content with other website?
Hello, I guest that my website may be duplicate contents with other websites. Is this a important factor on SEO? and how to check and fix them? Thanks,
On-Page Optimization | | JohnHuynh1 -
Cannibalizing link
this is what the reports states :Cannibalizing link"Red & Color Roses", "Red roses", "Black Baccara Roses\rBlack And Red Roses\rStarting at: $ 0.72 per Rose", "Black Magic Rose\rDark Red Black Rose\rStarting at: $ 0.96 per Rose", "Forever Young Roses\rRed Rose Flowers\rVelvety Deep Red\rStarting at: $ 1.16 per Rose", "Freedom Roses\rRed Rose\rStarting at: $ 0.66 ¢ per Rose", "Madame Delbard\rRed Wedding Rose\rStarting at: $ 0.66 per Rose", "Night Fever Roses\rTraditional Roses\rRomantic Red Roses\rStarting at: $ 0.66 per Rose", and "Red Paris Roses\rDark Red Rose\rStarting at: $ 0.72 per Rose"ExplanationIt's a best practice in SEO to target each keyword with a single page on your site (sometimes two if you've already achieved high rankings and are seeking a second, indented listing). To prevent engines from potentially seeing a signal that this page is not the intended ranking target and creating additional competition for your page, we suggest staying away from linking internally to another page with the target keyword(s) as the exact anchor text. Note that using modified versions is sometimes fine (for example, if this page targeted the word 'elephants', using 'baby elephants' in anchor text would be just fine).RecommendationUnless there is intent to rank multiple pages for the target keyword, it may be wise to modify the anchor text of this link so it is not an exact match. the questions is then why having those links on that page hurts, when we need the links to take the costumer to the color of red roses they want.
On-Page Optimization | | globalrose.com0 -
Duplicate content from category pages?
I have an ecommerce store with different categories for my products. Some products do appear in more than one category, is that an issue even if you end up on the same product page/link? Also, I have a "show all products" category, which I believe creates duplicate content too? What is your take on this? What can I do to solve this? Is it even an issue of duplicate content? All answers are very much appreciated!
On-Page Optimization | | danielpett0