Automated checking for broken links within content pieces
-
Hi, I am wondering if anyone can send me in the right direction on a system suggestion.
We have currently grown out amount of content pieces on our website and our manual checking if the links in the content pieces are still 200 status is becoming extremely time consuming. Does anyone have a recommendation of a system that will crawl your pages and check both the internal and external links within the content for a status code (404,200,etc)? Preferably something server side so it can just run on a schedule but really anything would be fine.
I have tried things like Screaming frog, etc and it just doesn't seem to be the right tool.
-
Try ScreamingFrog again Jonathan, it works great for these kind of things and should also be able to solve your use case.
-
Jonathan, I'm not sure why you're saying that Screaming Frog isn't the right tool--we use it with great success to check the internal links on the site. There are other tools that you can use, such as Integrity (on a Mac), or Xenu, which is an older link checker but still works.
-
Have you tried http://www.link-assistant.com/website-auditor/ as it checks for broken links and can be scheduled to run automatically. You can sit it on your own server or something like AWS. We ran it on a free instance of AWS for quite a while before upgrading and never had issues. We upgraded as we run quite a bit of software on there - still isn't huge costs involved.
Hope this helps!
Matt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does javascript generated content consider as regular content?
The website mentioned below, the content is generated using javascript, and content is something to do with Unicode char. The Unicode content creates as you scroll down. Will this content affect SEO https://www.myweirdtext.com/
On-Page Optimization | | teenmass423230 -
Product content length & links within product description
Hello, I have questions regarding content length and links within descriptions. With our ecommerce site, we have thousands of products, each with a unique description. In the product description, I have links to the parent category and grandparent category (if it has one) in the main product text which is generally about 175 words. Then I have a last paragraph that's about 75 words that includes links to our main homepage and our main product catalogue page. Is the content length long enough? I used to use text that was 500 words, and shortening it I still rank when launching new products, so I don't think an increase in text length will have any additional benefit. I do see conflicting information when I do searches, with some people recommending a minimum of 300 words and some saying to try and go a 1000 for category pages. In regards to the links, I noticed a competitor has stopped following this format, so I'm unsure if I should keep going too. Is it too many links to have each of the products link back to the main catalogue and homepage? Is it good to have links with anchor text to the categories a product is in? There are breadcrumbs on the page with these links already. There are already have heaps of links on our pages (footer, and a right sidebar with image links to relevant categories), so my pages do get flagged for too many links. Thanks!
On-Page Optimization | | JustinBSLW0 -
Comment Links and Pingbacks
I'm updating articles on a large site, some have links in the comments (WP site). The question is: 1. Should I remove the links in the comments, or does Google even care? 2. How do pingbacks affect ranking?
On-Page Optimization | | MichaelGregory0 -
Duplicate Page Content
Hey Moz Community, Newbie here. On my second week of Moz and I love it but have a couple questions regarding crawl errors. I have two questions: 1. I have a few pages with duplicate content but it say 0 duplicate URL's. How do I know what is duplicated in this instance? 2. I'm not sure if anyone here is familiar with an IDX for a real estate website. But I have this setup on my site and it seems as though all the links it generates for different homes for sale show up as duplicate pages. For instance, http://www.handyrealtysa.com/idx/mls...tonio_tx_78258 is listed as having duplicate page content compared with 7 duplicate URLS: http://www.handyrealtysa.com/idx/mls...tonio_tx_78247
On-Page Optimization | | HandyRealtySA
http://www.handyrealtysa.com/idx/mls...tonio_tx_78253
http://www.handyrealtysa.com/idx/mls...tonio_tx_78245
http://www.handyrealtysa.com/idx/mls...tonio_tx_78261
http://www.handyrealtysa.com/idx/mls...tonio_tx_78258
http://www.handyrealtysa.com/idx/mls...tonio_tx_78260
http://www.handyrealtysa.com/idx/mls...tonio_tx_78260 I've attached a screenshot that shows 2 of the pages that state duplicate page content but have 0 duplicate URLs. Also you can see somewhat about the idx duplicate pages. rel="canonical" is functioning on these pages, or so it seems when I view the source code from the page. Any help is greatly appreciated. skitch.png0 -
Issue: Duplicate Page Content
Hello SEO experts, I'm facing duplicate page content issue on my website. My website is a apartments rental website when client search apartment for availability. Automatic generate same url's. I've already block these url's in robots.txt file but facing same issue. Kindly guide me what can I do. Here are some example links. http://availability.website.com/booking.php?id=17&bid=220
On-Page Optimization | | KLLC
http://availability.website.com/booking.php?id=17&bid=242
http://availability.website.com/booking.php?id=18&bid=214
http://availability.website.com/booking.php?id=18&bid=215
http://availability.website.com/booking.php?id=18&bid=256
http://availability.website.com/details.php?id=17&bid=220
http://availability.website.com/details.php?id=17&bid=242
http://availability.website.com/details.php?id=17&pid=220&bid=220
http://availability.website.com/details.php?id=17&pid=242&bid=242
http://availability.website.com/details.php?id=18&bid=214
http://availability.website.com/details.php?id=18&bid=215
http://availability.website.com/details.php?id=18&bid=256
http://availability.website.com/details.php?id=18&pid=214&bid=214
http://availability.website.com/details.php?id=18&pid=215&bid=215
http://availability.website.com/details.php?id=18&pid=256&bid=256
http://availability.website.com/details.php?id=3&bid=340
http://availability.website.com/details.php?id=3&pid=340&bid=340
http://availability.website.com/details.php?id=4&bid=363
http://availability.website.com/details.php?id=4&pid=363&bid=363
http://availability.website.com/details.php?id=6&bid=367
http://availability.website.com/details.php?id=6&pid=367&bid=367
http://availability.website.com/details.php?id=8&bid=168
http://availability.website.com/details.php?id=8&pid=168&bid=168 Thanks and waiting for your response | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |0 -
Product links on homepage
Hi, I;m wondering if it really does aid SEO or even hinder it now to have lists of top selling products as links on the base of each page. This was recommended to us sometime ago by a consultant so we did it but is it outdated? I'm concerned it may penalise our pages now by creating too many links? opinions appreciated. wwww.gourmetdirect.com and scroll down to see what I mean
On-Page Optimization | | GourmetDirect0 -
Using content for cliche' terms, or content found on other sites
howdy, I have a basic question about using content found on other websites for your own use. I have started a pick up lines website for guys to search for pickup lines to use on girls. Anyways, my website has many, if anything a lot, of the same exact pick up lines as all my competitors are using. If I use the same pick up lines found on their site could i be penalized for this as far as SEO? thanks and hope to hear back
On-Page Optimization | | david3050 -
Percentage of duplicate content allowable
Can you have ANY duplicate content on a page or will the page get penalized by Google? For example if you used a paragraph of Wikipedia content for a definition/description of a medical term, but wrapped it in unique content is that OK or will that land you in the Google / Panda doghouse? If some level of duplicate content is allowable, is there a general rule of thumb ratio unique-to-duplicate content? thanks!
On-Page Optimization | | sportstvjobs0