10 Basic Rules for Clinical Algorithm Development

Keeping an Eye on the Prize: 10 Basic Rules for Clinical Algorithm Development

While at the HIMSS16 conference last week in Las Vegas, we were asked to provide some simple guidance for people wanting to develop medical algorithms for clinical decision support. Here are 10 basic rules that may be helpful.

1. Keep an eye out for potential traps when selecting an algorithm to implement.
   It is good to avoid algorithms that may be a problem before you start. Problems can arise from bias, overfitting, or poor performance characteristics. Sometimes problematic algorithms can be quite seductive, but why waste your time on something no one will use? An example from our own experience was a forensic medicine algorithm we developed, using several formulae for estimating the time of death from body temperature. We later found out after talking to a medical examiner that no one uses these in practice.
2. Keep it legal.
   Some algorithms are under copyright or have been patented.  Many people using the SF-36 and its siblings may not be aware that the creator has a company for marketing these instruments. It is a shame to put in all the effort to develop a clinical algorithm only to be forced to take it down or to face legal action.
3. Keep it compliant with government regulations.
   Government agencies tolerate many algorithms. However, they will act when certain kinds of algorithms are encountered. These include programs that bypass a clinician or an algorithm incorporated into a medical device.
4. Keep its use clearly defined.
   An algorithm is a tool intended to achieve a specific purpose. It may only be useful in certain situations. Users who misuse the tool may blame you rather than themselves. So it is important to clearly state when and when not to use the algorithm.
5. Keep it simple.
   You may feel the temptation to make an algorithm more complex than it needs to be. A simple algorithm is not only easier to program but also easier to troubleshoot and maintain. It is often better to break up a complex algorithm into several modules.
6. Keep the target population in mind.
   Some algorithms work well in one population but not in another. These algorithms may not perform as well if they are transported to another group. For example, algorithms developed to predict body surface area for Europeans will probably fail in Asia.  It’s important to initially evaluate if the algorithm can be geared to multiple populations or if it should be defined as only applicable with specific populations.
7. Keep an eye on the data.
   In general, the fewer the variables the better. Discrete data that is clearly defined is best. Most algorithms are derived from peer-reviewed, evidence-based, published papers.  Often, these papers give only limited information in the Methods section about how the data was generated, which can cause problems when transported to another site. In addition, the cost to generate the data may be prohibitive for some. Make sure you know the data source and clearly understand its origin, as well as the cost for generating it before you begin development.
8. Keep the documentation clear and complete.
   Good documentation helps the user understand the algorithm. It is also reflects on your workmanship. This can help the user to feel more comfortable in trusting your work. Thorough documentation should always include provenance information, formulas, the algorithm purpose, and a discussion on how the algorithm is used and its results interpreted in clinical practice.
9. Keep testing.
   No matter how well you may believe you’ve done your job, you can always find mistakes or areas of improvement by testing. We use a spreadsheet as a simple mockup language to test the logic before and after formal programming.  Our development process includes testing by a minimum of three separate teams, each of whom approaches testing from a different perspective.  First, our technical testers, who are part of our software development team, test for programming accuracy.  Next our medical team tests for medical and technical accuracy.  And, finally, clinicians test the algorithm with a focus on ease of use in the clinical setting and accuracy.  After going live, we continue to test using the same procedure described above.  Set up a standard protocol for testing to increase efficiency and reduce risk of error.
10. Keep the algorithm discoverable.
    No matter how good the algorithm, it cannot be used if it is not found. With all of the content on the web, being found can be a challenge.  If you post the algorithm on your own website or a hospital/medical center site, add keywords and SEO  parameters so it’s found by search engines.  If you need a platform for discovery and distribution of your algorithm, check out the Apervita platform. Apervita is a powerful, secure, self-service marketplace that allows healthcare professionals to acquire and author evidence-based algorithms and connect them to practice anywhere.