The Case Study Problem
This case study derives from an ongoing Volv engagement, which started in July 2017. A pharmaceutical company approached Volv Global to develop a prediction model to identify additional patients suffering from the rare disease treatable by its specialist medicine.
This pharmaceutical company faced four difficulties: the disease prevalence was one in a million of population; specialist clinicians were able to diagnose the disease with no more than 76% accuracy; only one in four patients were ever identified; and those that were identified, were done so generally after six years of misdiagnoses. To compound these four difficulties, the pharmaceutical company was unable to provide medical records for any already diagnosed patients.
Meeting the Case Study Challenge
Volv undertook to build a methodology that would learn about the disease automatically from publicly available data to derive new insights, so that we could try to be better at diagnosing than physicians.
We also committed to prove clear patient cohort selection performance above current best, manual practice. Deriving diagnostic models for rare diseases manually, based on input from experts, is problematic for several reasons, including the evolving nature of experts’ understanding, complexity of disease processes, and cognitive biases associated with human decision-making [Elstein, Schwarz 2002]. By September 2017, Volv had built a disease learning model, applicable not only to this disease but any disease. We called the methodology to build this model inTrigue. This prediction model is built on a research and evidence-based understanding of the disease derived from extracting clinical evidence from publicly available sources. This represented a true breakthrough in data science.
Secondly, using novel techniques with extremely small sample sizes, that are typical to rare diseases and personalised medicines, Volv built a predictive diagnostic algorithm that outperformed Human Clinical Diagnostic Performance by looking at data earlier in the patient journey and by identifying cognitive biomarkers, digital biomarkers and medical biomarkers that drive a completely new way to diagnose.
Our case finding represents a strategic approach that focuses on features in a person that are unusual in light of other attendant features. It involves actively searching for at-risk individuals using systematic methods, rather than waiting for them to present symptomatically. This approach is parsimonious in terms of costs given the availability of access to electronic health records, and offers significant uplift in accuracy when machine learning algorithms are deployed compared to a search engine approach, for example, or to screening. Screening tests are only useful in large populations to determine the suspicion of a disease. They fail to pass the cost-benefit threshold for rare disease detection since a large number of individuals needs to be screened to find a small number of potential at-risk individuals. Diagnostic testing is extremely costly and requires a high degree of certainty to exist, which further reduces the chances of finding at-risk individuals.
It is important to note that our approach directly addresses known reasons why patients with rare diseases are seldom diagnosed quickly. Our approach removes blockages in clinical reasoning which lead to clinicians retreating from a diagnosis of a rare disease in the first place (called the ‘Zebra Retreat’), arising from either a lack of curiosity or the erroneous belief that the correct diagnosis is something more common. [Miller 2013].
Testing & Results
Volv tested the prediction model on a database of 2.5 million primary care anonymised electronic patient records in the Netherlands, and the results were clinically assessed. The anonymised patients were being treated for other conditions and not the underlying real disease.
The assessment showed that the Volv model identified a significantly greater number of patients with the specific rare disease and significantly outperformed other prediction models including clinicians’ diagnostic accuracy.
The Volv approach lifts clinical accuracy for the detection of at-risk patients to world-class levels of precision. It outperformed the existing, state-of-the-art prediction models with a two thirds reduction in error rate. The sensitivity of prediction models to false positives is measured using Area Under the ROC Curve (‘AUC’). In terms of AUC, 1.00 is perfect accuracy (no false positives), and Volv’s prediction model achieved a rating of 0.935, which is well above the human performance level of 0.76.
- Volv model gives accuracy = 90.8% (standard error = 0.4%) and AUC = 93.5% (standard error = 0.1%);
- Kopcke model (L2-regularised logistic regression on full feature set [Kopcke et al. 2013]) gives accuracy = 73.1% (standard error = 1.1%) and AUC = 74.9% (standard error = 1.0%);
- Miotto (classification based on relevance scoring on full feature set [Miotto, Weng 2015]) gives accuracy = 74.2% (standard error = 1.0%) and AUC = 75.8% (standard error = 1.0%)
When the system and methodology were used to augment the physician the accuracy rose to 0.975.
Further work
As part of the project we also worked out how to help healthcare system providers utilise the methodology within the tight ethical and regulatory frameworks that exist in different countries and regions. It is now translatable to different settings for potentially thousands of diseases.
We have called this end-to end approach and solution inTrigue.
Benefits of inTrigue
There are five benefits inTrigue delivered to our pharmaceutical company client:
- Accurate Disease Insight
- Patient Finding
- Prevalence Insight
- Risk Mitigation
- Improved Financial Performance
Accurate Disease Insight
inTrigue identified the clinical evidence that is most predictive for a clinical assessment by clinicians that diagnose and treat patients with this rare disease. inTrigue also found a novel (new) predictor of the condition, which could be added to the clinical diagnostic assessment, to improve diagnostic precision.
Patient Finding
InTrigue identified patients hidden in an electronic records system. Given the rare condition, it would have been highly unlikely that these patients would have been correctly diagnosed until a serious exacerbating event had occurred to unmask finally the condition to a clinician.
Prevalence Insight
InTrigue redefined the prevalence of the rare disease, producing an empirical assessment as being more likely in the ranges of one in 300,000 and not one in 1,000,000 as originally supposed.
Risk Mitigation
inTrigue enables the pharmaceutical company to demonstrate to payers
and clinicians new ways to reduce clinical risk, by defining the treatment population with greater accuracy. Patients who would
benefit from the medicine are the ones who get it. inTrigue supports both payer and pharmaceutical company objectives to derive maximum value from their respective investments. And because inTrigue is so good at patient cohort identification, it helps in developing US FDA Risk Evaluation and Mitigation Strategies (REMS).
Improved Financial Performance
inTrigue delivered world class precision which will enable this pharmaceutical company to better calibrate its market access strategy with market archetypes and payer preferences. Volv is working with the client to demonstrate improved clinical impact to tertiary and quaternary centres and to their specialist physicians. Uptake with these physicians shows that the specialist diagnosing physicians become the real buyers owing to the accuracy of Volv’s clinical tool sets in meeting payers’ buying behaviours. This reduces the costs of traditional market access and sales approaches by shifting effectively to a buyers’ market.
Summary: What does our approach do?
InTrigue is a case-finding prediction model that reshapes clinical practice by helping clinical specialists make better clinical diagnosis for precision and personalised medicine so that they can treat the precise cohort of patients that will benefit from a specific medicines.
• inTrigue uses known genetic biomarkers, phenotype markers, and cognitive markers and where evidence is available behavioural markers. inTrigue identifies digital biomarkers and novel disease predictors from the electronic records and literature.
• The prediction model continues to learn and become more accurate as the results of clinical use of the model in reviewing patients are fed back to it.
• Having moved the prevalence needle, inTrigue demonstrates precision case finding and has the capability of undertaking cohort identification for clinical trials by identifying the patients that fit the cohort profile.
• inTrigue assists human decision making, and helps reduce human subjectivity which hampers current decision making for clinical diagnosis, case finding and cohort creation.
References
Elstein A & Schwarz A. Clinical problem solving and diagnostic decision making: Selective review of the cognitive literature, BMJ. 325;2002.
Kopcke F et al. Evaluating predictive modelling algorithms to assess patient eligibility for clinical trials from routine data, BMC MIDM. 13;2013.
Miller CS. Skin-deep diagnosis: affective bias and zebra retreat complicating the diagnosis of systemic sclerosis. Am J Med Sci. 2013 Jan;345(1):53–6.
Miotto R & Weng C. Case-based reasoning using electronic health records efficiently identifies eligible patients for clinical trials. JAMIA. 22;2015.
