Testing & Results

Volv tested the prediction model on a database of 2.5 million primary care anonymised electronic patient records in the Netherlands, and the results were clinically assessed. The anonymised patients were being treated for other conditions and not the underlying real disease.

The assessment showed that the Volv model identified a significantly greater number of patients with the specific rare disease and significantly outperformed other prediction models including clinicians’ diagnostic accuracy.

The Volv approach lifts clinical accuracy for the detection of at-risk patients to world-class levels of precision. It outperformed the existing, state-of-the-art prediction models with a two thirds reduction in error rate. The sensitivity of prediction models to false positives is measured using Area Under the ROC Curve (‘AUC’). In terms of AUC, 1.00 is perfect accuracy (no false positives), and Volv’s prediction model achieved a rating of 0.935, which is well above the human performance level of 0.76.

  • Volv model gives accuracy = 90.8% (standard error = 0.4%) and AUC = 93.5% (standard error = 0.1%);
  • Kopcke model (L2-regularised logistic regression on full feature set [Kopcke et al. 2013]) gives accuracy = 73.1% (standard error = 1.1%) and AUC = 74.9% (standard error = 1.0%);
  • Miotto (classification based on relevance scoring on full feature set [Miotto, Weng 2015]) gives accuracy = 74.2% (standard error = 1.0%) and AUC = 75.8% (standard error = 1.0%)

When the system and methodology were used to augment the physician the accuracy rose to 0.975.

Further work

As part of the project we also worked out how to help healthcare system providers utilise the methodology within the tight ethical and regulatory frameworks that exist in different countries and regions. It is now translatable to different settings for potentially thousands of diseases.

We have called this end-to end approach and solution inTrigue.

Benefits of inTrigue

There are five benefits inTrigue delivered to our pharmaceutical company client:

  • Accurate Disease Insight
  • Patient Finding
  • Prevalence Insight
  • Risk Mitigation
  • Improved Financial Performance
Accurate Disease Insight

inTrigue identified the clinical evidence that is most predictive for a clinical assessment by clinicians that diagnose and treat patients with this rare disease. inTrigue also found a novel (new) predictor of the condition, which could be added to the clinical diagnostic assessment, to improve diagnostic precision.

Patient Finding

InTrigue identified patients hidden in an electronic records system. Given the rare condition, it would have been highly unlikely that these patients would have been correctly diagnosed until a serious exacerbating event had occurred to unmask finally the condition to a clinician.

Prevalence Insight

InTrigue redefined the prevalence of the rare disease, producing an empirical assessment as being more likely in the ranges of one in 300,000 and not one in 1,000,000 as originally supposed.

Risk Mitigation

inTrigue enables the pharmaceutical company to demonstrate to payers

and clinicians new ways to reduce clinical risk, by defining the treatment population with greater accuracy. Patients who would

benefit from the medicine are the ones who get it. inTrigue supports both payer and pharmaceutical company objectives to derive maximum value from their respective investments. And because inTrigue is so good at patient cohort identification, it helps in developing US FDA Risk Evaluation and Mitigation Strategies (REMS).

Improved Financial Performance

inTrigue delivered world class precision which will enable this pharmaceutical company to better calibrate its market access strategy with market archetypes and payer preferences. Volv is working with the client to demonstrate improved clinical impact to tertiary and quaternary centres and to their specialist physicians. Uptake with these physicians shows that the specialist diagnosing physicians become the real buyers owing to the accuracy of Volv’s clinical tool sets in meeting payers’ buying behaviours. This reduces the costs of traditional market access and sales approaches by shifting effectively to a buyers’ market.

Summary: What does our approach do?

InTrigue is a case-finding prediction model that reshapes clinical practice by helping clinical specialists make better clinical diagnosis for precision and personalised medicine so that they can treat the precise cohort of patients that will benefit from a specific medicines.

• inTrigue uses known genetic biomarkers, phenotype markers, and cognitive markers and where evidence is available behavioural markers. inTrigue identifies digital biomarkers and novel disease predictors from the electronic records and literature.

• The prediction model continues to learn and become more accurate as the results of clinical use of the model in reviewing patients are fed back to it.

• Having moved the prevalence needle, inTrigue demonstrates precision case finding and has the capability of undertaking cohort identification for clinical trials by identifying the patients that fit the cohort profile.

• inTrigue assists human decision making, and helps reduce human subjectivity which hampers current decision making for clinical diagnosis, case finding and cohort creation.

References

Elstein A & Schwarz A. Clinical problem solving and diagnostic decision making: Selective review of the cognitive literature, BMJ. 325;2002.

Kopcke F et al. Evaluating predictive modelling algorithms to assess patient eligibility for clinical trials from routine data, BMC MIDM. 13;2013.

Miller CS. Skin-deep diagnosis: affective bias and zebra retreat complicating the diagnosis of systemic sclerosis. Am J Med Sci. 2013 Jan;345(1):53–6.

Miotto R & Weng C. Case-based reasoning using electronic health records efficiently identifies eligible patients for clinical trials. JAMIA. 22;2015.