Standardized Reporting is Crucial for Effective Evaluation of Digital Health Interventions

Reporting of patient adherence and engagement to digital health interventions would be more effective using a standardized method, a study finds.

Standardizing reporting methods would allow for more accurate comparisons between different digital health interventions, as well as across various populations.

“As the number of such studies continues to increase, it is important for clinical trials of digital interventions to align with a common set of core adherence and engagement metrics,” wrote the researchers of the study. “This alignment will encourage the consistent reporting of user engagement to make comparisons of digital interventions across studies more meaningful.”

The full systematic literature review study is published in JMIR Publications.

In the study, the researchers aimed to understand how patient adherence and engagement was reported in clinical trials, what levels were reported, and whether higher adherence and engagement was associated with increased efficiency.

The study included patients ages 18 years and older with depression or major depressive disorder and included clinical studies, feasibility studies, and pilot digital interventions for treating depression.

A database search was conducted for articles published between January 1, 2000 to April 15, 2022, in Ovid MEDLINE In-Process & Other Non-Indexed Citations, Embase, and the Cochrane Library.

The database search resulted in 1181 records and an additional 590 records that were manually searched. A total of 756 studies were screened, in which 94 studies met the inclusion criteria.

The researchers extracted data from all eligible studies, including:

• Study design
• Primary diagnosis of participants
• Metrics used to diagnose depression for inclusion in the study
• Number of participants
• Participant demographics
• Type of digital intervention used
• Number of days participants were given to use the intervention
• Whether the intervention was unguided or used human support
• Whether other forms of treatment for depression were used during the study period
• How care as usual was defined
• Type of control group used
• Adherence and engagement metrics used
• Level of adherence and engagement reported
• The primary efficacy outcome and any other efficacy outcomes
• The relationship of adherence and engagement with clinical outcomes, if available.

These studies included results from 20,111 participants in studies using 47 different web-based interventions, 15 mobile app interventions, 5 app-based interventions, and 1 CD-ROM.

Adherence to these interventions were commonly measured as the percentage of participants who completed all available modules. Although less than half (44.2%) of participants completed all the modules, the average dose received was 60.7% of all available modules.

Despite some differences in measurements of engagement, it was most commonly measured as the number of modules completed, with a mean of 6.4 (range, 1.0-19.7) modules. The mean amount of time participants engaged with interventions was 3.9 hours.

Additionally, 76% of studies that were web-based and 89% of studies that were app-based had substantially greater improvement in at least 1 outcome in the intervention group compared with the control group. Furthermore, of the 14 studies that measured the relationship between efficacy and engagement, 64% found that increased engagement with digital interventions significantly improved patient outcomes.

The researchers acknowledged some limitations to the study. The main limitation was publication bias, in which studies with statistically significant results and larger effect sizes are more likely to be published than a study with negative results or a smaller effect size. Other limitations include the potential for bias for studies of higher adherence and engagement levels, and the possible overestimation of the effects of engagement with digital interventions.

Despite these limitations, the researchers believe the study highlights the importance of standardized methods of reporting patient adherence and engagement to improve efficacy with digital interventions.

“For the research field, improvements could be made by using consistent metrics to report adherence (eg, dose received) and engagement (eg, hours spent using the intervention), through regular inclusion of control groups and patients of diverse backgrounds in studies, by always reporting race and ethnicity data in publications, by investigating the interplay of socioeconomic factors and the efficacy of digital interventions, and by measuring the dose-response relationship to make data-informed decisions about dose recommendations,” wrote the researchers.

Reference

Forbes A, Keleher MR, Venditto M, DiBiasi F. Assessing patient adherence to and engagement with digital interventions for depression in clinical trials: Systematic literature review. J Med Internet Res. 2023;25:e43727. doi:10.2196/43727