Although numerous assessment initiatives and frameworks exist, these efforts encounter substantial challenges, including the contextual complexities of healthcare, a lack of validation, incomplete criteria, and impracticality.
To address some of these challenges, a study titled “A sociotechnical framework to assess patient-facing eHealth tools: results of a modified Delphi process,” published in npj Digital Medicine, presents a validated sociotechnical framework that comprehensively considers various criteria for assessing patient-facing eHealth tools. The international panel that validated and co-created the framework comprised 57 experts from 18 countries and 9 stakeholder groups.
“Our aim was to contribute to an area that is still being shaped and developed by analyzing and aggregating existing criteria through rigorous academic research and systematic review of existing frameworks, combined with a practice-oriented approach through discussing, pressure-testing, and co-creating a relevant and accessible assessment instrument with a diverse group of experts spanning all relevant stakeholder groups,” says Christine Jacob, the project lead and the first author of the study.
A consensus was reached on 46 criteria, classified into foundational and contextual criteria. The 36 foundational criteria focused on evaluating the eHealth tool itself. They were grouped into nine clusters: technical aspects, clinical utility and safety, usability and human centricity, functionality, content, data management, endorsement, maintenance, and developer. The ten contextual criteria focus on evaluating factors that vary depending on the context in which the tool is being assessed and were grouped into seven clusters: data-protection compliance, safety regulatory compliance, interoperability and data integration, cultural requirements, affordability, cost-benefit, and implementability.
The classification of criteria into foundational and contextual helps us assess not only the quality of an isolated tool but also its potential fit in a specific setting. Criteria subscales may be particularly relevant when determining the tool’s strengths and weaknesses. This granularity enables different concerned parties to make informed decisions about which tools to consider according to their specific needs and priorities.
Taking the varying healthcare context into account
Research indicates that elements extending beyond the eHealth tool itself, such as implementation costs, clinical workflows, necessary resources, infrastructure, patient characteristics, and socioeconomic status, have a substantial impact on the acceptance and adoption of eHealth. The inherent contextual nature of healthcare poses a notable challenge when engaging with eHealth tools without contextual awareness. Despite existing standards for assessing usability and scientific evidence of tools, there seems to be a gap in guidelines supporting the evaluation of their implementation and processes. Consequently, this leads to a lack of contextual assessment criteria.
In contrast to several assessment initiatives and frameworks that focus solely on evaluating the tool without considering the healthcare context, the sociotechnical framework introduced in this study emphasizes the inclusion of contextual criteria. These encompass factors such as the readiness of local infrastructure, resources required for scale-up, cost-benefit analysis, reimbursement standards, and cultural aspects like the use of the local language. Extensive evidence underscores the significant influence of these contextual factors on the adoption and scale-up of such tools. The unanimous expert consensus (100%) on the inclusion of contextual criteria reaffirms their importance and relevance for a comprehensive assessment.
Balancing the different priorities of multiple stakeholders
Various assessment initiatives, often focused on curating, certifying, or accrediting eHealth tools, frequently utilize a single-score approach to provide an overall indication of quality. The objective is to simplify comparisons for potential customers, aiding them in distinguishing between low- and high-quality offerings. While these initiatives contribute significantly to assessment efforts, scholars argue that they might not offer clear guidance on the most effective tools for seamless integration into specific healthcare contexts. Consequently, a scorecard approach may be more suitable for context-specific evaluations involving multiple stakeholders.
This was confirmed by the strong consensus among experts who strongly supported the preference for a scorecard, viewed as a more balanced way to present assessment results, with 78% expressing a preference for this approach. Experts contended that a single score might obscure crucial details, emphasizing that the actual value of the assessment instrument lies in understanding the breakdown.
Addressing the subjective nature of some criteria
The inclusion of subjective measures has sparked debate in the literature due to its potential to introduce variability in assessment outcomes based on the subjective perspectives of assessors. Prior research has illustrated that certain characteristics of eHealth tools pose challenges for consistent rating. Despite this challenge, several scholars strongly advocate for incorporating subjective criteria, including factors such as ease of use and visual appeal, given their significant role as fundamental drivers of adoption. Thus, integrating subjective criteria into the review process holds the potential to enhance tool adherence and improve health outcomes. The consensus among experts unequivocally supported this perspective, with multiple subjective criteria meeting the predefined consensus level.
To mitigate variability in the assessment of subjective criteria, experts provided several recommendations. Foremost among these was the advocacy for assessor diversity, ensuring a well-rounded assessment that considers various perspectives.
The project team is currently developing a web-based assessment instrument based on the outcomes of this study, where the assessors can find step-by-step guidance on how to assess each of the evaluation measures, input their assessment of each criterion, and download a report comprising the assessment outcomes and dashboards. This work aims to empower decision-makers with an assessment instrument tailored to support their decision-making based on their specific needs and priorities within the distinct contexts where they are evaluating a tool.
This innovation project is sponsored by Innosuisse (the Swiss innovation agency), with the University of Applied Sciences and Arts Northwestern Switzerland FHNW as the research partner and Roche and KPT Insurance as the practice partners. For more details and resources, please check the project page.
