Search
Close this search box.
Search

Sociotechnical Framework Unveiled: Balancing technology, people, and organizations in eHealth tool assessment. With 46 criteria, experts advocate for context-aware evaluations, preferring a scorecard system.

eHealth success lies at the intersection of technology, people, and organization

The current technological advancements have facilitated the development of a multitude of eHealth tools, offering users a wide array of options. However, only a few undergo expert assessment, and a significant majority fail to progress beyond the pilot phase due to their inability to demonstrate value or face implementation barriers.

Although numerous assessment initiatives and frameworks exist, these efforts encounter substantial challenges, including the contextual complexities of healthcare, a lack of validation, incomplete criteria, and impracticality.

To address some of these challenges, a study titled “A sociotechnical framework to assess patient-facing eHealth tools: results of a modified Delphi process,” published in npj Digital Medicine, presents a validated sociotechnical framework that comprehensively considers various criteria for assessing patient-facing eHealth tools. The international panel that validated and co-created the framework comprised 57 experts from 18 countries and 9 stakeholder groups.

“Our aim was to contribute to an area that is still being shaped and developed by analyzing and aggregating existing criteria through rigorous academic research and systematic review of existing frameworks, combined with a practice-oriented approach through discussing, pressure-testing, and co-creating a relevant and accessible assessment instrument with a diverse group of experts spanning all relevant stakeholder groups,” says Christine Jacob, the project lead and the first author of the study.

A consensus was reached on 46 criteria, classified into foundational and contextual criteria. The 36 foundational criteria focused on evaluating the eHealth tool itself. They were grouped into nine clusters: technical aspects, clinical utility and safety, usability and human centricity, functionality, content, data management, endorsement, maintenance, and developer. The ten contextual criteria focus on evaluating factors that vary depending on the context in which the tool is being assessed and were grouped into seven clusters: data-protection compliance, safety regulatory compliance, interoperability and data integration, cultural requirements, affordability, cost-benefit, and implementability.

The classification of criteria into foundational and contextual helps us assess not only the quality of an isolated tool but also its potential fit in a specific setting. Criteria subscales may be particularly relevant when determining the tool’s strengths and weaknesses. This granularity enables different concerned parties to make informed decisions about which tools to consider according to their specific needs and priorities.

Taking the varying healthcare context into account

Research indicates that elements extending beyond the eHealth tool itself, such as implementation costs, clinical workflows, necessary resources, infrastructure, patient characteristics, and socioeconomic status, have a substantial impact on the acceptance and adoption of eHealth. The inherent contextual nature of healthcare poses a notable challenge when engaging with eHealth tools without contextual awareness. Despite existing standards for assessing usability and scientific evidence of tools, there seems to be a gap in guidelines supporting the evaluation of their implementation and processes. Consequently, this leads to a lack of contextual assessment criteria.

In contrast to several assessment initiatives and frameworks that focus solely on evaluating the tool without considering the healthcare context, the sociotechnical framework introduced in this study emphasizes the inclusion of contextual criteria. These encompass factors such as the readiness of local infrastructure, resources required for scale-up, cost-benefit analysis, reimbursement standards, and cultural aspects like the use of the local language. Extensive evidence underscores the significant influence of these contextual factors on the adoption and scale-up of such tools. The unanimous expert consensus (100%) on the inclusion of contextual criteria reaffirms their importance and relevance for a comprehensive assessment.

Balancing the different priorities of multiple stakeholders

Various assessment initiatives, often focused on curating, certifying, or accrediting eHealth tools, frequently utilize a single-score approach to provide an overall indication of quality. The objective is to simplify comparisons for potential customers, aiding them in distinguishing between low- and high-quality offerings. While these initiatives contribute significantly to assessment efforts, scholars argue that they might not offer clear guidance on the most effective tools for seamless integration into specific healthcare contexts. Consequently, a scorecard approach may be more suitable for context-specific evaluations involving multiple stakeholders.

This was confirmed by the strong consensus among experts who strongly supported the preference for a scorecard, viewed as a more balanced way to present assessment results, with 78% expressing a preference for this approach. Experts contended that a single score might obscure crucial details, emphasizing that the actual value of the assessment instrument lies in understanding the breakdown.

Addressing the subjective nature of some criteria

The inclusion of subjective measures has sparked debate in the literature due to its potential to introduce variability in assessment outcomes based on the subjective perspectives of assessors. Prior research has illustrated that certain characteristics of eHealth tools pose challenges for consistent rating. Despite this challenge, several scholars strongly advocate for incorporating subjective criteria, including factors such as ease of use and visual appeal, given their significant role as fundamental drivers of adoption. Thus, integrating subjective criteria into the review process holds the potential to enhance tool adherence and improve health outcomes. The consensus among experts unequivocally supported this perspective, with multiple subjective criteria meeting the predefined consensus level.

To mitigate variability in the assessment of subjective criteria, experts provided several recommendations. Foremost among these was the advocacy for assessor diversity, ensuring a well-rounded assessment that considers various perspectives.

The project team is currently developing a web-based assessment instrument based on the outcomes of this study, where the assessors can find step-by-step guidance on how to assess each of the evaluation measures, input their assessment of each criterion, and download a report comprising the assessment outcomes and dashboards. This work aims to empower decision-makers with an assessment instrument tailored to support their decision-making based on their specific needs and priorities within the distinct contexts where they are evaluating a tool.

This innovation project is sponsored by Innosuisse (the Swiss innovation agency), with the University of Applied Sciences and Arts Northwestern Switzerland FHNW as the research partner and Roche and KPT Insurance as the practice partners. For more details and resources, please check the project page.

Tom Xhofleer

ICT&health World Conference 2024

Experience the future of healthcare at the ICT&health World Conference from May 14th to 16th, 2024!
Secure your ticket now and immerse yourself in groundbreaking technologies and innovative solutions.
Engage with fellow experts and explore the power of global collaborations.

Share this article!

Door
Read also
Doximity_edited
Innovation Adoption: How to Traverse The Valleys of Death
Pioneering Cardiac Arrest Detection for Enhanced Survival.
CardioWatch Revolutionizes Cardiac Arrest Detection
Dr. Oscar Díaz-Cambronero, Head of Perioperative Medicine Department at La Fe Hospital, spearheads innovative telemonitoring initiatives revolutionizing patient care
Smartwatches Saving Lives Inside and Outside the Hospital
EIT 2024
EIT Awards 2024. Two European startups are revolutionizing the treatment of cardiovascular diseases
Bertrand Piccard, Swiss explorer and founder of the Solar Impulse Foundation
EIT Summit 2024. What are the trigger points that drive or inhibit innovation?
MMC pioneers wireless monitoring for premature infants with the innovative Bambi Belt, revolutionizing care with improved comfort and mobility.
Wireless Monitoring of Vital Signs in Premature Infants at Máxima MC
Data protection-critical incidents resulting from human error are often rooted in stress, routine, negative attitudes toward IT, and deficits in employees' identification with the healthcare facility.
How cyberpsychology helps prevent human errors leading to data leaks
What technologies will enter our homes in a few months? ICT&health checked it out at the CES 2024.
CES 2024: Meet the exciting innovations for health and well-being
An article on a new study on e-health assessment tools
eHealth success lies at the intersection of technology, people, and organization
Unlocking the Future: Professor Sylvia Thun, a trailblazer in healthcare interoperability, discusses the crucial role of seamless data exchange in revolutionizing medicine and empowering individuals with comprehensive access to their health data.
Seamless data exchange will unlock the long-awaited benefits of digitalization
Follow us