E-health assessment: a scattered and challenging landscape

A study, "Assessing the Quality and Impact of eHealth Tools: Systematic Literature Review and Narrative Synthesis," published in JMIR Human Factors, systematically investigated the literature to understand the different approaches and criteria used to assess the quality and impact of eHealth tools. The review builds on the growing body of research that explores the criteria used to evaluate eHealth tools and highlights the complexity and challenges facing these initiatives, emphasizing the need for a more comprehensive approach that reflects the complexity and interdependence of the healthcare ecosystem and is aligned with the factors affecting users' adoption to ensure acceptance and stickiness in the long term. Health technology evaluation efforts face multiple challenges   Although various initiatives aim to find new ways to assess the quality and impact of eHealth tools, these efforts face multiple challenges. The multitude of frameworks and initiatives attempting to address the topic of eHealth tools' assessment signals the lack of generalizability and standardization in this field. In many cases, existing initiatives are conceptual without granular guidance on how to use and apply them in day-to-day decision-making and sometimes overlook critical assessment criteria resulting in incomplete or issue-specific assessment frameworks. The lack of regulatory clarity and the absence of institutionalized quality controls in many countries make a comprehensive definition of the assessment criteria more challenging. Additionally, there are some limitations with some of the current certification labels. For example, although common labels may categorize a tool as a medical device, it may still include the warning in fine print that it is intended for entertainment only, showing a lack of accountability and creating confusion on the users' side. Not to mention that the validity and reliability of the existing assessment frameworks have not always been rigorously tested and validated by the different stakeholders in the healthcare ecosystem, dismissing the importance of involving diverse assessor profiles, including the tools' developers themselves. Novel health technologies can also be highly contextual, making it crucial to take the implementation context and use cases into account. However, the varying contexts and use cases make it quite challenging to find a standardized and generalizable way to assess them. And while most assessment criteria are objective, some of the criteria that are most relevant for user engagement are subjective, limiting the standardization of the assessment outcome. For example, given the importance of user engagement for the success of eHealth tools, it would still be crucial to include subjective assessment criteria that reflect key user engagement and adoption drivers, such as ease of use and visual appeal. Proper assessment is also primarily dependent on developer transparency and availability of information, which is unfortunately not always the case, making it quite challenging to address the questions needed to accurately assess a tool. [caption id="attachment_28887" align="aligncenter" width="640"] alt="" width="640" height="369" /> Source: Jacob et al. Assessing the Quality and Impact of eHealth Tools: Systematic Literature Review and Narrative Synthesis.[/caption]   Technical criteria are not enough in healthcare The findings from this study show that there doesn't seem to be a single framework used uniformly to evaluate eHealth tools. However, despite their different contexts and the different disease conditions they addressed, there was substantial overlap among the frameworks. Technical criteria have predominantly been the focus of existing initiatives, mainly because they are the foundation for the viability of any eHealth solution and for it to be considered at all by potential users; without this foundation, a tool wouldn't properly support the basic requirements for success. Therefore, ensuring a high level of technical performance, offering well-defined and useful functionalities and features, as well as credible, valid, and reliable content, proper data management strategies, and a superior user experience are the basics that every eHealth tool must meet for it to be considered by the relevant users. The inclusion of organizational assessment criteria may help address a key challenge with current efforts relating to the importance of the contextuality of eHealth tools, as these technologies are not used in isolation of the healthcare ecosystem. Therefore, a proper assessment of these tools' potential impact should consider the specific context. Healthcare technologies are generally more complex than tools that address individual user needs. They usually support patients with comorbidities who are typically treated by multidisciplinary teams who might work in different healthcare organizations. Hence, the importance of contextual and organizational aspects. Context-specific criteria such as implementation, workforce and resources, infrastructure, and the overall healthcare context don't seem to be fairly represented in the current assessment initiatives, resulting in situations where a tool may be of good quality when assessed in isolation. Still, they might not have the desired impact in a real-life scenario because contextual criteria do not necessarily allow it to be successfully implemented or scaled if not properly evaluated. The study authors equally advocate for the importance of the inclusion of relevant social assessment criteria that evaluate the potential societal impact of these tools. Notably, even though many frameworks included usability in general as an assessment criterion, more than half of the included studies did not specifically address human centricity through active user engagement and behavioral change strategies. This is concerning, considering the lack of reliable evidence about the ability of most commercially available eHealth tools to induce lasting behavioral change and address equity questions related to the digital divide. "In the next phase, we will validate the criteria with an expert panel that will help us define which criteria should be kept (highly relevant / must have), removed (optional/nice to have), or added in order to co-create an educational toolbox that helps raise awareness about important eHealth assessment criteria," says Christine Jacob, the project lead and the first author of the study. [caption id="attachment_28888" align="aligncenter" width="640"] alt="" width="640" height="629" /> Source: Jacob et al. Assessing the Quality and Impact of eHealth Tools: Systematic Literature Review and Narrative Synthesis.[/caption]   This innovation project is sponsored by Innosuisse (the Swiss innovation agency), with the University of Applied Sciences and Arts Northwestern Switzerland FHNW as the research partner and Roche and KPT Insurance as the practice partners. For more details and resources, please check the project page.