Testing the Effectiveness of mHealth Apps in Training Health Workers

While hitting the books for the 10-year renewal of his board certification in infectious diseases, Christopher Gill, associate professor at the School of Public Health (SPH), noted that this time around, the books were actually web-based tutorials.

As he had several technology-based projects brewing at the same time, this online-learning model for re-certification triggered an idea: could community health workers, many operating in remote areas but with access to cellphones, benefit from information delivered right to their phones?

This was the genesis of a two-year study to test the effectiveness of using brief text messages as learning tools for rural health workers in Vietnam. Mobile health applications, or mHealth, are gaining acceptance as public health tools, but there has been minimal assessment of their performance.

“Our pitch was that there’s shockingly poor data on the efficacy of mHealth,” Gill said. “There are little pilot projects around the world, but little data that it actually works. This is one of the first studies rigorously designed as a randomized trial with controls.” Apparently the National Institutes of Health reviewers who funded the project agreed. “I’ve never had a grant that was accepted so enthusiastically on the first go round,” Gill said. “It seems that we really hit the sweet spot with this one.”

There are about 300,000 health care workers in Vietnam, many of whom are community-based physician assistants (CBPAs), delivering almost all the primary care in the country. Their duties are similar to those of nurse practitioners in Western countries, but CBPAs have less training and do not work under the supervision of a licensed clinician. They are almost completely autonomous, and thus in need of support and training, Gill explained.

There’s an added systemic obstacle: Gill notes that outside of large hospitals and major medical schools, there are limited opportunities for government-mandated continuing medical education (CME) for re-licensing of Vietnamese medical professionals.

“Ultimately, we believe that health workers who know their jobs and are up-to-date on medical information will provide better medical care. In this project, we’re going to show that we can improve their knowledge on the board exams,” Gill said. “Down the road, we can potentially see whether this knowledge leads to actual improvement in care.”

A pool of about 660 study subjects are being recruited from the Thai Nguyen province in northern Vietnam, about 50 miles north of Hanoi, which has a high percentage of CBPAs. Study participants will be tested on their clinical knowledge before being randomized into three groups.

For six months, one group will receive daily SMS messages containing professional reminders involving all aspects of their duties. This group will be required to send back a reply acknowledging that the message was received. Another group will receive a similar SMS on the same topic, but will be required to answer a multiple choice question about the information. A third group will receive no messages and will have no interaction with the study group until the end of the six months. At the end of six months, the health workers will be retested to determine whether the SMS intervention had any effect on scores.

Using software customized for the project by a programming team within the Vietnamese Ministry of Health, the researchers will monitor how messages are distributed, track response rates, and tally the accuracy of answers. Working with the ministry from the start is particularly important for a project of this nature, Gill said, since the ultimate goal—if the strategy is effective—would be to expand the curriculum and scale it for nationwide implementation. The team also includes several key partners: Pathfinder International (an NGO based in Watertown, Massachusetts), the Hanoi School of Public Health, and the Hanoi Medical College.

 “There are so many things that we never thought we would need to consider that have become these amazing little puzzles to figure out.”

In addition, the project has engaged a team of six students from the Schools of Public Health and Medicine to develop the exam questions and SMS text messages for the project. According to Gill, “It turns out that medical students are ideally suited to write multiple choice questions. After having been subjected, and possibly vexed, by taking multiple choice examinations for so many years, med students have formed strong opinions about what kinds of questions are fair and sensible. This has helped us to steer clear of ‘gotcha’ questions, and focus on issues of particular clinical relevance.”

Gill said the project, while just entering its active stages, is proving to be a fascinating exercise in problem solving.

“We’re developing this brand new software using a series of questions that have to fit in 160 characters, so brevity and accuracy are extremely important. And then you have to put yourself in the mindset of these health workers in rural areas to create the type of information that they will need,” Gill said. “There are so many things that we never thought we would need to consider that have become these amazing little puzzles to figure out.”

Gill is the principal investigator on the study, which is funded through the Fogarty International Center of the National Institutes of Health. Lora Sabin, an associate professor of global health, is a co-investigator helping to develop the analysis of cost-effectiveness. Julia Rohr, who recently earned her PhD from SPH, is the statistician for the project, which is being managed by Ariel Falconer, an MPH student at SPH concentrating in global health.

“Right now we haven’t shown clinical impact, just feasibility. Proving that it actually works in a clinical setting is the big tuna, and would likely need an additional series of trials,” Gill said. “There are going to be a very long set of important discussions that we need to have.”