We investigated whether digital contact tracing actually worked in the US

In the spring of 2020, the first versions of covid-19 exposure notification systems were released to the public. These systems promised to slow the disease’s spread by providing automated warnings to people who came into contact with the virus. Now, over a year later, residents in over 50 countries—including half of US states—can opt into these systems.

But the big question remains: how well did this technology work? Some studies suggest answers, but despite such wide rollout, it’s difficult to evaluate whether exposure notifications were actually able to stall covid-19 spread. This is especially true in the US, where many states launched their own apps—a decentralized approach that reflects America’s fragmented pandemic response.

In an attempt to learn more about how this technology fared in the US, MIT Technology Review reached out to every state public health department that launched a digital contact tracing system and examined app reviews left by anonymous Americans. We asked two questions: who is actually using this technology, and how do people feel about it?

The end result of this analysis paints a picture of unexplored potential. Many of the country’s exposure notification apps are underutilized, misunderstood, and not well-trusted—and yet this technology may yet come into its own as a public health tool for future disease outbreaks.

How the technology works

Exposure notifications were first put forward as a complement to traditional contact tracing. Under the traditional manual approach, investigators looking for people who may have been infected ask patients to trace their whereabouts and activities through phone calls and interviews. The new technology promised to scale to cover entire populations automatically rather than just small disease clusters— a distinct advantage for tracking a fast-spreading disease.

You might remember the friend you met for lunch, for example, but not the stranger you stood next to in line for 15 minutes at the grocery store. An exposure notification system does the remembering for you, anonymously using Bluetooth to keep a log of nearby phones and alerting you if one of those phones is associated with a positive test result.

The first wave of this system was designed by cooperatives of developers, most of whom ended up collaborating with Apple and Google to create a uniform standard. The Apple-Google system prioritized privacy for users, anonymizing their data, and did not track users locations. With the backing of the world’s two most dominant phone platforms, this system is the one that’s been most widely adopted, and is used by the vast majority of US states.

The effectiveness of these systems has been notoriously hard to evaluate. Studies are just now starting to come out about apps in the UK and Switzerland, for example. In the US, evaluation is made even harder by the fact that every state is basically doing its own thing. But our analysis does have a few takeaways:

US systems were launched relatively late in the pandemic—when the country’s fall/winter surge was mostly already in progress
The technology has not been widely adopted, though some states are faring better than others
A lack of public trust in new technology—coupled with a lack of resources in the public health agencies peddling that technology—hampered both adoption rates and how people used the systems

Who’s using this tech

We tracked exposure notification apps that had been rolled out in 25 states and the District of Columbia. Virginia was the first state to make the technology publicly available to its residents in August 2020, while others are still only getting started now. Massachusetts began testing its app with a pilot in two cities in April 2021, while South Carolina is currently running a pilot program at Clemson University. The state actually started work on its system back in May 2020—but legislators barred the public health department from any digital contact tracing work last summer due to privacy concerns, holding back development.

Even in the states where such apps are available, not everybody can use them. Exposure notifications are only available for smartphone users; and about 15% of Americans don’t have a smartphone, according to Pew Research Center. Still, over half of the US population can now get plugged in. Whether they choose to join those systems is another matter.

As the vast majority of states do not publicly report user data, we reached out to state public health departments directly to ask how many people had opted into the technology.

Twenty-four states and DC shared user estimates, showing that, by early May, a total 36.7 million Americans have opted in to the notifications. Hawaii has the highest share of its population covered, at about 46%. In four more states, more than 30% of residents opted in: Connecticut, Maryland, Colorado, and Nevada. Seven more states have over 15% of their populations covered.

That proportion is important: modeling studies have determined that if roughly 15% of a population opts into the system, it could significantly reduce a community’s covid case numbers, hospitalizations, and deaths. By this metric, 13 states—which together represent about one-third of the US population—have seen some degree of protection thanks to exposure notifications.

The remaining 11 states with exposure notification apps fail to meet this benchmark for success. Of those 11, three states have under 5% of their populations covered: Arizona, North Dakota, and Wyoming. South Dakota, the one state which did not respond to a press request, shares use of the Care19 Diary app with the low-activation states of North Dakota and Wyoming.

Comparing states isn’t perfect, though, because there are no federal standards guiding how states collect or report the data—and some may make very different choices to others. For example, while DC reports an “exposure notification opt-in” number on its Reopening Metrics page, this number is actually higher than its residential population. A representative of DC Health explained that the opt-in number includes tourists and people who work in DC, even if they reside elsewhere. For our purposes, we looked at DC’s activation rate as a share of the surrounding metropolitan area’s population (including parts of nearby Maryland, Virginia, and West Virginia).

Another reason these rates are hard to measure: Several of the states with higher usage rates benefit from a major upgrade that Apple and Google released in September: Exposure Notification Express, or ENX. This framework made it much faster for states to spin up apps, and it also invited millions of iPhone users to avoid downloading anything at all. They could activate the notifications simply by flipping a switch in their phone settings.

ENX activation is much more convenient, and experts say it may seem safer than downloading a new app. It has seriously boosted activation rates for states that use it. Hawaii, for example, saw its users more than double from February to May while rolling out ENX.

The express system does mean we have less precise user data, though. States aren’t able to track ENX activations directly, and instead need to rely on Apple for their numbers.

Beyond the numbers

Even when a lot of residents have downloaded an app or turned that switch in their iPhone settings, the system still needs to be properly used in order to make a difference in covid cases. So we tried to understand how people were using the systems, too.

A recent study found that Americans were hesitant to trust digital contact tracing technology. However, this finding was based on surveys conducted before most states even launched their apps. As a proxy for public attitudes towards the US state apps, MIT Technology Review scraped and analyzed app reviews from the Google Play store. We only looked at Google Play reviews (from Android users) to get the most current and consistent data. (Most iPhone users can now turn on notifications without downloading an app.)

Looking at app reviews isn’t a perfect system. Users who chose to review their state’s app are not a representative sample of the EN-activating population—instead, they are those users who want to share strong opinions about the technology.

Still, here’s what we found:

Most of the state apps have average ratings between 3 and 4.
Michigan has the lowest score, at 2.6.
D.C, California, New York, Delaware, and Massachusetts have the highest scores, over 4.

Many 1-star reviewers appeared to misunderstand how their state’s app works, didn’t trust in the technology, or were unable to understand how the app fit into the broader public health system. This indicates that, for many Americans, the app wasn’t doing its job even though it was technically in use.

Lessons from negative reviews

Poor reviews provide a glimpse into common issues and misconceptions that the digital contact tracing system faced.

Small glitches made a big difference.
Over and over, reviewers stated that they got tripped up by needing an activation code. To help protect privacy, when you test positive for covid you don’t input your name or other identifying details into the app: instead, you enter a string of numbers that your public health department gives you. Some reviewers state that they don’t know where to get an activation code after testing positive, or that they ran into error messages. We’ve heard from developers in other countries about this issue.

Some US states and other countries have streamlined the process by automating how a code gets sent, but in many cases, users must wait for a contact tracer to call them. This waiting period can decrease trust in the technology, and it significantly slows down digital contact tracing.

“Trust” isn’t just about the app itself. It’s broader than that.
Many app reviewers also mistrust new technology, the government, or both. A Pew Research Center survey conducted in July 2020 found that 41% of Americans would likely not speak with a public health official on the phone or via text message, and 27% said they would not be comfortable sharing the names of recent contacts—both key elements of the contact tracing process.

Digital contact tracing faces similar challenges. Some reviewers felt so strongly about protecting their privacy that they came to their state app’s pages in order to boast about their refusal to download this technology. Many echoed the sentiments of this reviewer from Pennsylvania: “Open access to my wifi, GPS, and Bluetooth? Creepy. No thanks, Harrisburg.”

Low usage creates a downward spiral of mistrust.
One crucial aspect of digital contact tracing is that you need participation for it to work—at least 15% of the community, but preferably much higher. When people aren’t participating, the chance of getting a match is lower—even if covid levels are high—and so the system likely won’t send out alerts to those small number of people who do have exposure notifications activated.

A few reviews went as far as to beg the other residents of their states to opt into exposure notifications, reminding fellow reviewers that higher usage leads to higher effectiveness in a tone that seemed more reminiscent of a Facebook argument than an app store.

This may have been confusing for essential workers or other frontline staff who knew for a fact that they were being exposed. People who opted into the system hoping for protection—only to get silence—may have been discouraged into thinking the technology simply doesn’t work at all.

For example, one New Jersey reviewer who claimed to work in an emergency room expressed frustration at how they “hadn’t been alerted once” even after direct contact with covid-19 patients. Those who felt like their app wasn’t working may have discouraged others from downloading it—when in fact more users was exactly what was needed for the system to be successful.

Lessons from positive reviewers

What about reviews that rated apps highly? Here’s what we found:

Trust continues to be a big issue.
Positive app reviewers tried to combat mistrust with explanations and endorsements. Most of the state apps, in fact, have more five-star ratings than any other category—an encouraging signal that thousands of Americans were willing to give the technology a try.

Many people are trying to do the public health system’s job.
Some of these positive reviewers identified themselves as experts in the tech space, such as software engineers and security analysts. They posted explanations about how the digital contact tracing system works and reassurances that no, this app will not share your location with the government.

One security expert from Colorado wrote, “Please, please. Use this app. Control the spread.”

One security expert from Colorado wrote, “Please, please. Use this app. Control the spread.”

While these efforts may seem heartwarming, such reviews mask a more fundamental problem: it should be public health authorities doing the trust-building, not random commenters. And they should have the resources to do it in a more effective forum than the Google Play store.

Public health agencies lacked resources.
These reviews—both positive and negative—show that public health systems faced a lot of pressure to get it right. But in actuality, state health agencies could only do so much with the budgets and capacity that they had. Public health workers in New Jersey, for example, said they relied on a free marketing toolkit and ad space provided by Google. While the agency did work with county health departments and social media for more targeted advertising, its efforts were limited because most workers had to deal with other facets of the covid response at the same time.

“I believe that getting buy-in from communities for any new technology requires deep investment,” says Pardis Sabeti, a computational geneticist at Harvard and MIT who has developed covid-19 apps for higher education institutions.

One criticism of these systems in the US is that they were a patchwork without federal leadership. However, when implemented correctly, patchworks can be successful. Some experts have suggested tying apps to trusted institutions to build what experts have called “the piecemeal creation of public trust”—gaining trust in one local community at a time.

Sabeti’s work may be an example of this strategy in action. On campuses, her team has engaged with students and other community members to test covid-related digital tools in small settings, before they’re released to the broader student body. The process requires “constant communication between the individuals using [the tool] and the developers, elements that empower every actor in the system, and education throughout, so that communities understand how to use the available tools wisely.”

Long-term investment is especially difficult for US state public health departments, which were underfunded for decades before the pandemic. “Public health authorities have not been able to sustain the messaging over time in terms of attracting people to download the app,” says Jenny Wanger, who runs programs on covid technology for Linux Foundation Public Health, a software development network.

This type of trust-building is especially important for marginalized communities, who have many legitimate reasons to distrust the government. While we don’t have any demographic data on exposure notification users, Sabeti points out that they may also be less willing to opt into the technology, which “greatly affects the uptake and overall effectiveness of these tools.”

Still in a trial run

Even after our analysis, it’s still hard to answer that crucial question: how many infections were actually prevented by exposure notifications?

Our failure to answer that question is partially due to the fractured nature of the system. But it’s also because specific research to measure this technology’s effectiveness simply was not a priority.

Rafi Yahalom, a cybersecurity researcher at MIT, says that while the UK and Swiss analyses suggest that exposure notifications had a potentially significant impact on spread, he would like to see those studies replicated in the US and other communities with lower activation rates.

And he says that if states had wanted to do so, they could have devised a simple evaluation: when someone comes in for a covid test, ask them if they received an exposure notification. Such a study may help determine whether those who get notified are actually more likely to be infected. After all, there are still a lot of questions regarding how the Bluetooth system should even be configured—it was originally based on CDC guidance stating that a 15-minute, under-6-feet interaction posed an infection risk, and we now know the true infection patterns are far more complicated.

So why didn’t any US public health agencies do a study like that? It’s about priorities. State departments did not have the time or resources to look at how well digital contact tracing works; instead, they were focused on doing anything and everything to stop the spread of the virus.

“The ultimate goal [of exposure notifications] is for more folks to know they’ve been exposed,” says Hanna Sherrill, an Eagleton Science and Politics Fellow at Rutgers University who worked with the New Jersey public health agency on its exposure notifications system. “Hopefully some of them will take the advice to quarantine, and then they will stop the spread from there. Even if there’s one or two people who do that, that’s a good thing from our perspective.”

Other state public health staffers who responded to Technology Review’s data requests echoed her sentiment—and their attitudes suggest that digital contact tracing in the US may still be in its trial run. We have 26 different prototypes, tested in 26 different communities, and we’re still trying to understand the results.

“In the US, the existing apps and tools have never hit the level of adoption necessary for them to be useful,” Sabeti says. But such success may not be out of reach for future public health crises.

She believes that future crises may require us to closely pair digital contact tracing with genomic sequencing and other surveillance methods. Colorado Mesa University has pioneered such a system with the help of Sabeti and the rest of the Broad Institute. The school’s covid surveillance incorporates sequencing data from student tests, self-reported student symptoms, wastewater testing, contact tracing, and more—all compiled into geographic heat maps that administrators can use to pinpoint outbreaks before they become serious.

They are already thinking about how the school’s system may be expanded for other communities. If America invests in these technologies now, hones those prototypes, and builds community trust, it may be ready for the next pandemic.

“It is truly possible for us to actually get to the point where we have a real upper hand on containing viruses, by working collectively as a community to track viral spread anywhere in the world,” she says.

This story is part of the Pandemic Technology Project, supported by The Rockefeller Foundation.