Why The Accuracy Of SARS-CoV-2 Antibody Tests Varies So Much

Due to increasing demand from both researchers and policymakers, the number of antibody tests for SARS-CoV-2, the virus that causes COVID-19, has skyrocketed in recent weeks. According to a list kept by the nonprofit Foundation for Innovative New Diagnostics (FIND), more than 200 of these products, which are also known as serologic tests, are either now available or in development. Many of the test manufacturers are based in China, but there are also companies in a number of other countries, including South Korea, Germany, the United States, and the United Kingdom.

Only 12 have received emergency use authorization (EUA) from the US Food and Drug Administration (FDA), which gives companies permission to deploy a product without providing the same amount of supportive evidence as required in the typical approval process. In March, the FDA announced that, to expedite availability, companies could market these tests in the United States without EUA as long as they conducted their own evaluation. On May 4, the agency announced it would heighten the scrutiny of these tests after being criticized for the deluge of flawed tests that have become available in the United States.

“At this point we have more commercially available serologic tests for SARS-CoV-2 than any other infectious disease,” says Elitza Theel, the director of the Mayo Clinic’s Infectious Disease Serology Laboratory. “It is crazy.”

As researchers and policymakers around the world rush to put these tests to use, concerns have been raised about their precision. For example, when a widely publicized study on the rate of infection in the California county of Santa Clara raised alarms, a key criticism was the inaccuracy of the antibody test used by the group. In other cases, tests have turned out to be too inaccurate for use at all—for example, last month, the British government shelved a rapid antibody test it had paid $20 million for because it turned out to be inadequate.

Several groups are now conducting independent assessments of these antibody tests. Their work suggests that there is wide variability in performance, and that the accuracy of a test can depend not only on the test itself, but factors such as when it is conducted and how a user interprets the result.

How antibodies test work

There are two broad categories of antibody assays: lab-based tests and rapid tests. One of the most common techniques used in lab-based tests is the enzyme-linked immunosorbent assay (ELISA), where a sample—blood, serum, or plasma—is placed on a plate coated with a viral protein. If a person has antibodies against the virus, they will bind the protein. To identify these viral protein–antibody complexes, scientists introduce a molecule that will fluoresce in the presence of the bound molecules. These tests can provide a quantitative measure of the antibodies present in a sample.

Rapid tests instead use a lateral flow assay (LFA) format. The underlying principle of an LFA is similar to an ELISA, but these are carried on a device that resembles a pregnancy test. They typically require a small amount of blood gathered from a finger prick. The blood flows through a strip that has viral proteins on its surface, and if antibodies are present in the blood, they attach to those proteins. The resulting complexes will then bind to molecules immobilized on the strip, and if a threshold—which varies by manufacturer—is reached, the test will display one or two colored lines that indicate a positive result. Unlike ELISAs, these tests only generate a binary “yes” or “no” result.

Lab-based tests usually take a few hours to generate a result, while rapid tests can be completed in 30 minutes or less.

A test’s performance depends on two measures: sensitivity and specificity. Sensitive tests generate few false negatives and specific tests lead to few false positives. How accurate a test needs to be depends on what it is being used for, says Rangarajan Sampath, the chief scientific officer of FIND. For seroprevalence studies, which seek to identify the proportion of a population who have been infected with the virus, for example, he says that more than 98% specificity and more than 90% sensitivity are desired.

One of the antibody tests most recently approved for EUA by the FDA, a lab-based assay by Roche, boasts 100% sensitivity and 99.8% specificity.

Accuracy assessments

Several groups are now independently evaluating antibody tests for these measures. One study, dubbed the COVID-19 Testing Project and run by physicians and researchers at institutions including the University of California, San Francisco (UCSF), the University of California, Berkeley (UCB), Massachusetts General Hospital (MGH), and the Chan Zuckerberg Biohub, evaluated 10 LFAs and two ELISAs. To assess their ability to identify antibodies against SARS-CoV-2, the team used plasma or serum samples from three groups: 80 people who had shown symptoms of COVID-19 and had tested positive using a PCR-based screen, 52 who had a respiratory infection but were found to be infected with another virus or had tested negative on a PCR test for SARS-CoV-2, and 108 blood donors whose samples were drawn in 2018 or earlier, before the pandemic began.

Their assessment found that the ability to detect antibodies in people who had tested positive for the virus increased over time, rising to 81% to 100% when more than 20 days had elapsed since symptoms began, depending on the product. One of the members of the team, Patrick Hsu, a bioengineer at UCB, notes that this finding highlights why longitudinal antibody testing is important, given that a negative result may mean a person had been exposed to the virus but hadn’t yet developed a detectable level of antibodies. On the specificity side, the proportion of false positives found in the pre–COVID-19 samples ranged from 0% to 16%. The agreement between the findings of LFAs and ELISAs ranged from 75% to 94%. The team posted its results as a preprint on the project website on April 24. Alex Marson, an immunologist at UCSF and a coauthor of the report, cautions that some numbers, especially for tests’ ability to detect antibodies in positive cases, may be revised as his team continues to analyze the data.

“Some tests I would not consider adequate to be used for serology testing, whereas others were quite good,” Hsu says. A single rapid test, by the Chinese company Bioperfectus, detected antibodies in 100% of the PCR-positive samples—but only after more than 20 days from symptom onset. It did not demonstrate 100% specificity, however. For that measure, only one rapid test—an assay developed by the US- and Hong Kong–based company Sure Biotech—did not give any false positives.

According to the The New York Times, one of the tests examined in the study, by the Chinese company Innovita, has been ordered by Los Angeles–based start-up Scanwell Health, which hopes to market the product for at-home use, pending FDA authorization. That test identified antibodies in up to 83% of people with confirmed infections and has a specificity of 96%.

Another one of the tests included in the study, which is sold by the US-based Premier Biotech and manufactured by the Chinese company Hangzhou Biotest Biotech, was used in the controversial Santa Clara seroprevalence survey. In the medRxiv preprint about that study, the authors cited a specificity of 99.5% and sensitivity of 82%. Hsu and his colleagues found that the test’s specificity was 97%—and it identified antibodies in 37% of samples from one to five days after symptom onset and in 90% 20 days later (the Santa Clara study did not look solely at symptomatic individuals, and Hsu’s study did not examine samples from people with asymptomatic infections.)

A few similar assessments have been published in recent weeks. One of the most recent studies, conducted by a group at the University of Washington, found that a lab-based antibody test developed by the US company Abbott had a sensitivity reaching 100% 17 days after symptom onset and only registered one false positive in 1,020 samples collected prior to the pandemic. “These data demonstrate excellent analytical performance,” the authors write in a preprint posted on medRxiv on May 2.

Other studies have reported less promising numbers. One conducted by the National COVID Testing Scientific Advisory Panel in the UK examined nine LFA devices and found that at 10 days after symptom onset or later, sensitivity ranged from 61% to 88% in samples that had tested positive for SARS-CoV-2 using a PCR test. Specificity, as determined by whether the test would pick up false positives in pre-pandemic samples, ranged from 95% to 100%. “Our findings suggest that while current [LFA] devices may provide some information for population-level surveys, their performance is inadequate for most individual patient applications,” write the authors in their preprint, which was posted on medRxiv in April.

Overall, “the trend that we’re starting to see [with rapid tests] is that there is inconsistency in many of [the manufacturers’] claims” about performance, Sampath says.

Lab-based tests vary as well. Theel notes that her team has found variability in the accuracy of several ELISA-based tests for COVID-19. And a study conducted by a group in Denmark, which was posted as a preprint on medRxiv last month, also reported variability in commercially available ELISAs.

The assessments to date “underscore why it’s so important that we don’t just take these tests off the shelf and start using them,” Theel says. “We have to verify the performance characteristics, otherwise you’re going to be calling many people falsely positive and others potentially falsely negative.”

More evaluations are currently underway. FIND is carrying out its own studies of serology tests at sites around the world including in the United States, Europe, South America, and Africa. The FDA is collaborating with other governmental institutions to assess these tests—and may use these data to inform whether or not to issue future EUAs. And according to Hsu, several test manufacturers have reached out to him and his collaborators since their preprint was posted asking for their products to be tested as well—and the members of the COVID-19 Testing Project plan to conduct more of these assessments.

What makes a test accurate?

There are a number of variables that play into the accuracy of a test. As the evaluations to date have shown, antibody tests become more sensitive at later time points. How a test result is interpreted is also important. According to Alex Marson, a member of the COVID-19 Testing Project and an immunologist at UCSF, there was a point during their recent study when his group in California and their collaborators at MGH were interpreting a faint band differently—one considered it a positive result, while the other did not. Many of these tests don’t give clear guidance on how to interpret a result like that, Marson says. “If these tests are going to be deployed, whoever is going to be interpreting them needs to get really strict instructions on what’s going to be a positive and negative.”

Features of the test itself, such as the antibody it is designed to detect, can also affect accuracy. The two antibodies used in most available antibody tests are IgM and IgG. When a SARS-CoV-2 infection occurs, IgM appears to be the first responder, arriving on the scene within a week or so of infection. A second antibody, IgG, generates a more specific and longer-lasting response—but it shows up later, when the viral RNA signal may have already faded away.

“You’re looking for IgM so that you’re not missing a period in between when the RNA might be variable, and in some cases, undetectable,” Sampath explains. The downside is that, based on the assessments conducted so far, IgM signals seem to be less robust than IgG. Many tests combine the two. All the tests examined in Tsu and Marson’s study were capable of identifying both, and the team’s assessments showed that IgM detection was more variable than IgG, and that detection rates were highest when both were combined.

Another variable is the viral protein, also known as an antigen, that’s used to bind the antibodies. The two that have been most widely used in antibody tests to date are the nucleocapsid protein, which forms the shell around the viral RNA, and the spike protein, which studs the surface of the virus and binds to host cells. According to Sampath, because the nucleocapsid proteins tend to be more conserved across different viruses, tests that use them may be less specific than those that use the spike protein. For this reason, he adds, spike proteins have become the preferred reagent in newer versions of these tests.

In addition to those variables, how the viral proteins are produced—what type of cell line is used, for example—can also affect the performance of an assay. “It’s not just which antigen but in how is that antigen produced,” Theel says.

At the end of the day, there’s also a huge caveat to consider, says Kamran Kadkhoda, an immunologist at the Cleveland Clinic. “The elephant in the room,” he says, is the lack of robust science behind the usefulness of this test for clinical or public health purposes, given that there are still big open questions, such as what the presence of antibodies actually means about immunity and how long antibodies remain after an infection has passed.

The bottom line is that more independent evaluations are needed to determine how good these tests are—and more research is necessary to understand what their results can tell us. “For serology, I really think the best is yet to come,” Theel says. “Once we identify whether or not, or for how long, protective immunity lasts and how these antibody tests are related to such immunity, I think that’s when they’ll really be important.”

Source