Abstract
The recent emergence of the SARS-CoV-2 pandemic has posed formidable challenges for clinical laboratories seeking reliable laboratory diagnostic confirmation. The swift advance of the crisis in the United States has led to Emergency Use Authorization (EUA) facilitating the availability of molecular diagnostic assays without the more rigorous scrutiny to which tests are normally subjected to prior to FDA approval. The need to identify the COVID-19 positive cases quickly and accurately has propelled the release of a variety of assays intended to meet the urgent demand. Several Nucleic Acid Amplification Tests (NAAT) platforms are currently available. Our laboratory currently uses two real time RT-PCR platforms, the Roche Cobas SARS-CoV2 and the Cepheid Xpert Xpress SARS-CoV-2. Both platforms demonstrate comparable performance; however the run times for each assay are 3.5 hours and 45 minutes, respectively. In search for a platform with shorter turnaround time, we sought to evaluate the recently released Abbott ID NOW COVID-19 assay which is capable of producing positive results in as little as 5 minutes. We present here the result comparisons between Abbot ID NOW COVID-19 and Cepheid Xpert Xpress SARS-CoV-2 using nasopharyngeal swabs transported in VTM as well as dry nasal swabs for the Abbott assay. Regardless of method of collection and sample type, Abbot ID NOW COVID-19 missed a third of the samples detected positive by Cepheid Xpert Xpress when using NP swabs in VTM and over 48% when using dry nasal swabs.
Introduction
The novel coronavirus, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) causing coronavirus disease-2019 (COVID-19) was first identified in Wuhan Jinyintan Hospital, China in December 2019. Since then, it has spread vastly, reaching the status of global pandemic as declared by the World Health Organization on March 11th, 2020 (1). As of May 2nd, 2020, there are 3,356,205 reported cases, and 238,730 confirmed deaths worldwide, in 215 countries (2). In NY State and New York City, there were 316,415 and 168,845 reported cases and 19,189 and 13,319 deaths respectively as of May 2nd, 2020 (3, 4).
Common signs and symptoms with the associated respiratory illness are fairly non-specific and include fever, dry cough, headache, myalgias, arthralgias, and sore throat, with indicators of more severe disease such as shortness of breath and respiratory failure (1). Nearly 80% of the infected individuals are either asymptomatic or have mild symptoms leading to difficulties to diagnose this disease. COVID-19 has high human-to-human transmission rates (R0 = 2.0 – 2.5), and is stable in aerosols and on surfaces, presenting a challenge to mitigation and emphasizing the urgent need for a more rapid, accessible and accurate diagnostic assays (5).
SARS-CoV-2 virus is detected in various human sampling sources including respiratory and fecal, but testing has been validated for PCR primarily of nasal, nasopharyngeal, and oropharyngeal specimens. Positive samples for SARS-COV-2 were found in Chinese patients 1-2 days prior to symptom onset and persisted up to 2 weeks in severe cases (1).
While long-term mitigation and prevention goals include prognostic markers, therapeutics, and vaccines (1), the urgency of this rapidly developing crisis in the United States has prioritized the immediate goal of diagnostic testing, advanced by the FDA’s Emergency Use Authorization (EUA) (5, 6). The initiative to expand and accelerate testing has eased the usual scrutiny that new assays would normally undergo prior to release. This obligates clinical laboratories to more carefully assess the overall performance of the various rapidly emerging testing platforms prior to implementation. Several Nucleic Acid Amplification tests (NAAT) platforms are currently available. Our laboratory uses two real time RT-PCR platforms, the Roche Cobas and the Cepheid GeneXpert. Both platforms perform comparably with similar limits of detection (LOD) of SARS-CoV-2 viral RNA, 100-150 and 250 copies/mL, respectively. However, the assay run times of 3.5 hours and 45 minutes, respectively, are still too long for timely decision support in a variety or important clinical situations. (e.g. Emergency Department discharge). In search for a platform with shorter turnaround time, we sought to evaluate the recently released Abbott ID NOW COVID-19 assay. Abbott ID NOW uses isothermal nucleic acid amplification of the RdRp viral target with a claimed LOD of 125 copies/mL. The objective of our study was to evaluate the performance of the rapid Abbott-ID-NOW platform test by using the Cepheid-Xpert-Xpress SARS-CoV-2 test on the GeneXpert Dx as the comparator reference method.
Materials and Method
All samples were obtained from patients seen at the New York University Langone Tisch Hospital with diagnosis of suspected COVID-19. Nasopharyngeal (NP) samples were obtained using flocked NP swabs and transported in universal viral transport media (VTM) (Cepheid Xpert Viral Transport Medium; SWAB/B-100) or BD Universal VTM / 2250527) within 1-2 hours of collection.
For the assessment of performance of dry swabs, we obtained 101 dry nasal swabs of both nares as well as 101 NP swabs in VTM from the same patient on the same ED encounter to be tested using both, the Abbott ID NOW COVID-19 and Cepheid Xpert Xpress SARS-CoV-2, respectively. The dry nasal swabs, which are supplied with the Abbott assay (Puritan Medical Products 25-1506 IPF 100) were transported in a sterile tube to the laboratory along with the corresponding NP swab in VTM.
Instruments and Methods
ID NOW COVID-19
is a rapid test that qualitatively detects SARS-CoV-2 viral nucleic acids from nasal, nasopharyngeal and throat swabs. It is an automated assay that utilizes isothermal nucleic acid amplification technology that eliminates the need for denaturation by thermocycling. The test includes a sample receiver containing elution/lysis buffer, a test base comprising two sealed reaction tubes containing the reagents required for amplification of SARS-CoV-2, an internal control, a transfer Cartridge to transfer the eluted sample to the test base, and the ID NOW Instrument. The templates which are similar to primers, are intended to target SARS-CoV-2 RNA and amplify a unique region of the RdRp genome. The amplified RNA target is detected with the use of fluorescent-labeled molecular beacons. Target amplification starts after adding the samples to the sample receiver and transferred with the transfer cartridge to the test base. The instrument provides heating, mixing and detection. Positive results are available within 5-13 minutes and negative results within 13 minutes.
Xpert Xpress SARS-CoV-2 test
is a rapid, real-time RT-PCR test detecting ribonucleic acid (RNA) from the SARS-CoV-2 in nasopharyngeal swab and/or nasal wash/aspirate specimens collected from individuals with suspected COVID-19 diagnosis. The specimens were collected, placed in VTM and run on the GeneXpert Dx. This platform automates and integrates sample preparation, nucleic acid extraction and amplification, and detection of the target sequences. The assay detects 2 nucleic acid targets, namely N2 (nucleocapsid) and E (envelope) wherein N2 is more specific for SARS-CoV-2. The lowest limit of detection (LOD) for this assay is claimed by the manufacturer to be 250 copies/mL. SARS-CoV-2 genomic load is assessed using the number of amplification cycles needed for a positive PCR test (i.e. the cycle threshold [Ct] value). Ct values have an inverse relationship to viral loads and thereby provide a surrogate measurement of the viral genomic load (7). Consequently, the higher the viral load, the lower the Ct value of a particular specimen and vice versa.
Both assays are used only under Food and Drug Administration’s Emergency Use Authorization (EUA).(8)
The performance evaluation of the Abbott ID NOW assay was conducted in several steps.
Initial verification of the Abbott ID NOW platform
included two previously tested patients’ samples using the Xpert Xpress assay, one with high a Ct value 30.8 for N2 and another with low Ct value of 19.3 (i.e. high viral load). Both samples were diluted 1:2, 1:5, 1:10, 1:20, 1:50 and 1:100 in VTM. All the dilutions were tested separately on both platforms.
Assessment of performance of Abbott ID NOW on NP samples transported in VTM
This included 15 sequential NP specimens submitted in VTM by our Emergency Department from patients with suspected COVID-19 disease. Each specimen was tested on both, the Cepheid GeneXpert and Abbott ID NOW platforms.
Determining Abbott ID NOW performance according to N2 Ct values obtained by the Cepheid Xpert Xpress assay
For this analysis, rather than using specimen dilution to simulate a series of decreasing viral loads, a set of 8 previously tested specimens in the Xpert Xpress assay were selected with increasing N2 Ct values ranging from 32.6 to 41.8. All samples were tested on the ID NOW platform and the results compared to those previously obtained on the Cepheid GeneXpert.
For all the comparisons mentioned above, all samples were nasopharyngeal swabs transported to the lab in VTM. In response to our findings of lower sensitivity of the ID NOW compared to the GeneXpert, Abbott communicated to us that despite the indications in the package insert, their assay was not optimized for testing nasopharyngeal swabs in VTM; the best performance would be from dry nasal swabs obtained at the point of care (POC). They indicated that they would be modifying their package insert to reflect this change (ID NOW COVID-19 Product Insert, IN190000 Rev.3 2020/04:6-8.). Since the Abbott ID NOW rapid results feature was regarded as a worthwhile advantage over our other assays in use, we compared dry nasal swab (both nares) tested on the Abbott platform versus NP in VTM tested on the Cepheid platform. Both samples were obtained in parallel from patients with suspected COVID-19, seen at the NYU Langone Health Tisch Hospital Emergency Department. Although Abbott indicated that the dry swab could be transported in the same sleeve in which the swabs were packaged, we considered the safety of this modality and chose a sterile tube instead. A total of 101 paired samples collected during 12 hour shifts over 3 consecutive days were included. Ninety-three percent of the samples were transported to the laboratory within 1 hour for testing, the remainder within 2 hours. Testing at the point of care was not practical.
Statistical analysis
Qualitative Method Comparison was performed using EP Evaluator (Data Innovations, VT 05403) and correlation coefficient using Excel 2010.
Results
Results of the initial verification of the Abbott ID NOW revealed that it performed well with respect to the reference method when a specimen (Sample 1) with low N2 CT value (9.3) was progressively diluted up to 1:100., but this was not the case for Sample 2 with had an initial higher N2 Ct value (30.8) (Table 1). Decreased sensitivity was noted at 1:10 dilution level (approximated Ct=33.8) and beyond with only a 33% agreement at 1:100 (approximated Ct value of 36.8).
Comparison of results obtained on 15 sequential NP specimens submitted in VTM by our Emergency Department produced 5 false-negative results by Abbott ID NOW (33.3%) (Table 2). The Cepheid N2 Ct values of the discordant specimens ranged from 36.3 to 44.3.
In order to confirm that the lower sensitivity of the Abbott ID NOW platform was related to the Ct value obtained on the GeneXpert Dx, we used a series of previously tested NP specimens with increasing N2 Ct values (Table 3). Similar to the dilution studies, the performance of the Abbott ID NOW started to decline as the N2 Ct values increased throughout the series, although at a higher Ct value (38.0) than that seen in our previous comparisons (∼ 33.8, 36.3, respectively). That is, beyond a Ct value of 38, ID NOW performance became increasingly less reliable.
The distribution of the Ct values of 1439 positive samples previously tested in the Xpert Xpress over the previous 1 ½ months in our laboratory revealed that 15.1% of total positive samples had Ct values above 38 (data not shown).
For the dry nasal swabs versus NP samples in VTM we included samples from 101 Emergency Department patients (age 28-90) with suspected COVID (onset of symptoms 1 day to 1 month). Patients were sampled twice during the same encounter using a nasal dry swab of both nares and a NP specimen submitted in VTM. As seen in Table 4, the Abbott ID NOW using dry nasal swabs detected 51.6% (16 of 31 positive samples detected by Cepheid Xpert Xpress). The remaining 15 samples were falsely negative. The sensitivity of the Abbott ID NOW platform according to the N2 Ct values obtained with the Cepheid Xpert Xpress is shown in Table 5. All 6 dry nasal swabs with their corresponding NP in VTM having a N2 Ct value of <33.5 by Cepheid tested positive by Abbott ID NOW. But sensitivity of Abbott ID NOW was 45% (5 of 11) and 35.7% (5 of 14) when N2 Ct values obtained by Xpert Xpress were between 33.7 and 38, and >38, respectively.
There was no correlation between N2 Ct and days of onset of symptoms (R2 = 0.097). Sixteen of the 31 patients whose samples tested positive for SARS-CoV2 RNA by Xpert Xpress were admitted; 6 of these patients had negative results when tested by Abbott ID NOW.
Discussion
The urgency of the COVID-19 pandemic and the United States government’s subsequent use of the Federal Authorization Act has placed an unusual onus of responsibility on the clinical laboratories to substantiate the clinical value and performance of the COVID-related tests they introduce. Accurate diagnostic testing and subsequent isolation of sick individuals is crucial to mitigating continued infectious transmission. Therefore, assay sensitivity must be evaluated on the many newly available COVID-19 testing platforms to minimize the public health risks that come with false negative results.
Our study used several approaches in an attempt to validate the rapid testing methodology by Abbott ID NOW COVID-19 in comparison to Cepheid Xpert Xpress SARS-CoV-2. The appeal of the Abbott technology stemmed from our intention to provide rapid and accurate results in our institution, especially for our Emergency Department.
Our results showed progressively more false negative results with lower viral load (higher N2 Ct values) as determined by Cepheid, but this was not consistent; there were samples testing positive by Abbott platform even when N2 Ct values were above 38 when we did the parallel comparison of dry nasal swabs versus NP samples in VTM. A study from Northwell Health has shown similar discordant results comparing these two assays, with ID NOW COVID-19 producing 7 false negatives out of 107 nasopharyngeal samples, resulting in a sensitivity of 87.7% compared to 98.3% on Xpert Xpress SARS-CoV-2. Their study however, used NP swabs transported in VTM, which the manufacturer claims decreases the sensitivity of their assay. These authors also determined that the GeneXpert had the lowest limit of detection at 100 viral copies/mL, while the ID NOW had a limit of detection of 20,000 copies/mL (9) which is well above the limit of detection stated in the Abbott ID NOW COVID-19 package insert (125 viral copies/mL).
Using a larger sample size, preliminary studies at the Cleveland Clinic also revealed a lower sensitivity for the Abbott ID NOW, with a false negative rate of 14.8%. However, the samples were all stored in viral transport media (10). Although the initial Abbott package insert had stated that 0.5 to 3.0 mL of viral transport media was acceptable for use in their assay, it also recommended minimizing dilution of the sample is as it may result in decreased test sensitivity. The use of dry swabs is now recommended by the manufacturer for optimal test performance. To investigate potential improvement in test performance resulting from this change, we evaluated the use of dry nasal swab and NP in VTM obtained in parallel on ED patients. Contrary to manufacturer expectations, our parallel study showed that the sensitivity of Abbott ID NOW using dry swabs (51.6%), was actually lower than when using viral transport media (66.7%) as seen from our earlier study of ED samples.
The literature provides no strong consensus regarding the influence of transport medium and storage conditions on viral load stability. One study looking at differing viral transport media and storage conditions using the Quest and Roche EUA assays showed consistent detection of SARS-CoV-2 RNA after 7 days at room temperature, and up to 14 days refrigerated or frozen. (11) The SARS-COV-2 ID-NOW package insert also directs clinicians to test swabs as soon as possible after collection, and if not possible, to hold it in its original package no more than two hours prior to testing. Outside of inpatient hospital laboratories, such as in outpatient point-of-care testing, independent validation should be performed to assess performance in different settings, including pre-analytic variables.
Extrapolating from our results from 101 patient samples, if the Abbott-ID-NOW COVID-19 assay were to be used as a first step screening, confirmation of over 80% of the tested samples would be required to be confident that the negative results are truly negative. While there was one apparent false-positive result in our parallel study, the overall high PPV (94.1%) and specificity (98.6 %) do support the added value that in 5-13 minutes a positive result can be interpreted as a true positive. We could not confirm that the apparent false positive result obtained with the Abbott ID NOW was due to sampling variability.
Based on our findings we could argue that the Abbott ID NOW detects samples with high viral load or possibly viable virus that could be of importance for transmission. But, the fact that it misses positive samples on patients being admitted to the hospital with clinical picture of COVID-19 makes this technology unacceptable in our clinical setting. Recent reports indicate that mutations in the RdRp domain have been emerging (12). It is unknown at this point if the genomic region targeted by the Abbott assay has changed significantly as the virus is spreading throughout the world.
Limitations of our study include relatively small sample size, inability to control for sampling variabilities, time of transport of dry swabs and testing being conducted in the clinical laboratory, rather than point-of-care setting as Abbott ID NOW platform is intended.
Conclusion
Overall, our study revealed low sensitivity with high false negative results by Abbott ID NOW platform irrespective of use of viral transport media, which raises concern regarding the performance of the assay and its suitability as a diagnostic tool for symptomatic patients. The resolution could be to reflex all negative results for confirmation by a method with higher sensitivity. However, such requirement would, except for positives, severely diminish the value of the rapid results of the assay.
Acknowledgements
We sincerely acknowledge the help and support of the medical technologists from the Tisch Microbiology laboratory (Jiamin Chui, Claudia Funez and Alicia Lovell) and the Emergency Medicine Nursing Team including Patricia Baptiste, Jennifer Tiarsmith, Jason Garcia, Roselle David, Robin Keida, Jacquelyn Burns, Lauryn Clouden for caring for the patients and their assistance in collecting the samples used in this study.
Footnotes
Conflict of Interest statement: No conflicts of interest.