Tubes

Test Performance

Update: Jan. 27, 2021

Medical Test Paradox video presents examples of clinical decisions that are taking into account the probability of false positive and false negative results.  A quote from the video:

The paradox is that you can take a test which is highly accurate, in the sense that it gives correct results to a large majority of the people taking it, and yet under the right circumstances when assessing the probability that your particular test result is correct, you can still land on a very low number. Arbitrarily low in fact. In short, an accurate test is not necessarily a very predictive test.

False Positive and False Negative

Diagnosis and treatment of patients experiencing symptoms similar to COVID-19 cannot be solely dependent on the diagnostic molecular test result, since a certain fraction of the results can be false positive or false negative.

In case of a false positive, there is a risk that a patient will undergo unnecessary treatment or therapy and delayed diagnosis of the true infection. In case of a false negative, there is a risk of delayed treatment and spreading COVID-19 within the community.

Similarly, there is risk involved in making decisions about a person’s immunity to COVID-19 based on a positive result of a screening antibody test, which can be a false positive.

Virus

What are the false negative and false positive test results?
Are some tests more reliable than others and how are medical tests evaluated?
Discussion of these subjects in the scientific literature is complex and likely out of reach of a layperson.

We describe several test characteristics and demonstrate how they are calculated, based on the Guidance for Industry and FDA Staff Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests  [Ref.1].

Hypothetical Test Results

Suppose you are validating results of a study for a new diagnostic test, where the condition of interest is presence of the SARS-CoV-2 virus and the reference standard is the clinical diagnosis of the COVID-19 disease in subjects of the study.

For verifying presence of the SARS-CoV-2 virus in the patient test samples, you collected 51 samples from subjects diagnosed by clinicians with COVID-19 and you expect 51 test outcomes to be positive. However, you receive 44 positive test outcomes and 7 negative test outcomes. Also, you have 169 patient test samples taken prior to COVID-19 epidemic, assumed to be free of the SARS-CoV-2 virus. You expect 169 test outcomes to be negative, but you receive 168 negative outcomes and 1 positive outcome.

The numbers were borrowed from an example on page 23 of Ref. 1. Names and descriptions of the tests and the reference standard are not to be taken literally, as they are used only as an example, to simplify presentation of the concepts.

Definitions

Table 1 presents a common 2×2 table format for comparing test outcomes to the reference standard outcomes. It was borrowed from Ref. 1, page 22 including the definitions and formulas.  The new test has two possible outcomes, positive (+) or negative ().  Subjects with the condition of interest are indicated as the reference standard (+), and subjects without the condition of interest are indicated as the reference standard ().

TP = number of true positive results
FP = number of false positive results
TN = number of true negative results
FN = number of false negative results.

True Positive Result a positive test result for a subject in whom the condition of interest is present (as determined by the reference standard)

False Positive Result — a positive test result for a subject in whom the condition of interest is absent

True Negative Result a negative test result for a subject in whom the condition of interest is absent

False Negative Result — a negative test result for a subject in whom the condition of interest is present

Sensitivity — the proportion of subjects with the target condition in whom the test is positive; calculated as 100xTP/(TP+FN)

Specificity — the proportion of subjects without the target condition in whom the test is negative; calculated as 100xTN/(FP+TN)

Predictive value of a positive result (sometimes called positive predictive value or PPV) — the proportion of test positive subjects who have the target condition; calculated as 100xTP/(TP+FP)

Predictive value of a negative result (sometimes called negative predictive value or NPV) — the proportion of test negative subjects who do not have the target condition; calculated as 100xTN/(TN+FN)

Estimation

Table 2 presents how test results compare to the reference standard.

Estimated Sensitivity = 100% x 44/51 = 86.3%

Estimated Specificity = 100% x 168/169 = 99.4%

Estimated Positive Predictive Value (PPV) = 100% x 44/45 = 97.8%

Estimated Negative Predictive Value (NPV) = 100% x 168/175 = 96.0%

The values above are estimates based on a subset of subjects from the intended use population. If another subset of subjects were tested, the results would be numerically different.     

Last Updated on April 23, 2021 by covid

eLab

Nanomix Rapid Test

Berkeley Lab News Center published on May 21, 2020, a story featuring Emeryville-based biotech company Nanomix, Inc. and its rapid COVID-19 test, submitted for the FDA Emergency Use Authorization (EUA).

Carbon Nanotubes as Biological Sensors

Lawrence Berkeley National Laboratory (LBL) scientists Alex ZettlMarvin Cohen and their teams are researching nanomaterials for the Department of Energy.  They demonstrated ultrasensitive oxygen sensors built with carbon nanotubes (CNT), with possible application in gas-leak detectors or air- and water-pollution detectors.

carbon nanotube detector
Artist’s rendition of a carbon nanotube

In later studies LBL scientists found that CNTs could be used for biological and medical applications, to detect proteins or carbohydrates at the level of a single cell. Nanomix, Inc. is a commercial spin-off of this LBL research, and both scientists are currently serving on the company’s Board of Directors.

Assays for Portable Analyzer

On April 8, the company was awarded $570,000 in funding from the Department of Health and Human Services’ Biomedical Advanced Research and Development Authority (BARDA) to develop disposable cartridges that test for protein traces of the coronavirus – known as antigens – in nasal swab samples, and for antibodies to the coronavirus in blood samples.

eLab System

eLab is a simple and easy to use diagnostic system using a disposable cartridge. A proprietary nano-biosensor shaped into a disposable microfluidics high quality test cartridge is used for reliable performance. It was initially developed more than five years ago, in response to the Ebola virus epidemic. Since then the company developed rapid tests for sepsis and for kidney injury, allowing physicians to make a quick and informed decision for proper treatment planning.

After minimal training, an operator can take a patient sample, introduce the blood into the cartridge and will receive the test results in less than 15 minutes. The device is simple to operate, with an intuitive user interface on a color touch screen.

Results presented on the screen can also be sent a nearby printer via Bluetooth. Designed for an emergency room and a point-of-care setting, this robust system may be used in other situations.

Last Updated on April 23, 2021 by covid

CDC Logo

Serology Testing at CDC

CDC has developed a laboratory serology (antibody) blood test to assist with surveillance efforts, to better understand how much of the U.S. population has been infected with SARS-CoV-2 and how the virus is spreading through the population over time.

It is not certain yet whether the antibodies that result from SARS-CoV-2 infection will provide someone with protection (immunity) from getting infected again. If antibodies do provide some immunity, we do not known just how protective they are or how long immunity might last. CDC scientists are currently conducting studies to answer these questions. The CDC test is not designed to find out whether a person has been previously infected with SARS-CoV-2. Commercial tests are available to provide test results to individuals.

Lab

CDC’s serologic test is an ELISA-based test, detecting SARS-CoV-2 antibodies in serum or plasma components of blood, is described in a April 25, 2020 pre-print. The test uses purified SARS-CoV-2 S protein as an antigen, not live virus. The antigen comes from the Vaccine Research Center at the National Institutes of Health (NIH).

CDC’s serologic test is designed to detect antibodies produced in response to SARS-CoV-2, while minimizing cross-reactivity to the ones generated in response to other common coronaviruses that cause less severe illnesses, such as common colds. However, potential for occasional cross-reactivity cannot be ruled out completely.

CDC’s test has specificity of greater than 99% and a sensitivity of 96%, based on initial performance evaluations. It can be used to identify past SARS-CoV-2 infection that occurred at least 1 to 3 weeks earlier.

CDC Evaluating Performance of Commercial Tests

CDC is checking the performance of the commercially manufactured antibody tests in collaboration with the following federal organizations:

  • Biomedical Advanced Research and Development Authority (BARDA)
  • U.S. Food and Drug Administration (FDA)
  • National Institutes of Health (NIH)
  • Department of Defense (DoD)
  • White House Office of Science and Technology Policy

FDA provided results from the initial evaluation in  EUA Authorized Serology Test Performance and will be updating it, having performed more.

CDC Interim Guidelines  for Antibody Tests

CDC issued interim guidelines for using antibody tests in clinical and public health settings. The recommendations on the use of serologic tests should be updated as new information becomes available.

CDC Serology Surveillance Strategy

Learn more about CDC’s COVID-19 serology surveillance strategy.

Last Updated on April 23, 2021 by covid

Dropper

EVMS Prophylaxis Protocol

Eastern Virginia Medical School (EVMS) published on their website guidance for healthcare providers for treating COVID-19 patients,

this approach to COVID-19 is based on the best (and most recent) available literature and the Shanghai Management Guideline for COVID

COVID-19 Prophylaxis Protocol

Developed by Dr. Paul Marik, the complete EVMS Critical Care COVID-19 Protocol  is supplemented by a two-page Marik COVID-19 Protocol Summary. Notable excerpts:

Webinar on Clinical Protocols

On May 28, 2020, the M. Foscue Brock Institute for Community and Global Health hosted a webinar by Eastern Virginia Medical School and Sentara Healthcare doctors and then posted a video “EVMS-Sentara COVID-19 Update on Clinical Protocols”,

an overview of COVID-19 epidemiology, transmission and clinical manifestations; reviewed testing, infection control and PPE recommendations; reviewed the pathophysiology of the cytokine cascade and pro-coagulation in COVID-19 patients; and discussed current COVID-19 treatment protocols

Screenshots from the video:

Dr. Edward Oldfield

Professor of Infectious Diseases and Internal Medicine

Dr. Michael Hooter

Sentara Healthcare, Pulmonary and Critical Care Medicine

Dr. Paul Marik

Professor of Pulmonary Medicine

Last Updated on April 23, 2021 by covid

Apple

Apple Google Covid API

Apple and Google have been eager to participate in the world-wide COVID-19 response. They have jointly released a standard of sorts, for a Contact Tracing methodology, each providing an application programming interface (API) for their mobile platforms.

Apple documented the Privacy-Preserving Contact Tracing API, delivered via ExposureNotification Framework. Apple, in its usual manner did not release the source code. But they do provide a sample application and an outline for developing a matching Notification server. At device level, “Coronavirus exposure alert support” is included in the newly released iOS version 13.5.

The cellular phone giant has earlier released COVID-19 information and personal guidance app and website, developed in cooperation with the FDA and CDC. They also made citywide and regional charts of Mobility Trends (routing requests) within Apple Maps available.

Apple Google

Goggle published a reference implementation of a server, written in GO (Golang) programming language. Included are scripts for invoking Docker OS virtualization and Terraform cloud infrastructure deployment and datacenter provisioning environment. Even though the source code for the the sample Android app, Android API, Exposure Key Export and File Format implementation is available, the Goggle offering cannot be used directly by the innovative open source community,

Only approved government public health authorities can access the APIs (included in an upcoming build of Google Play services) to build an app for COVID-19 response efforts.

Still, the app codebase continues to receive updates from Google.

Mitre

There is an open source alternative for contact tracing, a server-side system from a non-profit and academia powerhouse Mitre, for some reason called SaraAlert. Implemented with Ruby-on-Rails (RoR) web framework and MySQL database, the choice of platforms would have been very popular a decade ago. But these tools have since fallen out of favor with the OSS community, due to a mix of perceived performance, scalability and security concerns. On a positive note, work on SaraAlert codebase continues.

In a related development, CDC has provided a hub of information about Covid data sets.

CDC specified reporting requirements for Laboratory Data, aimed at CLIA certified labs.

U.S. Department of Health and Human Services [HHS] has published a document bringing together links to Diagnostic Data Standards for reporting Covid test results, including related data formats [Mandatory Minimum Core Data Elements] and HL7 messages.

Last Updated on May 27, 2022 by covid

FDA Approved

EUA Process

Emergency Use Authorization (EUA) for COVID-19 Tests

On January 31, 2020, Health and Human Services (HHS) Secretary Alex M. Azar declared that coronavirus presents a Public Health Emergency in the Unites States. The Public Health Emergency declaration issued on February 4, 2020, provided liability immunity for activities related to medical countermeasures against COVID-19, as defined in Section III, Recommended Activities and Section VI, Covered Countermeasures. The Declaration sets out activities for which the immunity is in effect:

Covered Countermeasures are any antiviral, any other drug, any biologic, any diagnostic, any other device, or any vaccine, used to treat, diagnose, cure, prevent, or mitigate COVID-19, or the transmission of SARS-CoV-2 or a virus mutating therefrom, or any device used in the administration of any such product, and all components and constituent materials of any such product.

During a Public Health Emergency, such as the COVID-19 epidemic, FDA uses its Emergency Use Authorization (EUA) authority for establishing a process to provide access to critical products, when there are no adequate approved and available alternatives. FDA Commissioner may authorize the use of unapproved medical products, or unapproved uses of approved medical products. FDA evaluates products quickly, using the evidence that is available and carefully balances any known or potential risk of that product with any benefits to the public from making them available during the emergency.

FDA Logo

Public Access to FDA Documents

On March 25, 2020, FDA placed a Notice in the Federal Register announcing “Process for Making Available Guidance Documents Related to Coronavirus Disease 2019”. FDA Guidance for the process of COVID-19 tests authorization was implemented immediately by FDA Center for Devices and Radiological Health (CDRH).

Per the Public Health Emergency declaration, FDA issued this Guidance without prior public participation and comment, but it remains subject to comment, in accordance with the Agency’s good guidance practices. As stated in the notice, “Guidance documents related to COVID–19 will be accessible on the internet from FDA web page titled ‘‘Coronavirus Disease 2019 (COVID– 19)’’. This page is kept up to date and contains links to COVID-19 EUAs and Guidance, latest news from FDA, FAQs, etc.

Guidance Document for COVID-19 Tests

Final version of the Guidance Document titled “Policy for Coronavirus Disease-2019 Tests During the Public Health Emergency (Revised)” was published online on May 11, 2020, and is available for download.

Since its initial publication on February 29, 2020, this Policy was updated three times, as the number of required tests was increasing, and public-private partnerships were established to respond to this growing demand. At the same time, more experience was gained with COVID-19 testing process. Per the “Insight into FDA’s Revised Policy on Antibody Tests”, a group of government organizations (NIH, CDC, BARDA) helped establish a capability at the National Cancer Institute (NCI) to independently validate SARS-CoV-2 Antibody Tests, such as Lateral flow and Enzyme-linked immunosorbent assay (ELISA) tests. Letter for the Umbrella Emergency Use Authorization was issued on April 28, 2020, with an update on May 4, 2020.

Policy defines how FDA is working with test developers from laboratories and commercial manufactures, on issues of test validation, FDA notification, EUA request, and clinical testing. In Section IV, Policy, FDA presents four paths of its regulatory process:

A) Laboratories Certified under CLIA that Meet the CLIA Regulatory Requirements to Perform High­Complexity Testing Using Their Validated Diagnostic Tests Prior to EUA Submission

B) State Authorization of Laboratories Certified under CLIA that Meet the CLIA Regulatory Requirements to Perform High ­Complexity Testing

C) Commercial Manufacturer Development and Distribution of Diagnostic Tests Prior to EUA Submission

D) Commercial Manufacturer Development and Distribution and Laboratory Development and Use of Serology Tests Prior to or Without an EUA.

In Section V, Validation Study Recommendations Based on the Technological Principles of Tests, FDA directs developers to demonstrate that their submitted test validation is based on the underlying principles for the specific test type: molecular diagnostic, antigen detection and serological. Developers can use alternative approaches and discuss them with FDA through CDRH-EUAtemplates@FDA.HHS.GOV, to get feedback, recommendations, and help through the EUA process.

EUA Templates

Section VI. Availability of EUA Templates, FDA refers to a series of templates for various types of COVID-19 tests that developers may choose to use to facilitate the preparation, submission, and authorization of a EUA. Links to the templates are provided on FDA page Emergency Use Authorizations in the Section “In Vitro Diagnostics EUAs”:

FDA updates these templates over time, as more experience is gained with the EUA process for the various types of COVID-19 tests and as the COVID-19 disease is further understood.

FAQs on Testing for SARS-CoV-2

From the beginning of the EUA process, FDA has been providing online answers to Frequently Asked Questions (FAQs) related to the development and performance of tests for SARS-CoV-2. The list of links to the new topics is growing. At the time of this writing it contains the following:

Medical Device Webinars and Stakeholders Calls

FDA Virtual Town Hall Series is an open forum for test developers to interact with FDA officials and have their questions answered. In these weekly webinars, Timothy Stenzel, Director of the Office of In Vitro Diagnostics and Radiological Health in CDRH’s Office of Product Evaluation and Quality is usually starting with an overview of the changes made to the FAQs pages and answers questions from the callers. Links to transcripts, recordings, and slides are posted online and can be found at Virtual Town Hall Series – Immediately in Effect Guidance on Coronavirus (COVID-19) Diagnostic Tests.

Last Updated on April 23, 2021 by covid