Celeste Elash, Director, eCOA Sciences

With acceptance of personal electronic devices (e.g., smartphones and tablets) by the public, the use of computerized systems to collect clinical outcomes assessment (COA) data in clinical trials is commonplace and becoming the preferred and recommended method.1-3 The advantages of electronic data collection over paper are numerous and well documented in the literature4-6 , and when well applied, can result substantial trial cost savings1.  This movement toward electronic data collection has enhanced the quality and accuracy of clinical trial data and the and regulators are encouraging electronic instead of paper-based data collection.1-3

The 2016 Century Cures Act7, which includes sections devoted to streamlining patient input and using patient experience data in drug development, builds on FDA's work to incorporate patients’ perspectives into the development of drugs, biological products, and devices and helped focus drug development on patient-focused outcomes. In 2017, former FDA Commissioner Dr. Scott Gottlieb stated “Our aim is to facilitate the development and use of patient-focused methods in more parts of our regulatory activities as well as develop and elevate common standards for how to integrate the patient voice, as a matter of science, into product development. Among some of the long-term goals of these new efforts is the creation of consistent approaches to how the FDA develops clinical outcomes assessment tools such as patient-reported outcomes to inform our regulatory decisions.”8  With the reissuance of the Prescription Drug User Fee Act (PDUFA)9, patient engagement was elevated to be a core part of the FDA’s regulatory approvals process.

Electronic COA (eCOA) is central to patient-focused drug development. As the regulators continue to encourage its use in clinical trials, eCOA deployment will continue to evolve.  Hot eCOA topics that demonstrate the trend toward patient-centricity are described here.

Wearables – Within patient-centered and value-based care, wearables offer huge potential to complement electronic Patient-Report Outcome (ePRO) data for the identification of new signals, higher sensitivity for new endpoints or screening criteria. CNS indications, and more specifically, neuroscience conditions, may represent the biggest opportunity for sensor-based measures due to well-known challenges such as a lack of classical biomarkers in CNS diseases.  Potential benefits to using wearables in CNS, as well as other therapeutic areas, is the ability to generate a richer dataset while reducing the bias of traditional clinical scales.

As more companies explore new technologies, they may want to take into consideration some key questions related to mHealth and wearable device development. At YPrime’s Expert Community event in April 2019, industry expert Marie McCarthy advised to start with a clinical hypothesis versus allowing technology to drive objectives and to think about the data upfront. She posed questions for consideration such as: “What type of data will be collected? What decisions will be made?, Will the data/trial be blinded?, How will you ensure that the technology conforms to local privacy regulations?, What is a valid data set?, How will we deal with data loss?”.  McCarthy noted while there are tremendous benefits to collecting wearable data, Sponsors need adequate planning and preparation for management and use of the data, as statistical analysis and data management plans must be capable of handling massive amounts of data. Data management, particularly edit checks and cleaning of wearable sensor data, is likely be cumbersome.  Most importantly, though, in the context of patient-centricity, Sponsors must keep the patient in mind and minimize burden with the introduction of wearables data collection in a trial.  It behooves Sponsors to consider the feasibility of the wearable device by trial participants, given their condition and age, and whether the length of wear, frequency of device change, whether the wearable is waterproof, and the extent of available technical support will place undue burden on the trial participants.  All these issues can be addressed with forethought and careful planning during eCOA development.

Web vs. Paper Back Up – Backup and recovery are essential parts of eCOA deployment planning. The need for back-up to the primary electronic data collection device, e.g., hand-held device, can be minimized with comprehensive system requirements, good user experience design (UX) and user interface (UI), as well as thorough user acceptance testing (UAT), quality assurance (QA), quality control (QC) and thoughtful site and participant training.  Although uncommon (typically less than 1% of devices deployed in a trial)10,11, device failure, breakage or loss can result in missing data. The most primitive approaches to back up missing eCOA data revert to paper diaries, which is plagued with risk, 10-12  puts undue burden on the trial participant to remember when to complete assessments, and most certainly results in poor compliance with per protocol patient-recorded outcome (PRO) data recording requirements13 and data quality.  Just as electronic data capture drives better data quality, so do electronic back-up methods. Ideally, if a trial participant experiences difficulty with the primary electronic data collection device, the participant will have remote access to a help-desk service to trouble-shoot and/or provide a replacement as quickly as possible in order to minimize data loss.  In those rare instances when swift device replacement is not an option, the participant will have access to an alternate electronic method with which to record trial data.  In addition to remote troubleshooting and readily available replacement devices, YPrime routinely uses web back-ups in our eCOA deployments.

Importantly, The Critical Path Institute’s ePRO Consortium also does not recommend the use of paper back-ups.10,11 Using a paper back-up for your eCOA strategy is akin to backing up your playlist on a cassette tape. Think about it. If you wouldn’t use outdated technology for your tunes, why use it in your clinical trial?

Bring Your Own Device (BYOD) - Using participants’ own mobile devices to collect ePRO data is certainly an industry hot topic. The primary driver of BYOD is the convenience for the trial participant (i.e., patient-centricity), which the industry hopes, in turn, will result in higher compliance with per protocol PRO data recording procedures and data quality.  Allowing participants to use their own familiar phone eliminates the need for the participant to maintain a separate trial device and ostensibly reduces budget and simplifies trial logistics by eliminating the management and cost of provisioned devices.14

Despite the perceived advantages, the limited use of BYOD in regulatory studies to date may be mainly due to two primary concerns within the clinical trial industry.15 The first concern pertains to the practical aspects of using a participant’s own smartphone.

The industry has many concerns about how participants’ day-to-day use of their personal smartphones may interfere with trial data collection.  Although the typical, constant checking for incoming messages may, in fact, enhance compliance with trial data collection, updating their Operating System, running out of storage space for ePRO trial data, or upgrading to a new smartphone mid-trial could certainly impact a participant’s data quality.   A larger issue is the exclusion of potential participants who don’t own a personal smartphone.

With the prevalence of mobile devices, it is thought that most trial participants own a smartphone. In fact, in January of 2018, the Pew Research Center15 conducted a survey, the results of which indicated that over 95% of Americans across a wide range of demographic groups own a cellular phone of any type.  However, use of a smartphone exhibited greater variation based on age, income and educational level.  While smartphone ownership was common in younger (18-29 years, 94%), well educated (college graduate, 91%), affluent (income >$75,000, 93%) Americans, the rates were lower for older Americans (50-64, 73%; >65, 49%), those with a high-school diploma (69%) or those making less than $30,000 annually (67%).

In a 2019 international survey, the Pew Research Center16 demonstrated that people in advanced economies are more likely to have smartphones (76%) than those in emerging economies (45%).  However, smartphone ownership varies widely by country, and in both advanced and emerging economies, the global patterns are like those in the United States.  Younger people, those with higher levels of education and those with higher incomes are more likely to be digitally connected.

Without question, a pure-play BYOD trial with an inclusion criterion requiring ownership of a personal smartphone will systematically exclude people from key demographic groups and bias data and/or limit generalizability of results.  Additional logistic considerations, including site and helpdesk training for BYOD models, are harder to manage, and cost savings may be negligible except in cases of large studies. With standard devices, training for helpdesk staff is much easier than having to support any number of consumer devices. Site staff are unlikely to adequately support the range of devices that a BYOD trial may generate. Risk mitigation planning needs to consider a range of scenarios, including the inadvertent deletion of the study app from participants, the likelihood of participant’s changing devices mid-trial, device OS updates and insufficient phone space to install the study app or store eCOA data.

As such, it is recommended that a hybrid model using both personal smartphones and trial-provisioned devices be used and to be prepared with a risk-mitigation plan for hiccups that may occur with participants’ smartphones.   In these hybrid studies, sponsors need to be prepared to provision more devices than they may have initially envisioned, as determining the ratio of BYOD to provisioned devices isn’t an exact science. Over-tasked site staff may encourage participants to use a provisioned device for ease of support.  Fortunately, as the industry prepares for such hybrid studies, we have data to suggest that although some people may prefer using their own smartphone versus a provisioned device or vice versa, trial participants generally find both easy to use in the trial context.

In a recent methodology study18, patients with painful chronic health conditions (N=155, Mean age = 48.6; range = 19-69 years) participated in a single-visit, three-way crossover study during which they completed a PRO measure (PROM) in random order using a paper questionnaire, electronically using a standard device provided by the study site and  using an app installed on their own device (smartphone or tablet).  Forty-five percent of participants felt BYOD would be more convenient compared to 15% preferring a provisioned device (40% had no preference).  In another study using BYOD (Android or iOS) versus provisioned device19, 64 participants with COPD (Mean age  = 59; range 40-77 years) completed the EXAcerbations of Chronic pulmonary disease Tool [EXACT®], COPD Assessment Test™ [CAT] and Patient Global Impression of Severity [PGIS] during 15-day assessment periods on BYOD and a provisioned device, in randomized order. Of the 57 participants who reported preference data, there was no significant difference in ease of use or preference for BYOD or the provisioned device.

The second predominant concern about BYOD is an important scientific consideration that is currently being debated amongst industry stakeholders.  That is, that the psychometric properties of a PROM may be different when presented on different data collection devices.14,15,20,21 When BYOD is considered for a clinical trial in which the PRO instrument will be used to collect primary or secondary efficacy endpoints, regulators have strongly recommended collecting data to demonstrate measurement equivalence of the PROM across the various devices. FDA’s expectation has been that clinical trial sponsors collecting PRO data using multiple device types (or paper and an electronic device) will be able to demonstrate the data are equivalent regardless of the specific method being used to capture it.2 Measurement equivalence concerns can be addressed in well-designed studies, as outlined in ISPOR best practice publications.20-21 These studies, although important, can be costly in terms of potential delay to trial start if not well planned and conducted in a timely manner relative to pivotal trial FPI.  With the plethora of data demonstrating data equivalence across modes of administration,18, 22,23 industry eCOA experts and clinical trial researchers have recently begun to question the necessity of such equivalence data, particularly the need for cognitive interview studies that had heretofore been recommended20 when only “minor” changes to PRO items were made when migrated to electronic platform.

Industry expert Willie Muehlhausen has taken this concept one step further with BYOD implementation, arguing that it’s impossible to conduct equivalence with every device, and every device class. His research focused on establishing the equivalence of widgets instead of full PROMs on devices. “If a patient population understands the concept of a numeric rating or visual analog scale, that should be enough” Willie stated in YPrime’s Expert Community event in April 2019.  His research concluded the widgets were equivalent – with no discernable differences in the way participants understood how to use visual analog scales, numeric rating scales or categorical response scales, whether presented on paper, iOS or Android formats. Thus, the use of standard widgets, implemented on electronic platform in accordance with industry best practices 20, 24 meets regulatory expectations.

In a synthesis of findings from 53 formal cognitive interview and usability studies conducted between 2012 and 2015, Muehlhausen and colleagues25 concluded “With the benefit of accumulating evidence, it is possible to relax the need to routinely conduct cognitive interview and usability studies when implementing minor changes during instrument migration.  Application of design best practice and selecting vendor solutions with good user interface and user experience properties that have been assessed for usability in a representative group may enable many instrument migrations to be accepted without formal validation studies by instead conducting a structured expert screen review”.

Until the regulators formally announce the relaxation of the equivalence data requirement, trial sponsors are likely to take the conservative approach with key endpoint data and conduct the required equivalence studies, whether the COAs were migrated from the original paper format to a single electronic platform for use in a trial or presented on multiple electronic devices in a BYOD study.  In the meantime, industry stakeholders may ask “Who are these experts and how do they review screens”?

Expert Screen Review - In addition to advising to incorporate all individual instrument author requirements in a PROM migrated from paper to electronic platform, Muehlhausen and colleagues25 outline an extensive number of broad areas that an expert screen review should cover.  These areas include those related to faithful migration from the original, psychometrically validated format, e.g., paper, to electronic platform, as well as items pertaining to usability such as clarity, font size and ease of navigation.  Guidance on such reviews is found in familiar industry documents 20,24, but YPrime recommends taking a very conservative approach to screen review to ensure your “expert” has deep experience implementing eCOA instruments on a variety of electronic platforms in multiple therapeutic areas and is sufficiently familiar with industry standards to understand the principles of faithful migration and to minimize the likelihood of disrupting the PROM’s original psychometric properties.   In a personal communication with the author, Muehlhausen himself suggested that the Sponsor and PROM authors/copyright holders should acknowledge the expert’s competency, and the “expert” should feel competent to defend the decision to use "screen review” to the FDA.

Interested in knowing more?  Follow YPrime and be sure to stop by our booth at DIA to speak with our team of experts.



  1. Coons SJ, Eremenco S, Lundy JJ, O’Donohoe P, O’Gorman H, Malizia W. Capturing patient-reported outcome (PRO) data electronically: the past, present, and promise of ePRO measurement in clinical trials. Patient, 2015;8(4):301-309.


  1. S. Food and Drug Administration. Guidance for industry—Patient-reported outcome measures: use in medical product development to support labeling claims. December 2009. http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf.


  1. S. Food and Drug Administration. Guidance for industry: electronic source data in clinical investigations. September 2013. www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM328691.pdf


  1. Stone AA, Shiffman S, Schwartz JE, et al., Patient non-compliance with paper diaries. BMJ, 2002;324(7347):1193-1194.


  1. Shields AL, Shiffman S, Stone A. Patient compliance in an ePRO environment: methods for consistent compliance management, measurement and reporting. In: Byrom B, Tiplady B, eds. ePRO: Electronic Solutions for Patient-Reported Data. Surrey, England: Gower; 2010:127–142.


  1. Ganser AL, Raymond SA, Pearson JD. Data Quality and Power in Clinical Trials: A Comparison of ePRO and paper in a randomized clinical trial. In: Byrom B, Tiplady B, eds. ePRO: Electronic Solutions for Patient-Reported Data. Surrey, England: Gower; 2010:49.


  1. 21st Century Cures Act. https://www.congress.gov/bill/114th-congress/house-bill/34/related-bills.


  1. Gottleib, S. Statement by FDA Commissioner Scott Gottlieb, M.D. on new steps by FDA to advance patient engagement in the agency’s regulatory work. Oct 11, 2017. https://www.fda.gov/news-events/press-announcements/statement-fda-commissioner-scott-gottlieb-md-new-steps-fda-advance-patient-engagement-agencys


  1. “PDUFA VI: Fiscal Years 2018 - 2022.” http://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm446608.htm


  1. Howry C, Elash CA, Crescioni M, Eremenco S, O’Donohoe P, Rothrock T. Best Practices for Avoiding Paper Backup When Implementing Electronic Approaches to Patient-Reported Outcome Data Collection in Clinical Trials. Ther Innov Regul Sci. 2018, Sep 24:2168479018785160. doi: 10.1177/2168479018785160. [Epub ahead of print]


  1. Howry C, Elash CA, Crescioni M, Eremenco S, O’Donohoe P, Rothrock T. Best Practices for Avoiding Paper Backup When Implementing Electronic Approaches to Patient-Reported Outcome Data Collection in Clinical Trials. Critical Path Institute, ePRO Consortium Webinar. May 16, 2019. https://c-path.org/wp-content/uploads/2019/05/Webinar-AvoidingPaperBackup2019MAY16.pdf


  1. Fleming S, Barsdorf AI, Howry C, O’Gorman H, Coons SJ. Optimizing Electronic Capture of Clinical Outcome Assessment Data in Clinical Trials: The Case of Patient-Reported Endpoints. Ther Innov Regul Sci, 2015;49(6):797-804.


  1. Stone AA, Shiffman S, Schwartz JE, Broderick JE, Hufford MR. Patient non-compliance with paper diaries. BMJ, 2002;324(7347):1193-1194.


  1. Gwaltney C, Coons S, O’Donohoe P, O’Gorman H, Denomey M, Howry C, Ross, J. ‘‘Bring Your Own Device’’ (BYOD): The Future of Field-Based Patient-Reported Outcome Data Collection in Clinical Trials? Ther Innov Regul Sci, 2015; 48(6) 783-791.


  1. Byrom B, Lee J, Dennis K, Noble M, McCarthy M, Muehlhausen W. Bring Your Own Device for Trial Outcome Assessment. Applied Clinical Trials, 2016. 25 (6).  http://www.appliedclinicaltrialsonline.com/bring-your-own-device-trial-outcome-assessment


  1. Pew Research Center. Mobile Fact Sheet. Feb 5, 2018. https://www.pewinternet.org/fact-sheet/mobile/


  1. Pew Research Center. Smartphone Ownership Is Growing Rapidly Around the World, but Not Always Equally.  Feb 5, 2019.  https://www.pewresearch.org/global/2019/02/05/smartphone-ownership-is-growing-rapidly-around-the-world-but-not-always-equally/


  1. Byrom B, Doll H, Muehlhausen W, Flood E., Cassedy C, McDowell B, Sohn J, Hogan K, Belmont R, Skerritt B, McCarthy M. Measurement Equivalence of Patient-Reported Outcome Measure Response Scale Types Collected Using Bring Your Own Device Compared to Paper and a Provisioned Device: Results of a Randomized Equivalence Trial. Value Health. 2018; 21 (5) 581-589.


  1. Newton L, Eremenco S, Crescioni M,  Symonds, T, Griffiths P, Knight-West O, Reasoner D, Byrom W, O’Donohoe P, Vallow S. Comparability of a Provisioned Device Versus Bring Your Own Device for Completion of Patient-Reported Outcome (PRO) Measures by Participants with Chronic Obstructive Pulmonary Disease (COPD): Qualitative Interview Findings. 2018. Poster presented at ISPOR, Barcelona, 2018.


  1. Coons SJ, Gwaltney CJ, Hays RD, et al. Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force Report. Value Health. 2009; 12:419-429.


  1. Eremenco S, Coons SJ, Paty J, et al. PRO data collection in clinical trials using mixed modes: report of the ISPOR PRO Mixed Modes Good Research Practices Task Force. Value Health. 2014;17:501-516.


  1. Gwaltney CJ, Shields AL, Shiffman S. Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Value Health 2008;11(2):322–33.


  1. Muehlhausen W, Doll H, Quadri N, et al. Equivalence of electronic and paper administration of patient-reported outcome measures: a systematic review and meta-analysis of studies conducted between 2007 and2013.HealthQualLifeOutcomes2015;13:167–87.


  1. ePRO Consortium. Best practices for migrating existing patient reported outcome instruments to a new data collection mode. Ther Innov Regul Sci. 2014. 49(6) http://c-path.org/wp- content/uploads/2014/05/BestPracticesFor-MigratingExistingPROInstrumentstoaNewDataCollectionMode.pdf. Published 2014.


  1. Muehlhausen, W, Byrom, B, Skerritt, B, McCarthy, M, McDowell, B, Sohn J. Standards for Instrument Migration When Implementing Paper Patient-Reported Outcome Instruments Electronically: Recommendations from a Qualitative Synthesis of Cognitive Interview and Usability Studies. Value Health. 2018; 21: 40-48.