How to get the Most Out of Your eCOA Clinical Trial Data

How to Get the Most Out of Your eCOA Data

May 10, 2022 YPrime

How to Get the Most Out of Your eCOA Data

By Monali Khanna and Frank Menius

Adoption of electronic clinical outcome assessments (eCOA) for clinical trial data collection is happening at a faster rate than ever before as many healthcare organizations and pharmaceutical companies are now choosing to move away from paper data collection methods.

eCOA is a method of capturing patient experience data electronically in clinical trials. eCOA utilizes technology including computers, tablets, and smartphones to allow patients, caregivers, and clinicians to directly report outcomes, supplying high- quality data and near real-time insights. eCOA measures include patient reported outcome (PRO), clinician reported outcome (ClinRO), observer reported outcome (ObsrRO), and performance outcome (PerfO). When participants are more engaged, compliance levels tend to increase as it is easier to submit data entries. The data entries show exact timestamps, which provides confidence in the data quality and integrity, resulting in better information for regulatory decision making. By providing a more comfortable and convenient way to collect data, eCOA tools can also help increase participant retention. The decision to include eCOA should be considered early in the clinical trial design process. Once it is decided that implementing an eCOA platform is the right fit for a clinical trial, and the eCOA vendor is selected, overcoming challenges for the need to collect quality data throughout a study is critical. This article summarizes a few high-level upfront considerations for obtaining high quality, regulatory-compliant eCOA data in the expected period.

1: A THOROUGH PROTOCOL REVIEW SESSION

Fully understanding the ins and outs of a clinical trial protocol before eCOA instrument design, and database creation is critical. What is important for eCOA study data collection?

Trial objectives and design
Inclusion/exclusion criteria
All applicable visits (visit schedule, including windows); safety, efficacy assessments or other datapoints used as primary or secondary endpoints
Data criteria that will be used as part of each statistical data set/sub-group
Adverse events that can provide important insights into treatment tolerability, especially during treatment recovery phases at home
Concomitant medication usage

2: DATA TRANSFER/INTEGRATION KICK OFF MEETING

To conduct an effective data management kick-off meeting that ensures project team members walk away prepared to successfully execute the data management plan, all stakeholders from both the eCOA vendor and the sponsor/CRO should take part. In addition to eCOA data acquisition leads, considering including data managers, statistical programmers, and biostatisticians who would work on downstream analysis of the data. The data standards representative may be a critical person to have attend the meeting. Example agenda items to include in the meeting:

Milestones, deliverables and timeline agreement
Creation and ownership for data transfer/integration guidance documents like global data transfer agreement, study specific data transfer specification (DTS)
How metadata is to be presented and how metadata specification is to be used
Communication plans including data management escalation pathways
Roles and responsibilities for creation and maintenance of data management tasksContinuous Improvement

3: PLAN DESIGN TIME DISCUSSIONS WITH SPECIFIC AGENDA ITEMS

The US Food and Drug Administration (FDA) recommends qualitative, quantitative, or mixed methods to collect robust and meaningful COA data. Evaluating what questions to ask that are well understood by a wide range of patients, caregivers, and clinicians, and deriving that list of questions to show clinical benefit in a specific treatment trial is of extreme importance. Decision making by regulatory reviewers will require developing good measures for identified sets of impacts from treatment and incorporating clinical outcome assessments into endpoints. It would be useful to have clear research goals and questions established. Each clinical trial is unique and study teams need to plan for optimal data collection and data transfer. Expertly designed, collaborative solutions give you confidence by delivering cleaner data for well-informed decision-making. Some items to talk about proactively:

How to create logic sequences to guide patients through the appropriate questions, how to prevent patients from skipping questions, and entering inconsistent or conflicting data, how to catch data issues at data capture, discussions of adding unscheduled visits for patients who may need additional time at the site due to a change in their diagnosis, extending data entry windows to accommodate the availability, implementing a caregiver additionally.
Whether the eCOA vendor team has a questionnaire library and will the configuration specialist be able to pull in any previously configured questionnaire and translations into current study for a quick start. • How the eventual data transfer will look and has the team reviewed what data are needed for eventual analysis. This facilitates forward thinking to capture data correctly.
What specific business rules as edit checks are created for the time of data collection to prevent inconsistency in questionnaires answers and are there any cross instrument eCOA data checks.
Are rules and guidelines to be followed created and shared with the study team for sensitive data correction. Each study would have its own configuration for what data change types would be allowed and associated permission and approval workflows would be there to support the changes. A reference guide listing the example of data change types could be created. A few examples are “change questionnaire information”, “change subject visit”, “Change questionnaire responses”, etc. An audit trail should always be available and shared when required.

4: ENSURING AVAILABILITY OF FDA 21CFR PART 11 COMPLIANT AUDIT TRAIL REPORTING

A crucial element of compliance is to ensure is that electronic records are supported with an audit trail, where event, user, timestamp, and a required comment are logged (electronic signature). The process for obtaining audit trails for all content, transfers and reports should be understood. Being 21 CFR part 11 compliant also means having a good grip on the issue of data integrity. FDA uses ALCOA principles to define five data integrity attributes corresponding to good manufacturing practice:

Attributable
Legible
Contemporaneous
Original
Accurate

5: CREATING SUPPORT AND PROPAGATING IMPORTANCE OF USING DATA STANDARDS

Collecting data to conform with near SDTM standards accelerates the speed of a clinical trial by making the downstream process much easier since the standards are aligned. The use of standards would increase efficiencies due to repetition and standardized specifications and forms. Standardization will also make possible the use of commercially available industry tools, which can be used for data validation and conformance checks. Standardization also increases the value of the eCOA data transfer as it eliminates the need for the data to be manipulated by data management. The FDA and Japan’s Pharmaceuticals and Medical Devices Agency (PMDA) require the use of data standards for submission. The Study Data Tabulation Model Implementation Guide (SDTMIG) provides the organization, structure, and format of these standard tabulation datasets. “Flavors” of SDTM (e.g., SDTM+, SDTM-, SDTM-Inspired) are being implemented for eCOA in diverse ways across organizations that have well-structured standards governance and implementation strategies. Questionnaires, ratings and scales (QRS) supplements are vital to clarify how implementation of an instrument should be performed. Each QRS instrument is a series of questions, tasks, or assessments used in clinical research to provide a qualitative or quantitative outcome of a clinical concept, or task-based observation. The integration of data standardization in the overall process will gain efficiency, productivity, and cost savings in the eCOA environment. Requirement for the teams to follow guidelines for data standard implementation by organizations like CDISC and PHUSE is highly encouraged.

6: ADVANTAGE OF ANNOTATION OF ECOA SCREEN REPORTS

Different stakeholders are included at different points in the eCOA process, which may create variability between what is collected vs. how it is stored vs. what is expected. To decrease this variability, utilization of the annotated eCOA screen report is encouraged. Annotated eCOA screen reports provide clear mapping of the collected eCOA data to the corresponding variables or discrete variable values contained within the data provided, in the data transfer. Like an annotated case report form (aCRF), the annotated eCOA screen report facilitates clear and transparent review to help identify incorrect data mapping decisions that may delay the eventual analyses. A study’s eCOA screen shots can also be provided as part of regulatory requirements to submit an annotated CRF (aCRF.pdf).

7: DISCUSSING METADATA MANAGEMENT

Management of metadata is an important consideration, especially as more automation is introduced. The use of spreadsheets will be replaced by electronic metadata repositories, to which all systems will have access, ensuring accurate processing of data. Making sure that metadata remains machine-readable enables the use of automatic processes to create data transfers and facilitate structured review. The use of application programming interfaces (APIs) will ensure that eCOA vendors can easily integrate with other systems. APIs combined with standardized machine-readable metadata contribute to near real-time data transfers from patient entry to client review.

8: AWARENESS AND KNOWLEDGE THAT SDRG WOULD BE CREATED FOR THE STUDY SUBMISSION

The reviewer’s guides are to aid the FDA reviewer in understanding data by providing a single document that includes a comprehensive description of those data, including any conformance issues that those data may have. The Study Data Reviewer’s Guide (SDRG) describes any special considerations, directions or conformance issues that may facilitate an FDA reviewer’s use of submitted data and may help the reviewer understand the relationships between the study report and those data. Sometimes data collected has values that are not compliant to the SDTM standards or are anomalous but still need to be submitted, we must include those and annotate in the SDRG. Knowledge, and planning to record unavoidable data issues in the reviewer’s guides, will not only help expediting review, but also providing a place of reference for the entire team to track critical issues found and decisions made throughout the course of a clinical trial.

9: MOTIVATING TEAMS TO CREATE DATA TRANSFER/INTEGRATION GUIDANCE DOCUMENTS:

A secure data transfer process will ensure that organizations meet compliance requirements. It will supply a record of the transactions so that the organization will know exactly what information was shared and any anomalies can be investigated. An established process answers important questions that arise such as the mechanisms and frequency of data transfers and integrations, or the availability to review real time data. The practice of creating and following documents like global data transfer agreements and study specific Data Transfer Specification (DTS) removes ambiguity and provides clear expectations for data exchange. The DTS describes the requirements for any data transfers or integrations in scope for the project. To start drafting the DTS, a meeting is required including all study team stakeholders and the 3rd party integration partner, to ensure all stakeholder needs are met. Key questions to ask while drafting DTS include:

What data points each system needs to send or receive. (Example: variables DOMAIN, USUBJID for ease of integration with other collection data; use of QSSPID variable for questionnaire data; variables RNGTXTLO, RNGTXTHI, RNGVALLO, RNGVALHI, as these are implemented in some QRS supplements (e.g., EQ-5D-5L))
What file format each system requires whether provision is in ODM XML or CSV, etc.
How/where the data will be posted, location to post the data and what access credentials or rights need to be in place, whether to post on a sFTP site, or to integrate with a web service endpoint to push/pull data, etc.
Whether there is provision to provide data adhering to SDTM or near SDTM data standards and the regulatory accepted transport file (.xpt)
When and how often data are needed—daily transfer, weekly, etc.

10: CREATING AWARENESS AND RULES FOR MISSING DATA

Even with the best planning, eCOA data may be missing at the end of the clinical trial. Adequate education on the purpose of collecting the eCOA data to encourage patient compliance would help prevent and reduce the frequency of missing data. Missing data should be distinguished from data that do not exist or data that are not considered meaningful, due to an intercurrent event. The protocol and the SAP should address plans for how the statistical analyses will handle missing COA data when evaluating clinical benefit and the eCOA vendor and data acquisition teams need to be made aware of these plans. Rules for handling missing data should be specific to each COA and is usually determined during the instrument development process. The SAP should specify plans to assess the impact of missing data (i.e., a missing data simulation study) and all rules for handling missing data. For example, the SAP can specify the proportion of items that can be missing before a domain is treated as missing. A lot of time and energy can be saved by the eCOA vendor and sponsor or CRO’s data receiving teams if they are made aware of plans in the protocol and SAP for how missing data for a COA endpoint and any related supportive endpoints will be handled.

11: EMPHASIZING IMPORTANCE OF BLINDING

Human behavior is influenced by what we know or believe. Blinding (also called masking) is used to try to eliminate such bias by keeping one or more parties in a trial unaware of which treatment arms participants have been assigned. How a trial is blinded should be accurately recorded to allow readers to interpret the results of a study. A well-written communication plan is the key to maintaining the blind in a trial. An effective communication plan will detail whether a team member is blinded or unblinded, as well as the protocol for the handling of unblinded information. It is especially important to take appropriate time to review if the information being handled is of a blinded or unblinded nature. The study is to be treated as blinded unless documentation stating otherwise has been received by the team. If blinding is broken during a trial on individual patients, it needs to be statistically and/or ethically explained at the end. Careful consideration should be given when dealing with any information that could be potentially unblinding.

12: EVALUATE TRAINING NEEDS

Inefficient data capture can hinder clinical development and regulatory approvals. Using eCOA promotes data quality by taking measures for minimizing incomplete or erroneous data entries and uncertainty about when data entries were made. A well-planned training program for patients, study teams, CRAs, and sites will ensure the best outcome for data entries. How to use devices; training session on guidelines to answer questionnaires; demo videos; quick reference guides; hands-on training sessions would maximize data collection and completeness, reducing the need to follow up on erroneous data and will shorten the time to database lock.

13: SCOPING DATA MONITORING

Data monitoring can help identify compliance patterns that are below or near threshold levels, allowing for additional analysis by qualified staff to evaluate the scope and severity of the issue. Expert assessment can determine if the issue requires a formal data correction, or if it can be addressed in downstream statistical analysis. Early identification also allows for intervention and corrective actions to prevent problematic trends from escalating into larger challenges. For example, if a protocol requires 80% compliance, and data monitoring shows compliance has been varying between 79% and 85%, a risk mitigation plan may be executed at that point to evaluate the likelihood of meeting the final compliance target, and to identify reasons for low compliance (such as demographic-related factors, problems at study sites or problematic issues with the outcome instrument). Examples of data monitoring checks would be:

Identify any diary entries which have no answers
Site compliance report for all active subjects at the site and their overall compliance
Partially completed visit
Out-of-window visits
Missed any two non-consecutive diaries
Compliance of daily diary
No study medication intake alert report
Alerting duplicate subjects by specifying site and subject number, and similar ones.

14: CREATING PROCESS FOR DATA TRANSFER EXPERT REVIEW

Understanding of protocols, endpoints, study populations, subject’s exclusion/inclusion criterion, and questionnaire standards help raise meaningful data anomalies early in the data life cycle. Expert review of data before transfer aids quality and accuracy of data. Examples of how this review is done include:

Use of programmatic checks to confirm calculated variables
Metadata reviews based on pre-specified criteria
Comparison of data transfers to the protocol and eCOA screen reports
Analysis of value frequencies of categorical data, min/max/quartiles of continuous data
Conformance checks against industry data standards

15: CONFIRMING EXPECTATIONS ON STUDY CLOSEOUT

When a study is ready to close out, the expectations on the final data transfer package and archival should be discussed. The list of deliverables should be made and reviewed by each party, along with timelines of delivery. The example list includes the final data transfer, a file per patient for attributes, diary entries, questionnaires/answers, database extracts of DCFs, eCOA screen reports, and data verification documents.

CONCLUSION

Clinical outcome assessment data provides valuable information for compound development. The FDA recommends qualitative, quantitative, or mixed methods to collect robust and meaningful eCOA data. Although the benefits are clear, introducing new technology to sites and patients can create additional challenges for the anticipated data collection. By proactively considering eCOA instrument design and adopting best practices for data collection and maintenance, high quality and compliant data can be achieved. Effective data science teams collaborate with customers both early on and strategically, providing streamlined processes, building standards and necessary tools to create an efficient workflow that accelerates data collection and integration, and ensuring higher data quality.